pax_global_header00006660000000000000000000000064141214166010014506gustar00rootroot0000000000000052 comment=3fc5b4761c0e03122b053b81cc73b28ea38d3a92 qiime-2021.8.0/000077500000000000000000000000001412141660100130445ustar00rootroot00000000000000qiime-2021.8.0/.gitattributes000066400000000000000000000000401412141660100157310ustar00rootroot00000000000000qiime2/_version.py export-subst qiime-2021.8.0/.github/000077500000000000000000000000001412141660100144045ustar00rootroot00000000000000qiime-2021.8.0/.github/CONTRIBUTING.md000066400000000000000000000015131412141660100166350ustar00rootroot00000000000000# Contributing to this project Thanks for thinking of us :heart: :tada: - we would love a helping hand! ## I just have a question > Note: Please don't file an issue to ask a question. You'll get faster results > by using the resources below. ### QIIME 2 Users Check out the [User Docs](https://docs.qiime2.org) - there are many tutorials, walkthroughs, and guides available. If you still need help, please visit us at the [QIIME 2 Forum](https://forum.qiime2.org/c/user-support). ### QIIME 2 Developers Check out the [Developer Docs](https://dev.qiime2.org) - there are many tutorials, walkthroughs, and guides available. If you still need help, please visit us at the [QIIME 2 Forum](https://forum.qiime2.org/c/dev-discussion). This document is based heavily on the following: https://github.com/atom/atom/blob/master/CONTRIBUTING.md qiime-2021.8.0/.github/ISSUE_TEMPLATE/000077500000000000000000000000001412141660100165675ustar00rootroot00000000000000qiime-2021.8.0/.github/ISSUE_TEMPLATE/1-user-need-help.md000066400000000000000000000006111412141660100220620ustar00rootroot00000000000000--- name: I am a user and I need help with QIIME 2... about: I am using QIIME 2 and have a question or am experiencing a problem --- Have you had a chance to check out the docs? https://docs.qiime2.org There are many tutorials, walkthroughs, and guides available. If you still need help, please visit: https://forum.qiime2.org/c/user-support Help requests filed here will not be answered. qiime-2021.8.0/.github/ISSUE_TEMPLATE/2-dev-need-help.md000066400000000000000000000005641412141660100216720ustar00rootroot00000000000000--- name: I am a developer and I need help with QIIME 2... about: I am developing a QIIME 2 plugin or interface and have a question or a problem --- Have you had a chance to check out the developer docs? https://dev.qiime2.org There are many tutorials, walkthroughs, and guides available. If you still need help, please visit: https://forum.qiime2.org/c/dev-discussion qiime-2021.8.0/.github/ISSUE_TEMPLATE/3-found-bug.md000066400000000000000000000017421412141660100211430ustar00rootroot00000000000000--- name: I am a developer and I found a bug... about: I am a developer and I found a bug that I can describe --- **Bug Description** A clear and concise description of what the bug is. **Steps to reproduce the behavior** 1. Go to '...' 2. Click on '....' 3. Scroll down to '....' 4. See error **Expected behavior** A clear and concise description of what you expected to happen. **Screenshots** If applicable, add screenshots to help explain your problem. **Computation Environment** - OS: [e.g. macOS High Sierra] - QIIME 2 Release [e.g. 2018.6] **Questions** 1. An enumerated list with any questions about the problem here. 2. If not applicable, please delete this section. **Comments** 1. An enumerated list with any other context or comments about the problem here. 2. If not applicable, please delete this section. **References** 1. An enumerated list of links to relevant references, including forum posts, stack overflow, etc. 2. If not applicable, please delete this section. qiime-2021.8.0/.github/ISSUE_TEMPLATE/4-make-better.md000066400000000000000000000015321412141660100214530ustar00rootroot00000000000000--- name: I am a developer and I have an idea for an improvement... about: I am a developer and I have an idea for an improvement to existing functionality --- **Improvement Description** A clear and concise description of what the improvement is. **Current Behavior** Please provide a brief description of the current behavior. **Proposed Behavior** Please provide a brief description of the proposed behavior. **Questions** 1. An enumerated list of questions related to the proposal. 2. If not applicable, please delete this section. **Comments** 1. An enumerated list of comments related to the proposal that don't fit anywhere else. 2. If not applicable, please delete this section. **References** 1. An enumerated list of links to relevant references, including forum posts, stack overflow, etc. 2. If not applicable, please delete this section. qiime-2021.8.0/.github/ISSUE_TEMPLATE/5-make-new.md000066400000000000000000000015131412141660100207570ustar00rootroot00000000000000--- name: I am a developer and I have an idea for a new feature... about: I am a developer and I have an idea for new functionality --- **Addition Description** A clear and concise description of what the addition is. **Current Behavior** Please provide a brief description of the current behavior, if applicable. **Proposed Behavior** Please provide a brief description of the proposed behavior. **Questions** 1. An enumerated list of questions related to the proposal. 2. If not applicable, please delete this section. **Comments** 1. An enumerated list of comments related to the proposal that don't fit anywhere else. 2. If not applicable, please delete this section. **References** 1. An enumerated list of links to relevant references, including forum posts, stack overflow, etc. 2. If not applicable, please delete this section. qiime-2021.8.0/.github/ISSUE_TEMPLATE/6-where-to-go.md000066400000000000000000000100111412141660100214020ustar00rootroot00000000000000--- name: I don't know where to file my issue... about: I am a developer and I don't know which repo to file this in --- The repos within the QIIME 2 GitHub Organization are listed below, with a brief description about the repo. Sorted alphabetically by repo name. - The CI automation engine that builds and distributes QIIME 2 https://github.com/qiime2/busywork/issues - A Concourse resource for working with conda https://github.com/qiime2/conda-channel-resource/issues - Web app for vanity URLs for QIIME 2 data assets https://github.com/qiime2/data.qiime2.org/issues - The Developer Documentation https://github.com/qiime2/dev-docs/issues - A discourse plugin for handling queued/unqueued topics https://github.com/qiime2/discourse-unhandled-tagger/issues - The User Documentation https://github.com/qiime2/docs/issues - Rendered QIIME 2 environment files for conda https://github.com/qiime2/environment-files/issues - Google Sheets Add-On for validating tabular data https://github.com/qiime2/Keemei/issues - A docker image for linux-based busywork workers https://github.com/qiime2/linux-worker-docker/issues - Official project logos https://github.com/qiime2/logos/issues - The q2-alignment plugin https://github.com/qiime2/q2-alignment/issues - The q2-composition plugin https://github.com/qiime2/q2-composition/issues - The q2-cutadapt plugin https://github.com/qiime2/q2-cutadapt/issues - The q2-dada2 plugin https://github.com/qiime2/q2-dada2/issues - The q2-deblur plugin https://github.com/qiime2/q2-deblur/issues - The q2-demux plugin https://github.com/qiime2/q2-demux/issues - The q2-diversity plugin https://github.com/qiime2/q2-diversity/issues - The q2-diversity-lib plugin https://github.com/qiime2/q2-diversity-lib/issues - The q2-emperor plugin https://github.com/qiime2/q2-emperor/issues - The q2-feature-classifier plugin https://github.com/qiime2/q2-feature-classifier/issues - The q2-feature-table plugin https://github.com/qiime2/q2-feature-table/issues - The q2-fragment-insertion plugin https://github.com/qiime2/q2-fragment-insertion/issues - The q2-gneiss plugin https://github.com/qiime2/q2-gneiss/issues - The q2-longitudinal plugin https://github.com/qiime2/q2-longitudinal/issues - The q2-metadata plugin https://github.com/qiime2/q2-metadata/issues - The q2-phylogeny plugin https://github.com/qiime2/q2-phylogeny/issues - The q2-quality-control plugin https://github.com/qiime2/q2-quality-control/issues - The q2-quality-filter plugin https://github.com/qiime2/q2-quality-filter/issues - The q2-sample-classifier plugin https://github.com/qiime2/q2-sample-classifier/issues - The q2-shogun plugin https://github.com/qiime2/q2-shogun/issues - The q2-taxa plugin https://github.com/qiime2/q2-taxa/issues - The q2-types plugin https://github.com/qiime2/q2-types/issues - The q2-vsearch plugin https://github.com/qiime2/q2-vsearch/issues - The CLI interface https://github.com/qiime2/q2cli/issues - The prototype CWL interface https://github.com/qiime2/q2cwl/issues - The prototype Galaxy interface https://github.com/qiime2/q2galaxy/issues - An internal tool for ensuring header text and copyrights are present https://github.com/qiime2/q2lint/issues - The prototype GUI interface https://github.com/qiime2/q2studio/issues - A base template for use in official QIIME 2 plugins https://github.com/qiime2/q2templates/issues - The read-only web interface at view.qiime2.org https://github.com/qiime2/q2view/issues - The QIIME 2 homepage at qiime2.org https://github.com/qiime2/qiime2.github.io/issues - The QIIME 2 framework https://github.com/qiime2/qiime2/issues - Centralized templates for repo assets https://github.com/qiime2/template-repo/issues - Scripts for building QIIME 2 VMs https://github.com/qiime2/vm-playbooks/issues - Scripts for building QIIME 2 workshop clusters https://github.com/qiime2/workshop-playbooks/issues - The web app that runs workshops.qiime2.org https://github.com/qiime2/workshops.qiime2.org/issues qiime-2021.8.0/.github/SUPPORT.md000066400000000000000000000122421412141660100161030ustar00rootroot00000000000000# QIIME 2 Users Check out the [User Docs](https://docs.qiime2.org) - there are many tutorials, walkthroughs, and guides available. If you still need help, please visit us at the [QIIME 2 Forum](https://forum.qiime2.org/c/user-support). # QIIME 2 Developers Check out the [Developer Docs](https://dev.qiime2.org) - there are many tutorials, walkthroughs, and guides available. If you still need help, please visit us at the [QIIME 2 Forum](https://forum.qiime2.org/c/dev-discussion). # General Bug/Issue Triage Discussion ![rubric](./rubric.png?raw=true) # Projects/Repositories in the QIIME 2 GitHub Organization Sorted alphabetically by repo name. - [busywork](https://github.com/qiime2/busywork/issues) | The CI automation engine that builds and distributes QIIME 2 - [conda-channel-resource](https://github.com/qiime2/conda-channel-resource/issues) | A Concourse resource for working with conda - [data.qiime2.org](https://github.com/qiime2/data.qiime2.org/issues) | Web app for vanity URLs for QIIME 2 data assets - [dev-docs](https://github.com/qiime2/dev-docs/issues) | The Developer Documentation - [discourse-unhandled-tagger](https://github.com/qiime2/discourse-unhandled-tagger/issues) | A discourse plugin for handling queued/unqueued topics - [docs](https://github.com/qiime2/docs/issues) | The User Documentation - [environment-files](https://github.com/qiime2/environment-files/issues) | Rendered QIIME 2 environment files for conda - [Keemei](https://github.com/qiime2/Keemei/issues) | Google Sheets Add-On for validating tabular data - [linux-worker-docker](https://github.com/qiime2/linux-worker-docker/issues) | A docker image for linux-based busywork workers - [logos](https://github.com/qiime2/logos/issues) | Official project logos - [q2-alignment](https://github.com/qiime2/q2-alignment/issues) | The q2-alignment plugin - [q2-composition](https://github.com/qiime2/q2-composition/issues) | The q2-composition plugin - [q2-cutadapt](https://github.com/qiime2/q2-cutadapt/issues) | The q2-cutadapt plugin - [q2-dada2](https://github.com/qiime2/q2-dada2/issues) | The q2-dada2 plugin - [q2-deblur](https://github.com/qiime2/q2-deblur/issues) | The q2-deblur plugin - [q2-demux](https://github.com/qiime2/q2-demux/issues) | The q2-demux plugin - [q2-diversity](https://github.com/qiime2/q2-diversity/issues) | The q2-diversity plugin - [q2-diversity-lib](https://github.com/qiime2/q2-diversity-lib/issues) | The q2-diversity-lib plugin - [q2-emperor](https://github.com/qiime2/q2-emperor/issues) | The q2-emperor plugin - [q2-feature-classifier](https://github.com/qiime2/q2-feature-classifier/issues) | The q2-feature-classifier plugin - [q2-feature-table](https://github.com/qiime2/q2-feature-table/issues) | The q2-feature-table plugin - [q2-fragment-insertion](https://github.com/qiime2/q2-fragment-insertion/issues) | The q2-fragment-insertion plugin - [q2-gneiss](https://github.com/qiime2/q2-gneiss/issues) | The q2-gneiss plugin - [q2-longitudinal](https://github.com/qiime2/q2-longitudinal/issues) | The q2-longitudinal plugin - [q2-metadata](https://github.com/qiime2/q2-metadata/issues) | The q2-metadata plugin - [q2-phylogeny](https://github.com/qiime2/q2-phylogeny/issues) | The q2-phylogeny plugin - [q2-quality-control](https://github.com/qiime2/q2-quality-control/issues) | The q2-quality-control plugin - [q2-quality-filter](https://github.com/qiime2/q2-quality-filter/issues) | The q2-quality-filter plugin - [q2-sample-classifier](https://github.com/qiime2/q2-sample-classifier/issues) | The q2-sample-classifier plugin - [q2-shogun](https://github.com/qiime2/q2-shogun/issues) | The q2-shogun plugin - [q2-taxa](https://github.com/qiime2/q2-taxa/issues) | The q2-taxa plugin - [q2-types](https://github.com/qiime2/q2-types/issues) | The q2-types plugin - [q2-vsearch](https://github.com/qiime2/q2-vsearch/issues) | The q2-vsearch plugin - [q2cli](https://github.com/qiime2/q2cli/issues) | The CLI interface - [q2cwl](https://github.com/qiime2/q2cwl/issues) | The prototype CWL interface - [q2galaxy](https://github.com/qiime2/q2galaxy/issues) | The prototype Galaxy interface - [q2lint](https://github.com/qiime2/q2lint/issues) | An internal tool for ensuring header text and copyrights are present - [q2studio](https://github.com/qiime2/q2studio/issues) | The prototype GUI interface - [q2templates](https://github.com/qiime2/q2templates/issues) | A base template for use in official QIIME 2 plugins - [q2view](https://github.com/qiime2/q2view/issues) | The read-only web interface at view.qiime2.org - [qiime2.github.io](https://github.com/qiime2/qiime2.github.io/issues) | The QIIME 2 homepage at qiime2.org - [qiime2](https://github.com/qiime2/qiime2/issues) | The QIIME 2 framework - [template-repo](https://github.com/qiime2/template-repo/issues) | Centralized templates for repo assets - [vm-playbooks](https://github.com/qiime2/vm-playbooks/issues) | Scripts for building QIIME 2 VMs - [workshop-playbooks](https://github.com/qiime2/workshop-playbooks/issues) | Scripts for building QIIME 2 workshop clusters - [workshops.qiime2.org](https://github.com/qiime2/workshops.qiime2.org/issues) | The web app that runs workshops.qiime2.org qiime-2021.8.0/.github/pull_request_template.md000066400000000000000000000006121412141660100213440ustar00rootroot00000000000000Brief summary of the Pull Request, including any issues it may fix using the GitHub closing syntax: https://help.github.com/articles/closing-issues-using-keywords/ Also, include any co-authors or contributors using the GitHub coauthor tag: https://help.github.com/articles/creating-a-commit-with-multiple-authors/ --- Include any questions for reviewers, screenshots, sample outputs, etc. qiime-2021.8.0/.github/rubric.png000066400000000000000000007014131412141660100164060ustar00rootroot00000000000000PNG  IHDR,4\sBIT|d pHYs.#.#x?vtEXtSoftwarewww.inkscape.org< IDATxw|g3AޫEEQvgmKhQRD.#g$ G)9IIr~vkxr\7ޯ\b0 R1NE/zQ@G E/z'w:~9 .VhhD?^Kg7o78s7HG^{5]~]v ӧOլYD ^-ZٳgW_TB iٲeںufR)//_lRl߾ݤIs8#8ѣGرcٰakQ ___۷Of (p&I8p@sIO5ydXTRZ~ҦMk7oZpDO<1ӧO(8!?0c2dȠs,fݺu_6 8 2?sI @C ޽{uСfIZj& ilϟ߬EILR^DԬYq]]]`I&Ѿ6vUV5o\TD$&~<н{%6p^DGGk֭˓'O/Sڵkhh#^QF &:t\]/TՕ={ve˖M/[l8cǎ%.aÆI ￗWڴi#F)+)<<\&MҡC8$^dN>mR7n$Y*Toh?^@\UTщ'4tP.]Z3gͫ-[jӦMZf0=ŋMw!;֭ۗ`^q 7|iĚ3gNrN/p6mq$=iӦD Klc׮]K4(z1P ><Ȟ=j֬`gϞ%Y|֋رc; IMMd4k,$ zSLÇ; Id̘Ѭ#GԈ#e\~)z)׉''$w(z6{_uѣ6ͥ\rʔ)SIQh`7ne˖zyr8bŊY4\:t蠀UN/ N%ɝJ,iXQVR 4l0ruw*g͚k׮m+][-Zӧ;8\ !OddgϮGd|k׮ڵ-j9<{Q۶mu=O,XP+VN4ӧOu1*((Haaazː! (E\rJ>}2g`>}ZgΜӧ+W`0_q\\\%K+WN5kT9$ϘԩS:}N:]qSM0A3fPtt(z{xCpBmڴy`0hڿ6mhTMݻw~zmذAғ'OL%Jyj۶*Tę:K.i͚5ڼy;(ƹB jٲڵkB %q҃b[Ew/FGFFjŚ]t<$???EEEI8>vzRѢEuʕ$Uzu |ڷooҽfS5jO?5ڶdXª6m'o׮]2ev!k_?aÆ=zj׮m\֋}I' @;vɓ'M#Oҷ~kl9z:viӦiѢE*W\#""gIh@s(^x큁:tlbUѶm۴m65o\S\̚/oݺufIү_O>5y̍7ԡCi[bV\_:tf͚e.]d$I|'Ol; Kwv=zƍ?~ϯݻ믿Klٲ7oM ^v1UZU~m̟,[L˗iU7oVٲegƭ^Z~~~6/xܹS-Z0~;w֭[Z˗/5-Q'EӮq#""d/_^=z͛7,֊+;tD[ʕf]|Yyn*___b={L}_|amک`ѣյkW=~8...j֬vء7n(22RW^UlYݻwO5?l-v%kNϞ=x_ՆY'{R7'ojΜ9ѣcך5k4zh3Ʀw~-\P}1.q,W\ʕ+5j%KW^&7ne˦}iҘ1c#Gɓ13G*TP`N:PhhN:G***ʤ8UVUժUU^8m CK.Uv^z/_>uI۷ט1c4mڴD犈PvcլYӤչsgX5ٳgmV ,ŋSddU󹹹QF*U1c(,,L>ԑ#Gt9֑iӪv*VMs1X{, ՛0a&N9TREWV맟~RVL*R :T|MK}G&O&L~I/_N:%`0M6Z~IsΝ['OTk׮ԩSڸq:dϟ_|>C3fnܸϟ'8WZo߾ߵh"]~\,Y]&`0رcF>cmݺլ 2) *\Kn߾QF~0+͛7Ox͛75x߿&M$oooQHFi3gNmٲE+Vx /_^>LouIK.ѾϞ=Sum(UN8!wY~ʖ-4ￕ-[V'N0{ܫݻ9rmsuuՎ;۷QF&iժɅ/dɒ&رمM6K.hLƌg2eѣչs؝D HtTsxչsg?WZJH.]|rLlx.]TىbJKX6mژK@@/^hyjȐ!&oݳhܫ*, 6,ނ׃ԭ[7 ^4j(s*VTbr+Wȑ#&-Zhҥfy!,,쥂Wٵ~7L+O< {zzjŊ*UE:ɓ't5egbΓ1cd%  ֐!C}xﻱQFrcWX=c^UNO6ͤЀ>}z9v- qx^^^ 矛DZYyy v*lRժU3{ܿjFRIt)=<<4vXs4iEc9~Fڷoϑ /W/{6m(**1ϟ?7-IPʫxfpݛhٳ[nf-o6m2|ʚ5Ѷ˗/kܹ&LjoXBׯ_o1͛77{̿ 2DF֭X///w6k,~&SM74{ܬYnU g}j^`['%@dɢ ?P͚5%Çk&_j]frs{!wfaÆYtҥKM>јH\2sOOO?qӧOWddqyGFFj֭fDZ3w߿uY9r䈷=W\:zƌ*Upz7ճgO=zTY Zn%K=nڴi={y޻woI QuujnGh ѣG͚;gΜfcUvwĈf/IgVLLEc|_~{}c,aB.]#G}'Oj…*S,͑ZtY``֭[gQUVɓ'/=K>:vh| PbŊZ~4qD(Q"c.[,>7n4{^KW6{LttM[Z5ըQ_wEڵ+u&xwRJ/]dq\YfUjK:t`1-wy֮];eɒŢ@ҡ0[5n8;wN>e˖-Ib>}Z.\Hρ̞7$$侏?̙3GYw9Ң(`0y>dȐ%} :Tk׮5;γgt]YAVe5zh;vL~~~f9y݃m@D `*Uh֬Yy6mڤVZæ1lH޽{+cƌʚ5kx{{+s2d={fQ?6ofTxqcڵK&ҥKokv<<<4j(|_xxVXy>}̎ 샢fͪ:wFiAAA]ry ҥrav˗| <˔)zหW*Zhrz^{qv8EoOOOu>(zYlYY'!%JД)SѣG9֤g3f4hvOj…;}{v,Y$8?oKy{{%hWh?cGnݚ`Mj]t)ޝ@I;K,<./WWW 4(-=~pڴi]Ec͕?~ġf_њ:u͛og- ۢ<}P't>K5jΝ;o|_Bw4Yzz-Cյk_sm޼YZ2-_>>۷Oϟ~}%޽;N>}X^DrUw1 ()+V̤~E`NME#F=&""],>ԏ?_>}ڽU@sRiԨQfwzuX" `0ԧH"]MrI8$XQ$)yxx\rȑCٳg7;v@@@GJѼys/^q͋?ĹW]7k7x<9>%gΜ޽LEFFjɒ%qk yQ'ц/㏊C6x"""WNx\\\TJcĉfK6l_ 6H2~a˖-޹UP!̙\8p3fҦMk֘gj˖-駟tΝҦM]2E(z0eWӭ[n:;d;w9JU V%c`ھ}Ec}zMڵ|||7sLi5h rUcu%… ?~!e3(o|SNfꫯwyޢE ȑ;Nä#ƍ_tСDt)Ѽ}YƦŒuQƍS*UtY驁=?,I:y*U}ltګ;ِ!C,Sox IDAT̙?ƍho^ϟ$O*[n޼ia_~~~5c խ[Ws-ZTڵ3{ܫ"E^z쏢8tҙO>QPPPrƍDRرwNYr$M8Ѥ3{l}W/=ԛoi/[nVkYTx!cƌyYӧOe+W.Oy瞽kƍ/=H>HֽիIc@A IBCCѣGIxӧOO>Znŋ>Sg}f~ &˪_5l0_OV>}z͞=[7o8gԮ];-^8N۔)S,*R<~آ<ý{C㚲KU&LPDDDי3gV…͎oJR˖--&M}Q''O?~\M4I̙3-O>QM]|EL4ɢbbb4h E2eʨ哐bŊyO6guyѣg0ԵkW8q챏?{ｧiرj֬i,޽{q}9sqձ-AD]6qM6+s?;6m9sZ/^DrmVztMhʔ)EذaC97n?աCM4)ν>|ԩٳgiK.VXaSXټЮ];UL=|P5kԆ Lk׮]q (`)-.&&&G5cҤIo~z#o޼f9poΝ;3k˗Wƍ-ۧO싢8K^\\r5k?nqgϪaÆC_ʕ+-1|+Vq111?~jժk*,,,N(:tHzoN)SXT 2UUzu2fyKNk֬QƌnZmڴѮ]ӧOm65iDjR```>iҤ-:F޽{jժ._lXI5j(G4Gdd>3M>ݢϟ?׻ﾫ 6\}Uƍ4 jٲ[]rEu6lؠ GqܹTbQƌ71 R l:-R^,?~ 8P)h[nСCocǎD_,YRwz'R``jժ[nY< .ܹsMWPPP=6L_qMqFjʬ15jЁl4ib=W*^rȡhݿ_/^LpNWWW-]T:u2)u9׆ lr|g,YԨQ#UREEU TLeϞݢBBBt1ݹsGGѺut59Sҥ?~e͚U+V4il g<)R;̙3t!SjĈ6E/p"ǎ3wb˧ҥK+_|򒋋BBB#ԩSq5lP+WT֬Ym_@@{=]r&{Zh#&&F%KŋMfi&Iٲe!\\\4sL}&?~&MYϢEԣGƮ]Vm۶qF*Tn'O[oeՎhǎ*W޳g֭kRߌ3ڵk=OR&MGɽ^z_~w5>I>>3 6LnnnVBR}ڽ{5EǏg}f.Ç&ϗ/Zld4iDP۶mu…$#IZxڷoqe/.B iKj:uFOon(xq8wz+WNuQ5k,rI&wﮀ9Ҧrȡ]vi5kV^Z ^/tU>>>&8pݓw_˖-'NhĈJ6m(VG+ >\SLnnn۷>dƏhWWW <8II8ƍy/_>-YDGUʕ%S0ڼynݪ5j,~|4~x]pA/Vm61...8p._qY|ϒy{'ّԀ>}zԩSuyS6?שSTR% ˌ9RwVٲe-.]:mV-[6go4n8ɋo iq8ǏB 9pĈرc;sN::qݻgRӫ|]5kʕ+5~ٳgڸq6oެ;vfԭ[7KEM,Mw=(P@ӧ,X`Ǭ'$$D?l٢={$1%JP޽եKw>秥K/5!W\Y4h6m$6mJϝ;w~vE/p2A&LPcwuYƍp߿P`0(}ʘ1ͫH"I~bbbt=zT/^իWu-EEE)44Tʚ5gϮrʩjժz7hgѣw^˗OSb=_|ʙ3gr(IzN>G*((Hׯ_۷0e͚UYfUΜ9UR%UZU JaWҥKzٳ+w)⿣`Pٲeuԩ8mK_e#JmQSXhzw}g9H]R֯IСCq̙S;vLlQû{VX~(OOdE/8s*""gL[`G111u넇kĉX ,jժiƌ<5jҧO;rOĉuM͝;W>bbbԬY3>r~wkq|׺sKϊ-$y~\ j]矪Y"##վ}{-[LiҤIRu֩M6F۲e˦k׮tGŋUB>sqqy$N`>T)IZj5jR1ۼysm_c U6m|ݻw`SPPKvڥƍÇɔUsۗ/_.XÆ uɗ .)S$Y~ p!@[dwo{ ce˖͎YNJҙ3gmwqqQJt%dɒI&HP``*TG%د\rڹsgnRիСCfK6~Wխ[7 )$(u1т$8qB׽{Y[o=&C ڰa/E/$2~x$WWWm۶rww7ɓzwt{ 8P&/^8{/ )E/$pYM2ETL:tH۶mիyx 7OVݺuum{j+VLVR,YQFѣ*Wɉ;,mذ!NfjѢ"""-^vޭܹs#TΝ;Z|ۧ =}T3fT…UNuA>>>ɝ&#^I(*** nݪVZٳgFK,ݻw+gΜI"C`6lM6)]tF/\u͛v u(|X@ аaCmܸ(z,D `^) / Cr')**Jо}}/Yvޭ9s1K@25k,]zլA }xŋkʝ; 3H](z$VZ:w$)cƌݻ7nB Ç:~֭[;v(&&&޹E/dzi޽jժiʕ+ǏרQk׮xkr'؂W͚5{x ^T|yܹSӧOWڴi9w֭۷o'I);,88X ֣GSN)O<&?ԤIݿh;;3b-\P=$kά$UREW 8#^vq,dɒڻwrmEڵkP0;v,kk ,Xmۦ,Ym?wUݻwi;6l`ql=p&VdddVojUDDD7n^zzw+>|X+W*U1 E/; ygϪVZVYvm-_\ڵSLLLv۷k/=WkUl ՝],պuk >>>>Zl\]y=#77޶m}R{Fe5?QƌU\=}T]tQTT󻹹r2 Zn*VV`ACO8v֩S8:~d4iHO>Qn:e͚&1R^v֡Cyh"9RFqEIO?TAAA/p4,gΜٳѶiӦGu{n={V={9rh>`lSjf͚Zzrm>T*Uty)88XYdU);&NoTB-]T111SmVϟ$կ_p*ŋu]zUTP!URE 0kݻkɒ% TF-[*M4qڏ?={T^=rH(z`0hݺu:u8...jԨ&LU4j޽}TfM.]ZO߿_~ӦMYo P0{Ծ}{ouww_A4whhZn]vY^uRHDppj׮mRK4x` 6Lmۦݻ[g2e}v ^)QH@dd7ogϚ=v&ҤIŋk̙ʐ!ٱZh!˗]t)S8z5ydy.]:Ϙ1—$ 4H޽I1J(~I7nTLL83G^8+QR?XK/ܩ@&L5iҤدկ_?_zbbbt1} 2U@-^XoҥKզM+WN^^^ʕ+*W~~S@@6mj 8G^8+˸L0a&N(I{%sVWCՌ3$sڵkU~}}o߾^z%rjȐ!o$W0 ϮKvz'cFQ6mZmڴ)ނ$̙S?&NoQ_l+:d/'C ڲeԩ|II>l٢u&cVwJ>#ծ];1...7n;C /gX8+QRǿto /)y.\[ 7rHٳ{ _@2s a ^@ @ @j/d>s[ `PLL$z3۹sg _@2r ca^@27n\/B @/dΜ9oF4| 0@E$eΜy--|EGGkŊ86yXXl=ٸq_~eʽ1`OB`0h/ϟϟ+M4 jΝ;Kw83fPttO.777]rE|l٢;w*>lyXXlz Y|G/cΐ!nݪ~;?[2M/^TB|x _THɓGRddҤIիWe˖VWUҠ$W ^3gUjd r2*Yܹ`>}hVK/|^pWI;; p2ǏO%I}~7%vǗD 3WUb`GcǎɓcΒ%~W ^Rgx!sQUR%vוeUX1c/^X={`>gX8֫N/N(xH-̠A7xCTHHN8%J=5lP׮]:v=TJQ,ԏ>(zI`0hСq ^i %r2WztAUVMTT)޽[%K4:&((HuֵI+[lN 03WR7+xC   6L3f̈}o@rs2A...qߺuKΝ;gt\"E{n˗ϢoV3 ^q:^;$b/1V\riϞ=*U@ժUK1cbbԷo_ ^u^;$5sgT_Rھ}|}})x`Z^%vz6_kǎ8B&22R-[ԡC,#W\+00;6l/`&G^8֫N/  Yf>[o%cfׯ,X 2h֭z-Ν;W(P@{Q- ~^pWɋ^P8 I&i'OiӦVz״sxw|]rE7+::ZsUxxgz(z6`04x8;vP8 Q&Lx٣Gk\ݻjѢ֭[w8zq^ oXEkٱ^*W@\B_WDD̙3_Uj,q-;:wlٲ}z֬YO*M4+*X_\|YժUݻwg;`>G_8֫`!AiΜ9(xHLppW .HgG`Ǐǎ/` G_8֫; Wڹs'/)yZj .H"ڼy>|7naÆF򎯂 m+*e`0>x;wRJɘ /dÕ9sfըQC5k&Iʔ)6mڔ䅯>s ^aXR.75wgڱc/)31 TڴiEDDEھ}ѱ80} D 3WR7֫^(xHM테т$KN?4ibѣGjРgQSNQlԋ* GHx!Wڴivx:|䉚6mjQ111;v$ ^%XW0 8p͛EbŊɘ ϫB p IDAT @jzz HC @/d?A)$$ĪyҥKM6ŻѣG5yW%?k͚538z^.a04`͟??vڥ2e$cf#1 ֭-[J*iǎjNKw|iܹܹreUr caJ}A @j/d&Me˖I9w}n;c8p@+WѣhrWUN/&BfժUС^}}a_͛7׎;MV5kThh;&IU\9b+*KLLz%K> r2V:uS˗Ν;-[6ܺuK VDDI/^ݻ[p6^pW_LLzR+GHB&((HM4%IǏGʕK:t0o~(xfr `J(z_kҥ(xHLhh4iwJjԩS}z-M>ݪ8q `r *x.]:3LLLڴi3gΨxڲe"##u1-Z[jwTTb b-֨K"1D{KJWc^(*-5(bIٝ.lly>9-3lg&puu)S&[VhQر...zAdk^` Ȧ) ^Dd1laB#F@׮];j֬cǎ-rvv\rXr|988`ӦM(S^m"[WDdXB Bf~zi/"2Wyɓ'hѢݻvz OOOۮS.^B ˗زe ɓ'cҤIf^`> &i =UȈㄌ*c_B)R^wޅ/d;c""Kze!BA^ŋgEDf2ٕ*U*ץ۴i8 ǫWKWPXR^Y/^DDDDdS2?SVxq9rNhMպuk|RfΜ)ܢE 9HdSXR^Y7^DDDDd3x%Lr Ο?-Zٳg9`H,8eÇzDy^DDDDd  6H24RJ(;kQ(p9($$$ ##e˖76mSm_ؾ};jժ]3gbƌȜi߾=߯#1^Iݻw{F+<<ڵBÇQHIdnXlC/""""z ȒXݻw-[HҤzLj#k۹_...ѿ,X׮] pIB ҥK(Sn/FYs2%KoŜ9s0i$x~}͛g`EDDDDVYkIKKԩSpBtfϞ.cs r㣏>\"[dT_#%%|e .xxx9c-Y-B?Yky5ksx@TT={")))cK*'NF:쌍72"Ғ+SIMMxɓ1k,Y9<:tׯ_Fl^DDDDd27JJ(#G0""c2hܸ1nݺ]6??? 22ك\۫]6qqqq֭?kΝ;ѦM^zeJǏǂ 9s&LbpϟG6m >D% nR^&^DDDDdu4^GEʕpdDDYLFFڷoÇsرcao}љ]xYVR'N@…sԩ;w.Jcjժ 6jժڿ0"fʔ?VZiK׻|XbmY+ЋBi&i/"2WH=[W#G 888 ڵkɓ'# @cعs'n݊'*jsD"KzEY1"""""P(,Ld\\\qF#>Rݻwzejl|խ[zRK?QYte^\̍-M-[W6mxUɓ1k,sqq6*V,c$2'WC/""""8۷D'/"2W6! sIKw .>^2G}5k4^5u*3lc$2W C/""""( Ȓ̛7o_|)XU ))I>l-+seR 6u|DyrЋ,Fzz:2ZlY^DdluBt*̙ :t(TEFF>-ze2eԫW~)f̘-[HHH{|J3gDttmr~X"""""3}bҶ'GFD-OȔ+W'N_räIdi  P+Mdl^;P(?3/2 +ҖB 4IKKC>}_Ix 999aƌ8qʱQQQ(S Jv777=zT{\t kݻIdk^Y 6h Kɾ|+WQ`h(ezE=li ;̎NǣYf*>&MU{yyI&INNF6mj#}Gdk^Y3lڴI㒃'OƬYo޼'O2777ٳ銡?GFD5OȌ5 nҸĉ}ʶoF ԩ6l`И""";udP[Dƚ땵ӧ֬YNիW/.]=>cxyy4www۷Z2"m^>xO/""""2;iiiݻ7#mFXX/"2;|||b ѪU+$$$}ؿn/FBd?q8::GڵkmذAc Ϟ=CuٳgѰaC ȴXPҋBZZzꅽ{Jۼq1+W.FDN!`gg'=OHH~^l5j @ضm5j|u~g}FDDuԩL뫌!s\6lÑ???lڴWKd^Y[nr労mٳ'6lWWW@WFt OɁWzQcEDZ'dJ%tI&q@3xA0Jv{,X@˔)mۢt(Rw^\v PdIbŊ:XkѨ[.?~vnݺ\튯9s`ҤIZ+>>-Zݻwxɰ^\zQRx+Waaa XҥK1bxxxhܸ1mۆ}ʲի1|p{f͚ؽ{7} ҂5+k7xÇZU˗s5j9Db2o7֮]gHHH@ٲeQ^= 4ڵQЋLZZz`i/"2W!_ܿh߾=;whDEʕq-Z.XQZ5uۗL撆퓶1""se2BtAkĉhڴGFDze 6m!C`С? "##agg'{_x)^}b۶ms{{{m 63]իW1Oz%"""",-LȬ^Z x 6 ϟG|82뒒!D/#җ-+Kh"ٳCJ˕+g x{/9_ ȔXۥK}vy5aT^]帤$,YfBRRRvѱc< t[H)))ԩJ̒-L( ̜9P^=^QlYݼyǏ70رcѪUM/[W޽{6m@T/ĪUPreٳgF?} ȔXߒ%K{ 6mNxok? .vj ;f! ȨRRRХK>|XږxyyyȈ 3g`޽9s&D 0vxôhCΝ!@jpAGzehٲ%?+VƍqIDGGh@Uw^dJWOE˗(]4.^"Ez^bb"zPbC^DDDDd4 ȒpB ׫ ;v4,Ǔ'OPfM|RP^YKbĈjի1x`|򈈈L|) 888=zooou0`IKKC߾}{nM|qyC""""2d^Dd18!t?w) |L<2ˠT*g^-ZΝ;y42W!""?JvI1Riܺu <L|=|e˖=../_>[m6tM~S/uЋd.X"/"2KQk׮Mׯ_m۶ٖ"`ܹ Sh޼9]fQ5` JC AbbV_x֭3XRSS1h !xI^QF!** ;wFHH<<<>˧s{NNN_ B>5aEDDDD 9"mX"='dryŗ+)) ;vdſӧxGիY+o|Ep^dRWm޽سg͛7޽;n޼ ͛7zoumTV 6mһxO/""""Mrr2:wGJxℌrǗ;ۇ?#3/[,XBz<2tWnݺpv튱cǢ\ry&~/e˖8x d/3"`2oɨZ*>|C\\>}ŋGzz:=̙za߾}ߐUŋ@^DDDD$ MWXXJ,##"Κ'dӱc8q(VڵkΝ;\M&BbÆ jϟ!!!hܸT1ժUѣ 0'CK.Etts .CN:*YkWӧ_|]ٳgcԩ6l/_n!Gؼy_l%Ɲ;wвeKDEE_@:t 40Xk"U ^Y-[O>ԯ_͛7j*( 7nA}yu͛7UkbEDZ'dbbbЬY3/ B޽WXv-s<ܖG۶mqỶ+nXh}B|7M<*"^t IDATY+&O3g`ԨQXlv'''lٲCR4oWWWtIe1zΒ^#kIMMEqm@ձg %%h۶cpfhB~^1Xk"F ݗ|ZJJRUDzeY.]={`ٰw ۶m|YFƍ,7$""""d^ǎe^%JȻV?={ETT--Z`(]t^Hv7 'OָsرcKe˖ԡze*AAAׯ,QN}ڷo/;Q_:u¦M?~iìފDrbr/uk$''v/HDDDD6%)) ;vT j֬HW~p<ӦMS{M9D"„̉'hӦ|+h ݋^z!--M6.]K~~~ h۶-Μ9c⑑1B2w v͛KW˖-~z>}ºu0x`xzzVpp04i'Oh1c`aEzeN9m4QdIqѤI:yHLLB$ڵk% 3xbѧOQn]QH1gϞa l^[% 3,ׯ/vZǦUV2ea͚5EBB6l4}pttosZѣG (O^O|A___[շo_# 1g)J*#4cEDDDDdle3gtؿt% dpTzEʕ1tP}_3fDNٍ7_{Z2MW… z}aoo/ &|-}HcZ!xQ`2_GŊ3Y:tqqqZ[bŊ"**ȿzx|hU/ &J,)֭[p'd&NvB BTf;~RPls V*Ν+>Si[۶mMRΝ;wT&|}}ʕ+T~A*Tze ^w;v^~WϞ=u8>gggq֭\9y$/WkYzojԨ!F!ƌ#>QV-é*UkկW^^B0"""",^111y=4Rpp}gy=" cǎihx IIIK?6=z$˧$… 2NrrZ;OT{׫WĸqT`el^|nݺ&b ,UO< ѣAc}h۶ƿ~x~rr]6/29+u.ŋ`ݾ}[L4Ixxxdɒ޽{Z[eB؃@||<>cKjժÇpy82둔+WbܹqQ ~0JD`ʔ)={ hٲeJ>7n͛7k֬4hm{QHm6l@TT\\\W_iL29rVc+V ձ?3f={ƍ/_>,XÇ8mΝ3ʘI^LׯQZ5߸w_ܹsuԑźut#88B@R o>L۷#::Z۷oŋ`h i|EGGk׮HJJBҥqitQ~~~3g߿ѣGAcO>\Э[7̟?_+",, %KS^nDDDDxݺuK$e'Xzu'2y s!~(^(QEDD___wPD qΝlU^]]tɵgϞ GG ȫ )\\\J{hdzY&@bB>o߾SNjUT~ʕ:ѣG2nR)zowʕ9?ɰ^Ç /^ܰ0Qt?uMJ(Sʹ*U2+21""""qqqqaÆ (>>^TPA\ZJ־21˜~왬gn4ɫ 5kGGG}vxÇxzjtR5;`?߃+K.W=M2E=3F6D|YN /==]dddhe-[S 4ڵÕ$%%:udСl}ʼݽ{WmƍӫѸq͛7kVr6ٴxkgΜծ]K,00wUoųgd+%%E9##6lLk֬A?6QNjɝ9Ocs j ϟ?'f͚믿ƍ7p1,_v۷/ .m oÓ'O>ɋKctըQ~< iiiZ pٖvssCPP<<7d_6.oooO*۞>}S(G$7+'dooʕ+V|sN|jغuk0""""Ih۶-2 渿XbUzu瑑Xpl@ѢE%11Qֶ 9s* UTGrr2\"=vZ!pYxzzt ˗WhQh|޼yq_|ƍ>۷ԩSqq9.\ƍ 55Us˔)|̙]CUU*^O +åcҤI(\0+ٳg/K.!22Πl2:UN_l@׮]:zٜٳҶЫ H ͚5Syf͚ٶ͜9ϟKDEE.:y9!w/OePQQQBH7lؐMvޞry7n0vuoVXj.\(}O?xصkW3FNdd$V\y*ϯ\y4駟l+Plip ]tO?W^!::SLA׮]e n޼)پt8o-\0s6mڔc? ӪU"V^VT\Y(JYٷo_{-KۙN8idmzY1M׫Wzhg ѫW/#GLprrZ}%"tQNiB)S|~Me'k8{ptt*zիWe闲k߾}߹l_@ӧ UOIIReNʒxϟ!V%DRRRgddڵkkvvvbΜ9zqd' |j8wl}sݵkΩSd5ee\xe}x{{˗~e.+@̝;WsF֭[G ł ֭[O$c2_|)ѣG*V/6x\%KD6m n+1""""" ,BM4qRE+%%Eߋ!;;;Փ- 9A bذaFkDD8w]ׯE``С(Z...aÆok7oVJ*%o#Qxqf9z״ >ʕ+'VXq)fmIm-ZT$$$zR'NzE}cƌ믿eʔWPA<~ؠA+u}QN_X&N}Ϟ=+a0' īWDݺuU>6nܘdZM|)J1}t'2NNNbFzdmBwxTV^ĤIĺuDxx,KӮYFyvlBH"Kooox 9QFiGEEGx$^}wïUV~mVj履~ WWW@|z_I-+T*G}$J*|ĈbŊXg+JѲeK@ԨQhK4C/""""+"ZO| !ĕ+WDÆ uV8{^5*k1$ʜl(Zhڴ2d;wصkyf~oV]t%ۄǏM<ݶm[G.X s .,&N(>| i&i"핸Ν3ůXk2w h]n_^ZܹsN;{ݻ<^/>{dtWmժU*>ׁ}5G?Zy5 e%^DDDDu>|.Oծ];ѽ{weKTcǎ 777m;99͛?4$+k7V^WtA3F͛7?CL4IZNIa^*J.;tO uB ݻwgkwذaZogg'|}}E^D߾}EN >FgT^z%\u֕~"44To^ ty?CB={ژ={ίĉ?9֪UK y V+ ~>?_IMMGΙ2e/l !"EGGuָr労q Apd/_bȑ9s&ʗ/HOOǢE0i$( 7o^fdd͛|2ooo4jzW IDAT"uLٳgK-[7m4̜9S>_~%JFn݊ݻ}kƍptt1bIs׮]Xx1bbb4SN~6RcѢE?bp[?RL%,, ֭òemױpBlܸ:]|yЍ7PF ( *T}4*UO>U{%K z;t+9y6mV{Q\9|r>}y3g4+#2+cǎ\>W|m޼{ 4^戡be>v-M,k|ݹsG*UJ\zU~__dluBF%K޽{ ???0^˺tڵks=^PVZxe:~ GGG1a.?kZׯs]YӣB b9} jѦMƪo={L5uT̚5K>GGGT\Yk֬slѣGz*\˗/̙3x𡴟2޽{ѥK̙31e[j o%J+W;w ""BzгgOAcGPP5j ,,̠ϧ)))(_<>} [n_+VĔ)SЯ_?888H۟={___$%%@xNɈ 4hlق~=z@jj*vڅ@=G{|esv܉K.ݻx9 .bŊN:h߾=6m '''C_yԍlWx5mڔWx7oވRJ\a_K.1u[м,~YTC &ӧ+Whu^NKf>˧['O^28qܽ{VxB.]Z+'22Rl޼Y㣣EZ|C/""""3)~2GIl޼Y+SnWnDRRtR-~ M(uB&...-M>]vr u׳gDɒ%xOzO~7u W.ǐIOZL-keoo/n޼)k .ޣ}b̙p:_UT[lf͒M>|(ǰIG\L)k@GRR(^ *v؄/HWȺXELL쯁W$ʖ-u NiK~ ̔Yf ذa4JŊe2|`E`2^...m͘1C6H>*U '''lxerttϚ)xFY"pR9{+%%E,YDx{{~e>&L zw-[=zQ@s_^z뒟x䉡/f؃΋/ЪU+\vM֬Y3߿^rq_DDŒoݺu zln:888e,dLٳgK-[T{ｧoXbZ9sLL2E |'رcG}=z4NhҚ+S B~=..}d﫯jl`` bbb#Fݻ?s_K,AttA%U8pʿ7n1B(PBÆ u!!!UwG}Oܶ-bEDDDdf?VZ6^^tAAAFnݺcx1„ݻwq!̘1CcȐ!:MfiۥKЮ];ڵ+uꏴӠA7nĖ-[x@BB3Gӱze LG/z˗/ҧ;Əx5͛NNN4hnܸ͛7zZ4^"C^=l[p|FxB Ç5_nBV_7oD&Md//5#"""y}͛sIC 7dQ︤!.P(TϚ5Kz֭BT*ŋEƍȑ#ׯ/Qzu:dkE_ӧOӦMS=z4{*&M$J*eВ^*=~xC^iZ땩[0G֭ٳg ;11Q+VLZT*Ş={|՘ 3>Xٳgֈ-[,ix_b߾}"==ٻ(m. ҤXbcbA-F5j~1jbѠhL4jƆX5*"X( /ceβpkKvfΜqq3gիW/7o\~X2^f"!!kԨ˂UVMx=c˘ӧxA(2r;uǏ^ߘݝrQ)Be?<ժUXme2msJ*x13ZJhoɒ%z > ^hj̙3ܹsgE ze^d2/^\e;ʡ)Rxyyq||}D LixVZaJC r!$S FEE̘Sfdd()  ;;:ttB'N "eI$WT9lllhǎԣGJHH ???ɳ.22wP;Rک*{ה9իhVU~46;F7vŋjcܸq³V^Z-ZЁٳԹsgu}ʕW?ze~R)yyy\߹sg4TET'NP2eDE2uPة+--] R~}~32pذa񕝝k׮~M/0|r\\+VL܎?׭[}3f ِ!C4^믿* vssSAw[N}5djLP ׫$˹m۶LD'_~ .ӧ?u痿??^B#F0\=z`T&M2}m^=zeƬIII S\+[pW^LHYծ];NOO7u@sb_r7oŊ㤤$u۷oGFUd:uwޞ;5j`GGG4jRRRH222^{~{ T .ԹΞ9sF! Pv\GAW---gϞͯ_γ.==ׯ_իW+СGDDhݗa1o߾ `+++&"H$uVQPʼ bccgv!B/yBB&E /&"6mGFFSd6lؠw^xBhhK$?B \J<ﱲgiӦj4h/_֪/ׯE ZYYիW>e_̕L&{rƍ ڵkgΜX .8p>|8 yd\D j1c"B/xYa۷o3g"b32׬Y DWd 0 lZCuKxxQ~~~ Sk֭k.Gy۶m,նyMgiZӓZ6M6FPꕥq]~v޼y¶}dz0eh߾x]x?ܹm+99SSS0`/ G,ÇyŊܫW/c___8p XoܸUb_ׯ__lTa y r*% qF^ ĉ`n͛\|y([j֬ϟ?+ ʒEGG44JNNf777&k׮tze9BCCUr|EGG+wy]|Yx+/ '?^ӧ]u1!777̏(D0 r93F޽{MMqx(ɸUVJߣ!CQ͛ӧD;6+;ްaƻk-u;w9sL}JJ^L_Zjyꌳ3CСCLD\t<{Πs,lzQ\\0uVұcG^`ҤI{VlY^x1;w\5J+wڥ8ASeWBd "E0OMѺ}MСC˜m۶֗-[T* ںu+)SFaM2^J: ̬YhFr;w\.71ݻGVNJ'N`ڻw/xB$ /4rH322[ntĉ<233iȐ!`RUXʖ-r}Rn$Hhȑ#JOOOF>r)11Qi^x{{ƍ)22ƏOvvvz*۷/zd_PJJ uЁvMŋga^۠A?T~[HHy{{իiŊԠA:~~Ȑ!deeفԩ@A Wq/ r 8p徒S;#GUxϞ=+Qa+O<7ݻw`ݺu7j(YveFv` 5k0ŋ!x֬Y\hQرcZKSED666gQl߾]!⠠ j>}`@^JNN`.S[P^,Ⴈ޽{1mۦԬ666||WA  3Çr45a@WquѨQH.4a6mɉ^|w_e)wN:ENDDTH}мyݻw:M)SL&ӸD"UVѨQTnFDDdccC;v=zܷ?7nз~KǏsu҅ǎkhf)wrOiԨQvM63.\rи8>}z~([ڴiu=dr:!?~hipЌ^eo>^hX>M2%ߏs6m{yy+7i҄.]ʩޗbbb^,?֯_14bf`"bz d^DĥJ2I&1D"ᐐy)_c5h qu^"ҥ ;88xe4_C Q|effWD fz-+ˑ^n]#:%___zZ@@޽SZF)|UZxS>ʕ+GL};w׸8;w(]?yd RM6tY^ ͛7yޞ8@mڴ8/y3ԸqcQL]zer.^+V=xJ,)qH[l ҥKDkt;K[FU1B:////vLaP[z@… BUBڱcy^|I'OTѣG~F9pVTɠVPΜ9C͛7W~͚5 $[v-/>|XiEDN]t'Nr͛ÇNoq^h rY[[S͚5^KIII&z;>|(j^P֭)&&FxW^ mR:u>mڴISRR`kݺmb25jԠ:uPfҥKԷo_a֭[Ӊ'TR-"]'!9tEg777:zhgQxx} FݻwGeT=zmөk׮Muؼys #;;;۸ѣGx9+m^Y ym˖-"1%SuU>0bŊQxx88q><===iƍϗ/_0.h߾}T\<٣r?[[[ڻw//̜9SeVZѲe˔sqqpjذ(ER4A*W_ѱc n?%%_""*Qzjٲ/EL=hтn߾U{_Ν;Ӂ $6 D*wЁ+tυ?_pAN*QLڳgY[[+~eюiժUDDTjU $__tA1%J/_/czUqF/+RЀtn7>>իGDDD'O6?^"___a:"^ٳ4m4w}8::җ_~Is%777{ 5nܘ5nDM4%J==xnܸA YYYѾ}sڝ> 3j(Zf խ[]tzяըQ޽+ܱcG:|hz /_^b())/.1@ 6lm۶Mz1K. Be\^ :\.UԮ];:~m+ R222ϏΝ;GD.2eʈgB/\T^۶mS9 ЬY4NVT)ڰauQv߿OC !Hhʕ4f 2ӧL2TjUx UZ޽{GDDw咽={jߟ9ږ-[zcC?S9==]3Ә1chJӁM6:[jܸ1LԨQ#C J^f۷oM6њ5k0+󖖖FG7o5ժUNۃnݺj^:OԥKm=~zAW^^4hmذA``fﳻ;Ӈ߿oꮁ'NTx4-ּf7o\c|aĿ(l;c|ĉ|;ɓ'ޞ9&&Ơ&O,G:uX.kg̙y~ҥK%k m.N&СCUX>u:uE; ӱcG&"vssM6;&ze޽{sŋ+Vl e+++5jīVDyߋp%[n^!D"˗k},^x1;88|J*qDD1LNEDܬY30/^p>vҸו.]>|hPr?~\h~YXi;;w"2"S+Ntt4(Wݻw甔Sw ߠ^ԩ{W˖-9))I6WXwŋW+믿o H^*11;WWWyvvv_111~1^zrw 6T;͏?6uW|;vL^9KjѣGZRT- j0>^PEFFrٲe>1c{7l0~6AAA*\\\'?ޠ ,c"b[[[>tmr۷o9<<>fW^||} $+󕚚*\$HxʕJ;u+WNK.ƍAZ^[ .x ^P *Oe7n bUч>|X^ZhK+ggϞq`I&ZߙveЮ]zJΝ;^}]H|yNj3 4ҥ0 SwdPӅf޼yj}1רQC+ǵkxܹܽ{wd\2wܙgQ ƃ eLfkkw g},ZHЍd^VZ֯,RM4#˹o߾immͣGXmm߾͛7War%w˸L]@Ç3ч;`/^h yKJJ>GtA;MVZ_`$PDFFR6m(>>^x?͛7 {X|9M8ʗ/Oqqq:Mj۶-|2:DB˖- &U̚5ϟ/looOaaa/?uPzz?@>>>{˖-4p@"";w}222hgK$_>k׎ʖ-Kŋ8ڳg]tʔ)CN"OOOΥMҋ/ʊjԨAo߾*֖KW.S>}T...N 648hm۶t j۶-;v0 +O?7|C\BjDPٵbŊt)OE-%Sn޽{\L;,޽{XbzTz@Y;T-[Gd2]ކ oƌz;gU?|PVԼysu3{*mmmŋ}<˸L]@;Ϟ=UlM޽2u:uJx~wKHHEqժU5!Cpff@W!==L20MXbLDsiҤƟӾp-T-%K1cf|5mW>1u9rDxڷoq/H":hze9UI$yѢEݠޙӣGa5k_߫WK.S{ëWbJC#2ud2nѢ^uƆ |Ҷo޼Ç?W+++:Ŀ;?^6˖-SZo߮L&#0a޺uK &M$i CY5tP^… ɓLN:53g/0:SȜg󌯬,ްa(Q5k 3'TV899Ys.LBzeTQvmNMMU^&oƁrJ~}6A7of"RJqݺuu/9 wMa 8p 駜>aaaloorcǎZcӦM}U 9q{SfM~(&&& w{yyy ak׸W^.t"^n^x[r>v3vݻwÇowѻSLa"L|dz}G͔֩sV\|Y(ȑ#9X^`^Æ Ce޽Ea'''>{,337nܘ_s{gΜ-t=2/"Zj~J#Gd[[[ïF ?+WN;9Aw.\`;;;Xw|5J?g^z|H!!!yi„ Z33]tIɓ'vZq:JPn(##R)U\Yu}xٿt֍Onq@5S+PѣGԨQ#Zv-ry=m޼ﾣwމ/LFÇ,>|8& IDATى~ ^Yw n޼9]pY`N?~n̴f""j׮ e @W˲mݺUt9T߿mkחٵknZ|W |ǵSgٳgA܍s1c+VL;˖- $;;LJo߾̏?z펯L.@//<Z*yFӄ^z?r\+?^>3~hdHNN2eʨ 4/ggg:t(Ϙ1[h!ߋt2ܿӧOϟv܉' /[L}^~sŋ4xsA{gϞI&qff^2jUjU!lۿ|988[ =-P f͚xh ::Z-Z͛7CiWrr2Khwb:@Wƍx stt4O2Expl"( F/v&Mh0buN>^`,5Ǐ7xz m:6lP|U,]Ԡ#S'''u Rh&˅; ?]Ɲ;wV[?۷opG3sBB0p{)^8תUkԨ666 ٹsN@ze-Z_9uHGuq?.\PyCfdܴiSsssS|yatyf^L= cH{#G'OG||^ܼ,ްapdɒWi&+X; ֹ?S+ΥKՕܹ۷oW;u쓒~~~*f͚+P!_^H.ZhZ1a^z5;%\.ZjiCW *24 c&".R/Z߽{!!!,JU~94iׯ/7dDvvv#G>}^ᗝw~}B;s1ܶouQ%K~aֻwo&, }ꢦjժ+3e}Ndz[|ڀJ=իp@Y蝙_|} ʼd2g%ݛ% iF޺uK=<==իWB/0[׮]CU@s=رcy֭_^m5yd}^/e@ɓ'x ;88Dʦ#[`(m 2_^K.rJ *U(A@fe{X933SvmVVV*$M8Qf͚ܷo_LAə2PyڌU107ol1$z x;wN[ ׯ_k|+WD9VVV{zz L *ltٳ':t(Ϻkr"ETֲƍszzAP^͛ YZh/^й=1JJJ^ 0 sU TB+[R)Ӈ###=zB}u ._a!WFFK㸻+YsWŬY41B mרQCzN8VVV:}z1o۶M<(\l۷/Q SxM8-KWL*U׵kpWUa}6 S~ݻw?sů >NNNLN;ooon֬׮][PӐK.+zu҅E>k`.|ʒ|+ʊo޼)1NhB ԫ5g/(ʊ7oެq4_&L~B/0+W PS,ڵks]!*^^SVi a%K5Eadz^ݻwyȑ 2) ۷ogkkkJ77o֧۹uF cccWZŅ;v4A\,ь35koߊrܦM )SF6tz޾}*T:Zp~Zc! Lʕ+'7yd='NL&S&%%9ƍzqTT^|9::rxk : |s9sf -ZPΝ;>ɓ\H*\ڷo) rdeezeY?Ύleeŝ;w_~DcΝE]0`fE8ݠ^ի.zj9;; t>֑#G>S KtVX! XrLMWdJMM`.WNWo߾>˗D[li_`}JDD^^^t *S}Gj@p9^$ˉtRA3gΤ&MUX*WLTzu߅UH$ԴiSx1`@0-Zеk(00PٳgCxx8YYYx͝;f͚t]vv6ߟve10WC@@޽lmm?py{{ɓ'5{:txUPA9P̏T*%///;w,zEDԩS'ڳgʚk:u/ B/0sΑB5e^ѣGŸ_x>IIIxb-Z… ŋݽ{hԣGW8g^` g}F7n{Ul=zN:+nܸADBwܹsiJ!2-+ˑMoߦӱc… 3|=}h*xk׎oNDرt;!^RJ\e )ʁ tf`'''9LbnfϞ-Vyذalggӳ} ۷^`TxƄq_y~}||Lݽ#w=y(md2ݻ7eFP_jj*]u癶988++Kv5=+͛ѣm6ꫯC%F[@2wm۶͘޶mzyyq||ю B/0^ˮ]` zjnٲAˎ;ah0 ^xsJNDdnR(T_r!2W...:}}iu m/uK͚5F@2w2K(U0f D#af&EDD? M:,Yb^!222t TVR*VHO퐐6lVVVqF߿(ǃPӧOo߾t9Сmٲ/vR^4N [boɓ'};vk׎iܸq?|IhҥcxbgLi@CUp-\fΜiPգ!CPvjժLvoVnݢڵke0 Mf8rHEi;++J,pǬi֭ԧOQ ʼEGGS6m(..>L-[$gggJHHzVmUR8@^^^j&j߾=թS7oN;vEjRzB :tprvvpjԨmذ*z𕒒BRٳ'ݻxLz8gϞ3iL-ɻwQF:O#J[n?l;::˔)#v|<3(0DӦMS2M~~~3T}Lu(>+^:[YYo߾ͳyʔ)󓛛ߺuK5MuئMNJJ2i(ze9BCCՕ/^Kxuhvuu3gYw ^ 3g *={ժU*첷篾JgF>|XwٲeF>(0 Uhhhz\xqnݺ1qǎy֬YydܱcG&"P<'Lj?K-[N߾})--~'51cU^ShL/0‡Y(={FDTlYjԨf:uSN2P,X<תUTuy{{Q`` mܸfϞMS5P``N*U ^L򢘘jٲ%>}Z6޼yC{Ǐ&((HeM'ORҥu':WX:|0uԉJ.MO>U233UB9{,SZZ/:z(=z"""(%%ҨhѢ駟P׮]>pXl_ҥK%$$A : Ȁ%bfTJIIUR DժUW^Y駟ÇK~d2SNъ+믿֫ Jcǎ%kkkڲe QժU֭[J.P,ßIC%"-[Pj'++ wxT;!{GAJP@~"x$(Cӣ ( M"R DE@z( ded$3ILk3$3뙽s/_>j@ٺ"h?CpXpñc̋/hrֹ"""L.]̱co߾Fҥ_sL 8p Xb)r%̓O>i-[f\n?mڴTzv$+8 5GMu%44̟?듕QG-N;vj$o73p@J_+Wvy5k֘ Cz\}P2כVZ.TPtl޼٭=W^}hzKhxSy=)y%wf&!!繾;bo1!@ozmܪ5k46lp9Vddd۶i޽;</^4mڴqڮk׮>ޝ QǕ+W@tA;d7UThw_1& :Ԅ`3uT<dmԫ믿2O<۟7on>ӧOOk2ӧMBc M/` 9szj֬Mu_Ǐ7qqqng[nɟ?d:ug Uz\CBB;c_4ޙ3gLJ6G'4 ,0Ν3$?ȑ#ft)9ӧ / W%&&k>s^V5+fLٲe$^p*Zvˣӻ(P|g鎟^3_x4jqR%J^#^ݱ~o̜9ѸtٱcyL*UPTZ5sٿLɒ%Ͱa$%%':oJ2Ok SD kMdd4+W6AAAu .8~Sxq'r̙"_|sIJJ2< /PϪU&O쳱] ?k|ݸWעEK.mZje5jhnwܹsK'ߐ^RJf„ fԨQ)|ӿ z5]n[v- 5{ldiӦMͥK|%KL@TfG|=zԩqaڷooNj~Ws1ￛ)Sڵk;WJ- Ä 8*,,|)0 IDAT=xy]gϞ)Lrn|vd꣏>&U`3gӿ?OgϞ_ݻk|W?;/Z;wnՖ-[ZG|/^<ŗs_B+p;vΝH2 4pZj"-?Lwr]v)={y衇'Oӯ_?CÄ IRRcٟPߦ.[t]wǏɓE]G?UZzӿ]PP>z.wwQ2;*Qٿ={ qf͚W_O_aaaf͚5>x@hzt]4qcw,c…fӦM~_4Lܹ=+[lI3fʕ+e˖nS^=3asY_}o= ɬZ^jz)+U涇r,ub 7T7/SسgO9uꔹ;-\b"##9++{?qTcPPPG|}w&44HKD;vf͚}QHH>}Okٰa/znw9s[l߾ݱ7GxGzV\-[*..q5`SeMpyeTF Ƿz6nhy{W4c ܽ{~T۵vZ͛7O=Os!{d[lq|ڵti5nX{ŋ>*"""|rrze+::ZdI 6UV~WJr͛PB)nOHHPΝΝ$+V̫#8[Wׯ_O111jԨ.]^1cMm\B2+WUVN A6ʚΟ?SNzѣGzox>}ڣ% թS'͛W۷Wxx׹H3|7&gΜNKX 4XYʱc;c~#GḺt$$$x-[8ѤI>M6x;w>!)),Y4iZ?n6l^yǘ毿rSNի[^2|U^XCvXƌ3+VL\֭͹s<Xb1-ez彸83ydSf4Bxxy_ݺuOJJr۷"EX^0<88ٳǧW_uGÆ }:>&d۷o76{nN;)mcq#<3oK>Ku2]7ovٳ 8]nvn:\| 1oS=X?ܣzSܾg;wN ҪUTD ƽƍ+I*Yqo;wN6mryŋh߾}}Scƌh̛jJ?xKk׮Z~J*e9wΜ9G}T;wtVxq :T111ѣGuEm߾]/rʕK,ѯQ2eʨI&)nߺu_O ,PXXGlʕy/Lx:tݱ]:^:uK.ٖˎozɓ'W'LW^1Ԁ3 _KHH0:tpjݺuGSƚ-[ '|>,\дnڄ{dD…ڵkFdFHJJ2_|T[Gxlذ^yܹsVZ/88ؼ[?Zjiւ?<_~e5m$%%Y}իNuǀB l-\-[֜:uHo߾~ 53fjI{QboLג?4I5rHOLL4Zu|rꫩYvms K.Zi[^yo߾}{==EC GiӦ~m|ݰuVӧOӨQ#SlYd (`*UdZnm>cs H;/^4<˗/YrG?>MRRW^c>&>>q1_~q"""LllqB ld4Ne䃏??ޔ,Y2 {ҷ9&f͚Θ?+ߩ3{lHLL4QQQ.kjb,:w)^x\rYɓ1F5,eCW=z sy5m۶5|Yz_̆ ̒%Kȑ#MTT)RbfΜ~ndƎk1f~k|%&&Zw\b~aWpauVǹpႹ]֝R_~%#%z]vy4q$]Zd$^d^ݱ!C8^H8Yti_nkTHeRĄ o*sZҥKxx\p NsSNNX]~|G&$$ir6zk׮.]8^9s7|Ӝ>}:m_n-Zd6mꘔ NsO?ԩu_'O47h;W޹vѡ 6r)MZ[/9s47'OLsXK/}ٿl@F@6|C[W^u:o;R Æ K$SD 駟:iذdrՇ=VLF{勣Sunݺ^9ttkuPPyG͈#?~s1vZ˧橧"%n>j%,jG5.?ݗ$SNS@sY +$%%g}͛'{W^<^bbi۶m%SO=e&Ol~W~sQb ӷo_> d^dq˖-Kz펕̝;(P\vH.<{5'N4Ǐw>11,_y睎 onń aڴiNGPEEEl *8>^ٽ{wj5k/f+7-[>|1=ܗ'Io߾^eENRRٳ_DzGy?ޫ1\b4h<; dY˗/WTT^*I ȑ#էOe= .t|})4448qmۦnݺtҪP|A5lPe˖?={(88XcǎO<ɑ 0@v\ϕ+.] ژ nz畔nsޤI?~  Ǫ^/_Ʉd+:tbcc%It42_5nԞ'Ojǎ>zc^u92}s~n~/q|^?\C Qxx1[oiܹ :Qhz-[^4c޽:zlLRbbt[ǎguU9rp{P ڱc6mȦ?Ç_~.\gU޼y}2$mV;wT.]<:zPB8q/^,WrUM:UTdI6'Jm۶9~>t/^yF#nXd>cϒz"%DGG+&&F5hۊ+jŊ<d v|oҥNܼy;ݻݑ6mdSP!s ٿ2d^CBBL {g=Y瘀?z. z_7Ιl۷ה,Y2ZÇ .-GvF/:8izDVk֬֨Wqu/WP`o (`$%Kx=6nh^x7o^'w&22̜93ӞpGϺgv˖-S6m5jTlVr1'N)IJKÇkҤI)sw(::ZѺtݫC)w*Qʕ+R2!#n5h IJh>|X֭$uE+V4;*TCjС:|bbb_)_|*RjժE#o}k֬i_7 0@o={ŋׯ}%IaaaV^yˤsͮ_zJgVTT-[qKvڪ]&N{*&&FW\Q|TD ժU˯2J1xo…zꩧtuI42R5d5kj˖-6&r֬Y3-[L!!!ڽ{#L/f̘{έZ,-۷ٳU`AܹSԦM{nJLLtL$vmZ|A̙}?VHHO)SFeʔQڵUV-=>} FG(#l= Ă LhhR+G;Vѷo_e!S]J7/اO,>}Ӓ{y'LZ]H:;wcbŊEB }sι8Stifk奁WQ^=k_tDxbir̙cDDݻKaj9Ps뒆7xoӦ?ᗥ{ؾ\r&wnՕ˛cǺݻMHHM\\(4pO3fݱ[?`J2'O;ŋMXX`'&d_|UZ5qFSP!5~t'ӻ.\،1\z5;F޽P'22'c? >q=qfӦMn|z7|5ouϟߜ9sXv2=z0yo1""|WnƗz@ZWtRnMbb̛7'e˖;'Zjŋz뭷$Ӻuk^'  @͛7/Ekرvʖ/^GPPٺu ȶtR bBvرT'nuaKGs)S1b9y>]fڴixܥK|5M05n׮[G*ܐdkӤI䳿Ðl22?>IP3tP/6o>׏?u>Kٳgwm$'c@@?:tׯK4f IDATNpoÇ\r)noٲ/^\X`ڶm+I;v,+PTոqc>}: 0@t\?r6lHʕ+zK 6T"EtI=zT?,Y;vXʗ#GYrȡUViΝϞ= Z^e WwѣG+$$$Ξ=Yfi̘1ڽ{_͛W?Vco߮h}~ռys_իWK>}ϟ+!!!DŽjيrܶn:EFFҥKڵk_W:u?SGʕ+tR=zRիWOժUӕ+Wxb#7o27W_57˗/OEXvZӮ];}qiذaFOy!b޼yСc ;V=z9'|}O?> N=#I7n7;A pwׯիW+44}?^|F+Wx}uV/_m6}{9M6o?ׯ4i 1O?+xy|-)UR7{W5zhK:xy}JJJxUjƍ PW׬YTϱkر?,Y$A^VZ"EI&YjxIMCjϞ=z#G-R^~@ׄ Tn]۸qcYq^R.]hN/^̙3} z9mVfJ|7.\XÆ kƌںu{1ZVXA M/2shxqN;zϞ=` Llݺu`կ_߶A ҥKkҤIQ-O?v^ΨW˙3-Z뱂ԼysmذA~~aoذ{=y UnZd^VXUVVZ>ETT) 7 ;w~ ݻۜ nݺD>ѻwo}ᇎ7nHK؆zxh|_)&&F.\PrTF ^dB^ǏWnlN?~\w}Ο?cBBB4i$uyhʕ+>kw1!$&&F5Jջwo͠TUwY͘1C˖-Ν;ui\rUV-EEEf͚~I/:ZJ5˾Pׂ Ծ}TW9rh̙j߾}&^d2s3}Ժu4 ƒ%K4h IRկ_?2ȪUիW{d|>}dP*xzw:uzѢEmJz5iFfRhhhILLTǎ5k֬ Ld}4$nmxkԩ1'((HSLQ }ŋu_~iQڵSbb$i*]O du4ğ_|Qw}FӧOMBBf͚jժw4J*e˖)^=&d]UVMVroz\\:tu:"nA1FT^=5nDWC6m_($$$$&&SNZ`A9vZhG#&%7j֬YرAqڜ {nh<ȑ#իzHUTQbŔ+W.]pA֭[W_8z7ߨbŊ|Ƙ۷q~!7|࿿gcuQ۶mS2eo>;rDK.U%IaaaڴiWns*duԫgj߾㋯j)kuEǏ?e˖;.hz`iӦK.JJJD+UΝh"N:ZhJ, `BU]vʙ3֭[%K(11Q ʕ+ugPz}c||Tn]ǗƎ_~'c^e_4$qƊT\\kOʕ+ ^dS^pjxM:U;v92̙3̙3>788X={￯0  5jȭ%ho?~}7Seoԫ#))I/ .>׶mԺuk:tHԳgO9GIרWX`ڷoׯ{ݝwީ5k֨TR~J>^d ^̙3:tƏxǫUƍD*|  u6hx*{9scG}TSLQٲe=#!!A'Oo+WHz1c8-M  7xO?U׮]ix5i$M<~z7լY3-HLpE˿Wُ1FСC%Iѣw *d :Tv풔w;ިW/^hz\5Mgyd,1ڲe/_M6i:~PrTV-=j޼J.mcjdULۼy4h8s/^eocƌkx-I˗Wu=@ʟ?N8C)&&F?rB4uThŽlzL>];wN~x~F V%%%… ו?~!;`B\5dȐ/$iѢEСEUhQ?Q{L+WLq; /s,R 7U`A,X225M6Mq /^֭[kɒ%ʛ7o͑#W~[>c^;ѬY ;YՔ)S/y)ZQpƍkݺujӦۗ:2ed`2dw+eZ%^78 ?p>}: /2wVXQjժi֭j۶˖-K zO|׎ixޡM4ydKN 3f)1!xիzuy^~FԳgO\RŋwިQ#S!;^Ǐ׳>+c /Ghz#5:t`s2H {}~^~F?nݪzH9sOd1v M4Iݺu 0!Δ)STZ5^~B_UV)22(WAk׮4\5fΜۜ RbB@^+IhxȔ(W 2O>Dݺus4BCC5gnd2 @^@đ^x@„ @A(Wy1 / z PP scyC{N sUV6'(W 2? 'Ntjxi1! PP ^cȑѣSkΜ94dJL+z Hȑ#;8«e˖6ט(W  Gz ^ 2 @^@ #F `0! PP &7#FP^4oHo: /2 @^@cyC5|5oT2 @^@d1vNÇכoN @fƄ @AiKqqvM q7劈ɓհn X)1%RN)dk4& Le ]5@璴T @6`4p)2%ȶ F @`@^4 Ȗ zq=<<\ , Sb@^4 kbyCɓ5qDc@Ν;8B ʗ/3gСCXc_(pz-gΜ;FTItI%\`*z?-[ܖ{ܒݻwEY>}_||d*ϟܹs۶mB ٘\ٳףTHk+}[J7;!i=q~9ή]t]w9o߾]ժU1~p\߼yjժec"pz S4HzmSpNT!r BCN+]Μ;@ʙS x?$M[%&J $/#Vi`Ś5Сv[b/_T2v?d*2-7WƌΟ; hPFΞ; h@HM/HҔ)!<ew g۶I&%/{Le i,SݓLV>kWGNhz@zfϖ.;OS87O+S}jN[$$HӦI7ڝV'7);S~; W*U M/pDž ر޽v'ŋ'7)3ѴK҄ Ү]v'EHRXIenN ${:LN; T)Lxi(Y N!ii^}M/ܶmҔ)R|I`ETvpҤIҕ+v'-ZHO=ewlXrvU]J ڝwI}fw XեԸ)5^`ܹW_ٝVEGKUڝ…vU}H5j؝"ۢV%&JSJ?dwX-*ew#͘!I`UtT)%^K ]N+InRٝo/K'JfwXQԯ+I^୓'>; T)񕙜:%!>lwXQԷ)^ J#GJg؝V<+vpvpr)u=N_>Dr$eK驧Nߒ:|$Y3gNm_;Nt7;~O; IjM/ Nj԰;E NxC~Sdy4׌fL>Bo_lYS89SZo_\9Sdi4.]J>۝V,ܤ;߮\I>gܯڝV+EGKy؝$ˢrGv'+&7)2ӧ#CN+ʗ|SYM/ÇNٝVԭ+iw gGJÆI'OڝV9s4~Mcefͤg;ݻ DiDzYSd94 #]+͜)cwXѩcvpOҴiRRI`EǎRӦvRhz@FY RZvpܹvU&թcw,d35kNʕ;?*:ZPYM/HWH|"I`E޼M~G 2? N; YSz-S8OiYM/VƏbcN+ONl>i8~C U6;iӤNl&i5~A $פڵNliSKzS8ki,S~A nf͒_?;Nli2S>G nȝ;?Iv4yeI`ExxTBv'ԩYM/'7)21cN+JN>+3pA;Vڻ$[լ));& .?nwXQ۝Ұaɿ[@@ \y1Nls.\; hHzS8;p@=Z:w$hz@j:t7;M)SkN+ڵZ;[wI_؝_KfٝV!կow g+WJ_|aw +4 =Iwiw gs$7*W;s%KNXF ܤ(X$~=yMN+BCEڝo ԩ?۝L):.\Ǝ; (Q"w*($xQ?^ڽ$hzȺ|L^:~\6L:v$JL}[ IDAT=yR#QhHzS8;p@3F:w$ᇥnNihYnjNjζl&O^; h&l&}gw-4=;+/;-djNX-Ulw gsJK؝VEG'+3;WZ@hz1ֶ mo$$HSJ?lwX*Q$3F1CZ$@hz7JHnRٝo/JK{؝V-*T,.]&Lv; *^*Uɓ؝V}wߩ#GND |aC饗Ncﾣ(o"Ev.^'`4 0j4E}6ł 7XPUTDu0 3{g]a{ hIHcNJN(4XxZtl""""""""""J@~S(u XHJѧ0h֮E'!RQt Eg۷NA5 Et Ex{NAM/""""""""""yxR'{NAʚ1h@t ENA^DDDDDDDDDDRkkIdf[.NB*V"L&': 6TJ>K__t,5@`$ ss5ed$:ID^wNBĦ˛$2XQ|Ɨ:-BCE'!-Ǧ*k+:Py"*JtRFV€%KׯE'!-Ʀ $:;wuQg_N񢓐bӋ(t,:"__[t RȑΟnd2IH EDDDDDDDDDTT݁ DP?oS&MDPtgE """"""""͕gϞ˗x)}HOOG\\tuuaff055ER`eeXYY|򰵵EEXD L<.:L&eat(: ) "RHHΞ=gBd2 $$wT^ݺuÀбcG HD3~~~I7_bĉ]hD@^@Ts$Y<֬/X4 =VNLLD! Ʀi\7n@ODZ%>>'OqYQAAA oJ*aȐ!6lg/_ .#\ll,~7]]t+#:i#:%:I `?H /7czzӐbӋ\D={#G-֯IO>1|k3f'|Ԑ;}={`ݸqd2H;<Ο?F҇Djdxjhժ;;;ёH[˗|Tt,۶ɛ;NBbv\9It@lzQS+ IIIGgϞEffHE֭[4h4hիWM6#ѣʙTʔ<=卯Di䒒uM E!Ml 64 ^DDDTddd 00 ."*RSSqQ8q222DGh׮+,X#(99'Nđ#GeTXQt$6ի̙dy~VMto+VȯE! µ%XYz50b,_/^dË1ydTT ñcǴ/LM6N:سg8DDy:s 6lcǎBڨE `D)=} ,^ z%: i˗aӋHBÇGzdDFF1h ?)))͛7ݻ7VX!: iO? BÇ5@\$)?VbcE'! DG(6VZ֭[όZFF\]]51>Ct E/7NRzw -Mtlz0h޼9\": Q/_d2QHL 8:NaKouN?]D  nݺGt">>+ұIDE[Nc윩S33ibnbVQdʛg}ӵڸQ\ /7i FR{lzSzzzGӦMѬY34k 4@ɒ% U755wޅ/O?iiiΟ'O`С8~8ttttl"ʝڷoVZyhܸqflZn֭[c֬YǦMn:DDDI???]cǎ-:xP0^6oƎ D)0=]]I԰/B:tHt,=&LAFlz)IWWcSw'''tA%6I&hҤ &Ol߾ׯ-x9y$֯_o$"Eԩ:t耎;^mѕ+Wٳݻwc RsСCϢ%*2nn&ŕ+d9z(WNޔgx5u$YN,,#D'bBWt"""6l-[cݺucGݱvZXlzYd+ʔ)q֭[E6md\2e BCCl<"m-Z`Μ9z*^z={`ܸqW6>!CBҥU:ׯ1|A+wwzu) ?.:)][t Ev׬FRkEDDDY&6my0r A6m_`齎;… 8r&L0ѣG*H*U | >sCc300+sΩle˖ *(GJI@L4r%,,GGi 卯ɓ峾Az:ijRtRsEDDDZ xܻw[nڶmED ;w.BCCqy7N^p3F3?|||p)Amʔ)`˖-xۇCjLCvvv8{,O1i&'SJ&:V<ac~ԻwUǢL/"""EDTڴi7774^;v U6'uV,V#*\]]ZOOG C&I>1m4s%5Ҹ10}:`$Y""ŋ~7Qx__I| ,Z$lmE!5U|~""""*\022`llƍCZD)FpvvƋ/T2ƍ7pK%6f K^;88>>>prr6Qt ~-_ZP]ѭ[7zhӦ '''8:: M6/ּ~:.\X鉈T'66VZ""vijk׮գ"&N>zBt Ek 2CEP!f|MlzI0`U777亱L&C@@<==QJi?:t͛o޼QADDDD/^@jjdT"Y-"nޒjҤ %G$_/ۛ j{w)]l dfNBaӋ0sL4hCFFF322q㉉XbjԨ~z Dff&tu9IdJx7"* S=\N*Y-"SId-F%㴿?nɏիQ1e дZ'#3>8o길'-DPt({T@lz'p|(]k׮&_|,"""""UȀ5{S.3CDp=nZ####޽&&&%*6<=n!̓ _Fas6qwDPs'pTlzٳgVFbӋ ((HҚÆ i(L:M6ŵőҲeД3]HțmAyĉxXt16<<33I6P>ETwc*dɒEFFl<"""""dddয়~%$ID'883f̀-Zǘ9s&;;vv& w}036l@jqއJbbˁ'OD'|`ӋBBB Gjjtqqw7ũl,"""""lذ!Ǜ c000&i/_bÆ ޽;j֬  &`޼y*MT,5oL$:Hݻl=I m_,/56 ݻwَ%$$Oec`ĈWjDDDDDRzfϞ-iM##l vJKKãGc+Vѣq)dfflYfa*OTl} 0|;!!hߎEyNNȑS(zX /:Qq t~Sͦ^ *V(iM"NZZ<(Io(DEE!22AAA Ez/edd+V`E:.Q2l p$ 5=cp0zz#ܗ__GN5`F`xPt^D`lllϜ9׫o6m`aa(.HDDDDTXXb5 0m4Ik0p@1Trؽ{7Zl): 8QޤvMtǺc%̌E)8WW * |Yt,ǏE@rr=7~xڵK%a„ ڵ+Zj1 +)) CEZZujժIZctttlx`g':}?}=^e<=ZDP{7g5)6 N:KMMŐ!C0m4ƚ5kN>H^H x5MLLO?IZc4h???o033x167wG Ai':Jɯ)ssI6qVbӋ5jLCƍq"JEDDDD$ޅ C+9sFDD97n޼ۋCT|U"oRZKL*VEP \ <~,: }{ziElٲAQKݦ}s}f͚ݿ=z@1c :::Ep"""#z 000@2ePZ5Ԯ]%K,,wA|۷oqMDFFݻwjԨQp-<޽9,--ѰaC)SP#44111044D2e`cc"Co޼!C)i]{{{L0AҚDD9iڴ)\]]1x`2"I4iL,Z$:I.YVeLX!J$")%)iixHzgw503gį8Ө0c0$Y^_fs$ ɘ>}_4֭ NY5jܽ{7_?wΝ;ZjaԨQ1bT2233ݻwxi544D6mзo_ :TOpp0Ο? ϟǓ'OD(QBqqq߱eܺu 2,[=;;;`ѰWDlٲ;vKrF__-[ķ~CWWfm6ܹΝCzzzCݺu쌾}QC"<<\K,uB /0l04nXt"Խ;|85gʕѤfMԩT ml_*T)jfdfś7OVP?|̹;w0vrl:UhtDE6N%8X\ސ+[VtǦx‡IIIhso߾zΣG0m4̜9{ѣѥK7LrM6axI ___bƌ3f O 2>##wݻw1|o˗/GÆ %FDDD9sĉꫯ>:럈ׯO>={D۶ms=$4xIq$ #:6nu~j 3 [KKZZ©iǃ""p v MΪ)upן|RcڠAfd 7\]{0lzi͛7+|Xf->s*sSSSw^ݻvvv * inݺ#G͛Y[[e˖R J,x}dP;)) 111Ri6sss:u UT)tt?o߾Ir}CD$x{{c~EM"68[9~l\Lѥqc\X '~U+1DDL(6 TKKItItŦW1cƍh׮I:+W˗x +,, fBժU1rHɚ_7oƔ)S5eT>7Osmw^|7y>nݺ8y$^6lXYb> @GqF 85BʕQ^= 0Gݻsl̙3ػw/NwJ.͛7# nnnhѢ*W5kC_Ct5׺W\ӧY={fhXh?q0z͞=7nP:ѿ2220x`ܹsG7ou>&66֭zD$[[C%@*牱ʕ*ɥ͚j1F*o|TDk?+ ^ <|(:VґiؚW׮]ѣGDEE!::Hgf\\\(Ww)))r.]0˖-|icccL<ӦMR5ЪU+*PݻJ]kkŘ4iRk.\񜁁_`۶mZo}ݺuos=_n]<|(]4N86m|nbb"Zl<~)=:9ꫯecy& wPZ5DGGg;׼ys\r%2(00u} LD֭[1bݻwգ/((5jx7$Hƍի%kllW^.C{fffcHFGGO?T H 30s7x0)*gKR)5TBvvY@Ŋ0 7T:N&MpzR~~NSWm>#?s*MiL`̛7Wƞ={ .x"""bU5TbHp/^[n?saÆ4T :4[ 7ixҥKx+P z.-- Xvm7ooPX"_ /(UT!---_>tĉl /Xpa^`jj1cxڵk? o\ /^^ / +*d2ۇcƌHIIHt%IJ~A!^ X`°RW/G֭U:O@;1TcGDP ,_ps:5j֬):Z Akԯ_gAAAAСe SRR ]%ǏÇ6 ^JKK ШQ#.yH$A>}DPlYۤT)3 R8o/'R'nd9MqMZjV*zM6Ç9sJ*MNNƀpԩ|=>11K BWԩS.WXP/ ={~g5kRs=?9rAAAَ4l0}|}}T]"""^.\!y-Z`Œ%"R`` Zj 6v,жNv",5 ? 1n㔿ﾓR'g۷N54ebbkkk1FN|hTT gFpp0N<#Fz}t 4O֭[͛u666_bEz¢355Efr=oggΝ;+~i/%s`zz:NoF^ z~c׮]^ҙԏ.$EٲenL8GVMDZSGt E{GN#F`7QY"88Nh)PV-zJt d(QBt '''899!99'v؁cǎ!99bbb0~x:t(mڴ)s-[,^z~ÇB~_SN\|9s7Vzjժzm} ٳgs}:6oތk׮۷8w~GmV%qA8;;+u3+|yyBU".XxHtltttqd8桵GnMUuYpK"bK pCdjjÇ㈈ҥK tZ*s~~~n:inn^ହ =F^ )L(/Jӧs=sN<o߾Wtt4nݺ///9.DDD1֭$kע&"bddva֬Y8<ñl24iҤH?s ʥ oRWŋgD'ɦtɒ7kJ`/"]^_Sᢓh,6@ ;j'ND`` vؑMǏϵuʕ\'eӫVZ[ =FaLR5Ke?猫S*W7n'"...ϱT'iX>TI 믿VIm"n+V gggڿ?\]]U:V [)^^@q/ +U_J^7*6|ڵƎBQh(t,a4Up4^Ň.Kܻw={^£\W?~8,YP?TD TT) U5w DlHD8;;͛*)S6ч:v#GhSV^͛7t "ҿ?Яn֭#13u+Ws$)L>AS({X+FBD5LQ255EݺuQNQ4£Gϴ͛jٽ133Çѷo_>|8Lkv<444礤H=KK\NjtbF|Ϧm۶ׯJSfff$""/99}.\~-~'&"M&Mp9l߾&L@LLJ7n6m뫤>;Ot,gQ(0#G9=p"ָAKTQ(⯿{bN#^%KDŊW\Y`!!! tuuQF TR+WFʕadd333D7@}- !!HMMEXXr!R~}CݺuQY~mqF,\|*U);vAΘʭٔ "^y}}|)5mۆ*UQ"""v6lΞ=ǪU$"\::::t(ڵk!Cŋ!C'J&ݻdٻWsI|޺5V}x>>>>IyL~Oazzz]4Y!QQo>۷/vD$\Vf IDAT*UWWWYFwE.ym"&^N# l"o|A0|I^wO^)?DN&Tǎh ]n3ϟ//[2 C 6kԨ<0̟?|dKm۶?>BBB0{lY0fIƢ,;=֫W/T\|0yF\*QD猍y7o.>lῳ=T)33#GΝ;UR[nرcgիO?y!$$DDZJ>KKk(Ա#ML$y}Iss5{ӬbLc^5kvUVlcǎիَO0ׯ_G޽U~7!̙ooo7k֭ڵkU:pW@@@>,s+f?pɒʫUB4ʕ|ddd%!"""m%0vXlݺU%۴i(&"RX @ʏ?(y]"fo/oRH`b <\t y6ּFZ_$* X(4V57o.99>|8/_^w2<_±)S܇3N>"1㖖9^P/^(td\UVMq4ǚ^EL&Uv\˖-qq>N^۶mÇ%KڵƎBQh(lY&I<^PD^JZSmjN(<tItbO^ c咍QPnnn C(ʖ-_zӧOVVV9xݺu(Wyܸqc4˗ϳ9~d2b*߰aC;v,ŢEХKIkfdd`…$"}N`:tjSÆ(%,OJZO|嗢S(a+%*mze@Y_N:혋dv)![dxP˖-+sjiӦ9>e˖y֓ȫպuk4Jʳ)xE$%%a""""3fʕ+URv8ydDDJWW[lukΝUD5 Yt E}}gaHIt 6 -Vtid;֭[7+VZ gffbhgz'pȑ";ԩ 0''<7}d^|RJSNڶmwaϞ=EfԨQgϞu5""uUbE_sNIk?݁ DP?o p5u*Ф$mz͛7kܸq vc(fڵ+}(r?E0}ڵkَuc^[[[j*>>>Iq#ڷoy̔IDDD̙3믿vJpiب>}W֭[%GD*V"L&': ׮-i=/@ >ߧ覗 ʖ-HLrؠB WVjjjc/^t'meccCCCc.lpAjՒ$:7ce$]p._>mO YIڟzuSKmC]JHHݻws=M:Cq\RN:3f8֦~<<~i3 b۶m9reN'ka?>OH2e/FD$>3Yk9rDzD6m)4I/A3=-QNz))2-['NAEr۫?$ڵk|.gw$kZ;&N( W⣏>QECСCѣG\i8:u*5jTZO&Bg3gyOdddmjժ3kvލQFiZh T:0 ?ghFӦMOD$'|"kK.ZBݻ% h>4 +gJ%JϞNAZbKKr(?߇Jķ~y+++)?ر#4͛ѶmM%kkk]PK,#?,twoqm۶^:'11NΝ;y>)) ˖-+t]غukB9s&ի5k׮Ev |oz ƍ_lDDDdVX.]4:͛+RH$WWWT\Yz˗/eG uKtֈ@׮Sh:}Xg]VlzgݻNAZ`KKիWqٳg֭[@C'''&* 93gP^=;6rsMt 14ΙbÆ / {Aٲes դ{w76שSww7/ލG}׍ĠA: ;w,p]Z%K`:t( \ ;v쀃Cם;w5ȑ#qͬ{!Ν WWW,Yq[[[ܹeUV/|o>lRDDcǎ͛7eG {(s3SMNi.`6+kӫ-oo?epR<~^~Z֭[s} -/_~@ƍڴs#֒m3'==K.Eѯ_?Z QQQ9Sոw,YmۢaÆ8tPlll}vO܍7ƩSPj碢Ю];;vuΞ==8ުU+5vZL6 uƍz1b-Z `ٲe5jj׮ׯnJJ 6ms4ܹL6 4q/+v,kƱcP||KJJB@@:uKKKTV m۶EfP|yKncc}zI"""2iժBCCeG [ 06ǏtH+^YYH+^(3a&ĉQN <|7o^%j5޽3gɓػw/>7k(y{ԩ׿>={`Ϟ=RJrʰϟ;fӦMvڷ.6Ǖ+WcvSTT:w.] ͚5Cr吙\r;v5T*xxxyrܽ{y?WիoL/>rƎ[^K.۷c֬YZ]f ֬Y{{{ywq% <gϞ}iiixQ&UZ{AÆ Ϻu0rHE>|gnn۷C&"75jԐ^xxHaҥ=Pg1$@24CUIN/)=IIKJScc pZZneڛ/_(UlmmaffkkkXZZK|111ǣG\M4qI$#'''XZZ}/QFARaݺuӧѣ9Kܽ{@cUX>>>=z4LLOlٲرc///zc#Gȑ#ռys,X 0^J%K`9u+(###1sy벉DDDd6mڄ#F(ڳg>Ck飚5kZ/""Bzy@ =SԤ>]tltO`rBKZCRV U&5SM M/-o߾Oӧ ]Νu>fqRPZ5_uZl#GΝ;عs'?أ+7NNNh߾=^zxTa =zl2>|laa={bM4Kzsilٳgjiidd?>|8zj\v@ﵶ<<<иqcmٱcLkb֭lxQRreYfgR6²_g"el L,X :IpOJΦMɓ__I-*c%J޽EG(VV\ :֭ubDxx8BBB/^~(QL899Ut民>} !!Nµk8 Y&5km۾~4mT"#Ch PB Ə#""sBCC4XYY5j@˖-{FH^"""ҽ]vaСSacӧ쵉 ,,,r-mV!=qԡ'P!$@Ϟ@LX4 (UJaz%[-+>o>P v$6yԩ;&:J6mFQh1Ujz:%uSʕADDDz:t(eeMDdlll"I`ر#_NY7P`K DVRZ°arNBy0~Y'Odv! 0622š5k0tPk 333j)ұ]mD mMȰ%nUgϐ)>ӄ @VSP* b̙cI&pssdtORd 6LDD$11QZE&Ć Sgg)4m>,{ِ'OdWM 7P 6dnݺ VӧO5/^ѣG^Dr2bp$ I>Z Z\*kًZϹ|yYP$vDd`ll۷kדR|}}Q1rCRpB;VDD&)) ճ 6U$.NBBd+ymj@56VL%¦Lq!CtL4 cLO^RܹsGDDѣGLb&<D'!m4o.O"#{"RqBeLhk)_ ooo||Jo:q9t /_T?S*RZtҲ#=p>d IaݺÇN)(XHH(R !^IU;#GNA`Kf&&&pi4o\g:99?3t6&)իѣb ^`r7eGzY` 3Stưa@^Sh:^ӵ. /]M2Wt ^ypylٲE旅L[nCDDDDDu5tԟ9s&OHm""CvyYըQCzGm5aЪlYj5uݺPT,־ hFtM/T*… 믿W_rʲԮS[,uHׯsΈS'fϞHm""CVq)YkUm9":i ^]t M7m.\XY[Wz%7P%%EVЪU+미}6.\/ÇǓ'OrgҥQreTT CƍѮ];N'""""*ݻ?2?'ODD͛xlJ*'''JIVƍE²||OOsi$ii@@tOb&{lz4 DbKT*ׯcĈ9gdd˗DDDDD$ѡCDEE)RS6QqEe*k=Sqq?0s&": V*RbTI=,Z̘`7p-Y#9g,ibEDzvvvlx0AAAر#>}H#F`ٲeܗ(X~5۷o/k=c/Ia_-:Ǐ??|d}NB%FRtM/""""""ѡC<~C m}Xb^DD.k_z炂ŋ/D'!mt 2P….?۲=dYtNQEDDDDD$PXXtHE껹aʕ02_򒕕ٳgZڵ&sի IHC}N%`*i7Dcʕifb[^Drs Dz |PE47n"M6ʕ+lԨʔ)#kM2&:ihFt M7kFW׵Y3^;r:c":@I{ի Ǐ/^ôiD$"""""@CE6m GD8xyy^w޲$y3`ot.: i$oIT^WKjh-||X@(I ZGbƍطoaÆnzaΜ9 ?mقWɺcG]7c!矕ѯ)o+:EŦLn߾ኍadd'Ncǎ9ΥsQl|"""""ѽ{wܼyS[Ea~ `۶m^z!55USLc%`*i/*L[.~7n3]4܀'zZq*98s@G HɎM/<}~!bcclٲ8tƍ IDATZ7f͚x"""""*t/_V~Vp*UJDT|ܹsnnnxqJ>>?iii4iFtEprrM͢S f  _ -ر !C:kر(ee5wwC)6HVcĈ٘&&&Xx1<==s=waӦM:CDDDDDKHH@׮]q%E7j;J.H}"*ѵkWɺZ"L7nUV7ox{{Z1m(5JImh=aLe#KeU ] #FNBqqx':mmqQQt"E饥t[1fM4:j믡Ve:{&<<c(NRaʕh׮(T\l 88}NBŘ- W&: bҍ7r=ާO[WT2e .][ʚJ*bbb xDDDDDDDD%J?>Q8֬_[NC^ EG!R*eK7o۷nݺɞL2Wոw$&&&Xb<<|Xڵkc֭8r^leAx{ˋNMfN"J%:^za'M'N(H$6:55syW^}ƖUTQ""""""""'ǴiPF Q,Y[nᣏ>Drt||76+`RIٲ%[ơs&031I68 Kaߜ$wܱcKKspB,áCЬY3ܻw/ʖ- *(̙3AAAuf͚:uꈎ5je˖Ç3f LJCt4KDGN"K п?mߎm3f^Pret<OHЩ(bb ɍM/-n:1ZQFaҤIHIIQl@ݻwG\\[oҤ TKDDDDDDTիW| ܹcҥ EGSSS 8NիW1zhXZZEm[૯DԤD/Z[cPX:a^MvT|޵kjYw4xX<{&WE"-iHMM8V1o}O>>}'OĹsp5EWQBʕѮ];t{F2eDG"C4j ܱcd;u ppNy/]=[Dϖ-_\7>Dpd$?WdT]4>Ϟq#.GM">|xMKHH別ٙ򂗗DDDDNٲeѷo_[nҥKtnܸ;w AOfMvhݺ5ڶmvڡjժcQq1m \&:I;&ŠAJ-ZhNH@pd$B|8/^˗/äI`oo/:@6662T|Ԭ)5)f$[L 0>0cP4TX..R3GtlqqtONCTh\n:+&O,:Q{Sh gD'!mh!'/$DƦ ֭ŋCR@ڼuڵxc)4ݻ5E{wD$S,GKd(ٳgXt)Zh!:QNY`:@@ϞSh/`j #CtcKF3f?(lƗJ_Ę<hDt Mv[NAښ8QZPl":Q%3///lڴ Kذa&NqJ,oojU)4m;&:ipqBCS^ 2d];d&MŋX&""""""""*lm&$ْˁkD'!mX[K$RS+˗E'!z+6cǎahР"cTT ˖-ŋQNE """"""""|HM }!!6Vտ{s`BIŦׯСC8p T۷իѣGȈDDDDDDDDD´hxzN)"D'!m4i"O"#{Id":@IRеkWt8qΟ?/ CJJJ-_<ѤIj ;wFuQwE')@RPa}! ^-:I `"`TZlz阥%z=zhGRRRSSK ,,,D$""""""""3 &8p@tl%5MƍL8|tO':I UD!rzvvvcQQxzJs.\$۾}='6Ə_%:I{jpI4p3(4o<4m /GDDDDDDDDDT*ݏ ~l >,:iUKt M[׬F"e6l؀+W`׮]:u*W/ҧ=RS+˗E'!m^^>.-sOcKK8zhs):nhh(^#˚5kаaC?~\񉈈V$>Xx@tFR3U$$AA`Kk5jt5fE KS###ѥKlٲE DDDDDDDDDTM'N)2<Ѩ0m<|}{H06K.B|_/FFYYY>|8g|?Bt MAA위IH:FN)$DE/: plzA۶m" NBr4Υb^cwo)4]H2<_t M/KƥNB%^E,:ڶm@-[VxLL f͚%(i?hJt Mq3ru0iTUڵkcٲe9/]Q@͚Shڼ8p@t Җ7*:[}DM"AA222vZAHԤ$[z:j!SSpt$[Fz5矢P ĦW1ңG6lZ- P' @P${J$˗ߢP æW1RjBCCqi(WӦNcO O{*"Bt*A*Fs=~'!""""""""|u):_E'!mt|BB .Nt*!*FnݺӧO8 Ր!@߾Sh|XHM1h0p]/E'Mbڵkؾ}{ݻ4DDDDDDDDDT _}m+:ÇMD m|=E:a":>HJJB\xycPݻy.o>@zShںؽ[t ҖShڱؾ]t *&ʃ &L˗/iӦhppp D'ɖ] 9#: iTItlj4II`-ԩgb;1H[I/Iz,] ܻ': ilY27$[Rۢcӫͱe|駢ܹsQfM1(֕$* E'!mԪ%-uOyPIȀU@&&&@.]DG)3gDDDDDDDDD$v퀱cE ̟ĈNBhBSX ^`bb-[raΝ={(DDDDDDDDD$77)4ݼ)-K$: iO`P)4ݽ ,Y"-ITHlz6m=YCժUCݱvZc݁Et~#]Et4nVNBF?:7Yf]ŋѲeK={ȀU.5)IL 0>*: iZ5Kt Mqq?": 6dpelwcǎa޼yE/88qqq9ǣG|r """""""""ֲ%0a__ *JtF_$2R>^E޽{#11@CVi̺uڵkhڴis/^@=\1 &:%K/E'!mt .: `" !AtlzٻضMt ք @VShڿ7)H06`ZQ KƦM`nn?0""""""""""T&:#GD myyիNif)H 6c>|sqq;#k =t̛7Oֱ*x{66dKIV^aeeʈN-- .^aKK@f!לUT;w"y&MJʕ+ Yj|铸8 QSϟlziܹsV}ԬYfff{G)Zjh֬Y㉉8t"ci\KDFӧ67Zt M~~үT饥+W8ɓ￑O˗?ѽ{wE3c]qȀt .: `b IH;#FN)8XPE%^Zxmoo~ FFJMLL dF'ODDDDDDDDDd zBӹs@F$C>}Dtjlzi)..N!C`ii)(&M(l"""""""""-Eд?oS<<6mDt y#lzɤu֢# 111joq"""""""""{@Shڼ8p@t Җ7PlT饥7guUTIPl>ܳgt )#:I4 |Yt҆rp$[z:f p$06dmm:))IPlgϞ7$""""""""\U"5)6*VDД,ZNB bKK*Tx}5AI$7n+W<4DDDDDDDDDdP7LBǀ+wտӧ/!: )M/-U\Y둕%$ZɓJGi u !:``Biww)4@\$6TF wŏ?($95:JCDDDDDDDDDkPO)4]Z%E 0@t M׮ITj$$3 Uƍs9s&2220k,T*3cԩX`[ZyIIF )ӘSDSF%KASaH<}[n… Z7ڴiHDDDDDDDDD%TJ0i$Ef\͚Pa99I/OO +KtɫW%RPj.V/Ec޽{ѻwovZXBgVrr2\uF`mm-0QRRRpׯ6l}Qaac樃;9)4YXUff6^Edn{htΘ| gzAѸq/3(B2eйsgJDڵk#EG "zDD21RR )8IM~O5:z=hԩ#c5.EDd8ӫO?,+++L0Ah:aÆׯ rs_!CQ`""l.^ΝH6jBuE.NV-ww5_!bIQD-ҢjPE[ZFդZrί۴k,.:Lcߗ !BBV ?>SK}q]^N~{|ŏ M/.y ѡCIX>}:>⺞DD C&Mxꩧ,<<5'NE .xADHWW#+/7x]_KJƏ.!g+q~k_}=]vxw3,d2Ig=qtL&Z5<`*djՒ0+(V.B:OqIO)(_<֯_eJQIWt+,. T 89I؟ҥkɞaÜ5!;'"#u\{uؼՁW~t,\wlX C]S@Nt ,^ڸ|K'شiv튓'Ovlܸ͚5k """"""g@`0 ۴k0ctY^zMd{~s дi@rʸ(_L*s__5S8 0POթ#]CC`,uMխ+]Crسg:t r~|7իHGݻÇKWX ;wKH]Q+,ED%ENر^nݒ.!-ڷ&Lt0{q襳+bCaÆؿ?Sl$"""""""+{5h?rXΖ.!- tlZ֓l/HWX:sXHO.!-^z P9-JjZt >P,!=׿uk6T%f9^=.oHDDDDDDDdKvF̛$'K:cJWXzU]SII%E "#Ր"!Ah8Qj@/]B4O҉e@Vt iNRKeI3*^RUFrjy:=}&]a),L-ɚ*]BЋ0]+H1cS_FuըQT#ٷX8Q !]a)4X ϗ.!pEDDDDDDDdL&۷KWV&H[+HS~FeEidOҶml0mPtYn.r%pt i_5jH's.!L&SҚ5޽մi@z֭SO(AVV~G|ӧ|}}888L2TG$j8a2aQ=N.^.!-VU7KRS^Laa%E@@PtYzΝ.!-*TPהtYFڇ)C/+:y$Fjժ^ϟGRR2~ۄ3''o~>EDDDDDDT tSrOxC6_?` KΩeӥKH>}^dMM.!-zxCRxRRKJt4g+HNNv빹>}:z4lر/#g`: j(kW KWV9Á=+,eCᗑ<\ J(zdѢE?|TRXp!K8=`#ѝ;wsθuLj/#ټ[ jTys K|l"]AZM l)]aiV 8XWM% ^: |o~GTPᾯ=z{Ffff88W%fW_"]BZkKWk{JV&PtuөdL&K҆ ?JWzQJJ f'sغu+ڡC0hQZU )KRSE9d{TQ7p?JLz9ҢbEuMJed}O.!-\]5UtYVlptIWEݵkW]KkW0||?#ץKHFiӤ+,%&g%ާyҢn]]SoWH=ݻ3ghÀM2{t:KWXTCҢ];`D KׯfUmڨ$*J KH-վqFrㆺbbK^EvZJg`ĉѣ}ߺu P ttZlO>kIWXpA-iuAϞС.],RRKH=å+,AA;%vC"XftsARW_ݻUDDDDDDDd(F]HWXڳ0`p{w Kz5PP ]BZ~ʕ@nt ikjHo$G˗%vC/nܸÇKgnݺzjj*6n(PDDDDDDDDƌWWW4|oMԮ-]aVPڻW2u+,[tie2^^6lv풮 M5a~8ȑ#ڐ!CpY\z7nħ~)S`„ Xx1;777ݛy~Q Q,]b,Y"SAoM22ԞqOKԊK̲eˀǥKH gguMU,]b9|Ĭ C/.]tkXj֬Y__߇7&L{SjzRRu?٨Fԧލ$!=.!- e#u ;zUWxTr20>pt iFrn8ڵk6~x 6~Ez'lٲZTT#"""""""֮_fR0=<L$]a)* 9.!-ɓ+,ݸ11%Eԩbbԍ%vC/-~_LL+'ʗ/w$GDDDDDDD6o_W+,],ZJzCJWXt XHI.!-zιsG5JRDz03SqQFF[n5k (/^|5#""""""l{w Kz5=mo={JWX:xX ˓.!-^{ Gґ#j_&=/$]aq4ӛPٞ={ҥKc ٔZu+ytii#]a;+Hm+,qti5aоt˗ l^+Wgdd 88_s/n] Kk!!U@Ptۥ (L&aC K6;vHWV&-]A:K?uYxG!>>_\r1ͩTI )\\K22%KӧKH K̲ (ʕSהu˗89ӦUJN8ҨN:x"BCCEZ6oތ/^"""""""4hn(ɭ[ܹի%Ez2^<=>u`U5jkAtF 4{ڱd :O\;V]S{JziTT) ,x\r ZJPt@LJJB%ٳ."""""""0]nub ^Q@%ɤ%f6)GMOIUFz<&П;']B `رp(}6nݺƻ9 IDATd^ХKxzzv<""""""B,N.!-ʗW7*T.1/Wˈ)WN]S+KeT#=>'' @ "7XXO5spPoЫ|}}M:Ǝ+@DDDDDD ;.!-M̟DDHꆲܾ /Kj{W]S.Iժ#̳%z_|ҥKKg>ƻ8*͛㭷ޒx?Jb""""""2Eaä+,')ܑ.!-uFt 0o,]BZt #]a)"B=휔$]BZt/]A̚5 ͚5ΰ0dK:~U@^t i1dлtÇ~L:Nh`ॗ+,?,b嗥+,:,] dfJJWcKe˖Epp0ݥS 6ļy3` wm+,iti5aоt;+Hc+, [']AZ t"]AC/x{{ￇh;;T\Y/_ti4l(]ai` * hXƍKWVӦMJWX V zJC/m9͚5oooJf&lpt inU$]b;zT(S0\`J!ҢT)uMU.]b|/%մi@zGA݋:t#GDDDDDDdKsҢNuCHn˗KHZwMݽKKHj5(]b ,^ IUkYC/+Y&v܉%K?EP|y| {ŒHnfbcKH-)S+,ݼ7KHf͌7hǸTTt iѨzڙ Ǯ^غu+^{5VZ?OX=z4.]O>5j0a0e8 DDDDDDDsG 9Rҕ+/=;GKWXPO&%I:IWXfKH~[n^IIIԩ 6󈉉1g4ogΜzMHcqk'''t .ĵk0|ڡDDDDDDd/VR{<Wұcj/=KWX:u XTIHu]ٳ%@zt iѷ/vxPNNz#G<{1p@;wXrvv+W^y ñcpDFF"&&YYYHKK*UWWWԫW^^^hѢZjrYH!%on%f;wa%Ÿq)KvVהўn3zBBKQ#]BZ۵K`Cs>r/D^ʬTR񁏏O&lؠn)]BZL?ptƍ[6MP>ȇr!]#Rq PbX mTjN.)nyÂ,X౿V!""""""BV.!-ʔQ/\tC%ERꚪ^]Ch|૯T#N@PtYAzj ԑ(nuEDFF>)Sz1DDDDDDD@Ppt iQRɝ;@xt iQ\@3SSE0COpZx.O5Jԓ..%%ǩS |+WÈAAυуܼ iL )$69.!-6U7$.NOEEIo@̚2 wM0v7JLL|X"v\?ڶmZjlٲ_>|M#""""""?|?_=EK`h Kܹ@Rt iѱ#0ntH`lٞ~[j ]BZ< 0itEewC&M}ݺuyfT\Yو,fz_dɒ{'CDDDDDDTdǎ˗ /]a)`R 3S_]WFrӥKH}W_tZ!z (=֮] ???$&&"//uRJE>V\+V# `Nʰa%Ÿq)KvVF.!-FVTHtYHt i1bzjn55n ]C5lzڹSD'\I\rТE ./Ct~զM+]AZ=$8; * @g$_ |ti5e }dغؼYz]uk^\1*&&F::rr+ÇKH GG5VM,/O]S%Ԫ%]aVP^mpL&N KkF* __İWݺu{-77_rrrEu"""""""s åKHͿR% ,ZIzȖ'"^LK*ӦXQLLF3YҢ|y>UtI`ЇIp9]äDEE!::Z:Ⱥbc365mn(I\DEIF̞ \&]BZ4h`k*1Q]S%Ezig:zUR^^^v>Cff.2ړ^[lN """"""*@Rt iѱ#0ntH`- +,] %3#]a)*J}#.NhZEVewC/h۶_OMMdBÆ  K!i蕒YfIg'ˁ,Ң` KgK%K/KWX:^-u&]BZ "]aE$kJt i av.^}yףQFhԨFe˖!44IXDD}\v }E'"""""f.`: jsg K!!ڵȑ@n ((.!- S # VKH!C@Q:zr!##/7<<X|*U ʕ+J*sn{="">>>p)o߾k׮!77XGDDDDDƖdܾ}?Ґ;wX| *|UUe i xx}J&ګYVT%Ŕ)jʓ'K̶nU%ŻkQV5ӍxyGwtݱˡ^u,[L1n߾۷ok .h>/ѣ$&&~]v FLL nMJPfMԩS>>>A&M*Ur"+Wj_=&|tRPY;Jא&@2uԐhO7ڸ\DǾi#F11@t֮Jڷ/֨R.Vާ\.+v9wy+V@~~t ٰL\xaaa77n >>YYYsp]F*VRJB ]6WOO{lڴ)P]8MDE3gܹs8{,Ξ=ֲ?xڵk^^^ر#yt^^^DHjqc*,";[FIMU{1yx>>5TX* AgB=6`4gp~v;[anܸGȑ#8upڵBœv~+|||O駟FN_lҝ;wp19rGѣGmOIDD"""j*@ݺuѯ_?+h׮d䊋Ӂ:uk7VC?|0@15YZ6TӥKٳ>Փ!"iuߗ.v;O>۶mCbbt L~~>N>ȑ#qFwݻ5kk׮֭^xT^ػȺo>ݻ{ůڵk3g̙:u`=z44i"F%ի9t V @Pc}VZHOՐjUkֶ;g\(`L5Qj//. vN5zj~ʐ.]‚ 0`TV -[ɓuV`ڵx7QvmX~=ӥӈH޽SNEVPF +X`Ο?o?ٳcvG6M/ vYF SKIO{CHWXxQ-ɚjȖt"'gϞOa2Sؾ};mۆK.I'Z^^v؁;vB 8p }% -//۷o޽{g;wD7}iӦ裏п$*IBBԓ^o%]BZ۽_0@Nq6߯3mϰaڹS,4T5~<}~W-HnSJgQ11l0T^;vČ3lrg)))X|95k[,HDr]sř3gJΟ?_~m۶Łs$k`V jy~3NjFٲE"ݶ  "xW"^gaɒ%(St ,//?F5jo߾Xz5nݺ%fغu+ڴi=zIDDh߾=|Mܾ}[:J`j& x) dj5k={$ (Ǝ.ߺuO?IWم3ѣGh۶t ĉxQV-t+V@rrtVڵkZh{l@AA+ Tii-[Ԑbz 0mJG۶M]SK&}.1{2DȦ'E`OUZDDDd0YYY8p VFoѳgOdaoL&SҚ5= H%)ׯ~Y2 +,mtMˎ⋨\tL^^FkJ= )ʕ rKӁ%K>_dXvǥ3OFT#=> @=jYYeJHNNFʕv0͛c?FDDD#Ff͚ڵtlٲ_>֭ڵkzpww2e"%%HLLDBBbccK.!<<Vɓ'{a >.1KLfOՓ?y,]*Q8IIܹ򒮡ª[W L.1}7O]SO>)]CM"WBB>3lٲׯ_zV_K.ui"T*Uʕ+lٲpss  ::QQQst2rrr0p@8p7!BV|}}Ѵi{lРjԨ p|?TpIIIݻ7?77b??=777M6hӦ Zn:u98vZ QPP#F… (sR1G=a~tYXp!0y2Pt X{78 ]x8`lO[%f/kq-[eFzz5* ?:U-CLd'r蕘^z=p<[v믿_]:&9997k~~~ *UN:SNbؼy3֮][O]|XlY ;v?N:E(]t=hݺ5Zn>| f̘ŰHbb">c̝;b4|%f0~"Ttim~!>$l߾'OFV 53''' 4GŦMkѢEyM IWXڶ ؼYuo Urfv`& ղFcati5~<Сt]u+tcwCl^?""B*WcǎYfIDDDWfM;8t"""`兖3rqq|ח_~iyG۷~."RT!C`M6r=3g`ĉpppڹrrr`㓠n] K?,]Qbm޿[??X4]XKm`pM5iJWV&#]a)8XO5 N>]7^#DDDd˗ȑ#h̙3 BllUAT… ѭ[7899IgYUq!-wFnIR0w.Pa&.Xd?zu`6:m`|eҢvmėܹ,Xp_TzH7MI..^.!*JɅ K.믿ij>ڵkh@xWc„ XtV 0WΑ?V;>QIv~Ȯƍc޽V{Zbrrrrl֦ /]a): KJ>!!ye˔Q/ ݸ̚<>d@O=L*]a)&9S][d{|}Hԟ}QQ%Dz>}0zh4nO>$^u޽e~! tGAAaҥ8p_-!""*Ne˖Ÿqp%sVc8,صkW8ܹSA tKԓ))%vNZΝ[c~Eti\"+ S:d{vF̛)Iԩ0vtW9s[K4Wr d!<<k׮Oµk_FIVSfMWZ%:ٗcذaw׮]o?ہM+RȩSXYaFe+; +H+,_/]AZt,]a)$$z=fWZZZz\X^\×Jebɸr OnXC GGGݏ}%8p@Q3<<W^d@&%]aiO*-3g*ozmXU6mR/M&#]a)8X 6}$8غUPnUn^+((… u?,\xQ:j1rHcƌpwwN*v;wɓruYDTrժU ?t?u?&\tYVZqҥn..X;:uM9"]BZ8:!Eժ%fj9V~698?IAz5t cW ?k׮\Fz;w*OAӦMqY,[ 9qƺwݺhر(UJ߿z~%Dݺꆲ$'WHؼcqH[m͝;@P.]BZԨ%f))…MUӦF ,^ t c |hNN^z… hC\c옏3 lٲ1c u?.lGgЫf7n@P*ҳ/²5_ԱJPL 0sũ(xTB0{6pt _Z?|}}ѹsgZ c˗1| """*&}A6mt?}t?&Q^t=^%L9wHؤV!g'',4 Fz""yӄd{:uƎtKH +,EFs AAAz~~>݋{3ϠI&Fj *B \.1336mXYGľ}0yd9} 0@c۷CDD]vxQzk@b"PtuڏiLq8, sHǘ>dqLn(<(]bcy,Y99|hV>8P*ٵ 0ӍxƌQS}~Y]SFIFk知KSԘ1ώ7vb"""")Sus%]GDx'u;^vv6u;ِaC K6֬H?__NcWT#ɤ2``V jTys K[_DdC/?~<ʗ//ADDDdU=zxH^Ddz>Ub+n(W,]b9mTdc""s$ҢC`x KjH ]BZkL(]au5L.!`C/~ ggg""""{蕚 ##Cc)SF!٨A^t8l%]"./?nn;V*td (~{;KO.!-^{MRXZ$t 6lt jI6^Dd )))vpppxd&Lڷk~t͛q{R?*UҩȆ3nti5jz:H֬ KHÁ=+,W"q%bQ^=""""]UPA4u\Sc09F l.]!Rt4ߚ5E:Fz.:ف͛o NUɖ-٦ɓ-+,mݪ#Tb^s'st\2!<<\c=䓺섓0mPtYn.jptI/(YQ%]ߟOuQ~>W/HV@fi{KH+ [WҺuT"!%jBfŋ1f.yHDDD6/33SQvv6"""t;^K}F uCHKKoΞ-1>>^5kTdGRSE9d{TQC ggt׹s%EŊE!|_~%ƍg}+VDFPPLq`ܸqhذ!ʗ/&MUVxgO =6o\,=Dd|z-$$DcUPC/zΝ1c+,EDs窛vo͞"X44 Kȑ"#Ր"!Ah8QYlU6t(`Lb(`$KFVЪU~O^^ݻwcҤI}Ϸ~;Z3բwTTW.BT+VPĮwE &` {Î{DEPDqH&Rso-ֻ8{q'55U9&&&r?6)) J^bŊPb#=zÇx%իWLܸq7nIn:%$HU$/LBÇղ>zDC+\$ÇrONN}wɓ8TMXZۛwO7mߡ~zLN aZA88cjnIr~ [͕̚DEUH:::J.kӫL2jF(]vhݺ5tJ]vaܸq#55U/nnnXjhY\]]dtaL8?|/_^dׯw.R:B:(Y^XLB۷o㉒{[nݘ"n * |w~~f 1[*U aHct%Lz|F^Ν$ǵk90q"; )aÄ>N$͛I.#$6W(JWW˗/'I5Fe˖}8p N>;w`ڵȵv%' ---iSBSj_~"Bp(M\1 ?_KK `h`0\\y :;QԬY@ƼS:rDx"itaC)9~\Z3eF֪iڴ)*TPQ `ڴiƓ'O7600@ƍaccSSS܎;"++ @``{T+WZBrT' 7ƤIG{>DVV,\/^f͚z{ӧOEHgϞ!%%Y5j0E!w!b @yR8ڀpP*{de;w 3)lmyabs8vR5v F%0x'dg ˙NC ̘{;I{1eg;$ 8<?ǏARj*SSՅ!uuaT `Z*. +33T(]&&k6?}oވS_ ;w杄h8jzizQK"7o"::woB2001qD*GѣǷ?~V¦M iԨ/_{{{ޜEԲeKl۟CCCkILW\uf;k^ ;B4\fdS:u#]4i7ZCKK SNeR1K\PF cnԪ;R>}7*UƌaHLMs.(wv$% {ƙuNC dɜ1M=́ x&":^ (a_0 +ssԫRMQLFFӁx뱐 l",ڤ 4DtҼ#<`ԩx)b)ccc\znnn 7rSF _Ǐ/FݻwǍ7ЩS'lxB صkFsfffbʔ)6VZm_CBBrՄ>(RA`B#+ݻǬ^^Р_t#JSG,%?߸:y|U)S`bl(Ԩ!H`J[I"VR^ q=BefeƳgpݶ Əe>oڅn1kx@ZFB>}±7p.?Vu<"cc?A{֬y'!^>>>~&N;KKK:uJeYV^-3c]]ZZZb޽(Vhyx5kε\%lԨZjsGo1GQ>}M%7---k׎Y=BHѵo>{`ѢE"M`$)d}+4)^DU%˗qujto?i(j8:N!{` FO3BCww,+DXc~<4৯_s ',CKGGw/YRI`L!&#$DCQK4nӴiSޑH.Z|6lyƏɓ'k 37 NNNS*W}" !Eff&zk׆%zĉ1b% Z!yߟw YO%%NR(QqqpZ^%u_gO` )dx!,; QDn/N!k`:,' |QffqK߱5F@Iu$ j.ȑTD` v- D3PK ZZZ011GWJ#ݻwԩJƓ> |۰accc$ȑ#ҒX9sϞ=cZsR&8̜97ofZWWW?.!pq|wз/$:q_Ta\nIY˜wGқH3}֝;8vLS11c~9z{ˠA(idN>uK5!q065a= //o?W\AGG>DEi&n~/{yիWs=-C) U-[2I|8r֭˼;Znͼ.!77ʊw=@"61X]^zz쬒6TTw Y{/NAT;{sx /!9˽a=|8V9f⹺֪ 'ӧy ^&88wRWQϥ,۹f͚*NBQW݃ӚCռ Odgg̙3hٲ%~'2?ѣ1{lu T,:ƍ,M۰"#1oԮXQ" )UJSŋN#9Y3#I"&f<[h2q"@CCaLF[JkV#Qk yD_"***c˗WqB:ƴiӘ/!D߿amm D>}i&jլ)\H`J[Iv(9ã^*=`DTTTz5; QDT)֬{y-Z8:b-y*_^zSu@P$DPK!%%w BH>򛍩, !ׯ_gZwިVӚgbhԨ*Us犺 A֭GG)d{X!4$"1%VDvv5 |3BCww "w&M3yxx>NB##3˽S6Um Sye&DN)hٷowTR*LBQGɘ+5 ! HIIA\\|/_ << ‹/š6mݡMgEWD*^:6MX⌳6ǏJ՘ڧgc(u&̰ڽw_63fHk<".]FΝ ֮fJ䝆G@` ZLC CÆ?NS[>\^OO`i-H 54D@@:;!oY$}@$`ɒ%xӚ?#6mʴ&!D.\@Nx!֮]QFBQ&1u*`fLp\Gt5' IDATR5*.#F . .(9;IaLMЌZ3t:u$9nmLx!5>΅ɘأ8PU./¾qS5DiB 1z#'X|9R[˝R4o"0kШvT]BYkalh(1h֌w Yǎ>>SE99-[N!))H2231sl|Vs@QKb̙W޼y;!D#BPRR6"VP>}ЦM5 !D^FFFXbnܸڵkCHWWB)d\n۶'١.ҫ+P ֬FR85N杂OooL۰!@Z Tij$j^j&-- DNPfMxxxEtB{%(bܹxӚ`ZB䡧qט6mthy-"5Jnn@⼓HNGTz[/^`ѣJ0+Q+&L`HM VjσE cJJ{ ޽; #G }\lܺ; Q3R_ܹsQ|y.\P҄6xG cڵΞ=Uv1!DaҤIx6mڄrDHުU.(KITz5>,c2rD.UQ*5VL_1E&kAA.L${`YnQ+'8aL~; Q#/JMMűc???jr"#88!!!D||D (N7$9rT>xhlm6E кn|gm&hbm AC׮aۙ3rŒP`߾˜Ui|=|l,.Vw"a˗رclق(qQ{aĈxYǍaaaDFF"#/\o޼֭['Y.pB͛7}%-7AHo>lذy*U`ŊB?|ӧc]6u놾}y4ۃ3&ݻ8v 07 bZ{@xլ ~bHC99 EIaNBS)Sשy'q:5|8:hW>zn^_3x)el1aq7eh(ƭ\}5a \,rpASGNB$Lw(%%DNPvm,_^Hٿ?ڶm+*Q&Lg"11x glٲVVVԩSqEUAi }0## cǎe^W[[;wD%&ܼxhٲ%*U}w,B T;}癕{?W69;CG.9BӼSE5kN!nW a~{TTijvgeK8Wc QIGNA$fzϱk.l޼_~9sٳhӦ 4hҥKT 111Ell,"""p-ݻQѻwo$%%1=k,iӆy]BGhh(-[?]vŌ3о}{ޱ)cc@B4`w%Ah)9`vhTR5sTL 44aI:ssY3iHa 3Rl.-E?!uꄱ]ajO U,-mLL c<<08XsN߸͛jk cjt -;T*+K坆H5DǏK%Hݻwcݼ0; <ϊ+lx8Fa@@.]uVZUVywpp<Xd :v(V4Bǣk׮M4E%‰'p nK,Avx"*KVV"\m(^~V,VURQ0oj9*ilIrkT?4)".NS@ {ˆΝQHsZkkY.\ aQZq^~N鎩2"^ M4AӦMhޱl߾kiiDž?y߿!7ggg;vy]---l߾UHVQUA4ށPvmlٲwBS ֭bc lYɩJ~\bHraSp0ft^$ӡ ^JyFx]hkv.٘;v6mE;B޾VrDhyC}!A*za``-Ze˖UahhTTÇxΞ=0Fs~ |!6_nܸqG!Z k֬[cegg@JDGGcر8y$v؁"LCQQu ر<Е#R45r]\WG7C)=`VQ/שÇy'!y3w(K㲇:ϝ;/_rׯ#(<˕?$)Qί'O//Y+yRP\\87 5~x,]'33!9KBpt.(Ki3g}s6gDaիpASRݨJܲe52k~ر˜xv7~<$DUHڻw/y!M67a/͛zkkkhiizǺYj,dgB|}}1rHdee1]re8p $D4ocR+>>HMMEdd$m% 39Waz 8uQhUWW`ta)8ؿ_C1nJ$$'+un-Z6mA1$9TE gx'q0zD42©%Kld(c~=Æ<..B㫐[f"^r7֯_/ׯ|rܿB˫"Hڅ 0ddff2maaӧOܜymB_%J@&MTrt~o˗q^%(ѣSy QHӥQZm0Y\IP4%F''j\ 3OMFtТ4s23;1պ54jǬD Y:IJ_I:u @u*< %m; Hw) aeeSË%ڵ /FFPBm[nů;!̭[ЧO<(QΜ9CKB5 v»wS*{IHH@Ϟ=q T \P`Z (Ca_eVO|XPErS˜SBB[B t VŦiD}ҥdn.NIiD`Fij$*'W9iʕ+QN4m^^^_ưvڰ`޼y>޿+W`ԨQЖ.BǏѵkW$$$0mhhǏqkB`…x]>}9.)) z+)-EH^5fBVx8]cWDGZթqݺ)U^=I!%>E3S$"Bx D- CV}YSz H` [I't:֗/_Ɛ!C`eeӧŋcFXh^|ϟŋR X&==dž 0rH$%%D!.] ::ymٳmi BxBKUƗ/_гgOʼnzB5w YAA5ߖ^~,N߹TI==l>ZZ,t;7oի_y'!h8w Yo+WJg9O5fǡW{`ր#+{`*F"􊎎ׯ:`߾},5Ċ ƍk׮˗X`j֬ @X~;m&J4Hvv6h"ڢdɒhҤ &M;v ##wDB[hh(:uO"l2͛7'ڬ5qU4h@~cƌ>!L ;{['ظQr AJ#rg͛I"~IWR0&BX4 Ay^W`왰ԡWq#Wvv60`XZZbx)Xy244qqDDD`ӦMV.wSuԉCBBŋ1vXXYYiӦXp!;!(ӧOûwD鉑#GRBզMܹs3f F Q GGw)9s]]䲆u+Wl.L(Б }x ;VI(%/{N{FI##5 03FfPիΝ@v6$DL+""˗/G5`oo+mmmn6m§O~tzzz>v*JH[QQQXbjժ;;;lٲ?~v;vļyo>ܾ}䘘B.] P;-[)SRB K__8y$J,)9̙0Qjœ׉DpD=slBqqBpDQ..qRrp0j&0{œ9Rrp D4镕sΡo߾PΝ ޱTNx=_qơDr?r6d,]+Wƌ3oʔ)ggg\xŋ1h 4oBHbccǏR3g( !DG ׎3󺄈B__lj; `/DZwoƆA%]]Z4$923I\]rxȑ %!ػ7t^/t Ɣg0 Zt)W.]#x f͚+++jrl"cǎvpssC¿([,֬Y\:t.EHQ3qqqܹ3޽+J}''',ZHڄ‚ _.Cʕ+" ++͍w LYK#!J)SF¶R /y'!07Ɣ>$9=D55E ּHӫT)ὯxqY, $D$NDD;.\mXU !ѧOݻ7?~)SАSBBQNbb"z;wRԨQXr( !+(-ŒjR0㋓Yt(D3UFD(&fM%%ʕH{#k3 (ɩbj\r0޾坄L#^R[PWW;vɓs=^zufqQӋHYFF뇹s"++KѣqasJG!KJJBqU8~l޼ZZZ'֪W___3/]Dz;V姍0A;G˖ *fڶ&oU(I"Z_wB"2wѧukh36[ea"4pwDcid+88w’BCC燥Kf͚1;ݻgj,GGG~{{{{lܸ.BZrr2w˗/R~¶mۘ 'U3֮]˼&!' @5JJ5Wয়}y1$%NBѫ0p =:LLD-(KUAԽ;0t(0,z%,; Fns1 c~ }-[e˖eVVΞ=7~{---xzzҾ]}ҥKҥ G5o<!.<<|'B6,Gs]|ADt 0ǎ~Æ qIRBxpuueZ͛+ӚD2e A#F($˜$GB缓E*%4) y'ɑ$ $V7|aS))}MIlY'kzEGG#::y]o^x9šyH%)xDdg+8FH>gff֭[A!ddd`4g IDAT8z(4h???RBx5z8Owuec#\Pf%_ְaŊڧ8DjeL)%2XxwkkI!%_+WoN"Yf%JD%$'3*U7~V޼ᝄ0qM@jԨݬ.OXx}M-9?/_xzzzͫy-{ܹSR_O !D2331b:tHjٳgajj*J}BIKK #F`Z֭[LR&N!CVBak L;w&Ed$$D-[NNS {Mi(+ssfSR 5cۚ  W֤I;v Shf>zqΎ\DV:|0.] }'HMMŁ |7\kTRyx"222[Qzz: l?zQQQf$%%Ȩu-eбcGBĕ#Gb޽ԯQ.^2eʈRBk׮psscV/ Y-BgaH7)eԩ,ޑݡ/Vy.U kՓѫ02Ϟ KN(pн;;IW 3\D70`V+95Yo~Qxҍ큁uYth\KGG=z@=c_{Abb51c hBĤ(t/_^BBB憿K,prrkײeгgOUTXhh(֯_S[#%%ƕ+W.ϣzGӧѷoBG`` ߿qOOO :NA'I&aݢԯ^:.]_'_>ŤÇmF"DeƏ.(_; >8? ?LUdZ;G /\C cIq+-`v`dD4ՠ~ظq#ñi&/ԟ:u -[D&MU!T9pwwСC!Ǐk׮شi\CvsB-צM|Ϙ1K,ZBBQnoMÇysqq͛R4DNA4ɜ9@ÆS~z1h҄w Y N!9Ɔjed =#Y=΀&8!ЊDD7nƍׯGAFX߿c֬Y8p Qn]&͛7G\\q}8rw^8ph޼9V%KbrC>>hݺ\GQR%̝;"'%D34jǏG~6qy4nݺزeKc6!$/...Xz(˔)sΡjժ')c"V-2!X[KoL}\)BD~UHoL} xzoN" ,^ S lg)TP$L200@~Я_? {w6 |˗/;;;;hF7m8}t :T,rl۱cG|>>>8r߿oǵQR%ԫWvvvݻ7*VoaÆVZXx1.^߮)ڴicǢ[n=?ٳ ]nmLMMq=&,HQyaٲeԶj׮-J}B%K2W7BԎ-0e v-$DSl 89"ХP?_MAKf̜+x',XqJ%0YX1fո10k_9'33'N'PvmL4 K_ω'sssM [###9#Gh7v7oǏ~!=BWWMh3!jk577lllDO!ؘid,EKz ah݁H`>Ir| _̜ 0~o * N;I@`:yR4\dggC^й3%,-(AA59@R9 caas"88GEN 8::|0a=z$s<22ٌ7/???\pϿ"DLLLd211)E!߰xbQj*U gΜA=ڻR1GM/Fx dHK)d;vYYE ;\ݺlʬ*^0hгjY{[Q+:::իΝ;/^`ԩr-ÑM6aÆŁ\Ϫ{{{tۏڄBHQ_U%KĹsh(!@R1U_!D6䝂h3}&oLJw (ggE )d8QdgʾiLhZϙ3S9PKN5kիlذA;1h o߾>&&&I#BHZ 3g?f͚RBMBBzg"9@SMT;}??)\]jxup,*z r2LQϹ <;)5 &LǏqezzz>/11_|ؓ'Od!OOOL6Mŋɓ'ѦMQB:bbG!cb"\P64䝄h~Ɣ^?//CI"r2-@@$*uS-˴ )SS>Mztۼ|PK ڵ޾} /{y"""A!5[l( q kN>}Ĵ"EpAVTޘXxwJ&@P$*üeiɴ^XY 3S$6Xc+#5(W-Zw;++ ]v'>p{)\B!(ڶmƏlǡCСC !D1DM/RdjL;$͛ӧN!`|QƍYxxxB##myԯ/Oԇ\PӋ!=== 0W^ŋ/0uT|}NNN([,6m ˗:'M!I;vرcż>|}}ѵkW !D23ʊi=B$G``)&qp BV` n; QDQS Tl,$|i===Ԑž;cN!`jzVZXz5>|+WZeee! -BڵQ~},XyoƵkXD'B47ƌ#JKOODݙ&Mj_T^i=B$oHK)&[7)dݺddNB1hг'nx' WckkgZSa}N!}a߸TIȿJ*ggg899ܹsXnN<)E'Oɓ'Xx1*V޽{W^hӦ ?ƀDYB4СCɼvލRrG!ʛTJ*3g_T=D++RX^Jl!_ׇ.h&&j1 DF 8~03 ᝄ(QS7oNiaL ; sBCqk5[֮ʹ&M49aLImvcFM/B.]ХK`ƍغu+{xzzƨX"222$ʝB!ݢ:::صk 6!hW2fy"1xۯ7_ xחY Փ,WWa7ox'ɱ?`nNՕ07c-)nTka^Si{缓+ !UTnݺ5s~^B9AƎ;0 ދ^H+YR'~* xyNBad$%y'ɑ*,p0Q!aLIiVvz:};p6$|ĮוL/TҼcqwjzqe``~wƍCCCޱ!si 4kkiiaÆ :t(ڄiôf-#DT*\_ի5[ȯR%鍩` (w++aƗ01ӦreX3L2lgm 7Wx')$4*&M`ӦMǪUPreޑ!pYOHacY---[ƍc^B4ôf#D-5o.,&%>@D$DfN!+, Xwsy#.-5g@vdn]%%{_h($E5$TRprrBpp0Ο?ݻCKKw,B!D-?{FJJ ZZZĉ&M?iM333Hq Bxpp BLxI":wFBV` v0K;;`)dyxzѼ($+;==)7ڶe^ xZDENRdQKaooǏիW3gL-!"q׮]C>}Dix)SRB4ю;35۷om)-kCo"4MaU%*0x0Уnx'!0Ӈw Y Ɖ:ؖۇ/_2[JԮXy]Q3з/?LJ❤Hoj˖-ûwaԖBk׮HLL%K0gQjB&… ׵g^7}ܡ?x{NA5u*ЪN睂(j$M)d;;Es)Jm/m_:N!ES^j&LӧOq!TZw$B!Drnܸ$$$R~6&|>J^B5gy rsjB"Sw Y>>B^ CYkϗGNQPK ikkӧO1vXq!ɸu~Gċo̙31|QjBڷoünϞ=annμ.!H\$$9RS%x'!00.(KitaLݹ; Q0,,x'ɑ!,z$z.s"Z=GvK)[gi;Il`.UIjz1CCCxyy~Bڵh iӦ᯿6!h7obܸq1b(u * $:X杄(|y鍩X`Z 0wKKI%Ã7_L y#N uIoa!L$GBq#9$E541l01!n>|{{{DGGR +V6!D3#U 7?Ç e˖E.]%$EJQ&̙S <<y'!h;w Y?"n]%l MD:*Y uvϟE;GvP\9DZr)V޿睤HXz5LLLx BTǰׯ_E?f\Rڄ kkkxyy!33wѹsgƊR6!K`H)d sDz "ƌB֛70 zh&]j XBB1%r4:̜"żERcêA@4+h%c5ֱ^{0&$73KL,b&F{FEQ"Hc%9٣&(>x+/p8;ך,]a_%^Dr0lADD.^Ν;#>>>_裏wQ6DFFbxk.X`q{CͧW\ya~6t(гtѱcK%C S.P @S59zduSVt 3/G!C0wÅ yѠZ51@_S۹X1Ν.)t8""""p k]vŖ-[xl޼'Zn_8~8233{_s=QFѣÿ{=Bqcݓ̟ܺҭ` ?|Ԙ` *Jtԭy  8wW˝;ԍZ(+ ?ݸak4<>qo+U ;z \\ EԮf,\ugLDDDvڵkuyT IDATp7ʔ)#ALxBBBpqqAӦMѢE h͚5CժUP?G_{ 6lg QAyum*WŋsmHӮZS}$]bቛt T6Iꮽ|`pz)jAmuN'}=8E==Q %]\0=iiILDt|@|?Qti899YYYHKKjݻw8TDo݁QKHDŽ jM}tjM+]B:ƎUCK잃Mɽ{K5Jkuw\5 D$&&Jg䙿xץ3  0szY!EϞ%bQC 3mmZS/$]B:]S[V_^SN1ٳΧNIp\+"@DDDDDD[>>>x3΀!]b ] 9"]B:9Z^^%6YYuj MOK-'GG,<eUJW9#]`8"""""""SpttIJeA:奆frt)pt Tk*gNd 0Pt.-]`˖ä^ŞlY5*YR&#CqEDDDDDD+͛7cҤI)DϮ ?3KHG[SDEoo_ _|Q:żj0)HTݺuc)DԾ=0~tQx8$$H6m@988`R^> *T1ٖT(pEDDDDDDb'NE<0 VRSKHG>AOtQ]2?X6e 9;K؏n݀å+(pEDDDDDDbŊ/~z49 DLk']a l,]AƎ:tx_H b8r%7n,cFP/kzQqqq.^9T\. ]@FAAgIW.__Q# O> V cuÅիֈ($dfTЋ(jժwww S+RF˗/G2e99"]A:,LdeK.??+lV`oKT6Eoo888H+^(&+c̙<+/Y,@C/"""""\۷/nݺ}aN2bŊ^éSn:T\Y: ZZt)pt Tt@r2|9pt psSkX1` yk z5 ۶Eg4U<<7t(m؀%'tRS3tiQprrBǎ"::ĬYP^=4+Vo7n`͚5xHRL 0>%]B:WC 3sGHQLbc /77C~=>.Ŋa@۶;q#5 ^nnY[j=E¡QsttDv . <<˖-CQct2doߎ̛7wMx8$$H6m+""Ԑ".Nt20etjFvFŊ'#htF-v놭b?Ag˖p,KiS`l !ݛNDDDDDT8U^&M¤ISNСCow!))I:Q+zs(^tYh(r%0mbだ gնtӧ%JHPNꥆ%6?Ϝ9(UJr{w8ǣ h Q}P 2d*S5*VDڵѡqcҤ jV$EХo& 7o͛cFHH>'ODJJpUX[FVкuk4iEcʉ`3FtH</޽%qFV.;{jDArj{j_p?\p M1q[ҨX<+_ WG5РZ509wd4tZS.]B^Kg`0dHnS֧GHЋL>>>DDF>} ߻w_ܼy񈏏GBBC$''###Vwޅ+dɒ(Z(QT)B 򂇇<==QbExyy,._YYur5O]TySDZOOm[TdtƍTX2_ԵkZ?5M7mRΝYX,Y@Xt H2eаaC4lP:(w?T֕rs]PNOQWժa$$5.]B:*WVC 3{XrEtTd?Qzح[ ǏܹsdN$""""""7nٟ-9f 11%iSuƗܼ%IG(:ZhѰ!_&$@pfϞYYYO}\%зo_L6 /RV3g%KJPNڸQ%`r5<1vy=]qqu%6aa9sek(:vTkjjWŋ@L[0Ro#g^P\xBTT>| $''۷oGƍ|>|Mk`ƍظq#zTZ5oC@ĉt ȑ=%6!!jMM8:JPN .(tѣ<ɓ>d_Vv.9yR)SEk(Wkjvg^-Z :u iiiے?O;wXf DDDDDDDc:~@1kR?.]bkZSCHi#GKl PC9?'SK٣~O)]B:&LP[4:$]^Ξ=~!G߳n:/߿`ٲeyDDDDDDDd6HW.^]hf`> Ԭ)]a-]A|}:u+l1]3~~@4h"Ϟ˘"""""""SHIVΞ.!J ʥJI<|Zt (QB]P.W.G?-tW#e_Ѣj@.]b Yc)QkBg ^ŋg˗ׯ|||~=+/XV3W\ɳ$"""""",\DDH?3IHpQJה5R~u.t)rR%wO˗KHZSNlٲǍ7L2bg>)SѬ:LnTkmѤ #]aTtt h|`| *JtԯvW.hѢO~q%̚5 %Jxc/^ 5))]4u놑#GSN(Vcٳg~-DDDDDDT\,_ܿ/]B:vF0 -KHGǎ1FWK;o']aLMH.!m'JW<8ʅ={ ==;::bѢEضm*~III k:::Ý;wW_aڵػw/_nݺ=BHn%]B: z0:rXVCg`W/ 'իY_d0 U[>|(]B:U^뱯988`Ŋ6m͛ __P>S4im۶>FDDDDDDΝ@LJ&2mТt_[HW)SV++Hĉ3 6m ]:HWjzitN>x =:[ϱrȑ#1bĈ~x !""""""6IW.YS(0ؽ[tY,@:F[*'??A mۀ> ]@F^Ǿֺuk̛7/[?…%K:w 777ٓ!""""""+V<>(.(+']b|1pt (ZT ݥKl23֙GJ"EԚcl ܣGjߐ T$]Q(qرc`8::fԩSQt?|#7onZhhh^Ȯ$$@xt RE]P6wK+WKHGJ[SIIjMIOO5(bŋ%]*ZT1J˙3g /AfͲ9z*U z_V-wet͛?pt h3G(:?_OÆjHa&oSQQ%AugĨS%N 4>իW655r9ͿlٲcbbDDDDDDDv+, X wOtt ds dOqFԄѶ-0qtQDp!?׫NjL"]Qp̽{"555ۏݺuQ\py͚5sDDDDDDDv-- X 8yRt-.(ϛEed3.!NNjZtͣGjMH.TIj֯.!]f+q襩J*lܺuڵk>|g/9 wŋ+WKHs~~FIIҥ@6L&S0atQr2|9pt pwWEKl<>8^t/.]`8Ժuk{{~g"""in/m׮*Tk>vW/KDDDDDDTP?4j2۷ *Jtx{K<.&X.!u ׯK84i{hѢy!333ۯ1x53|3w*WK;t']aN5`" >^tjL,]at|J(45n~݋M6eK,~e2d"!>>>pvvJ'N K}+NRƥIPaq b"]B:zrxC;^mut QK#FeZFFzEܹs'՚7n\]]H/} _Z*N$""""""*v+Hĉ#n^ or&z%+Hט1@ǎF96Md8ʅ3gTRaƌpwwG:uPJHJJ:;;cƌ4EFpȑǾ'Ndu򜛛t֭_JW.__((صKt4l(]a'JW.I ;ۥ+ ^PBO^ZZ._(CM'L?OԮ]q){geeg_(ϼ_5ѣ%I <=Kl=Rk*$DtY,@J6V+aCT"]aaptQ+N>}UV c_駟ZBZ[oƍ|{EtttHҀtQRt)&]B:*TPVr2|9pt pw|}EKlYL.HrԚrq.IIQgƝ=+]BLo6mB޽s\KƶmPlYGcܹst6XVV>6"""""""mfr6ܼ)]B:40ߚ"#KHGݺ[S%[ $.X.!7z|s+WKO8|mڴ֯_@--; A<<" O:ZI~+ΜQ[H޽+ΟW[>x ]B:z 0tImɚ,]B8cq!|5kիyzzwزe N>&M[Mn@DDDDDDf1i7滽{M+H@F7JW1c+6lrp ȑ@׮F!!ڵ@Vt Qr(ڶmmŝ;wPD =UBԨQ㱯)SbŊ|~TT 5jԅ IDAT@͚5DDDDDDDb,`,mgOajΞ.vw#eZS%6۷滻g,MٹS)H z'''TX+V˗q ''59sGF֬ʗg% _T۶5SQQ%665e){Ԛ.ٴINK7$""""""WR88Hܿ|:쏇,]bb2]}j"]bΌ3]}K5UtÇl.!z?I]3`| 2Rtԭk5 ,X\.]B:ͷEue_[S @xt QqGۘ?>KܻwO:ȼڵ&L0vM]P.!['KW]Tlt h6M 573͚m$* 7D8ʥ˗/{R 9s7ٳѣGxyyҩDDDDDDDԯпtљ3ʕ@jt &550iҤ,#""""""3ǫd~`& 5f +F~ _X%c(K `Z +Kt t.]atf )]BC/M5o̜)]aoK_̑0VKC/M9z|-жm|!""""""* 0:w#=ٟ=C+.^>@mI[7` 0`2=ѹ30ztѕ+jM%%I!4UP![srr€! cW^0[`zj.!F]HWkYY%cp{w #GԚ̔.!C!;^ K.45lߛ}+NRƥI+Oxxxoȑ#޾}PfM=gϞͷՊ{/ߞtQۈɎ@Pt9Smwh&l*]ANU۲@`t4 hFh^`f "z];w{iiiXf 5joooL>}ѣ\vzz:M\?]XU+6m ] PtQ` ][hV5"dKW!=0'{??~uؾ};Oʕ+@@@XbX"ʕ+WWW\rpr5qqq8|0\""""""z-.͜ P:uk 58;;ÛGF?#]B:6fϖ0y?uKt4nlA*֒Z[d'52{*2RQzijРtVF/˖w5J(,Lݝc)g^y;V(<\ݙ(]B:ڵ&L0vM)lI47ADDDDDDDt&]a꜓L1t(гtѱcyp 4G)`*u~ٟ~ԇ+Vs. ;(_<|_ ݃oމKD4-::{8gdLuֱcO瀇0dt :SL+|w7RLKݫ֔n0A7%65ePK^DDDD/_756I(wd^5\oش (_Yt¤KleHc!ŋ%6AAjMFE ϝ. RF*&ooo+VL:T)5(]Z&-Mmvt pqQ]]%#X\w5R9;~~.PT۱~t ppPKlV`z!zFp DDDDDDD4ժ f,Y\*]B:*WV$) X\wQyy5%uWܿ|pt P9;K$'}dРA3ӭ&n:\R:RՑ#GdɒB5DDFEF;Qa/Ξx9Mنߺy1C P!+QQ6nmTQxqߨ\QyrM///s;wٳ]bc ksC\^Ov Hge[hhtѯ:H80TޙCyC) WyqMQ^KKM no z>t=3xW.pd͛7ǔ)S3*117n!C]Ȧ#DDyvx~} #G۷+Ud+Փ0dyQlY'8~\[7kIsmujtԽt ' ơW.ԬYŊCZZSSrei?<ʕ+<<2==111ؿ?..]K.DDOuEkܹhԨ`o^DTxq ѣ95f^U:ODfp*Х гtܼy08p jժ%XVZSI\ t t 30W/]b.7rKz傓ԩ3g<=777,_G"jŲe0m4deek5bQk ㏁ͥk(??bckL`ZhRrUk*&FF֭֭݁kH/k,wZ jMk']C^ԠAdž^%J L<ׯ_& +ZQtҀ #''Uה3P3.=##"~UQ//{*WV9sKl-ԖOd_l=`RjזTfRCL3HN/WC njlo TJ GjMqC#^.=\3fl5k,,\=*&"""6p 0rtK`" Ly kH_fz_Fx Ĥѣ#5j,ʆƍ.Vk7@KÆ{I?*W__ߖ.sǶ^rvmugoH 5Ut 31b@ 3i.!}JW=KI.!zCHW>39Ytt .]at2lpt  jűcǰh"khӦ jժ*U ^^^miӀ-+زEtM j%]ag(]A&Nڶ0 6m ]:HW<l ]AF:w0;`zj.!#FݺIWzsڣG%d8Gʕ+E1c֭[ÇիBbb"bbbpl?ӧ /`ǎXODDDDT|}5+ݻ+Hԩ#]ae}40 >LtIWm|t3xE ;mۤ+H̙@Fv?^ ..'ND:uyxxqJJ 3gΠo߾hݺ5.\gODDDDTJP˕.IOW[?.]B:U/wwL`QZS*Hde!!%x9 ظ8tHtUHWm8 ]A|}ե+6m +QFg̚50 K/>V2LDDDDJuAL.\.!*oMݻΣ .!jM99I$'ss.^.!nnrѢ%6~?/]B:ʕS%KlRSՙqOKRԟ}KKؤV'OJq蕇֭[nݺ֭[YYY{c_OMM믿!C $=j0W$Ӱl&o@Tt _|k*&F.!uo@ ,XDDH5ͷ぀5QIb"d pt ^yd=zt"E`xQmݺݺu󭁈(tc7+W_3:IW %m[`D `B .NtjL"]at|J-ӥ+"#Հ>&Ft4m ̞-]atZSoK q9^{ YYYZQxǾ7_oDDDDDfW/ 'իY_d0 V>.!}ue&gϪ-RRKHG^?3'%{w`p KԖ|}9R(,LmG$]B&áW.bذa(}WGGǾ3fhQ2EL+Hĉ3 V}7NIh&7JWѣΝ+֭x}1U($XxHt !]at(@_'s+-ZBXbҥKuV*""""<`NdV/+HРtѶm]Wg&||t3Gqi&;vUdf50ڵ  ]Ӧ-[JW}et^? G[o͘1+ """"P|Kddw'=*]B:QC OOG5k;>Y,@J6V+~CT"]aq#ptXե+6m ] -]ae {t^&8mڴyn;#PDDDDD*URC 3IJRgI ԅ"&pr:ETѢ%6Ο.!ʩ;SK.IMUgƝ>-]B:JRJ.IKSt pqQkU&=]msxt M_^:%K)]B:Q$6X.!5koM\SZ5TBZS%re󭩻w՛Ӯ\.!aziAx 3'%%a^&""""{֡0ntQxP ]B:ڶ&N0-KHGV)Fׯ5+]B:ZPgIdzGLt hLe&7o5ut h0Vk*:ZqoţG~1ԩ&L[0`Ӷ={|׷oߞ/GDDDDT`0 U[I}պ23g+az0:^mut 6L%%%kW`H 0uwNRt 3FU`b 1Q8Iثcǎ={_VZ?~ڵm[lį9r$O_?{wU{! IDAT`n(\TMKrռ.2d{ݺR.i*Z{(\Pr Qm:̙~xs>z^gf<߇DV:$ln|y`p$ȑj́IWsNBzS{J'1;tHͩ1cGl@7tSvI'1 VsjX׽ R*wNbvڏi8H4dy*(H:YhSkM\Sc63g{m/h޼9ƎkL+W~㉉f\""""r>>@)mۤS^&%Bk&`哑l p5ejMg$۶?O'y^?6H ƍS~d. @: `K؇?I&͚5u$/^FDDDDUbE$f+W;>L@ժ)rrի,c2իK7֪FԪ%Bk:`)B+ عS:e2uJZX)_S:;;?K.mHH~Bۛ`EJKu!}IMU{<#SFjsڋY$G2/Pt4`R)$*RH'1wX8qB: Q:O)#,#C;zT: Q<*,+Kݜ }K/wެY3ԨQC(ؿ}DDDDH4Pj$!=NBzԭ.(IbS4Bvxsu``SZꂲܼ ̛p[ TTRp!%Zxsm5""P>aKt/駟7'nVKDDDDm[`hZ.s i0XUxى-9F̚ d7W{|jNK'!=7V{ɕ+'//ԜtI: tzpW~巌 lxf%RVNuj!&x5ZgΨVwH'!=w Nujɚ":T:Vd$hZQAC`ZQQjٟv퀑#ShEG襓=ϟ?/D˗CFR_d~`Z׈:|-#:T*$8Xͩl$ǠAj$!!j,$G@Ϟ)VP{}[:VXlړ^:i~?{,;&e׮]9sSz|JCDDDD$dRTd&`V哑l l,2hT:VP(8Nc~t kxukZv)H1cTKr#ٻNAyE/<<<zlƌY6n܈}O|#""""S&<(rrի O>>7<[8p@: Ԭ)BسG:e2);S^&Pt UA쓯/Рt mۤSPaK&M<ؾ}0|pdÒ۔L:o&<;ʔ)癈D"E"IRS^LgJ'!=ʖUYTZ3)$GRjN=e{ˁ'%J F| ptңhQuruNbZ>,(THͩJݿڱK'^.]NBz4hVIB:O=p^:9;;W^O|Nhh(|}}Ѻuk.]...Dݺu1}ts]oDʕϢL2h֬???>}ZW֮]zz5Zg_ӑ4H:Vx8xSO.!)"#En%;ѩ0|t (`ϫ#GJЊS+Tm #B+&Fݜvt0j(DGG#""… FBBRRRlM6UM"""""+#FFzO@.ڔ  t,B+8Xs> <M:VH$+K: 1`гt cǀ+^_dzN,[$$ӻWFrꔚS9!eM⥗^DG@DDDD$g Q#Z7OSF79( NAzMx{KڱX^:5~<кt ]ט1j ['9hN:ڵ)X_|'''Tvm3Z՜(Lt 5kS^@͚)={S^&)BkzS^&Pt ￗNAzK mNAzLt M-[SXRV0h}駟H"1ʮ-T/.,- XTR!SS..I/W-(.()#,=]ͩP$GѢjN+',3S<|X: *VNbvSIH/ ZU:YNjE~t^6`գG1vmH_UgUK]P67yh$TR0>%VML5d`B 2R: Q'YYIH=Sh;| !K:j߸t$G@߾)NT{I'!=^{ _:֙3;I,z٘7BBB흯}_|1_KDDDDdN6Nl(4 Ocat kxU+Z;w)Hc^N{7/5 hN:ڵ)H#Sh:|-;m򀧧'~g|xgXE;＃ԩS'OEDDDDP|}5ShI ||Z?J |}}߰{ԯ/B+0ضM:e2^^)6mlNAzM֭)(X#Θ>}:"""0yd1p@:u _|+f񉈈^RHQtwU[0$Gɒ_IUP$3Ϩ9UtL:a$S+J'1_ͩ`$T*,'XF!d2իKZ8p@:^yRJŋ_s(Z5jO>QQQ."""""kxx/Fr&0o-pw7ޜJJժ_F ,ZDFJ'!=*URB, ,Y;'puU7YqRS^LgJ'!=ʔQsxq$fiiSSͱ.]?k.ܺu 1g7ݻwGfаaCxxxM6K/!CGPPp ]?1hL,B%`, >^: Ѥ 0mt +WٳW?.իfT+gpҋV/#WIVtcGZG+VIH7zNu*ĥK'!=Nuj!&x5ZgΨ;IH݁ASh;ZJ'G`ы&OVdv`ׄ @V)vS^c/$Bkn_:5jЮt uS^#F:H:tXɑNBz  t,B+8X ΖNB`ыL&C:V@)H/__NZ6/2ShmܨdL&K:V` ut ktQ#Z7OSMJ R*2#9%?#yڇJ'!=yFͩr夓ef֙GH'!=bE$fYYjNt\Y:YNk!$IZ8p@: Ԭ)BسG:^i5j@b gggxzz^ /񸻫 5F,X?/VM]P6d`B 2R: QSNNIRRŋ^'d*TPNbb:{V: Q%%ǏG #88qqqHcC#::۶mÄ PF <GDDDDTp4iڨˀptңQ#` ZWfEaCy*.N: QTb"0{6+]xsu`\ &F: Q͛@tt^eddØ?>z-4m+VDQV-4l͚5CVбcGtCr 6m ";effbڵhذ!}dee?:w NV$'K'!=ڷ~[:VtPstWѣSh].(?_26mqShƪ9(h8Q:ŋ@tңys`dZ.ؙM^:-[ĉn:?~V޼y:txbQI233GsHJJҝ0p Уt #G+ $ǛozI:~\ٙ>}}Sh< ,[ܽ+x5u2ӧVt>SIxjɚ*2D:Vd$hptE/իgrrr0|pV~t )))6HFDDDDD VSh l ;x%ZwI FRd>_:juz5#6L7``* ]`uN\ #tڵů)Z(J,+WDPP-b=#lSG:V@*~}QI` }t dR|I` ut k lެ~>M 4m*B+(H>Mx{Kڱ7 bK_~ sAL2 [FF># 7|cq gQEeef6GH'!=Q eeE: _+K'1Q+sNBzLt 5kS^@͚)֭SS>)NQ SB0gΜ'>p5j"##gjժ=g͚5͓.ُv?#IN.T{\Y)''$f))js礓* EH'1KMU{1=+([VNb,]"S*R<#w՜ NBz,TI~I gyR ۇ 믿KƴiӰsNbǎ:t( ڵk-Q#H^f._NBz4lhbjB0{6ptң^=ͩk׀9sk FrS11IH㝧nƛSII@TtE/+?yJ#44/_GZtF!<<ХK4iݺuêUyf.\Xٳg[4>BoKЊ,nݒNBz 0zt huM$G6ر)bbT"1Q: Ѫ0at ??UT%Ӣ0yt KԍDIH&MiӤSh]SWH')0X޽{XtCתU {A*U:Ʋe,:fcT~Lo_ƍd oVO?W99IHaÀΝSh9-<M:VHj%0@` #C:cK}=،3ЬY\H+_jԨ hۻwoCDDDDD29Fl.2>_Fl*1CGh$[7K N6Nu+qt k$Ho6lNAzMڲΝ@@t ,^8qB{r0eʔ\x3,( OXEDDD@ZZ._W"..vp-ܺu IIIAJJ rʔ)pqqAQbETT UTAŊQZ5ԩS'W+ĉ|}՞F٧&+Kݝ\;Zt YKܑNܽ!,_xJTsj>J^^eJORdkV-yg穛7(ʕ󔷷tĢN&l׮]_{- dɒ߯_l*nDDTp%''ѣ8vN<ӧO#<<믿_g}mڴA۶m^^^ (6T?L:YBM{%pvvf l'K'!GѢZE1{tK| IDATTJӐ4M ?s$fWfTji +!:-[V%6mڔ>WYv5o< *T@ƍ1qD]Νף?5j+W"/m`N=ݽkoĔʣPkzo)Bi`2՞O]ekW``Zj%EJtңS'`0Hڷ~[:Vt -$E/,zvߏ˗/ù>N.],Wh=Rb=MNNN> O>X"^xL4 }nWw0|p?Dbbt$"zѣW^NonE/ׂɋb8Y#6L]T&!CTH~Uc2"=H/IaK IN6mdѝ]u|X4?ADDdTgFJ &`˖-qt|>pD||>_F|]l<D1C1͛ Lo)ȑL )H T[V"[;V6ݻ-9E/^x{yk؂>+VD֭-kk֬Ecӧ/RRRz߮89BTH{dg~ {%sׯP;3o\&ʽVS=^Ӧ7˗oNAz5j$ WLMMN`X„ zl—_~7nDorQeNOO: 99YjժA_]"-ʈW/_?ZNKt&.^)iSDTu &B+&F:Y}{S#y`hdC,zY7@z<22w~aֆs={9/^<ޥIDDT۷mڴAllt"#W_Nuf ?{H!K t$a۷99)zNA7DDҧ!j*/^ܦΙ3Ώ:''GĉQZ5 8-.\ȂgΜa;'^:͛1㏑"}6 hD:@`t "2 T[V"[;`k=Z"Ǣׯu=HLj#гgOc?3F777x{{cHLLx/^DhhMt:vk׮IG![9yg/Pt??Lt -U""MkKxXptdՓNAdR|]c^u|(SUt .|ٳgcҥr劮qCBBEDDD!22=z@*/l; ૯p$GrK3H'AKCJVKNbv3ߤ/ʖNw#)H"EԍDI'!GS+K'!+e#:uBHHZh{͛QX1'Ξ=k1 2eʠvhڴ)6m:h~ZlM...qr;V:JBI'!=QE AiF`SC.ɍySZQfTj*p!)\Y# T1H$mznݺ ֭[駟O}Mjbȑ(\p|rKDDhQvmXL 0g*,պ50~<`MDLy4 ;W:Y\0k{@JiHZFK'1zUH{@ժiRFlI^=usGI'!X1'''{F\\~={ q-eˢAhժڶmBlp¨^:<<#""rdKFӦMѢE _>իE8Ŋ'<==ѥK?}6b۶mضmmzܧ;v,ZjOO|=.ST IڌrgOu _uS+֬Nb,YL*%@3Gi'ٗD` $fӧ# lrFx祗cŋXCիWǨQgF…>… t(_?*TE;"""{Vn]+xѢE ԩS'On<ɭg}ݺuCnݰd۷/eXJJ a׮]y~,'6IWEݻpFYoJbULݹS:Yp:O Q*`7mQ<$,4X0n S7M#ҫat^vW^쎇:v숗_~ T" *;cǎￏk";;;O?bԩSͪ}IHiſ\Io0ySG}mۦZ ?95xtc8uA知ޭ԰aIQۧoK'!=~[}ڷO: DDDD6TX1tsŹspy|W0` ^Y&[;Ϗ7mڴVNBzի!NmA}6 "5 UK:j" X2&?HۥS#ٸ;׌U%n/_?7n`׮]8q"ԩ#j^^^駟9 s+$G2o͖0ӊ1kZ(9uAHhaIQdf6FZHWzPA:YVj"Ev6ZHUNAE>6A;vĴiӤ# TR׿w^cʕٳ'JbI0Lعs'fK,ɳIHb"0{6#TE< J/ʕ,9c*嗚5e#u XNB") XNBzT.(jNEDH'!G ,YK'!=ʕSѹߟ9lɓػw?DDDD6駟b…h߾}2:_~AʕdCϓI''یs"0w*&M'1f*JCiS`Z/fIQ\ԕ+IH/:T:VdZI,j5!ٟvQShEG7oJ'!GÛYV ) Lڵk8t """"ݼ~z*dp[ldrrCBp̊ח }N8ӠA@)V1eZ_}ӥ}}H -ݓNB)5Ҥ=zH(0E/Xft""""tS=ŋq9Ku+qt kys/?#+#|l\n$fDeKZ?_/"@kn){6z4+)S#9pj5|8Щt zUZh3ٹO>6w6 dZ`~Ԫe˲ _0Q`Kf2kKZعS:U Pt ;S^>>@)ﾓNAdf`&״i@&)uu7nܐBDDD[ѢE|}|L2wU S..HQE/ g+ El+F&qŋ eH'1PmNBz89󔫫t,`J $D: QS*I'1Vmݕ/rrj^&.@_EfͰzjddX3z 6`}IGt:0g $fW{" //uϤ%$̙tTjN}d6t횚Sｧn Vb"0{:Oլ),nN{]$Z;w`5(W5j&Mm۶h߾=J(! `РA6-zegg#22^^^6 9u)#E>e V R_Aͩ+?,XAOv_K'1S%˕NCz%`(bb z$uUt70i0wt777o1k,˗G>}phDDD$5m|`[UIHAfX9pLkI:vLŭ S߾@)I'!=zNujK&￫n5ILn) <̽{ehѢ~WHDDD$]v6͛6 n6`Fĉ@˖=< @V';6nкt ]ט1@۶)S^#G*B`H_`j 'G: 1dеtE/;vQncǎEzzt"""РA; `e\f2uJZرC: ﾓNAz?/Bkf`&Hnild/``ɒ%hӦ \"Y:ul:^J5C.ݻ,_8!(QB]P.SWLrxȎ-ji+$GBjNU$,;[,UNa] :$Q`׮NAzL@Z) $gѥKԩS(^H,֭[HII۷qҥKرch׮<*Uqb"""25jt<*n{NC^0w-+yWDjUufd$fokK!KUԔ)ijߜzӐʕS+&O6ξw_}V7R,5mw=ʗ^xA: Yy7YT0p@899Iyׯصklق>7"":uBHHJ,)HM3g#C/fWiBnV+7CK]Pǫ̙tTjN}tOUԭ.(tD`w5Ӑ OW us̙\5djT+Sg̐NRlfޫR ~'[vqQ:uٳ 4k?}4Ǝ HRRl:w2Hm$d%K%ѶCW#ShEG7oJ'!=ڶƌNׯK'!=ZƏN ̞ `dIShũ}BtңiS`Ml25ڦTP!G+>yWFPPP>#""" 6D6PH;(`%%~}Q+PR-B+, XLIHwo5)B(-2={Hw$Gn)ΝS-Y&>u *(PkڵknݺIǰcǎ1{.L:FiLDDDy{pU2&$g`z CBOVaP6qQV޽t kHH֮NAz t$B!o$ǐ!@.)UԾcd wNQ `UTvm:tUV}oQQQXb@*"""Om܆Zj6X@st zT7Ϫ1+@)6nmNAzd&`״i@)nU*OS͛Kڶ  NAzMl)W#\jհi&*y!w9WF Gv,# 4T: =ŋqي=h X%"V*RT(}Վ58X: e2WLNzZCdWNv-t djՒN-2ڵS8S*[^iժFΝnZDD*Qc['S̟DEI'CΗcڵ &MBdddF ,$"ul=z d6lPnݤ83/y+V0vΞQ** e2 5NI'1۴IzNBzL.(8!lV5xm>/F IDATMѣI̶mSsj$ĉ/$D:?*ou#θqjNK'qްx] 9Ei,^lY;..ΝV,xݘ3gMYu9L`JuG)_Ϟ"+lרv颊+(K=3ڜVV+t.ZR^Eի4jnUC57֪FԪ%Bk:`w\Eg`<"ْԭ+!pqss.@\nP"@izjcGAXX5j0p@#66ϟ1zꅚ5k"11NB {lA%$$/xh߾ƣ 9YPPt~~ȶag?899*/Փ'Fiߜ8)_WO: Y5 c$fw\]xYY8㑑r\? \IO$\ @ypP㏟xƃiij8WW+F⢊'s=`r57NCOq?;/xT"/_V籜wr\|F+ uuus<:nv͛[;巢Eg1c?Xr0...=vU$h֬Y۷o?ҥKx>>(T BY̙t6s*~Uc|8d}Gam8nU+7-UiS7}$$rT/͛#FHк|Yͩx$^Xxڧ당[࿧NYUʍ;,0 jk}8 ˒\?T$`>)NNNxwj[o "ʭcԩ6?)b19~\IONp20/u1 ƊSQ1M>=~8yRc[3gTC K0ib=~Y]> lsTKVķOmJ'xXd$h;RS1j\lWZZHGs(usZRR#2.Z=F[-zߗ-[OIVN.lu'zt|"XnlӵkW6 ݻj)}]5 hΪڵ)-[J':t[]P kBh`)jNeoD!!ʕpј) r-4Z8v XBET`Ο?ݻIQ\rAAAO|Çqĉ|JUpnn)֬NA`AHسVHB ;S }azUP%* f̘1ϟpAHLb|%CFFF[]w#&_SA8xwЫW/KdIuA>ҧNH/6l@hdUc 4EjNiTٳIHeU1xq$fiijϸSAAz@S{[ u?!>(SF:YF7\D _+9LιG+??}o\YmRIl,L gN"Q+W`4pg,ǁF 0猋3!"\ 6}*>fφӧu'n,)x_;S9sUG6]fϞ}ߋ+… ̞ B!,,)SШQ#T´iӸqSԩ-) M47ԝڙ30}:\ۯ-^2[9x0r a2YI;Gݺ0lΟW/N"LĂ*uX 0y2\dx.O? GNa-*Jݧt'nfP o\S.>.e6""" a&}+F63n!441p@ZlIrxBᖢٽ{73gΤ[n+W+2j(Թիv>ުG\(V?wy{WXG}DΜ9]U!0c 9;+ƚ5khԨ(UƍS ǎN!$ vt'1t>M}د% JcfN_6)vQdΨQjA!I2|vQb gM_~tȜCϾ}t'ɰmO;pC[Q= 74nY1%K(Zj̙3ڧz'Yd I!ױcG>,^Yj^$,Xv%˘i?8^reA %J8gl{XaS6C2SX[ Vw aR;P 6X\mR%)] 7N!>p/TR^,X@w DGGLժU&~i'bp»Q 4zM~9ƢEM!ըQ;vB``8B BLΗÉ٘-Y,9l*?ZwHJR=՝D#wnuMN-X\stQd:A_$RSŨu'n"_ᙿp玺tQdM/aÆl21lre֮]K=(S ~!l`O:uPx[rԩ~PIжm9?#vJ>!OrJ~W^xqn+e3iYILb0`tnsFUgP|ԕ+0}:DFN"QZP681""t'&uh aL8}SzL`l ӝD?ɓB a, 55}pB:vH 7񤦦ùsx"'OѣX,lԨ9!9s{|ڵ^Kݙ6mCYt)3fpJF!4hah߾=ٳgGIÆ0dZ1HINeqKm_~qhJ%K2U}օR fqL&@B[ծ #Fԩdp&Ow1WX7V@ G ʗy!/~~w_w#);wHLNr|<^\vԴ4k?oLDEeܧ-ȓO ?ԝ$åK>5a{y)ZAy޽C{]o޺EJj*))s>6Wx SS5 6vbb dIل0Bˁ8pNצM g۶mwܷo_r׾Z|9~!9s4<BduJk׮ӇJf5 ̥m[ujZI2; ,9!U޾=yyx\Y >sݜs$̝ F\Ȝ-5|$`=ѝ-xyyQhQjV@ X|bV Hsp<'HsǠ 2@i5 ƌ|L8GӦ>eV;V!(U=K{JX8}"ᄆaniz_g)3g2N+XP[.!e7^{517oe}mʕ[C7ٳS !DVӰaC^z%ZlI5rVy^]-ܩ;I~@0?'7W.^mԈW5b|w/Kmcv@&P5oغUy S{NavuM;[KkJfMivX l\y=p+wߩk7\Bc#pˋQF!g}F\\?<ҥ s~DZc 'Y˗2d+WZj{?~\w4.a {)\JB#sxoSfƞVT͸qPV6שF ʛڵcϴiD]ˢ#h] 5/v8B;~8V>, .$99Yw,afeI2$%^Y!3g]]cdULEʥT^.oaN2~>>ڨ_>׬aƠA4VnP: pp,T9> tN5VBt'p玺!v! @r1cnJ*VT_fr L8Ńs3}@y81""t'(]Z-I|<̜ ONTE `L΄/_Cz1cԮ\gda3!!́0'%NUOi#81̓?НDrVv^|kMgt_?`  2`_HL .LtwF3f63gwޙ~o_ޑxB!2ҥK > *duGfTjxn&)p$Nsuޘ6q&@ax.pIG[xᰚ5ah)EEԩ;fxέZ_x`HZqx*_ީs%M>u Z|ĨSNryrd?Iڵ42 dʕGGӧCXlwgXaWNٞk)յkWl{c/ZȮlB!s^u5j$=ܑW^g?/k~g8yUOzssCc4Q͛(  Sԍ{4o}Na-, fVt#$3/;}`r6LhO?rNcшxevST(>'ii{t6䧟`HIѝD W?N~îF{p# :NaRZMmK}ǹ qi/UÛ78ƑM/˺u뤬IZZ .g}ZjU'Z`ƾ~:QQQ>}'NoqQӝ6ӰaCmF5\:0yB͈)Y,O=;ݾ eɶm=;KF^;.s*I<;_},SSU̠ ][wa^Hco€ 1-M]S B.0nO];b Bƺ{…}M$~v?ۓ&NzþN3өF!#^n(00sw^ʖ-;pSs"̤I(R(P Q^%fϞ=k !ȟ??/2ӧOٳ۷iٲ%NKڵՓfr>ٌ_[țcb}VTlbcaTsWC)o3 <鄙plzҥKf"kq+Vxoݺŵk2Ojjj^ϊ!DE={6ϟgĉ-Zԩs^t_|8в%;S`s7nth ///~ Sst'xxu)ED׮N"lTlYFth|AK5lCNaYuM9gThR. o>p”7G>wNm_مp욘X@@jf׮]3n8sJ>S_mBc̘1>}w}0ӧѣׄSm~IZ$u;5~SW7{y~IJIѝDأsghNw k q­ӣePС^GEϥp?m۪{; tn o :aNT%Y0˲ݐk׮MX"r.???C)^8^^^# cX7oײgNF(S 6ȹs8piYz*ׯ[n6#YIΜ9yݻ7}[Flٲ>f+%1B=]OdشI56g-[94F`޼LߠD⡶nUWvQdΠAҞ=dؾOI rNcC]/vTהN7yuuڹSw ?@O>37X}9~6)#W SԠA UɄIdM/ooo.\H*]vq˗gTT׿Ŝ9s ʦBdR2eعs'3gd6ͬ &ТE z)n,8Xm~jysI觓'~2y H$kjPcӍ"sƍSNԝ$CH^~YwanM5)"PсBBCf;(2gXI;I}CI=FRY}y>S~%ȾJ]S]8rۯ3f̐ /ݿ!5c C6@={6iϞ=4/Bar^^^ 6ܹs>~ZZ$==˕K-(;ᚳ۪|}_e 8KkӹIcBKMUetQd6"-MceGl>ތtA#X`rؽۈфPV'U.0~<)/qT3>3שFeM5jo!6ϟg=M .L- gȑbaņ+y|qB߉h"nLK\fAx$Vr%"##wΜ>ܘ@"`l ӝDأHl2]7n';Af1tL/?Nb$}ʉzm >_έNeOkSe>)3jYV߿?٤ϟos}dϞy֭KʕO>[n>Bx5ksNoIf~<ʨŋ0e DGN&8<·QL'Vhcǝ4iթ`!z50ۦO?ҥ/&^!z{dH<[Vuw$:Kh*$'‹>k`YcTä$G3'ה;Zb=M'dT|ɕ#齃b/cFɄ'a|yb`k"-[+t'p*=z4\.O8_@߸aX`lxmUS'M/!<ԁx <<*U,Kzxbz-J(a؉ͮP;|O:nqܩDtȜ_W;wNa.(X|{nz٣~ h"szT)3߿-S'f#Q\w q#]R^6{pO#FrfiNʊL1zbhիQ+%y&&MA\r塯/iԨ[ly))GƲ`d ݻw[fj#fذa|͛oɞ={25?~#uF||]s!jժexgϞ5t<||`8YНDxtSsj?ם"Ţʙ}$ƵNZeS6PVoѝBX:`kךT(r.Uw?fiժTX@=z4<Ξ=˷~Kxx8'Oȑ#9sŶgAÞ={ȕ+գqtޝeoq.^ݻoIOOx1114nܘuҴiS/NPP5H"VMMM믿ŋ9s/f^n۶mUV<*T"EТE Bᨲe˒={vN?ސqDRZ1S3UbEi'HLyN7+P >ec IJ 5U4H20 Uϸ@QCG\}jHHHНF}[ ;֌%ݺeX k`AnnO6'OdOq%''sn߾]^7ofvex7ocvA Bg" )o(BA?ׯ6b НZDZRQ"#atY!Oz0tΝS!6Vw,-3yz$~]1 .2vj*736 3N9s`hȝ[wa_T-ӝ$CX:ې74!n0l,"")1c ~g&(QK"CjYe5klH!7Չݻu'ɰcGѯ$SڥQtȜ}ՂdسGݧ//i11؛Q7jd @kj~&I2,] C};|fL6B!u'x);Vw k2a;w_z8 7'^B!"&&˛7ac qWЩ֎U%Ēt'aRьrvQdNp:sFw VAм$e-|߿~|B!WN7>u$BB (|=ܺݻ3 i׭SW^ѝDB!o6lz XIIѝFySm,U>%Krɒ.f> Ϗf5k3X2KdU`AQCwaYiݾʇ,<;G/4|zϸ(5U9 4Fx>>0~bcuQ`2uMtX,Lt©M<敞~)n# B!<ƍILL4l\w kÔ);3Ϩtfr"L .Nntǖ}i3y)^|}Ĩ}.NR^q_[?]S/k9IɦB! o5t{x6mkW)?J޼;GVнC=l2y6-Z@^SX S')t'q[o/ZD:eTfЯs;$<f4OEa矇tN;;3L^g>n瞃F` vHdz8IIɦB!׿Ezzc6iȔ}ޭX,{6*d>sGwaݡuk)<1Nv?-[ LԪT)s/ڶ5CU=SUlz !BÇb~~~ԕB1cT'3ٰ֭ӝBkHUKw k7nBÆ~NnN(b]GjԤ ^^^Ð!а9}\S7qc)N؜ʒoߦK6[׭K92д8d.XBw aB%B0`᧼Zlc a`(YRw k+WwN! eNaҴ…B)Y[NRrI}_gax80WYZw ajU)_N1M¯aaN /O:e>_;0B!pS&LlJD֓/ZəSw ɪ,ѣ{Mܹu'p** IDATo{ſ|t'ɐ$n0`t~8r)(W7T`앖˖{d˦…u'ɐFݫ;{orV唱T)Zթoŋ;e^X,N"LD6B!Ж-[4iK/>6+_^-(ɕ+0mj-O2滦` НDأdIl&0{6>;)Y,̚ŲoqctŊ~]]SNN"Qu'ɐɓIko'ӹ3V@uM9m~%%^ǏN"LB6B!́ԩe ^{5|}} Wԭz0y2N"Qe&/k%I=~Z"4(uMEEY}y70zB.k Wڝ;1m6mru*WI R6bb`pAwa*UwM]mz۶ԩDfa/gNdPKVhbc4qlz !´ܹC~8o/C楗^͛-[6x ![CSX{hf+NR$$N"Ѭ;p5K[nz`\Bەb'u #:*_ڴ]NKLԝDh$^B!0.\HJxwb,RSSywر#I6BY>>>{N[a󳺑#Ui:3ٸ֭ӝBk0xXv]֬ѝBk仙lW?[L *?ƌaӁX,t۷x1 㬓?;z{3{P>p 4nl옎ڱVҝBثx\:W۵ ~C.:u ݺѾaCc+_|ի4 ?#>&zDD!W^e͚5YlٲQfM{9֭K:u(i IIIرM6i&b`A *Ă \>h\Q", Q} ѝFتY3ضt$NYԆJ0iTݧ-ҝP 7ovO ŊQD *,I%XO+Fqh89oO[x8lݚ-Zvƍa@u,"#a4Ӡ ~֘Źsjk r7oÇ@%RrRwcE p4畄EFoO˩S8{ѿZթûF?Y ӧ;#ΟW&TpB!GKOO_~UX1ԩCթT+VbŊq"͛7 ѣi !Bx8___V\ɘ1ctG¹jքѣuS'ͷuL .N"QZPn!{lۣ&L 6UZP6X6 "#u'(_|ի0cNhՊ&Q8~=ʔ1穸8uMEDN"@6B!ooajR=E ѥKɦB!?Ahh(M6Eڴѝ/2*)){tNa-4T]SoN"Ѿ)kЀ gtGɼ6mK)?J޼;Gн.^ԝm<6>7%CЫTIdK!ȑ'm64tzLmիu85ҝpON!Q`AM7F5ӝ`N"ѫڨn#@ ɓ\8׽P5aRHM՝DD6B!ŋSXKHPt'(\X-4mb"̛'ON")__NB@~fnծ/gAYQS 9sN!) .Gu'PT@F ~7=5,=%Jd9g|t'ɐ$AY B}|g\|/ݻL5[.} 8ooyQzIPc&.Ґ]UjԘ:ΟםDأR%ܧr 4>^ټhRF&jf DQ36Z w L{-65 ,ΧKBLlߞ}\9rhN|*Ub֐!D]1cPH+YהxpӺȦB!B9h۶-+V 66~cO29s[nT;Ν2sf=M-ZT.?n`7lV ˔D\*lR%tv{mtoB_=NoL0>/穏>N̜cǢmӦh۴) Dzͱ쎏˗WBӢn*uUu$mjl$ӧKA3(ծ]O>$|I@||<֭[7_~ݻ;ŋGӦMѮ];t ]tA"0MQyz$V+ÂE-65X ^]h0P^V;Ú5ҦzJ; zNB&^xAV nFE, GSRp$) RRp"5Gqlz:^8 ,BBBPLX\9 CԊTAԎDuQҡmv/}Qof$˗E?rŊo-7ވ;;C'NDj*8~4RΝ/-U,W!*-+zDjV*V*Q0-g8\))~ *,zQWT)n[}ѣػw/<#G->>O?\jժjժVШQ#4n 6dHCh)FNN#9sNC&1cgΕοӐh9O=a^;ÂҦxr?O}?nviS.i'!X""""""d MS-^ ,]LhFieKv_~)* NF[k[XH;zYSح\ ,\L=4оv UxqZcыBBd!յ8XY^; jNa7ovv 25~Z;7Nax1lv rE/"""""\ҡ!=x`^$d""B:KN␑!kڥL//m\9$.fɔO|ʔp$/KڶM; (YRTJI23e͛PyZU;Cv5X""""""ZFGu IN&OMSIDzҡHRSSC:u]F~Q`PvI:sF; h>\;2i$\2^vsL t额nF` 'G; 0U;ݦMҦ~ݵS +ID>@Ϟ)~U֍|Y; [;Νf*/NSǢ;-Zh3Qp3hR;ݗ_Kh SF[k+OSgS;݊…)3i[ X@;6 Q;ݚ5)<<jNa7o)Tt4Pv  NA(v ~L4hONAbbƍS-^ ,[L͚iSϵSi,zy!%Kj'qȐvNB&åM-Egv$dlY NpLmv2QJ8df~lެLJZU;CvLǺqv2Ԩ뵓Y,zyA='IDTtT`T!$dNkSgӦh'!5k{_Z0s&LT&b-}<}ID*r!QXvtw{Itv!""""" pw .!88uJ; h =Z;ѣ ɓIcj;~4 8qB; h\:ɉr:vL; h$TR|JLNB&5 >$E^DDDDDD _;]l\ML<0pv 8୷w >;if̐Q:|2D;ݡC2$dC`pvSOk'!m#Fh;rDfeHNNREDDDDDA.]Sm̝+P0@_e휬,$d)-[ٳ+WGzNa믲nID^@)v-32=}S#^`ыĘ12X YXX;=Z; $_}|v 25rLHV.NAyhN;ݪU)԰a@ǎ)֬>X;<I;ݺuy8EDDDDDd*&SG;݂k S11@Tv oNAƏ4Na'_k S11@),"=hY3vD^xhB;gɍ^DDDDDDålY$/ʴ<۷k'!eJ NpLImv2QJ8dfԙ[h'!RN␕%m꧟iSիk'q,6zz$d*&UK;ݼyڵ) ="*J$i ;:uM9L)ISIDFצI#GMSI -A<]hۿ9S~ӥ 0hv 8L{/0dv Cdd3ID)6uv2Ѷ-0bv #GHLi.!A Ni')X""""""NaϲIVv2ѿ?Эv -[d+WGzNaY~^޽SmsIG9Wݻwߕ(t&IlLɚaыȟFW_|L yv +ئ3i[ cdj0HX@;2DFu뀹s9B4HF9s$ ^DDDDDDDEi[P:)8EG˺dB)~QpsR#|ŋKSqd=@gr4f вv /s ^DDDDD$$D;Cٲҡ\v˗em۴ҥC9"B;CfLsev2DFj'qʒXI;  TI,KF_Lji7djxn]v,S/X""""""uՁ`tA$dV- $ii#IDҦ邋e}Hy N␞.k1j'!r*]Z;CFݻⴲe8\GHyƢQ~iRQ $ǎ''Nh'!-ZtO SML=L4j$ʁ)yKHNB&6L k'!QQޗ L*QEDDDDDtI\0sҡs!)P^t !KMNB&ڵyF;]|)ئӝw#GjKHbSIDV)6nNX""""""oݺiۼYFdfj'!> 衝 ޽^Sm.=]L)@{7pv 2խ| $2%kzv2)NNX""""""*#Gmh`"djm[T . h SC@evʋAd} ٸ;,$dbkWG,zY",\XL9f:d*&F $K_|L'˘1k[XX;5J;BE/""""`tv2Qt(GDh'qf*, ?Nbliv 2"յ8}v2 Ԭ Nv T N11@Tv Ѷm)DZGu IKfNB&W$IDdP$o9|*U%\Sxe8\(n߮L-+m*<\; ^DDDDDy`D V-Z/q`OfR$IIr:zT; h8 NIJHNB&4f >^; Tj*0mpv2QN)^DDDDD*.NFRi'!:k;xP:jΜNB&:v Naw0utRixv@rv2qȑ)N&Mb VZGkKL}Rv2qn-ȺL|zNam0kpv2ѻ7Ыv e=GNa{7@Fv2ѭпv 3ӵ NaL}v2ѥ 0hv 2ĢQ`"djm[vVL t蠝٦ِ!=h[7O;4Y;݆ ܹei'!J+3@Vv2ѿ)EDDDDT,ZXLi[X\;4Nax1)Tt4мv >>T;;V $Kʹ2a +OS#eZV *,zW4[j'!aaҡ\v,:y&$dX1)V!'#ず5S8X$d*&SG;݂2:SL nBEOixA g35&;'mj~$djUiSIeݜX$dR%iS%Kj'q}W.mlY$/ʚq;vh'!eʅDI._RmNB&J6ĢQar80aKY+|%%'Gj'!^:uJTBv2Ѡ\HRRɓx$d"*J:ԩIDݺצΜONB&j S^DDDDDi5aôS:$IDv3hLb;}V;]B0iLn .1Q.$JJNB&ZƌNaw\qv2Ѣ0nv ^DDDDDѶmו+ID@^)o.]NB&zNak{2=݁~SLLt n~`Ly$dK` vqq2utZv2q!) +el NÇk[X@;:vki SO> tn`\NaYYID@n)l23GzNAnEDDDDT-Z,_L4inb/Shysv}|v 25v,p).,NAFZNaWlSlHMv|#)8m\`ы0ʒ7mNB&EUN␓|I@͚), ?X^; Na7>fv 2 ԫn`*d*&__;…)TL Шv rE/""""9Yb~$djU:4T;Cz:;@lv2Qt(,iS{h'!ҦʔNpcv2Q/e`,Y7OҦ""8dfgt|BH$t 'GƍC9$%IJLNB&6 6 Lk'!7$ʁi`Ta$dn]P$gӧj'!jy*-M.NNB&W6¢QQqtj'!O?.><HINB&xYv RJNNB&ZFNaL z >-[ch;vL.8yR; h7N;ҦNB&6 8ہ.]NB&~G;ݮ]2XFv2ѽ;Яv ~[Ѥӵ+0`v M?Lt ..1CFP^``vӦhB >;Æi?EDDDDTԬ^-Sp:vkk SO> tn`\Naq#0gLu( 衝n6Y7e$dwoW/%Ke˴Shysv~ |v 25v,pm).sC;ݲe)smh`"djm[vVL t蠝cы(>P| ԨddITL Pv 5kS^=v H2~}v +Vh S11@F)//NAbbd/RâQQ.kj'!U%Jh'qpx`$dbE`xL$/ʚq;vh'!y|y$/g4b|J6!3S9ܺU;  "#8dei6i'!!!ҦUNRdEDDDDT%%'IDÆHIx$d⦛M> L6֕bj IMONB&jՒ@r,0cpv2QF൩sM߯LT*BCI,zu@Jv2qjKHL֭QS%&&HO˖W 9vLɓIm/qiSǏk'!͚E=L4nxm`ы]22݁}S#S^Lt D;1hV;ʕE)@)V,NAG;E9X0wСԮn|IH)&WO;}LEGk[XB;6Nah|v 25~<Фv ŋ/NA͵S ,z݅ Ӟ=IDŊQSveZ]r夣\9$.fL)#mbE$WȔt[j'!%JHJ$YY6i'!ŊIVM;CNLI@͚) =z#IM7Ir II6Lԫ'$5:8tH; ];ٳIDצΝ6v2QP$rqZlv2Q%j,zsGH"9Y; hF $ @Rv2qW 9vLɓImZqiSǏk'!͚^+) 88zT; h8pTbv2ѠA]VȰEDDDDD#W^L< )~8^; ~'SH4$dS'``vӧ/ >;Æi;tHFj'!O?.>^.NKINB&xY^DDDDD~ei'!O{?nvY%$dᇁ>}SLt망P I!! [TuꤝLt$${j'qKKYrOkvҹpUwHk,_(<)z*mNr%PTO;=$-[l.ҦxByAR矵8]+mœQ8Pw#S4]DDT[nw؁͛+&""opw""""""r[;E@DDDDDDDDDDDDX""""""""""""ǢPDDD-44իWTZ'"r.$D;QPbыo'NЎADei' """""" Jސ^DDDDDDDDDDDDX""""""""""""5 7-[j "ΝSKRB$IJR67NoHDDDDDDDDDDDDAE/"""""""""""" z,zQcы^DDDDDDDDDDDDX""""""""""""Ǣ^7o)>K,K;  _nۖ-!+gΨ+RSg幱,re)ŕ+=Ϟul "BQٶͿiiunq"#|Y_v @T?k+W+u++^#) XXؽطھV2@˖@vw:xWKO5 ѯ?Qlsp"p̕+u/ɑ8Ko67J{` 23QH/"""""MHv%[nڷ:u:wJNEqHqh0kyR+? 4mz99ԩJ( ,\h(Wr aaw]}uj'*8? ,]jX"?EDDDDD ׵*TF4NC Y``tO;&M#ٷ+_8}Z:"^W g`X3?tcY"?EDDDDDG9y[!X1U+vmvp0p:x5o IDATY? ݻh`N !ǰiWΔ, 4kvLCZ<HIcceJ <42k>ٰh??]>*S8!]];[䟗_D}{X"?EDDDDDykx10c}AqOT/Zv]f,^mX:AתYS^cxbSѣc+~]P6ަ}{`{w <>/_u> GZcܱ,_›JdVW֝;v.Y-S0֮^yE.pE/"^DDDDD'O=$x㣏}?ŋ^\hb鄧).h,z…siS̏o H7z,1?&;Zx}xncl*⡡Cߖi l` `<۱ED`ы))I9}vŋKtic][oy6,Lցjx^y?]Ϣ˒TZ!!qK/I~x1瑩__:Kl,pW^8/J˛QO> ̚Udnp>n.<T vy)x@ٲ̙z̔<(x$rrd7p^*UJ}utv;AQ#GpljyXR.͏L*TDD^`ы˛Nnz xM۝<)W`STFaYRz}(!kp?ǏiIǎu]No_d "աwըcwra'11jQ^DDDDD_tJGy>)p +2m+|t钿BB=jDDժy]dת0a231]QƑE/""""" ~!!)9O/LZrrG 5k _|_ . ] 﨣75ڵM;.Qcы =od ڵe-ğJUOHԙoN-LŊs渞ȑCDx ~}sg`24G{Go" ¢y1`ϢeZ̞z 8w'e\\+Ӧ9/!`wOM bEg!" ,zQq}ޭysgѲzv Vs+Wukws:vG|".pjT"иRS?Q`ы  *v;w G?ŋ@Lm &;/pN.D$-.]NQ5hvǎo" .5jxަNx뵘ܙ:}i:@nǕΝ[o/5H… …AD$X"""""¥re$%v Fof@Xt!cؿ !*ʎZR!,̻ʔDDA">Q*mr`>Μq#Lo-k:YiӀ,Yn!s p,dfv F̘~Zf &7J:u,DEՉ]\PRRy?GA9yx,M&W]cы] ̙hHMPj D!?Qzm€{q.]> XP:*ϟin[ÆI?\$?myΝGVI+w$+%JNSAu4_+5S/Xz6+K1{G M@TTgrt8_IO8r.۾\ܖ-rۺU:rG*z9#=+T5FNuodek>7˖-As^JJ JZ*?Ęw_̝+M<)[nG9_;+Kn?&#Cs/K\?eɱ}ֻ K [F}<4\"9s:;W\tiM) bN`t/{ 4l(ˀ;//y1ŋK9/תLGӦKz^."B>O ~|oj7}j>劏f̐zGx޾V-,Яй}Je x!/""""""|5iZ:S'oӽtZHO^xG'P駁G:Wqm7gؿ_%xWm&#~ܭg<1w.0bw&LmsٹSK<)R^m2``yyZN0{LG'^xAFe)~7k4^"&HqWw|!Рdns}3s옌8{,[A[xۥ潘 yeK#}y5˒"ʸqލvL)sW(QkgJNƏs­+M ^DDDDD_ZE}dOW(%KQd;iBj'dR:p@?k^_xK:-ҀÇ_~qbŀ[Fuڑke4D^DUKggVANޭ6n4+R=[: MRjVF awm/ =k浝3gYѕ+tLH 7:A)Δ//={w7n, oք_)n~d輸8amwuL`,ٗՓ%nv:߬,ߝ[-,kڻײRS-%JL%K,kWϙg9'mzW^of<*]Fs\?vF˪]u-eYVޮWc,;ֲBB{+]ڲ~-dI?[6I%'= '"W{%>52ɓ}>㏖U/^ܲzʲv1YYzeOaY[z>f?#߹ӲV b,k,~w._ߗ28IӊLǸxѲOwzRŲ?~ͲjԸ!!5`eΖ`v3-kYk=_y?eI1hв>Pk;fY/hYJ׿ZVNY9;ҤerۻײRR8zԲ>̲zsmʲ\=ו+u]}Y֫ZVFefZ\g?WU5ye9sRS-k4\YZR2珩Yxa}з-[ukIU/qPd6~ݻ}˒k{]}V=lMk3ҥiI]u#""""BGz}ez>ܹ{diTCWs}fv} Ov\ j_{4+;[F\zZвbc-)JOxZJF+}`L=zh'7QZ}|+9YF-gYrpսee*Vt g9{VTɻ,'OZ։^եeedF42,|W-#W9ڴIzVFxx]27(Zs,kx uݏ??ƫmۑ^չe *Wߞx²yIJuA5kzR<Сd;2RF b@?K&y!#rr,ǝ}ƍېs}Ϝ9{7@?ܷL%yy__>~פIgDDDDDTT++˲|S:p=ϗBL}\9ٖլ۷Ѳ ;n׷"UFNߟ׸qdjt=xPڬvV%W_yy iۿ쏢eYV^cRc-Q|]w_uؤ,qqBɓyuӦ{sVʽU:ǹsuM?&,̲vkYRwVȽM>Ms;} y=X[r`AOvL|mbŤM9f맞p:[1nY#ITQQ_Sڦ,G^1c=Vdӕ_ΞmÇNYrն-`y&3G﫯d:WS? q6ՕOM\0v߯T 2lcIL&Mn?S4תV)BCB't/GM-ޠor(,Ybz03>ի;iiYP__=)932}'>,k&4i)#]ȜFɚt0aT̳Vd A2~}:wvMﳠuzu|MʮvZ[ 3xqۭ\iWA> \ߤW(Yz 1#UT)|;wWYl UL 5,/, h='<Ι3RU˅ bZ'(6yAX1+Ov\]弰4twYgʗ5\6ͻ  kʟGK|0j5k^ٱc=嗝׭pfyBBbW>77+HWNVΕΐ^ƍZÆղSG-H'?{YW UWۼY:]1MWm'O:=[Ώ?eey^\9onKM79&$}Kq͓P;2j܏<իck=ިRhy<47߷}^˛,K˨]w^}U.rhBFT.شIt9'-Za_~VZS&Vҙ yƁƛb Eߞ 7%seȫ]zՒ%}ɑ#z!!Wkp*bVb"z>P:_+;[7ɋs]o{HHp]ݹ}|hSi~iOvъGtFJ2{w}fΦM,zNoHDDDDD̳uaa@F2ާ IDAT2(Ym;'J{< g?|Y%11rPXT)44L]]OQp钌p_~\˛u_I_hq7/ zg)*deA/]ߗs}m?{>z)Nbaы{w]UuB0 DhQDD<%BE}P ڊRUh>dh@ & S@ 㷲2{>{o.|?kE=g}=So&""""ozHR?/~=0iRkX=5r^'6t/ |+0,z8S󟻟ͩnn ȟ?< E ffze`[#*۴q֭y-^l=VMys*37{:J8x]^,] eM3fWoT0xM`~%%˫KIo=~~TsmU+ ED{|f) """"""rRϤaazM7X)66nwo>s.eecD? n w)!rTO? wr$8tȷv68_)Ds?۷޽HII~ ̜_OwCwM/;ŨQ5./X4l(ϣJ;keב#%Á7h ? կo#ysguҸi"""""Ką _{Z`Qx<%818uU  M7l*+v}}sK7J|<^Z&N4TYlqK#o 2୷>}~gjIt.(ۓ'_m/^yyՊ =z7}€%A/""""""Skݻ?|ue믾,F"(Uun$^+&-=s)4iʕ{~%dh]N59mnE7OٳN=]i?iiP)`""""""frWׯaТ~ڽ)t=do.vr11@RRBצ~d׃M͛-$ave.F/~XV[O}ohܪUeǵʨcǀU$_O)I+)5g۶~eN~Ǡ'۷u߻v`B ,, X2GP<h@Ւ>ؿeK$j?ޔ;R#fH[zc_mlw2c}Lڴɝr;]J={ԅ- n헯;oRSݻj?,Yx׿ii/]hYlz".Ngsz_~EA/"""""ʬ.eb8[7e**>L}?bce R79*JJ,=cpz%'{o$ lR鮻̖o/h2CQ@ˬZ>UTX|y=x~ 11{RMYopu Qu;wOLtVny9' ;= ѯKLƏS܆ 2+ML3Ѩh|U =/rs<8z/%k֟GOGد~^ˁck"5U2S7Zq1oW)2X>*ll**wLxiϿrϿ7o| R_"k""""""w>_tA˭|iSzܹ2squIݒ7x1С~Sgw }AyMet}+4n Wӧݓ'ufffe[~(s)/`W1OF3 $(Q)2~u )S̶o*/ny'rUp<:{6;2fgJzݹ8rٺ_}߽;=.nn-[̚%cƸ[6A/"""""rI`;(/8wYQXhɓϽ{|{w;BgL 3|?.J|/ɜYƴ"i뫯Gå]xQW$b <^Aߪ*Х))k>)XLG:6Cgy=<ϗԹS<`um:wf㏁ɓ}o~|69{0GPʯP֭>utJIiǒ%fi`R_Z/oo۰C-W v>:TƴiiB8nz;/[w}Zcy}\Fpk^#@^[SSec}k7Xs..y X9=ON ^?l6z6o7}JtxCt|{ET9E (y(ݬ,z;=v?WU+|`ڝfd(5rdmiw^~AR͚YDRRe9u5ʺ^xAӧ}ի߉s;vҳR.jv6h_ɚ5J<}~{+ӓwߵu׉26.7R[ltR'O:K̚evlJ)j{~QmJ .ٳJmڤT||{}T?ryԼT5Y| w~*թپ&Vj,_nާ)zZ/>zYsaaJf{vmbb:|X^i|7&W+l^BNVYN+DDDDDt+?4kFV yzHiI2?f7JCmJK̔F+a?I Sj6TxCk\^uwoS)uRGJYO<9Ҧ,oQ_^zImZFDG+ηQdd(d}+u= +yR)ըcQ>?~&)r>ٸQ.zIIJkϢ:T3ZHG_;j¬Q/))JkgV3g~mJJPn uРnTZ:UWNfVo%Θ1_CO=ԁJ;'Ad lYs'pY-7WȈWdR?/:_9瘼:w~)rz@KV粊%v]u͛]iiխQ#V_~WJ=R_~=3x,d @V@orNth޼j>}:lu_Ujy̔g>}>U׏YڵfY盯x@y:atF=yR;tȑ=O_)TGQ9rD~> |_k TxfGeKIҽkնe(˵l)nV%)JL |Y#0c<1+}KWwe p-e?_Ni#scΫWۛsgסwoڵ.WR"s:%CIhOI ٱh| x%Igz5m*t5HZ¦M<8wNy9{VR</@k Tӧ (/\R(z2qK.#C-ɹn[Mw  cl_Uwꔜ{ii*jj{Y4Pʹز#T&%eDD5sgaCI{ x4ol&*}2OnY%ArjMΡv\ׯ7?6%`^rv*LJΪU6I:[O+rsÇ%\judnI4c=) *9v:t{gͫi壏$}'J5o3//<\ΙD筸XRR<#5yjX$%E1u4]V=O^ٺv~s_hB%Z3_NK1N,Zdw1C@Rܲo__:kӎG.…ԛ1c$WNI{@f̰==]RrIQܤ pr?={Acn8QӓޗWbb>MխR[,;/<]\e zik5{}lq{K~nsxIKsw;sQjR#FM7IJYVz{U/<謌6m2}{挳}6guIMkB?A +1^zM7ϓW^ 7FUƌ1HIV_,]V^w;ifYa];3iM_11CuG vR{Mlե9QQU2j>d85;&5ĩSJ wANGRp5"""""ڮx} kKH  ]vxL~D2g(r""BFpC}U);{/\6LFsmV5B;%ˁSZ[(C74i"rwF&_1!ٳx1{' ѣ@bp͚I:8`͒[} RXp! bH',L.=oAr ̽40V6zHR오$(kmyӯ0aBggqˈKlI%f\ٌF{XpKAo%ǖizN$7yr"I׶+_:ȿKG˗~jyL,X,] n]f͒FJ'Tge$%N4=& n5 /~$nHJ~|s瞓{ɹߠ/1unHNԽv]Tr;ް8sr[KZ8O6 Ν+iC:uuNk["#_wnZ<._.)jg I8{omD+_w3y ׫s;qUQR)r2~2曒NJttvy)q*3Sow3G""""""),Q[))ȟ/#;7?8JNbbeN '.\[ʨ'aQ#"!A:2DpKE|Ud\x )Kk~i^>҉íFWV Bolyen E, Rideout JR, Dillon MR, Bokulich NA, Abnet CC, Al-Ghalith GA, Alexander H, Alm EJ, Arumugam M, Asnicar F, Bai Y, Bisanz JE, Bittinger K, Brejnrod A, Brislawn CJ, Brown CT, Callahan BJ, Caraballo-Rodríguez AM, Chase J, Cope EK, Da Silva R, Diener C, Dorrestein PC, Douglas GM, Durall DM, Duvallet C, Edwardson CF, Ernst M, Estaki M, Fouquier J, Gauglitz JM, Gibbons SM, Gibson DL, Gonzalez A, Gorlick K, Guo J, Hillmann B, Holmes S, Holste H, Huttenhower C, Huttley GA, Janssen S, Jarmusch AK, Jiang L, Kaehler BD, Kang KB, Keefe CR, Keim P, Kelley ST, Knights D, Koester I, Kosciolek T, Kreps J, Langille MGI, Lee J, Ley R, Liu YX, Loftfield E, Lozupone C, Maher M, Marotz C, Martin BD, McDonald D, McIver LJ, Melnik AV, Metcalf JL, Morgan SC, Morton JT, Naimey AT, Navas-Molina JA, Nothias LF, Orchanian SB, Pearson T, Peoples SL, Petras D, Preuss ML, Pruesse E, Rasmussen LB, Rivers A, Robeson MS, Rosenthal P, Segata N, Shaffer M, Shiffer A, Sinha R, Song SJ, Spear JR, Swafford AD, Thompson LR, Torres PJ, Trinh P, Tripathi A, Turnbaugh PJ, Ul-Hasan S, van der Hooft JJJ, Vargas F, Vázquez-Baeza Y, Vogtmann E, von Hippel M, Walters W, Wan Y, Wang M, Warren J, Weber KC, Williamson CHD, Willis AD, Xu ZZ, Zaneveld JR, Zhang Y, Zhu Q, Knight R, and Caporaso JG. 2019. Reproducible, interactive, scalable and extensible microbiome data science using QIIME 2. Nature Biotechnology 37:852–857. https://doi.org/10.1038/s41587-019-0209-9 qiime-2021.8.0/ci/000077500000000000000000000000001412141660100134375ustar00rootroot00000000000000qiime-2021.8.0/ci/recipe/000077500000000000000000000000001412141660100147065ustar00rootroot00000000000000qiime-2021.8.0/ci/recipe/meta.yaml000066400000000000000000000014621412141660100165230ustar00rootroot00000000000000{% set data = load_setup_py_data() %} {% set version = data.get('version') %} package: name: qiime2 version: {{ version }} source: path: ../.. build: script: make install requirements: host: - python {{ python }} - setuptools run: - python {{ python }} - pyyaml - decorator >=4,<5 - pandas >=1 # tzlocal 3 is currently broken - once this is fixed drop pin - tzlocal <3 - python-dateutil - bibtexparser - networkx - dill test: requires: - pytest imports: - qiime2 commands: # TODO don't require devs to remember setting this env var before running # tests. The value can be anything. - QIIMETEST= python -c "import qiime2.plugins.dummy_plugin" about: home: https://qiime2.org license: BSD-3-Clause license_family: BSD qiime-2021.8.0/hooks/000077500000000000000000000000001412141660100141675ustar00rootroot00000000000000qiime-2021.8.0/hooks/00_activate_qiime2_envs.sh000066400000000000000000000002241412141660100211210ustar00rootroot00000000000000#!/bin/sh export MPLBACKEND='Agg' export R_LIBS_USER=$CONDA_PREFIX/lib/R/library/ export PYTHONNOUSERSITE=$CONDA_PREFIX/lib/python*/site-packages/ qiime-2021.8.0/hooks/00_deactivate_qiime2_envs.sh000066400000000000000000000001051412141660100214300ustar00rootroot00000000000000#!/bin/sh unset MPLBACKEND unset R_LIBS_USER unset PYTHONNOUSERSITE qiime-2021.8.0/qiime2/000077500000000000000000000000001412141660100142325ustar00rootroot00000000000000qiime-2021.8.0/qiime2/__init__.py000066400000000000000000000023511412141660100163440ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from qiime2.sdk import Artifact, Visualization from qiime2.metadata import (Metadata, MetadataColumn, CategoricalMetadataColumn, NumericMetadataColumn) from qiime2.plugin import Citations from ._version import get_versions __version__ = get_versions()['version'] del get_versions # "Train release" version includes . and excludes patch numbers # and pre/post-release tags. All versions within a train release are expected # to be compatible. __release__ = '.'.join(__version__.split('.')[:2]) __citations__ = tuple(Citations.load('citations.bib', package='qiime2')) __website__ = 'https://qiime2.org' __all__ = ['Artifact', 'Visualization', 'Metadata', 'MetadataColumn', 'CategoricalMetadataColumn', 'NumericMetadataColumn'] # Used by `jupyter serverextension enable` def _jupyter_server_extension_paths(): return [{"module": "qiime2.jupyter"}] qiime-2021.8.0/qiime2/_version.py000066400000000000000000000441011412141660100164300ustar00rootroot00000000000000 # This file helps to compute a version number in source trees obtained from # git-archive tarball (such as those provided by githubs download-from-tag # feature). Distribution tarballs (built by setup.py sdist) and build # directories (produced by setup.py build) will contain a much shorter file # that just contains the computed version number. # This file is released into the public domain. Generated by # versioneer-0.18 (https://github.com/warner/python-versioneer) """Git implementation of _version.py.""" import errno import os import re import subprocess import sys def get_keywords(): """Get the keywords needed to look up the version information.""" # these strings will be replaced by git during git-archive. # setup.py/versioneer.py will grep for the variable names, so they must # each be defined on a line of their own. _version.py will just call # get_keywords(). git_refnames = " (tag: 2021.8.0)" git_full = "a8f76ee62861bab529caa3554567b8670aa5f7b4" git_date = "2021-09-09 18:35:32 +0000" keywords = {"refnames": git_refnames, "full": git_full, "date": git_date} return keywords class VersioneerConfig: """Container for Versioneer configuration parameters.""" def get_config(): """Create, populate and return the VersioneerConfig() object.""" # these strings are filled in when 'setup.py versioneer' creates # _version.py cfg = VersioneerConfig() cfg.VCS = "git" cfg.style = "pep440" cfg.tag_prefix = "" cfg.parentdir_prefix = "qiime2-" cfg.versionfile_source = "qiime2/_version.py" cfg.verbose = False return cfg class NotThisMethod(Exception): """Exception raised if a method is not valid for the current scenario.""" LONG_VERSION_PY = {} HANDLERS = {} def register_vcs_handler(vcs, method): # decorator """Decorator to mark a method as the handler for a particular VCS.""" def decorate(f): """Store f in HANDLERS[vcs][method].""" if vcs not in HANDLERS: HANDLERS[vcs] = {} HANDLERS[vcs][method] = f return f return decorate def run_command(commands, args, cwd=None, verbose=False, hide_stderr=False, env=None): """Call the given command(s).""" assert isinstance(commands, list) p = None for c in commands: try: dispcmd = str([c] + args) # remember shell=False, so use git.cmd on windows, not just git p = subprocess.Popen([c] + args, cwd=cwd, env=env, stdout=subprocess.PIPE, stderr=(subprocess.PIPE if hide_stderr else None)) break except EnvironmentError: e = sys.exc_info()[1] if e.errno == errno.ENOENT: continue if verbose: print("unable to run %s" % dispcmd) print(e) return None, None else: if verbose: print("unable to find command, tried %s" % (commands,)) return None, None stdout = p.communicate()[0].strip() if sys.version_info[0] >= 3: stdout = stdout.decode() if p.returncode != 0: if verbose: print("unable to run %s (error)" % dispcmd) print("stdout was %s" % stdout) return None, p.returncode return stdout, p.returncode def versions_from_parentdir(parentdir_prefix, root, verbose): """Try to determine the version from the parent directory name. Source tarballs conventionally unpack into a directory that includes both the project name and a version string. We will also support searching up two directory levels for an appropriately named parent directory """ rootdirs = [] for i in range(3): dirname = os.path.basename(root) if dirname.startswith(parentdir_prefix): return {"version": dirname[len(parentdir_prefix):], "full-revisionid": None, "dirty": False, "error": None, "date": None} else: rootdirs.append(root) root = os.path.dirname(root) # up a level if verbose: print("Tried directories %s but none started with prefix %s" % (str(rootdirs), parentdir_prefix)) raise NotThisMethod("rootdir doesn't start with parentdir_prefix") @register_vcs_handler("git", "get_keywords") def git_get_keywords(versionfile_abs): """Extract version information from the given file.""" # the code embedded in _version.py can just fetch the value of these # keywords. When used from setup.py, we don't want to import _version.py, # so we do it with a regexp instead. This function is not used from # _version.py. keywords = {} try: f = open(versionfile_abs, "r") for line in f.readlines(): if line.strip().startswith("git_refnames ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["refnames"] = mo.group(1) if line.strip().startswith("git_full ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["full"] = mo.group(1) if line.strip().startswith("git_date ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["date"] = mo.group(1) f.close() except EnvironmentError: pass return keywords @register_vcs_handler("git", "keywords") def git_versions_from_keywords(keywords, tag_prefix, verbose): """Get version information from git keywords.""" if not keywords: raise NotThisMethod("no keywords at all, weird") date = keywords.get("date") if date is not None: # git-2.2.0 added "%cI", which expands to an ISO-8601 -compliant # datestamp. However we prefer "%ci" (which expands to an "ISO-8601 # -like" string, which we must then edit to make compliant), because # it's been around since git-1.5.3, and it's too difficult to # discover which version we're using, or to work around using an # older one. date = date.strip().replace(" ", "T", 1).replace(" ", "", 1) refnames = keywords["refnames"].strip() if refnames.startswith("$Format"): if verbose: print("keywords are unexpanded, not using") raise NotThisMethod("unexpanded keywords, not a git-archive tarball") refs = set([r.strip() for r in refnames.strip("()").split(",")]) # starting in git-1.8.3, tags are listed as "tag: foo-1.0" instead of # just "foo-1.0". If we see a "tag: " prefix, prefer those. TAG = "tag: " tags = set([r[len(TAG):] for r in refs if r.startswith(TAG)]) if not tags: # Either we're using git < 1.8.3, or there really are no tags. We use # a heuristic: assume all version tags have a digit. The old git %d # expansion behaves like git log --decorate=short and strips out the # refs/heads/ and refs/tags/ prefixes that would let us distinguish # between branches and tags. By ignoring refnames without digits, we # filter out many common branch names like "release" and # "stabilization", as well as "HEAD" and "master". tags = set([r for r in refs if re.search(r'\d', r)]) if verbose: print("discarding '%s', no digits" % ",".join(refs - tags)) if verbose: print("likely tags: %s" % ",".join(sorted(tags))) for ref in sorted(tags): # sorting will prefer e.g. "2.0" over "2.0rc1" if ref.startswith(tag_prefix): r = ref[len(tag_prefix):] if verbose: print("picking %s" % r) return {"version": r, "full-revisionid": keywords["full"].strip(), "dirty": False, "error": None, "date": date} # no suitable tags, so version is "0+unknown", but full hex is still there if verbose: print("no suitable tags, using unknown + full revision id") return {"version": "0+unknown", "full-revisionid": keywords["full"].strip(), "dirty": False, "error": "no suitable tags", "date": None} @register_vcs_handler("git", "pieces_from_vcs") def git_pieces_from_vcs(tag_prefix, root, verbose, run_command=run_command): """Get version from 'git describe' in the root of the source tree. This only gets called if the git-archive 'subst' keywords were *not* expanded, and _version.py hasn't already been rewritten with a short version string, meaning we're inside a checked out source tree. """ GITS = ["git"] if sys.platform == "win32": GITS = ["git.cmd", "git.exe"] out, rc = run_command(GITS, ["rev-parse", "--git-dir"], cwd=root, hide_stderr=True) if rc != 0: if verbose: print("Directory %s not under git control" % root) raise NotThisMethod("'git rev-parse --git-dir' returned error") # if there is a tag matching tag_prefix, this yields TAG-NUM-gHEX[-dirty] # if there isn't one, this yields HEX[-dirty] (no NUM) describe_out, rc = run_command(GITS, ["describe", "--tags", "--dirty", "--always", "--long", "--match", "%s*" % tag_prefix], cwd=root) # --long was added in git-1.5.5 if describe_out is None: raise NotThisMethod("'git describe' failed") describe_out = describe_out.strip() full_out, rc = run_command(GITS, ["rev-parse", "HEAD"], cwd=root) if full_out is None: raise NotThisMethod("'git rev-parse' failed") full_out = full_out.strip() pieces = {} pieces["long"] = full_out pieces["short"] = full_out[:7] # maybe improved later pieces["error"] = None # parse describe_out. It will be like TAG-NUM-gHEX[-dirty] or HEX[-dirty] # TAG might have hyphens. git_describe = describe_out # look for -dirty suffix dirty = git_describe.endswith("-dirty") pieces["dirty"] = dirty if dirty: git_describe = git_describe[:git_describe.rindex("-dirty")] # now we have TAG-NUM-gHEX or HEX if "-" in git_describe: # TAG-NUM-gHEX mo = re.search(r'^(.+)-(\d+)-g([0-9a-f]+)$', git_describe) if not mo: # unparseable. Maybe git-describe is misbehaving? pieces["error"] = ("unable to parse git-describe output: '%s'" % describe_out) return pieces # tag full_tag = mo.group(1) if not full_tag.startswith(tag_prefix): if verbose: fmt = "tag '%s' doesn't start with prefix '%s'" print(fmt % (full_tag, tag_prefix)) pieces["error"] = ("tag '%s' doesn't start with prefix '%s'" % (full_tag, tag_prefix)) return pieces pieces["closest-tag"] = full_tag[len(tag_prefix):] # distance: number of commits since tag pieces["distance"] = int(mo.group(2)) # commit: short hex revision ID pieces["short"] = mo.group(3) else: # HEX: no tags pieces["closest-tag"] = None count_out, rc = run_command(GITS, ["rev-list", "HEAD", "--count"], cwd=root) pieces["distance"] = int(count_out) # total number of commits # commit date: see ISO-8601 comment in git_versions_from_keywords() date = run_command(GITS, ["show", "-s", "--format=%ci", "HEAD"], cwd=root)[0].strip() pieces["date"] = date.strip().replace(" ", "T", 1).replace(" ", "", 1) return pieces def plus_or_dot(pieces): """Return a + if we don't already have one, else return a .""" if "+" in pieces.get("closest-tag", ""): return "." return "+" def render_pep440(pieces): """Build up version string, with post-release "local version identifier". Our goal: TAG[+DISTANCE.gHEX[.dirty]] . Note that if you get a tagged build and then dirty it, you'll get TAG+0.gHEX.dirty Exceptions: 1: no tags. git_describe was just HEX. 0+untagged.DISTANCE.gHEX[.dirty] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += plus_or_dot(pieces) rendered += "%d.g%s" % (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" else: # exception #1 rendered = "0+untagged.%d.g%s" % (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" return rendered def render_pep440_pre(pieces): """TAG[.post.devDISTANCE] -- No -dirty. Exceptions: 1: no tags. 0.post.devDISTANCE """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += ".post.dev%d" % pieces["distance"] else: # exception #1 rendered = "0.post.dev%d" % pieces["distance"] return rendered def render_pep440_post(pieces): """TAG[.postDISTANCE[.dev0]+gHEX] . The ".dev0" means dirty. Note that .dev0 sorts backwards (a dirty tree will appear "older" than the corresponding clean one), but you shouldn't be releasing software with -dirty anyways. Exceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += plus_or_dot(pieces) rendered += "g%s" % pieces["short"] else: # exception #1 rendered = "0.post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += "+g%s" % pieces["short"] return rendered def render_pep440_old(pieces): """TAG[.postDISTANCE[.dev0]] . The ".dev0" means dirty. Eexceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" else: # exception #1 rendered = "0.post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" return rendered def render_git_describe(pieces): """TAG[-DISTANCE-gHEX][-dirty]. Like 'git describe --tags --dirty --always'. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += "-%d-g%s" % (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render_git_describe_long(pieces): """TAG-DISTANCE-gHEX[-dirty]. Like 'git describe --tags --dirty --always -long'. The distance/hash is unconditional. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] rendered += "-%d-g%s" % (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render(pieces, style): """Render the given version pieces into the requested style.""" if pieces["error"]: return {"version": "unknown", "full-revisionid": pieces.get("long"), "dirty": None, "error": pieces["error"], "date": None} if not style or style == "default": style = "pep440" # the default if style == "pep440": rendered = render_pep440(pieces) elif style == "pep440-pre": rendered = render_pep440_pre(pieces) elif style == "pep440-post": rendered = render_pep440_post(pieces) elif style == "pep440-old": rendered = render_pep440_old(pieces) elif style == "git-describe": rendered = render_git_describe(pieces) elif style == "git-describe-long": rendered = render_git_describe_long(pieces) else: raise ValueError("unknown style '%s'" % style) return {"version": rendered, "full-revisionid": pieces["long"], "dirty": pieces["dirty"], "error": None, "date": pieces.get("date")} def get_versions(): """Get version information or return default if unable to do so.""" # I am in _version.py, which lives at ROOT/VERSIONFILE_SOURCE. If we have # __file__, we can work backwards from there to the root. Some # py2exe/bbfreeze/non-CPython implementations don't do __file__, in which # case we can only use expanded keywords. cfg = get_config() verbose = cfg.verbose try: return git_versions_from_keywords(get_keywords(), cfg.tag_prefix, verbose) except NotThisMethod: pass try: root = os.path.realpath(__file__) # versionfile_source is the relative path from the top of the source # tree (where the .git directory might live) to this file. Invert # this to find the root from __file__. for i in cfg.versionfile_source.split('/'): root = os.path.dirname(root) except NameError: return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to find root of source tree", "date": None} try: pieces = git_pieces_from_vcs(cfg.tag_prefix, root, verbose) return render(pieces, cfg.style) except NotThisMethod: pass try: if cfg.parentdir_prefix: return versions_from_parentdir(cfg.parentdir_prefix, root, verbose) except NotThisMethod: pass return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to compute version", "date": None} qiime-2021.8.0/qiime2/citations.bib000066400000000000000000000053031412141660100167060ustar00rootroot00000000000000@Article{Bolyen2019, author={Bolyen, Evan and Rideout, Jai Ram and Dillon, Matthew R. and Bokulich, Nicholas A. and Abnet, Christian C. and Al-Ghalith, Gabriel A. and Alexander, Harriet and Alm, Eric J. and Arumugam, Manimozhiyan and Asnicar, Francesco and Bai, Yang and Bisanz, Jordan E. and Bittinger, Kyle and Brejnrod, Asker and Brislawn, Colin J. and Brown, C. Titus and Callahan, Benjamin J. and Caraballo-Rodr{\'i}guez, Andr{\'e}s Mauricio and Chase, John and Cope, Emily K. and Da Silva, Ricardo and Diener, Christian and Dorrestein, Pieter C. and Douglas, Gavin M. and Durall, Daniel M. and Duvallet, Claire and Edwardson, Christian F. and Ernst, Madeleine and Estaki, Mehrbod and Fouquier, Jennifer and Gauglitz, Julia M. and Gibbons, Sean M. and Gibson, Deanna L. and Gonzalez, Antonio and Gorlick, Kestrel and Guo, Jiarong and Hillmann, Benjamin and Holmes, Susan and Holste, Hannes and Huttenhower, Curtis and Huttley, Gavin A. and Janssen, Stefan and Jarmusch, Alan K. and Jiang, Lingjing and Kaehler, Benjamin D. and Kang, Kyo Bin and Keefe, Christopher R. and Keim, Paul and Kelley, Scott T. and Knights, Dan and Koester, Irina and Kosciolek, Tomasz and Kreps, Jorden and Langille, Morgan G. I. and Lee, Joslynn and Ley, Ruth and Liu, Yong-Xin and Loftfield, Erikka and Lozupone, Catherine and Maher, Massoud and Marotz, Clarisse and Martin, Bryan D. and McDonald, Daniel and McIver, Lauren J. and Melnik, Alexey V. and Metcalf, Jessica L. and Morgan, Sydney C. and Morton, Jamie T. and Naimey, Ahmad Turan and Navas-Molina, Jose A. and Nothias, Louis Felix and Orchanian, Stephanie B. and Pearson, Talima and Peoples, Samuel L. and Petras, Daniel and Preuss, Mary Lai and Pruesse, Elmar and Rasmussen, Lasse Buur and Rivers, Adam and Robeson, Michael S. and Rosenthal, Patrick and Segata, Nicola and Shaffer, Michael and Shiffer, Arron and Sinha, Rashmi and Song, Se Jin and Spear, John R. and Swafford, Austin D. and Thompson, Luke R. and Torres, Pedro J. and Trinh, Pauline and Tripathi, Anupriya and Turnbaugh, Peter J. and Ul-Hasan, Sabah and van der Hooft, Justin J. J. and Vargas, Fernando and V{\'a}zquez-Baeza, Yoshiki and Vogtmann, Emily and von Hippel, Max and Walters, William and Wan, Yunhu and Wang, Mingxun and Warren, Jonathan and Weber, Kyle C. and Williamson, Charles H. D. and Willis, Amy D. and Xu, Zhenjiang Zech and Zaneveld, Jesse R. and Zhang, Yilong and Zhu, Qiyun and Knight, Rob and Caporaso, J. Gregory}, title={Reproducible, interactive, scalable and extensible microbiome data science using QIIME 2}, journal={Nature Biotechnology}, year={2019}, volume={37}, number={8}, pages={852-857}, issn={1546-1696}, doi={10.1038/s41587-019-0209-9}, url={https://doi.org/10.1038/s41587-019-0209-9} } qiime-2021.8.0/qiime2/core/000077500000000000000000000000001412141660100151625ustar00rootroot00000000000000qiime-2021.8.0/qiime2/core/__init__.py000066400000000000000000000005351412141660100172760ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- qiime-2021.8.0/qiime2/core/archive/000077500000000000000000000000001412141660100166035ustar00rootroot00000000000000qiime-2021.8.0/qiime2/core/archive/__init__.py000066400000000000000000000011631412141660100207150ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from .provenance import (ImportProvenanceCapture, ActionProvenanceCapture, PipelineProvenanceCapture) from .archiver import Archiver __all__ = ['Archiver', 'ImportProvenanceCapture', 'ActionProvenanceCapture', 'PipelineProvenanceCapture'] qiime-2021.8.0/qiime2/core/archive/archiver.py000066400000000000000000000313071412141660100207640ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import collections import uuid as _uuid import pathlib import zipfile import importlib import os import io import qiime2 import qiime2.core.cite as cite from qiime2.core.util import md5sum_directory, from_checksum_format _VERSION_TEMPLATE = """\ QIIME 2 archive: %s framework: %s """ ArchiveRecord = collections.namedtuple( 'ArchiveRecord', ['root', 'version_fp', 'uuid', 'version', 'framework_version']) ChecksumDiff = collections.namedtuple( 'ChecksumDiff', ['added', 'removed', 'changed']) class _Archive: """Abstraction layer over the archive filesystem. Responsible for details concerning manipulating an archive agnostic to its format. It is responsible for managing archive UUID, format, and framework versions as those are designed to be constant throughout all future format implementations. Breaking compatibility with that is a BIG DEAL and should avoided at (nearly) any cost. Example filesystem:: / !--- 770509e6-85f4-432c-9663-cdc04eb07db2 |--- VERSION !--- VERSION file:: QIIME 2 archive: framework: This file is itentionally not YAML/INI/An actual format. This is to discourage the situation where the format changes from something like YAML to another format and VERSION is updated with it "for consistency". To emphasize, the VERSION (filepath and content) and root archive structure MUST NOT CHANGE. If they change, then there is no longer a consistent way to dispatch to an appropriate format. """ VERSION_FILE = 'VERSION' @classmethod def is_archive_type(cls, filepath): raise NotImplementedError @classmethod def _is_uuid4(cls, uuid_str): # Adapted from https://gist.github.com/ShawnMilo/7777304 try: uuid = _uuid.UUID(hex=uuid_str, version=4) except ValueError: # The string is not a valid hex code for a UUID. return False # If uuid_str is a valid hex code, but an invalid uuid4, UUID.__init__ # will convert it to a valid uuid4. return str(uuid) == uuid_str @classmethod def setup(cls, path, version, framework_version): uuid = _uuid.uuid4() root_dir = path / str(uuid) root_dir.mkdir() version_fp = root_dir / cls.VERSION_FILE version_fp.write_text(_VERSION_TEMPLATE % (version, framework_version)) return ArchiveRecord(root_dir, version_fp, uuid, version, framework_version) @classmethod def save(cls, source, destination): raise NotImplementedError def __init__(self, path): self.path = path self.uuid = self._get_uuid() self.version, self.framework_version = self._get_versions() def _get_uuid(self): if not self.path.exists(): raise TypeError("%s does not exist or is not a filepath." % self.path) roots = set() for relpath in self.relative_iterdir(): if not relpath.startswith('.'): roots.add(relpath) if len(roots) == 0: raise ValueError("Archive does not have a visible root directory.") if len(roots) > 1: raise ValueError("Archive has multiple root directories: %r" % roots) uuid = roots.pop() if not self._is_uuid4(uuid): raise ValueError( "Archive root directory name %r is not a valid version 4 " "UUID." % uuid) return uuid def _get_versions(self): try: with self.open(self.VERSION_FILE) as fh: header, version_line, framework_version_line, eof = \ fh.read().split('\n') if header.strip() != 'QIIME 2': raise Exception() # GOTO except Exception version = version_line.split(':')[1].strip() framework_version = framework_version_line.split(':')[1].strip() return version, framework_version except Exception: # TODO: make a "better" parser which isn't just a catch-all raise ValueError("Archive does not contain a correctly formatted" " VERSION file.") def relative_iterdir(self, relpath='.'): raise NotImplementedError def open(self, relpath): raise NotImplementedError def mount(self, filepath): raise NotImplementedError class _ZipArchive(_Archive): """A specific variant of Archive which deals with ZIP64 files.""" @classmethod def is_archive_type(cls, path): return zipfile.is_zipfile(str(path)) @classmethod def save(cls, source, destination): with zipfile.ZipFile(str(destination), mode='w', compression=zipfile.ZIP_DEFLATED, allowZip64=True) as zf: for root, dirs, files in os.walk(str(source)): # Prune hidden directories from traversal. Strategy modified # from http://stackoverflow.com/a/13454267/3776794 dirs[:] = [d for d in dirs if not d.startswith('.')] for file in files: if file.startswith('.'): continue abspath = pathlib.Path(root) / file relpath = abspath.relative_to(source) zf.write(str(abspath), arcname=cls._as_zip_path(relpath)) def relative_iterdir(self, relpath=''): relpath = self._as_zip_path(relpath) seen = set() with zipfile.ZipFile(str(self.path), mode='r') as zf: for name in zf.namelist(): if name.startswith(relpath): parts = pathlib.PurePosixPath(name).parts if len(parts) > 0: result = parts[0] if result not in seen: seen.add(result) yield result def open(self, relpath): relpath = pathlib.Path(str(self.uuid)) / relpath with zipfile.ZipFile(str(self.path), mode='r') as zf: # The filehandle will still work even when `zf` is "closed" return io.TextIOWrapper(zf.open(self._as_zip_path(relpath))) def mount(self, filepath): # TODO: use FUSE/MacFUSE/Dokany bindings (many Python bindings are # outdated, we may need to take up maintenance/fork) root = self.extract(filepath) return ArchiveRecord(root, root / self.VERSION_FILE, self.uuid, self.version, self.framework_version) def extract(self, filepath): filepath = pathlib.Path(filepath) with zipfile.ZipFile(str(self.path), mode='r') as zf: for name in zf.namelist(): if name.startswith(str(self.uuid)): # extract removes `..` components, so as long as we extract # into `filepath`, the path won't go backwards. zf.extract(name, path=str(filepath)) return filepath / str(self.uuid) @classmethod def _as_zip_path(self, path): path = str(pathlib.PurePosixPath(path)) # zip files don't work well with '.' which is the identity of a Path # obj, so just convert to empty string which is basically the identity # of a zip's entry if path == '.': path = '' return path class Archiver: CURRENT_FORMAT_VERSION = '5' CURRENT_ARCHIVE = _ZipArchive _FORMAT_REGISTRY = { # NOTE: add more archive formats as things change '0': 'qiime2.core.archive.format.v0:ArchiveFormat', '1': 'qiime2.core.archive.format.v1:ArchiveFormat', '2': 'qiime2.core.archive.format.v2:ArchiveFormat', '3': 'qiime2.core.archive.format.v3:ArchiveFormat', '4': 'qiime2.core.archive.format.v4:ArchiveFormat', '5': 'qiime2.core.archive.format.v5:ArchiveFormat' } @classmethod def _make_temp_path(cls): return qiime2.core.path.ArchivePath() @classmethod def get_format_class(cls, version): try: imp, fmt_cls = cls._FORMAT_REGISTRY[version].split(':') except KeyError: return None return getattr(importlib.import_module(imp), fmt_cls) @classmethod def get_archive(cls, filepath): filepath = pathlib.Path(filepath) if not filepath.exists(): raise ValueError("%s does not exist." % filepath) if _ZipArchive.is_archive_type(filepath): archive = _ZipArchive(filepath) else: raise ValueError("%s is not a QIIME archive." % filepath) return archive @classmethod def _futuristic_archive_error(cls, filepath, archive): raise ValueError("%s was created by 'QIIME %s'. The currently" " installed framework cannot interpret archive" " version %r." % (filepath, archive.framework_version, archive.version)) @classmethod def peek(cls, filepath): archive = cls.get_archive(filepath) Format = cls.get_format_class(archive.version) if Format is None: cls._futuristic_archive_error(filepath, archive) # NOTE: in the future, we may want to manipulate the results so that # older formats provide the "new" API even if they don't support it. # e.g. a new format has a new property that peek should describe. We # add some compatability code here to return a default for that # property on older formats. return Format.load_metadata(archive) @classmethod def extract(cls, filepath, dest): archive = cls.get_archive(filepath) # Format really doesn't matter, the archive knows how to extract so # that is sufficient, furthermore it would suck if something was wrong # with an archive's format and extract failed to actually extract. return str(archive.extract(dest)) @classmethod def load(cls, filepath): archive = cls.get_archive(filepath) Format = cls.get_format_class(archive.version) if Format is None: cls._futuristic_archive_error(filepath, archive) path = cls._make_temp_path() rec = archive.mount(path) return cls(path, Format(rec)) @classmethod def from_data(cls, type, format, data_initializer, provenance_capture): path = cls._make_temp_path() rec = cls.CURRENT_ARCHIVE.setup(path, cls.CURRENT_FORMAT_VERSION, qiime2.__version__) Format = cls.get_format_class(cls.CURRENT_FORMAT_VERSION) Format.write(rec, type, format, data_initializer, provenance_capture) return cls(path, Format(rec)) def __init__(self, path, fmt): self.path = path self._fmt = fmt @property def uuid(self): return self._fmt.uuid @property def type(self): return self._fmt.type @property def format(self): return self._fmt.format @property def data_dir(self): return self._fmt.data_dir @property def root_dir(self): return self._fmt.path @property def provenance_dir(self): return getattr(self._fmt, 'provenance_dir', None) @property def citations(self): return getattr(self._fmt, 'citations', cite.Citations()) def save(self, filepath): self.CURRENT_ARCHIVE.save(self.path, filepath) def validate_checksums(self): if not isinstance(self._fmt, self.get_format_class('5')): return ChecksumDiff({}, {}, {}) obs = dict(x for x in md5sum_directory(str(self.root_dir)).items() if x[0] != self._fmt.CHECKSUM_FILE) exp = dict(from_checksum_format(line) for line in (self.root_dir / self._fmt.CHECKSUM_FILE).open().readlines() ) obs_keys = set(obs) exp_keys = set(exp) added = {x: obs[x] for x in obs_keys - exp_keys} removed = {x: exp[x] for x in exp_keys - obs_keys} changed = {x: (exp[x], obs[x]) for x in exp_keys & obs_keys if exp[x] != obs[x]} return ChecksumDiff(added=added, removed=removed, changed=changed) @property def _destructor(self): return self.path._destructor qiime-2021.8.0/qiime2/core/archive/format/000077500000000000000000000000001412141660100200735ustar00rootroot00000000000000qiime-2021.8.0/qiime2/core/archive/format/__init__.py000066400000000000000000000007371412141660100222130ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- # NOTE: Don't import anything here. Importing one format shouldn't import all # of them, that is a waste of the computer's time. qiime-2021.8.0/qiime2/core/archive/format/tests/000077500000000000000000000000001412141660100212355ustar00rootroot00000000000000qiime-2021.8.0/qiime2/core/archive/format/tests/__init__.py000066400000000000000000000005351412141660100233510ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- qiime-2021.8.0/qiime2/core/archive/format/tests/test_util.py000066400000000000000000000055151412141660100236310ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest import tempfile import os import zipfile from qiime2.core.testing.type import FourInts from qiime2.core.testing.util import ArchiveTestingMixin import qiime2.core.archive as archive from qiime2.core.archive.format.util import artifact_version from qiime2.sdk import Artifact class TestArtifactVersion(unittest.TestCase, ArchiveTestingMixin): def setUp(self): prefix = "qiime2-test-temp-" self.temp_dir = tempfile.TemporaryDirectory(prefix=prefix) self.provenance_capture = archive.ImportProvenanceCapture() def test_nonexistent_archive_format(self): with self.assertRaisesRegex(ValueError, 'Version foo not supported'): with artifact_version('foo'): pass def test_write_v0_archive(self): fp = os.path.join(self.temp_dir.name, 'artifact_v0.qza') with artifact_version(0): artifact = Artifact._from_view(FourInts, [-1, 42, 0, 43], list, self.provenance_capture) artifact.save(fp) root_dir = str(artifact.uuid) # There should be no provenance expected = { 'VERSION', 'metadata.yaml', 'data/file1.txt', 'data/file2.txt', 'data/nested/file3.txt', 'data/nested/file4.txt', } self.assertArchiveMembers(fp, root_dir, expected) with zipfile.ZipFile(fp, mode='r') as zf: version = zf.read(os.path.join(root_dir, 'VERSION')) self.assertRegex(str(version), '^.*archive: 0.*$') def test_write_v4_archive(self): fp = os.path.join(self.temp_dir.name, 'artifact_v1.qza') with artifact_version(4): artifact = Artifact._from_view(FourInts, [-1, 42, 0, 43], list, self.provenance_capture) artifact.save(fp) root_dir = str(artifact.uuid) expected = { 'VERSION', 'metadata.yaml', 'data/file1.txt', 'data/file2.txt', 'data/nested/file3.txt', 'data/nested/file4.txt', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml', } self.assertArchiveMembers(fp, root_dir, expected) with zipfile.ZipFile(fp, mode='r') as zf: version = zf.read(os.path.join(root_dir, 'VERSION')) self.assertRegex(str(version), '^.*archive: 4.*$') qiime-2021.8.0/qiime2/core/archive/format/tests/test_v0.py000066400000000000000000000044031412141660100231740ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest import tempfile import uuid as _uuid import pathlib import io from qiime2.core.testing.type import IntSequence1 from qiime2.core.testing.format import IntSequenceDirectoryFormat from qiime2.core.archive.archiver import _ZipArchive, ArchiveRecord from qiime2.core.archive.format.v0 import ArchiveFormat class TestArchiveFormat(unittest.TestCase): def setUp(self): prefix = "qiime2-test-temp-" self.temp_dir = tempfile.TemporaryDirectory(prefix=prefix) def test_format_metadata(self): uuid = _uuid.uuid4() with io.StringIO() as fh: ArchiveFormat._format_metadata(fh, uuid, IntSequence1, IntSequenceDirectoryFormat) result = fh.getvalue() self.assertEqual(result, "uuid: %s\ntype: IntSequence1\nformat: " "IntSequenceDirectoryFormat\n" % uuid) def test_format_metadata_none(self): uuid = _uuid.uuid4() with io.StringIO() as fh: ArchiveFormat._format_metadata(fh, uuid, IntSequence1, None) result = fh.getvalue() self.assertEqual(result, "uuid: %s\ntype: IntSequence1\nformat: null\n" % uuid) def test_load_root_dir_metadata_uuid_mismatch(self): fp = pathlib.Path(self.temp_dir.name) / 'root-dir-metadata-mismatch' fp.mkdir() r = _ZipArchive.setup(fp, 'foo', 'bar') fake = ArchiveRecord(r.root, r.version_fp, _uuid.uuid4(), # This will trick the format r.version, r.framework_version) ArchiveFormat.write(fake, IntSequence1, IntSequenceDirectoryFormat, lambda x: None, None) with self.assertRaisesRegex( ValueError, 'root directory must match UUID.*metadata'): ArchiveFormat(r) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/archive/format/util.py000066400000000000000000000014621412141660100214250ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from contextlib import contextmanager from qiime2.core.archive import Archiver @contextmanager def artifact_version(version): version = str(version) if version not in Archiver._FORMAT_REGISTRY: raise ValueError("Version %s not supported" % version) original_version = Archiver.CURRENT_FORMAT_VERSION try: Archiver.CURRENT_FORMAT_VERSION = version yield finally: Archiver.CURRENT_FORMAT_VERSION = original_version qiime-2021.8.0/qiime2/core/archive/format/v0.py000066400000000000000000000046771412141660100210100ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import collections import uuid as _uuid import yaml import qiime2.sdk as sdk # Allow OrderedDict to be serialized for YAML representation yaml.add_representer(collections.OrderedDict, lambda dumper, data: dumper.represent_dict(data.items())) class ArchiveFormat: DATA_DIR = 'data' METADATA_FILE = 'metadata.yaml' @classmethod def _parse_metadata(self, fh, expected_uuid): metadata = yaml.safe_load(fh) if metadata['uuid'] != str(expected_uuid): raise ValueError( "Archive root directory must match UUID present in archive's" " metadata: %s != %s" % (expected_uuid, metadata['uuid'])) return metadata['uuid'], metadata['type'], metadata['format'] @classmethod def _format_metadata(self, fh, uuid, type, format): metadata = collections.OrderedDict() metadata['uuid'] = str(uuid) metadata['type'] = repr(type) metadata['format'] = None if format is not None: metadata['format'] = format.__name__ fh.write(yaml.dump(metadata, default_flow_style=False)) @classmethod def load_metadata(self, archive): with archive.open(self.METADATA_FILE) as fh: return self._parse_metadata(fh, expected_uuid=archive.uuid) @classmethod def write(cls, archive_record, type, format, data_initializer, _): root = archive_record.root metadata_fp = root / cls.METADATA_FILE with metadata_fp.open(mode='w') as fh: cls._format_metadata(fh, archive_record.uuid, type, format) data_dir = root / cls.DATA_DIR data_dir.mkdir() data_initializer(data_dir) def __init__(self, archive_record): path = archive_record.root with (path / self.METADATA_FILE).open() as fh: uuid, type, format = \ self._parse_metadata(fh, expected_uuid=archive_record.uuid) self.uuid = _uuid.UUID(uuid) self.type = sdk.parse_type(type) self.format = sdk.parse_format(format) self.path = path self.data_dir = path / self.DATA_DIR qiime-2021.8.0/qiime2/core/archive/format/v1.py000066400000000000000000000020641412141660100207750ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import qiime2.core.archive.format.v0 as v0 class ArchiveFormat(v0.ArchiveFormat): PROVENANCE_DIR = 'provenance' @classmethod def write(cls, archive_record, type, format, data_initializer, provenance_capture): super().write(archive_record, type, format, data_initializer, provenance_capture) root = archive_record.root prov_dir = root / cls.PROVENANCE_DIR prov_dir.mkdir() provenance_capture.finalize( prov_dir, [root / cls.METADATA_FILE, archive_record.version_fp]) def __init__(self, archive_record): super().__init__(archive_record) self.provenance_dir = archive_record.root / self.PROVENANCE_DIR qiime-2021.8.0/qiime2/core/archive/format/v2.py000066400000000000000000000012761412141660100210020ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import qiime2.core.archive.format.v1 as v1 class ArchiveFormat(v1.ArchiveFormat): # Exactly the same as v1, but in provenance, when the action type isn't # import, there is an `output-name` key in the action section with that # node's output name according to the action's signature object. Also has # pipeline action types. pass qiime-2021.8.0/qiime2/core/archive/format/v3.py000066400000000000000000000011641412141660100207770ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import qiime2.core.archive.format.v2 as v2 class ArchiveFormat(v2.ArchiveFormat): # Exactly the same as v2, but inputs may be variadic where the UUIDs are in # a YAML sequence. Additionally `Set` is now represented as a sequence # with a custom !set tag. pass qiime-2021.8.0/qiime2/core/archive/format/v4.py000066400000000000000000000027331412141660100210030ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import qiime2.core.archive.format.v3 as v3 from qiime2.core.cite import Citations class ArchiveFormat(v3.ArchiveFormat): # - Adds a transformers section to action.yaml # - Adds citations via the !cite yaml type which references the # /provenance/citations.bib file (this is nested like everything else # in the /provenance/artifacts/ # directories). # - environment:framework has been updated to be a nested object, # its schema is identical to a environment:plugins: object. # Prior to v4, it was only a version string. @property def citations(self): files = [] files.append(str(self.provenance_dir / 'citations.bib')) if (self.provenance_dir / 'artifacts').exists(): for ancestor in (self.provenance_dir / 'artifacts').iterdir(): if (ancestor / 'citations.bib').exists(): files.append(str(ancestor / 'citations.bib')) citations = Citations() for f in files: citations.update(Citations.load(f)) return citations qiime-2021.8.0/qiime2/core/archive/format/v5.py000066400000000000000000000021001412141660100207700ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import qiime2.core.archive.format.v4 as v4 from qiime2.core.util import md5sum_directory, to_checksum_format class ArchiveFormat(v4.ArchiveFormat): CHECKSUM_FILE = 'checksums.md5' # Adds `checksums.md5` to root of directory structure @classmethod def write(cls, archive_record, type, format, data_initializer, provenance_capture): super().write(archive_record, type, format, data_initializer, provenance_capture) checksums = md5sum_directory(str(archive_record.root)) with (archive_record.root / cls.CHECKSUM_FILE).open('w') as fh: for item in checksums.items(): fh.write(to_checksum_format(*item)) fh.write('\n') qiime-2021.8.0/qiime2/core/archive/provenance.py000066400000000000000000000407341412141660100213250ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import time import collections import collections.abc import pkg_resources import uuid import copy import shutil import sys from datetime import datetime, timezone import distutils import yaml import tzlocal import dateutil.relativedelta as relativedelta import qiime2 import qiime2.core.util as util from qiime2.core.cite import Citations def _ts_to_date(ts): time_zone = timezone.utc try: time_zone = tzlocal.get_localzone() except ValueError: pass return datetime.fromtimestamp(ts, tz=time_zone) # Used to give PyYAML something to recognize for custom tags ForwardRef = collections.namedtuple('ForwardRef', ['reference']) NoProvenance = collections.namedtuple('NoProvenance', ['uuid']) MetadataPath = collections.namedtuple('MetadataPath', ['path']) ColorPrimitive = collections.namedtuple('ColorPrimitive', ['hex']) LiteralString = collections.namedtuple('LiteralString', ['string']) CitationKey = collections.namedtuple('CitationKey', ['key']) class OrderedKeyValue(collections.OrderedDict): pass # Used for yaml that looks like: # - key1: value1 # - key2: value2 yaml.add_representer(OrderedKeyValue, lambda dumper, data: dumper.represent_list([ {k: v} for k, v in data.items()])) # Controlling the order of dictionaries (even if semantically irrelevant) is # important to making it look nice. yaml.add_representer(collections.OrderedDict, lambda dumper, data: dumper.represent_dict(data.items())) # YAML libraries aren't good at writing a clean version of this, and typically # the fact that it is a set is irrelevant to tools that use provenance # so add a custom tag and treat it like a sequence. Then code doesn't need to # special case set vs list in their business logic when it isn't important. yaml.add_representer(set, lambda dumper, data: dumper.represent_sequence('!set', data)) # LiteralString uses the | character and has literal newlines yaml.add_representer(LiteralString, lambda dumper, data: dumper.represent_scalar('tag:yaml.org,2002:str', data.string, style='|')) # Make our timestamps pretty (unquoted). yaml.add_representer(datetime, lambda dumper, data: dumper.represent_scalar('tag:yaml.org,2002:timestamp', data.isoformat())) # Forward reference to something else in the document, namespaces are # delimited by colons (:). yaml.add_representer(ForwardRef, lambda dumper, data: dumper.represent_scalar('!ref', data.reference)) # This tag represents an artifact without provenance, this is to support # archive format v0. Ideally this won't be seen in the wild in practice. yaml.add_representer(NoProvenance, lambda dumper, data: dumper.represent_scalar('!no-provenance', str(data.uuid))) # A reference to Metadata and MetadataColumn whose data can be found at the # relative path indicated as its value yaml.add_representer(MetadataPath, lambda dumper, data: dumper.represent_scalar('!metadata', data.path)) # A color primitive. yaml.add_representer(ColorPrimitive, lambda dumper, data: dumper.represent_scalar('!color', data.hex)) yaml.add_representer(CitationKey, lambda dumper, data: dumper.represent_scalar('!cite', data.key)) class ProvenanceCapture: ANCESTOR_DIR = 'artifacts' ACTION_DIR = 'action' ACTION_FILE = 'action.yaml' CITATION_FILE = 'citations.bib' def __init__(self): self.start = time.time() self.uuid = uuid.uuid4() self.end = None self.plugins = collections.OrderedDict() # For the purposes of this dict, `return` is a special case for output # we expect to transform this later when serializing, but this lets # us treat all transformations uniformly. self.transformers = collections.OrderedDict() self.citations = Citations() self._framework_citations = [] for idx, citation in enumerate(qiime2.__citations__): citation_key = self.make_citation_key('framework') self.citations[citation_key.key] = citation self._framework_citations.append(citation_key) self._build_paths() @property def _destructor(self): return self.path._destructor def _build_paths(self): self.path = qiime2.core.path.ProvenancePath() self.ancestor_dir = self.path / self.ANCESTOR_DIR self.ancestor_dir.mkdir() self.action_dir = self.path / self.ACTION_DIR self.action_dir.mkdir() def add_ancestor(self, artifact): other_path = artifact._archiver.provenance_dir if other_path is None: # The artifact doesn't have provenance (e.g. version 0) # it would be possible to invent a metadata.yaml, but we won't know # the framework version for the VERSION file. Even if we did # it won't accomplish a lot and there shouldn't be enough # version 0 artifacts in the wild to be important in practice. # NOTE: this implies that it is possible for an action.yaml file to # contain an artifact UUID that is not in the artifacts/ directory. return NoProvenance(artifact.uuid) destination = self.ancestor_dir / str(artifact.uuid) # If it exists, then the artifact is already in the provenance # (and so are its ancestors) if not destination.exists(): # Handle root node of ancestor shutil.copytree( str(other_path), str(destination), ignore=shutil.ignore_patterns(self.ANCESTOR_DIR + '*')) # Handle ancestral nodes of ancestor grandcestor_path = other_path / self.ANCESTOR_DIR if grandcestor_path.exists(): for grandcestor in grandcestor_path.iterdir(): destination = self.ancestor_dir / grandcestor.name if not destination.exists(): shutil.copytree(str(grandcestor), str(destination)) return str(artifact.uuid) def make_citation_key(self, domain, package=None, identifier=None, index=0): if domain == 'framework': package, version = 'qiime2', qiime2.__version__ else: package, version = package.name, package.version id_block = [] if identifier is None else [identifier] return CitationKey('|'.join( [domain, package + ':' + version] + id_block + [str(index)])) def make_software_entry(self, version, website, citations=()): entry = collections.OrderedDict() entry['version'] = version entry['website'] = website if citations: entry['citations'] = citations return entry def reference_plugin(self, plugin): plugin_citations = [] for idx, citation in enumerate(plugin.citations): citation_key = self.make_citation_key('plugin', plugin, index=idx) self.citations[citation_key.key] = citation plugin_citations.append(citation_key) self.plugins[plugin.name] = self.make_software_entry( plugin.version, plugin.website, plugin_citations) return ForwardRef('environment:plugins:' + plugin.name) def capture_env(self): return collections.OrderedDict( (d.project_name, d.version) for d in pkg_resources.working_set) def transformation_recorder(self, name): section = self.transformers[name] = [] def recorder(transformer_record, input_name, input_record, output_name, output_record): entry = collections.OrderedDict() entry['from'] = input_name entry['to'] = output_name citation_keys = [] if transformer_record is not None: plugin = transformer_record.plugin entry['plugin'] = self.reference_plugin(plugin) for idx, citation in enumerate(transformer_record.citations): citation_key = self.make_citation_key( 'transformer', plugin, '%s->%s' % (input_name, output_name), idx) self.citations[citation_key.key] = citation citation_keys.append(citation_key) records = [] if input_record is not None: records.append(input_record) if output_record is not None: records.append(output_record) for record in records: self.reference_plugin(record.plugin) for idx, citation in enumerate(record.citations): citation_key = self.make_citation_key( 'view', record.plugin, record.name, idx) self.citations[citation_key.key] = citation citation_keys.append(citation_key) if citation_keys: entry['citations'] = citation_keys section.append(entry) return recorder def make_execution_section(self): execution = collections.OrderedDict() execution['uuid'] = str(self.uuid) execution['runtime'] = runtime = collections.OrderedDict() runtime['start'] = start = _ts_to_date(self.start) runtime['end'] = end = _ts_to_date(self.end) runtime['duration'] = \ util.duration_time(relativedelta.relativedelta(end, start)) return execution def make_transformers_section(self): transformers = collections.OrderedDict() data = self.transformers.copy() output = data.pop('return', None) if data: transformers['inputs'] = data if output is not None: transformers['output'] = output return transformers def make_env_section(self): env = collections.OrderedDict() env['platform'] = pkg_resources.get_build_platform() # There is a trailing whitespace in sys.version, strip so that YAML can # use literal formatting. env['python'] = LiteralString('\n'.join(line.strip() for line in sys.version.split('\n'))) env['framework'] = self.make_software_entry( qiime2.__version__, qiime2.__website__, self._framework_citations) env['plugins'] = self.plugins env['python-packages'] = self.capture_env() return env def write_action_yaml(self): settings = dict(default_flow_style=False, indent=4) with (self.action_dir / self.ACTION_FILE).open(mode='w') as fh: fh.write(yaml.dump({'execution': self.make_execution_section()}, **settings)) fh.write('\n') fh.write(yaml.dump({'action': self.make_action_section()}, **settings)) if self.transformers: # pipelines don't have these fh.write('\n') fh.write(yaml.dump( {'transformers': self.make_transformers_section()}, **settings)) fh.write('\n') fh.write(yaml.dump({'environment': self.make_env_section()}, **settings)) def write_citations_bib(self): self.citations.save(str(self.path / self.CITATION_FILE)) def finalize(self, final_path, node_members): self.end = time.time() for member in node_members: shutil.copy(str(member), str(self.path)) self.write_action_yaml() self.write_citations_bib() # Certain networked filesystems will experience a race # condition on `rename`, so fall back to copying. try: os.rename(self.path, final_path) except FileExistsError: distutils.dir_util.copy_tree(str(self.path), str(final_path)) distutils.dir_util.remove_tree(str(self.path)) def fork(self): forked = copy.copy(self) # Unique state for each output of an action forked.plugins = forked.plugins.copy() forked.transformers = forked.transformers.copy() forked.citations = forked.citations.copy() # create a copy of the backing dir so factory (the hard stuff is # mostly done by this point) forked._build_paths() distutils.dir_util.copy_tree(str(self.path), str(forked.path)) return forked class ImportProvenanceCapture(ProvenanceCapture): def __init__(self, format=None, checksums=None): super().__init__() self.format_name = format.__name__ if format is not None else None self.checksums = checksums def make_action_section(self): action = collections.OrderedDict() action['type'] = 'import' if self.format_name is not None: action['format'] = self.format_name if self.checksums is not None: action['manifest'] = [ collections.OrderedDict([('name', name), ('md5sum', md5sum)]) for name, md5sum in self.checksums.items()] return action class ActionProvenanceCapture(ProvenanceCapture): def __init__(self, action_type, plugin_id, action_id): from qiime2.sdk import PluginManager super().__init__() self._plugin = PluginManager().get_plugin(id=plugin_id) self.action = self._plugin.actions[action_id] self.action_type = action_type self.inputs = OrderedKeyValue() self.parameters = OrderedKeyValue() self.output_name = '' self._action_citations = [] for idx, citation in enumerate(self.action.citations): citation_key = self.make_citation_key( 'action', self._plugin, ':'.join([self.action_type, self.action.id]), idx) self.citations[citation_key.key] = citation self._action_citations.append(citation_key) def handle_metadata(self, name, value): if value is None: return None uuid_ref = "" if value.artifacts: uuids = [] for artifact in value.artifacts: uuids.append(str(artifact.uuid)) self.add_ancestor(artifact) uuid_ref = ",".join(uuids) + ":" relpath = name + '.tsv' value.save(str(self.action_dir / relpath)) return MetadataPath(uuid_ref + relpath) def add_parameter(self, name, type_expr, parameter): type_map = { 'Color': ColorPrimitive, 'Metadata': lambda x: self.handle_metadata(name, x), 'MetadataColumn': lambda x: self.handle_metadata(name, x) # TODO: handle collection primitives (not currently used) } handler = type_map.get(type_expr.to_ast().get('name'), lambda x: x) self.parameters[name] = handler(parameter) def add_input(self, name, input): if input is None: self.inputs[name] = None elif isinstance(input, collections.abc.Iterable): values = [] for artifact in input: record = self.add_ancestor(artifact) values.append(record) self.inputs[name] = type(input)(values) else: self.inputs[name] = self.add_ancestor(input) def make_action_section(self): action = collections.OrderedDict() action['type'] = self.action_type action['plugin'] = self.reference_plugin(self._plugin) action['action'] = self.action.id action['inputs'] = self.inputs action['parameters'] = self.parameters action['output-name'] = self.output_name if self._action_citations: action['citations'] = self._action_citations return action def fork(self, name): forked = super().fork() forked.output_name = name return forked class PipelineProvenanceCapture(ActionProvenanceCapture): def make_action_section(self): action = super().make_action_section() action['alias-of'] = str(self.alias.uuid) return action def fork(self, name, alias): forked = super().fork(name) forked.alias = alias forked.add_ancestor(alias) return forked qiime-2021.8.0/qiime2/core/archive/tests/000077500000000000000000000000001412141660100177455ustar00rootroot00000000000000qiime-2021.8.0/qiime2/core/archive/tests/__init__.py000066400000000000000000000005351412141660100220610ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- qiime-2021.8.0/qiime2/core/archive/tests/test_archiver.py000066400000000000000000000334041412141660100231650ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import tempfile import unittest import uuid import zipfile import pathlib from qiime2.core.archive import Archiver from qiime2.core.archive import ImportProvenanceCapture from qiime2.core.archive.archiver import _ZipArchive from qiime2.core.archive.format.util import artifact_version from qiime2.core.testing.format import IntSequenceDirectoryFormat from qiime2.core.testing.type import IntSequence1 from qiime2.core.testing.util import ArchiveTestingMixin class TestArchiver(unittest.TestCase, ArchiveTestingMixin): def setUp(self): prefix = "qiime2-test-temp-" self.temp_dir = tempfile.TemporaryDirectory(prefix=prefix) # Initialize an Archiver. The values passed to the constructor mostly # don't matter to the Archiver, but we'll pass valid Artifact test data # anyways in case Archiver's behavior changes in the future. def data_initializer(data_dir): fp = os.path.join(str(data_dir), 'ints.txt') with open(fp, 'w') as fh: fh.write('1\n') fh.write('2\n') fh.write('3\n') self.archiver = Archiver.from_data( IntSequence1, IntSequenceDirectoryFormat, data_initializer=data_initializer, provenance_capture=ImportProvenanceCapture()) def tearDown(self): self.temp_dir.cleanup() def test_save_invalid_filepath(self): # Empty filepath. with self.assertRaisesRegex(FileNotFoundError, 'No such file'): self.archiver.save('') # Directory. with self.assertRaisesRegex(IsADirectoryError, 'directory'): self.archiver.save(self.temp_dir.name) # Ends with path separator (no basename, e.g. /tmp/foo/). with self.assertRaises((IsADirectoryError, FileNotFoundError)): self.archiver.save(os.path.join(self.temp_dir.name, 'foo', '')) def test_save_excludes_dotfiles_in_data_dir(self): def data_initializer(data_dir): data_dir = str(data_dir) fp = os.path.join(data_dir, 'ints.txt') with open(fp, 'w') as fh: fh.write('1\n') fh.write('2\n') fh.write('3\n') hidden_fp = os.path.join(data_dir, '.hidden-file') with open(hidden_fp, 'w') as fh: fh.write("You can't see me if I can't see you\n") hidden_dir = os.path.join(data_dir, '.hidden-dir') os.mkdir(hidden_dir) with open(os.path.join(hidden_dir, 'ignored-file'), 'w') as fh: fh.write("I'm ignored because I live in a hidden dir :(\n") archiver = Archiver.from_data( IntSequence1, IntSequenceDirectoryFormat, data_initializer=data_initializer, provenance_capture=ImportProvenanceCapture()) fp = os.path.join(self.temp_dir.name, 'archive.zip') archiver.save(fp) root_dir = str(archiver.uuid) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/ints.txt', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertArchiveMembers(fp, root_dir, expected) def test_save_archive_members(self): fp = os.path.join(self.temp_dir.name, 'archive.zip') self.archiver.save(fp) root_dir = str(self.archiver.uuid) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/ints.txt', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertArchiveMembers(fp, root_dir, expected) def test_load_archive(self): fp = os.path.join(self.temp_dir.name, 'archive.zip') self.archiver.save(fp) archiver = Archiver.load(fp) self.assertEqual(archiver.uuid, self.archiver.uuid) self.assertEqual(archiver.type, IntSequence1) self.assertEqual(archiver.format, IntSequenceDirectoryFormat) self.assertEqual({str(p.relative_to(archiver.data_dir)) for p in archiver.data_dir.iterdir()}, {'ints.txt'}) def test_load_ignores_root_dotfiles(self): fp = os.path.join(self.temp_dir.name, 'archive.zip') self.archiver.save(fp) # Add some dotfiles to the archive. with zipfile.ZipFile(fp, mode='a') as zf: zf.writestr('.DS_Store', "The world's most beloved file\n") zf.writestr('.hidden-file', "You can't see me if I can't see you\n") zf.writestr('.hidden-dir/ignored-file', "I'm ignored because I live in a hidden dir :(\n") # Assert the expected files exist in the archive to verify this test # case is testing what we want it to. with zipfile.ZipFile(fp, mode='r') as zf: root_dir = str(self.archiver.uuid) expected = { '.DS_Store', '.hidden-file', '.hidden-dir/ignored-file', '%s/VERSION' % root_dir, '%s/checksums.md5' % root_dir, '%s/metadata.yaml' % root_dir, '%s/data/ints.txt' % root_dir, '%s/provenance/metadata.yaml' % root_dir, '%s/provenance/VERSION' % root_dir, '%s/provenance/citations.bib' % root_dir, '%s/provenance/action/action.yaml' % root_dir } observed = set(zf.namelist()) # Not using self.assertArchiveMembers() because it accepts paths # relative to root_dir, and we have extra paths at the same level # as root_dir. self.assertEqual(observed, expected) archiver = Archiver.load(fp) self.assertEqual(archiver.uuid, self.archiver.uuid) self.assertEqual(archiver.type, IntSequence1) self.assertEqual(archiver.format, IntSequenceDirectoryFormat) self.assertEqual({str(p.relative_to(archiver.data_dir)) for p in archiver.data_dir.iterdir()}, {'ints.txt'}) def test_load_ignores_directory_members(self): # Directory members aren't created by Python's zipfile module but can # be present if the archive is unzipped and then rezipped, for example, # using a command-line zip program. fp = os.path.join(self.temp_dir.name, 'archive.zip') self.archiver.save(fp) # Add directory entries to the archive. root_dir = str(self.archiver.uuid) with zipfile.ZipFile(fp, mode='a') as zf: zf.writestr('%s/' % root_dir, "") zf.writestr('%s/data/' % root_dir, "") zf.writestr('%s/data/nested/' % root_dir, "") zf.writestr('%s/data/nested/foo.txt' % root_dir, "bar") # Assert the expected files exist in the archive to verify this test # case is testing what we want it to. expected = { '', # Expected path: `root_dir`/ 'data/', 'data/nested/', 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/ints.txt', 'data/nested/foo.txt', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertArchiveMembers(fp, root_dir, expected) archiver = Archiver.load(fp) self.assertEqual(archiver.uuid, self.archiver.uuid) self.assertEqual(archiver.type, IntSequence1) self.assertEqual(archiver.format, IntSequenceDirectoryFormat) archiver.save(fp) root_dir = str(archiver.uuid) expected = { # Directory entries should not be present. 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/ints.txt', 'data/nested/foo.txt', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertArchiveMembers(fp, root_dir, expected) def test_load_empty_archive(self): fp = os.path.join(self.temp_dir.name, 'empty.zip') with zipfile.ZipFile(fp, mode='w') as zf: pass with zipfile.ZipFile(fp, mode='r') as zf: expected = set() observed = set(zf.namelist()) self.assertEqual(observed, expected) with self.assertRaisesRegex(ValueError, 'visible root directory'): Archiver.load(fp) def test_load_dotfile_only_archive(self): fp = os.path.join(self.temp_dir.name, 'dotfiles-only.zip') with zipfile.ZipFile(fp, mode='w') as zf: zf.writestr('.DS_Store', "The world's most beloved file\n") zf.writestr('.hidden-file', "You can't see me if I can't see you\n") zf.writestr('.hidden-dir/ignored-file', "I'm ignored because I live in a hidden dir :(\n") with zipfile.ZipFile(fp, mode='r') as zf: expected = { '.DS_Store', '.hidden-file', '.hidden-dir/ignored-file' } observed = set(zf.namelist()) self.assertEqual(observed, expected) with self.assertRaisesRegex(ValueError, 'visible root directory'): Archiver.load(fp) def test_load_multiple_root_dirs(self): fp = os.path.join(self.temp_dir.name, 'multiple-root-dirs.zip') self.archiver.save(fp) # Add another semi-valid root dir. second_root_dir = str(uuid.uuid4()) with zipfile.ZipFile(fp, mode='a') as zf: zf.writestr('%s/VERSION' % second_root_dir, "foo") with zipfile.ZipFile(fp, mode='r') as zf: root_dir = str(self.archiver.uuid) expected = { '%s/VERSION' % root_dir, '%s/checksums.md5' % root_dir, '%s/metadata.yaml' % root_dir, '%s/data/ints.txt' % root_dir, '%s/provenance/metadata.yaml' % root_dir, '%s/provenance/VERSION' % root_dir, '%s/provenance/citations.bib' % root_dir, '%s/provenance/action/action.yaml' % root_dir, '%s/VERSION' % second_root_dir } observed = set(zf.namelist()) self.assertEqual(observed, expected) with self.assertRaisesRegex(ValueError, 'multiple root directories'): Archiver.load(fp) def test_load_invalid_uuid4_root_dir(self): fp = pathlib.Path(self.temp_dir.name) / 'invalid-uuid4' zp = pathlib.Path(self.temp_dir.name) / 'bad.zip' fp.mkdir() # Invalid uuid4 taken from https://gist.github.com/ShawnMilo/7777304 root_dir = '89eb3586-8a82-47a4-c911-758a62601cf7' record = _ZipArchive.setup(fp, 'foo', 'bar') (fp / str(record.uuid)).rename(fp / root_dir) _ZipArchive.save(fp, zp) with self.assertRaisesRegex(ValueError, 'root directory.*valid version 4 UUID'): _ZipArchive(zp) def test_is_uuid4_valid(self): uuid_str = str(uuid.uuid4()) self.assertTrue(_ZipArchive._is_uuid4(uuid_str)) def test_parse_uuid_invalid(self): # Invalid uuid4 taken from https://gist.github.com/ShawnMilo/7777304 uuid_str = '89eb3586-8a82-47a4-c911-758a62601cf7' self.assertFalse(_ZipArchive._is_uuid4(uuid_str)) # Not a UUID. uuid_str = 'abc123' self.assertFalse(_ZipArchive._is_uuid4(uuid_str)) # Other UUID versions. for uuid_ in (uuid.uuid1(), uuid.uuid3(uuid.NAMESPACE_DNS, 'foo'), uuid.uuid5(uuid.NAMESPACE_DNS, 'bar')): uuid_str = str(uuid_) self.assertFalse(_ZipArchive._is_uuid4(uuid_str)) def test_checksums_match(self): diff = self.archiver.validate_checksums() self.assertEqual(diff.added, {}) self.assertEqual(diff.removed, {}) self.assertEqual(diff.changed, {}) def test_checksums_mismatch(self): with (self.archiver.root_dir / 'data' / 'ints.txt').open('w') as fh: fh.write('999\n') with (self.archiver.root_dir / 'tamper.txt').open('w') as fh: fh.write('extra file') (self.archiver.root_dir / 'VERSION').unlink() diff = self.archiver.validate_checksums() self.assertEqual(diff.added, {'tamper.txt': '296583001b00d2b811b5871b19e0ad28'}) # The contents of most files is either stochastic, or has the current # version (which is an unknown commit sha1), so just check name self.assertEqual(list(diff.removed.keys()), ['VERSION']) self.assertEqual(diff.changed, {'data/ints.txt': ('c0710d6b4f15dfa88f600b0e6b624077', 'f47bc36040d5c7db08e4b3a457dcfbb2') }) def test_checksum_backwards_compat(self): self.tearDown() with artifact_version(4): self.setUp() diff = self.archiver.validate_checksums() self.assertEqual(diff.added, {}) self.assertEqual(diff.removed, {}) self.assertEqual(diff.changed, {}) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/archive/tests/test_citations.py000066400000000000000000000071351412141660100233610ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest import qiime2 from qiime2.core.testing.type import IntSequence1 from qiime2.core.testing.util import get_dummy_plugin class TestCitationsTracked(unittest.TestCase): def setUp(self): self.plugin = get_dummy_plugin() def test_import(self): data = qiime2.Artifact.import_data(IntSequence1, [1, 2, 3, 4]) archiver = data._archiver expected = [ ('framework|qiime2:%s|0' % qiime2.__version__, 'Reproducible, interactive, scalable and extensible microbiome ' 'data science using QIIME 2'), ('plugin|dummy-plugin:0.0.0-dev|0', 'Does knuckle cracking lead to arthritis of the fingers?'), ('plugin|dummy-plugin:0.0.0-dev|1', 'Of flying frogs and levitrons'), ('transformer|dummy-plugin:0.0.0-dev|' 'builtins:list->IntSequenceDirectoryFormat|0', 'An in-depth analysis of a piece of shit: distribution of' ' Schistosoma mansoni and hookworm eggs in human stool'), ('view|dummy-plugin:0.0.0-dev|IntSequenceDirectoryFormat|0', 'Walking with coffee: Why does it spill?')] obs = list(map(lambda item: (item[0], item[1].fields['title']), archiver.citations.items())) self.assertEqual(obs, expected) with (archiver.provenance_dir / 'action' / 'action.yaml').open() as fh: action_yaml = fh.read() for key, _ in expected: self.assertIn('!cite %r' % key, action_yaml) def test_action(self): data = qiime2.Artifact.import_data(IntSequence1, [1, 2, 3, 4]) action = self.plugin.methods['split_ints'] left, right = action(data) archiver = left._archiver expected = [ ('framework|qiime2:%s|0' % qiime2.__version__, 'Reproducible, interactive, scalable and extensible microbiome ' 'data science using QIIME 2'), ('action|dummy-plugin:0.0.0-dev|method:split_ints|0', 'Sword swallowing and its side effects'), ('action|dummy-plugin:0.0.0-dev|method:split_ints|1', 'Response behaviors of Svalbard reindeer towards humans and' ' humans disguised as polar bears on Edge\u00f8ya'), ('plugin|dummy-plugin:0.0.0-dev|0', 'Does knuckle cracking lead to arthritis of the fingers?'), ('plugin|dummy-plugin:0.0.0-dev|1', 'Of flying frogs and levitrons'), ('view|dummy-plugin:0.0.0-dev|IntSequenceDirectoryFormat|0', 'Walking with coffee: Why does it spill?'), ('transformer|dummy-plugin:0.0.0-dev|' 'builtins:list->IntSequenceDirectoryFormat|0', 'An in-depth analysis of a piece of shit: distribution of' ' Schistosoma mansoni and hookworm eggs in human stool')] obs = list(map(lambda item: (item[0], item[1].fields['title']), archiver.citations.items())) self.assertEqual(obs, expected) with (archiver.provenance_dir / 'action' / 'action.yaml').open() as fh: action_yaml = fh.read() for key, _ in expected: self.assertIn('!cite %r' % key, action_yaml) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/archive/tests/test_provenance.py000066400000000000000000000260311412141660100235200ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest import re import unittest.mock as mock import pandas as pd import qiime2 from qiime2.plugins import dummy_plugin from qiime2.core.testing.type import IntSequence1, Mapping import qiime2.core.archive.provenance as provenance class TestProvenanceIntegration(unittest.TestCase): def test_chain_with_metadata(self): df = pd.DataFrame({'a': ['1', '2', '3']}, index=pd.Index(['0', '1', '2'], name='feature ID')) a = qiime2.Artifact.import_data('IntSequence1', [1, 2, 3]) m = qiime2.Metadata(df) mc = qiime2.CategoricalMetadataColumn(df['a']) b = dummy_plugin.actions.identity_with_metadata(a, m).out c = dummy_plugin.actions.identity_with_metadata_column(b, mc).out p_dir = c._archiver.provenance_dir new_m = qiime2.Metadata.load( str(p_dir / 'artifacts' / str(b.uuid) / 'action' / 'metadata.tsv')) pd.testing.assert_frame_equal(m.to_dataframe(), new_m.to_dataframe()) with (p_dir / 'action' / 'metadata.tsv').open() as fh: self.assertEqual( fh.read(), 'feature ID\ta\n#q2:types\tcategorical\n0\t1\n1\t2\n2\t3\n') def test_chain_with_artifact_metadata(self): metadata_artifact_1 = qiime2.Artifact.import_data( 'Mapping', {'a': 'foo', 'b': 'bar'}) metadata_artifact_2 = qiime2.Artifact.import_data( 'Mapping', {'c': 'baz'}) m = metadata_artifact_1.view(qiime2.Metadata) mc = metadata_artifact_2.view(qiime2.Metadata).get_column('c') a = qiime2.Artifact.import_data('IntSequence1', [1, 2, 3]) b = dummy_plugin.actions.identity_with_metadata(a, m).out c = dummy_plugin.actions.identity_with_metadata_column(b, mc).out p_dir = c._archiver.provenance_dir m_yaml_value = "%s:metadata.tsv" % metadata_artifact_1.uuid mc_yaml_value = "%s:metadata.tsv" % metadata_artifact_2.uuid # Check action files for uuid-metadata values with (p_dir / 'action' / 'action.yaml').open() as fh: self.assertIn(mc_yaml_value, fh.read()) with (p_dir / 'artifacts' / str(b.uuid) / 'action' / 'action.yaml').open() as fh: self.assertIn(m_yaml_value, fh.read()) # Check that metadata is written out fully new_m = qiime2.Metadata.load( str(p_dir / 'artifacts' / str(b.uuid) / 'action' / 'metadata.tsv')) pd.testing.assert_frame_equal(m.to_dataframe(), new_m.to_dataframe()) # Check that provenance of originating metadata artifact exists self.assertTrue((p_dir / 'artifacts' / str(metadata_artifact_1.uuid) / 'action' / 'action.yaml').exists()) self.assertTrue((p_dir / 'artifacts' / str(metadata_artifact_2.uuid) / 'action' / 'action.yaml').exists()) def test_chain_with_merged_artifact_metadata(self): md_artifact1 = qiime2.Artifact.import_data( 'Mapping', {'a': 'foo', 'b': 'bar'}) md_artifact2 = qiime2.Artifact.import_data( 'Mapping', {'c': 'baz'}) md1 = md_artifact1.view(qiime2.Metadata) md2 = md_artifact2.view(qiime2.Metadata) merged_md = md1.merge(md2) merged_mdc = merged_md.get_column('c') a = qiime2.Artifact.import_data('IntSequence1', [1, 2, 3]) b = dummy_plugin.actions.identity_with_metadata(a, merged_md).out c = dummy_plugin.actions.identity_with_metadata_column( b, merged_mdc).out p_dir = c._archiver.provenance_dir yaml_value = "%s,%s:metadata.tsv" % (md_artifact1.uuid, md_artifact2.uuid) # Check action files for uuid-metadata values with (p_dir / 'action' / 'action.yaml').open() as fh: self.assertIn(yaml_value, fh.read()) with (p_dir / 'artifacts' / str(b.uuid) / 'action' / 'action.yaml').open() as fh: self.assertIn(yaml_value, fh.read()) # Check that metadata is written out fully with (p_dir / 'action' / 'metadata.tsv').open() as fh: self.assertEqual(fh.read(), 'id\tc\n#q2:types\tcategorical\n0\tbaz\n') new_merged_md = qiime2.Metadata.load( str(p_dir / 'artifacts' / str(b.uuid) / 'action' / 'metadata.tsv')) pd.testing.assert_frame_equal(new_merged_md.to_dataframe(), merged_md.to_dataframe()) # Check that provenance of originating metadata artifacts exists self.assertTrue((p_dir / 'artifacts' / str(md_artifact1.uuid) / 'action' / 'action.yaml').exists()) self.assertTrue((p_dir / 'artifacts' / str(md_artifact2.uuid) / 'action' / 'action.yaml').exists()) def test_with_optional_artifacts(self): ints1 = qiime2.Artifact.import_data(IntSequence1, [0, 42, 43]) ints2 = qiime2.Artifact.import_data(IntSequence1, [99, -22]) # One optional artifact is provided (`optional1`) while `optional2` is # omitted. obs = dummy_plugin.actions.optional_artifacts_method( ints1, 42, optional1=ints2).output p_dir = obs._archiver.provenance_dir with (p_dir / 'action' / 'action.yaml').open() as fh: yaml = fh.read() self.assertIn('ints: %s' % ints1.uuid, yaml) self.assertIn('optional1: %s' % ints2.uuid, yaml) self.assertIn('optional2: null', yaml) self.assertIn('num1: 42', yaml) self.assertIn('num2: null', yaml) self.assertTrue((p_dir / 'artifacts' / str(ints1.uuid) / 'action' / 'action.yaml').exists()) self.assertTrue((p_dir / 'artifacts' / str(ints2.uuid) / 'action' / 'action.yaml').exists()) def test_output_name_different(self): ints = qiime2.Artifact.import_data(IntSequence1, [0, 1, 2, 3]) left, right = dummy_plugin.actions.split_ints(ints) left_p_dir = left._archiver.provenance_dir right_p_dir = right._archiver.provenance_dir with (left_p_dir / 'action' / 'action.yaml').open() as fh: left_yaml = fh.read() with (right_p_dir / 'action' / 'action.yaml').open() as fh: right_yaml = fh.read() self.assertNotEqual(left_yaml, right_yaml) self.assertIn('output-name: left', left_yaml) self.assertIn('output-name: right', right_yaml) def test_output_name_visualization(self): viz, = dummy_plugin.actions.no_input_viz() viz_p_dir = viz._archiver.provenance_dir with (viz_p_dir / 'action' / 'action.yaml').open() as fh: self.assertIn('output-name: visualization', fh.read()) def test_no_output_name_import(self): ints = qiime2.Artifact.import_data(IntSequence1, [0, 2, 4]) ints_p_dir = ints._archiver.provenance_dir with (ints_p_dir / 'action' / 'action.yaml').open() as fh: self.assertNotIn('output-name:', fh.read()) def test_pipeline_alias_of(self): ints = qiime2.Artifact.import_data(IntSequence1, [1, 2, 3]) mapping = qiime2.Artifact.import_data(Mapping, {'foo': '42'}) r = dummy_plugin.actions.typical_pipeline(ints, mapping, False) # mapping is a pass-through new_mapping = r.out_map new_mapping_p_dir = new_mapping._archiver.provenance_dir with (new_mapping_p_dir / 'action' / 'action.yaml').open() as fh: new_mapping_yaml = fh.read() # Basic sanity check self.assertIn('type: pipeline', new_mapping_yaml) self.assertIn('int_sequence: %s' % ints.uuid, new_mapping_yaml) self.assertIn('mapping: %s' % mapping.uuid, new_mapping_yaml) # Remembers the original mapping uuid self.assertIn('alias-of: %s' % mapping.uuid, new_mapping_yaml) def test_nested_pipeline_alias_of(self): ints = qiime2.Artifact.import_data(IntSequence1, [1, 2, 3]) mapping = qiime2.Artifact.import_data(Mapping, {'foo': '42'}) r = dummy_plugin.actions.pipelines_in_pipeline(ints, mapping) right_p_dir = r.right._archiver.provenance_dir with (right_p_dir / 'action' / 'action.yaml').open() as fh: right_yaml = fh.read() self.assertIn('type: pipeline', right_yaml) self.assertIn('action: pipelines_in_pipeline', right_yaml) self.assertIn('int_sequence: %s' % ints.uuid, right_yaml) match = re.search(r'alias\-of: ([a-zA-Z0-9\-]+)$', right_yaml, flags=re.MULTILINE) first_alias = match.group(1) with (right_p_dir / 'artifacts' / first_alias / 'action' / 'action.yaml').open() as fh: first_alias_yaml = fh.read() # Should be the same input self.assertIn('type: pipeline', first_alias_yaml) self.assertIn('int_sequence: %s' % ints.uuid, first_alias_yaml) self.assertIn('action: typical_pipeline', first_alias_yaml) match = re.search(r'alias\-of: ([a-zA-Z0-9\-]+)$', first_alias_yaml, flags=re.MULTILINE) second_alias = match.group(1) with (right_p_dir / 'artifacts' / second_alias / 'action' / 'action.yaml').open() as fh: actual_method_yaml = fh.read() self.assertIn('type: method', actual_method_yaml) self.assertIn('ints: %s' % ints.uuid, actual_method_yaml) self.assertIn('action: split_ints', actual_method_yaml) def test_unioned_primitives(self): r = dummy_plugin.actions.unioned_primitives(3, 2) prov_dir = r.out._archiver.provenance_dir with (prov_dir / 'action' / 'action.yaml').open() as fh: prov_yml = fh.read() self.assertIn('foo: 3', prov_yml) self.assertIn('bar: 2', prov_yml) @mock.patch('qiime2.core.archive.provenance.tzlocal.get_localzone', side_effect=ValueError()) def test_ts_to_date(self, mocked_tzlocal): q2_paper_date = 1563984000 obs = str(provenance._ts_to_date(q2_paper_date)) exp = "2019-07-24 16:00:00+00:00" self.assertEqual(obs, exp) self.assertTrue(mocked_tzlocal.called) def test_prov_rename(self): viz, = dummy_plugin.actions.no_input_viz() viz_p_dir = viz._archiver.provenance_dir self.assertTrue(viz_p_dir.exists()) @mock.patch('qiime2.core.path.ProvenancePath.rename', side_effect=FileExistsError) def test_prov_rename_file_exists(self, _): viz, = dummy_plugin.actions.no_input_viz() viz_p_dir = viz._archiver.provenance_dir with (viz_p_dir / 'action' / 'action.yaml').open() as fh: self.assertIn('output-name: visualization', fh.read()) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/cite.py000066400000000000000000000045251412141660100164660ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import pkg_resources import collections import bibtexparser as bp CitationRecord = collections.namedtuple('CitationRecord', ['type', 'fields']) class Citations(collections.OrderedDict): @classmethod def load(cls, path, package=None): if package is not None: root = pkg_resources.resource_filename(package, '.') root = os.path.abspath(root) path = os.path.join(root, path) parser = bp.bparser.BibTexParser() # Downstream tooling is much easier with unicode. For actual latex # users, use the modern biber backend instead of bibtex parser.customization = bp.customization.convert_to_unicode with open(path) as fh: try: db = bp.load(fh, parser=parser) except Exception as e: raise ValueError("There was a problem loading the BiBTex file:" "%r" % path) from e entries = collections.OrderedDict() for entry in db.entries: id_ = entry.pop('ID') type_ = entry.pop('ENTRYTYPE') if id_ in entries: raise ValueError("Duplicate entry-key found in BibTex file: %r" % id_) entries[id_] = CitationRecord(type_, entry) return cls(entries) def __iter__(self): return iter(self.values()) def save(self, f): entries = [] for key, citation in self.items(): entry = citation.fields.copy() entry['ID'] = key entry['ENTRYTYPE'] = citation.type entries.append(entry) db = bp.bibdatabase.BibDatabase() db.entries = entries writer = bp.bwriter.BibTexWriter() writer.order_entries_by = tuple(self.keys()) owned = False if type(f) is str: f = open(f, 'w') owned = True try: bp.dump(db, f, writer=writer) finally: if owned: f.close() qiime-2021.8.0/qiime2/core/exceptions.py000066400000000000000000000006731412141660100177230ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- class ValidationError(Exception): pass class ImplementationError(Exception): pass qiime-2021.8.0/qiime2/core/format.py000066400000000000000000000021441412141660100170250ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import qiime2.core.path as qpath class FormatBase: def __init__(self, path=None, mode='w'): import qiime2.plugin.model as model if path is None: if mode != 'w': raise ValueError("A path must be provided when reading.") else: if mode != 'r': raise ValueError("A path must be omitted when writing.") if mode == 'w': self.path = qpath.OutPath( # TODO: parents shouldn't know about their children dir=isinstance(self, model.DirectoryFormat), prefix='q2-%s-' % self.__class__.__name__) else: self.path = qpath.InPath(path) self._mode = mode def __str__(self): return str(self.path) qiime-2021.8.0/qiime2/core/path.py000066400000000000000000000113771412141660100165010ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import pathlib import shutil import distutils import tempfile import weakref _ConcretePath = type(pathlib.Path()) def _party_parrot(self, *args): raise TypeError("Cannot mutate %r." % self) class OwnedPath(_ConcretePath): def __new__(cls, *args, **kwargs): self = super().__new__(cls, *args, **kwargs) self._user_owned = True return self def _copy_dir_or_file(self, other): if self.is_dir(): return distutils.dir_util.copy_tree(str(self), str(other)) else: return shutil.copy(str(self), str(other)) def _destruct(self): if self.is_dir(): distutils.dir_util.remove_tree(str(self)) else: self.unlink() def _move_or_copy(self, other): if self._user_owned: return self._copy_dir_or_file(other) else: # Certain networked filesystems will experience a race # condition on `rename`, so fall back to copying. try: return _ConcretePath.rename(self, other) except FileExistsError: copied = self._copy_dir_or_file(other) self._destruct() return copied class InPath(OwnedPath): def __new__(cls, path): self = super().__new__(cls, path) self.__backing_path = path if hasattr(path, '_user_owned'): self._user_owned = path._user_owned return self chmod = lchmod = rename = replace = rmdir = symlink_to = touch = unlink = \ write_bytes = write_text = _party_parrot def open(self, mode='r', buffering=-1, encoding=None, errors=None, newline=None): if 'w' in mode or '+' in mode or 'a' in mode: _party_parrot(self) return super().open(mode=mode, buffering=buffering, encoding=encoding, errors=errors, newline=newline) class OutPath(OwnedPath): @classmethod def _destruct(cls, path): if not os.path.exists(path): return if os.path.isdir(path): shutil.rmtree(path) else: os.unlink(path) def __new__(cls, dir=False, **kwargs): """ Create a tempfile, return pathlib.Path reference to it. """ if dir: name = tempfile.mkdtemp(**kwargs) else: fd, name = tempfile.mkstemp(**kwargs) # fd is now assigned to our process table, but we don't need to do # anything with the file. We will call `open` on the `name` later # producing a different file descriptor, so close this one to # prevent a resource leak. os.close(fd) obj = super().__new__(cls, name) obj._destructor = weakref.finalize(obj, cls._destruct, str(obj)) return obj def __exit__(self, t, v, tb): self._destructor() super().__exit__(t, v, tb) class InternalDirectory(_ConcretePath): DEFAULT_PREFIX = 'qiime2-' @classmethod def _destruct(cls, path): """DO NOT USE DIRECTLY, use `_destructor()` instead""" if os.path.exists(path): shutil.rmtree(path) @classmethod def __new(cls, *args): self = super().__new__(cls, *args) self._destructor = weakref.finalize(self, self._destruct, str(self)) return self def __new__(cls, *args, prefix=None): if args and prefix is not None: raise TypeError("Cannot pass a path and a prefix at the same time") elif args: # This happens when the base-class's __reduce__ method is invoked # for pickling. return cls.__new(*args) else: if prefix is None: prefix = cls.DEFAULT_PREFIX elif not prefix.startswith(cls.DEFAULT_PREFIX): prefix = cls.DEFAULT_PREFIX + prefix # TODO: normalize when temp-directories are configurable path = tempfile.mkdtemp(prefix=prefix) return cls.__new(path) def __truediv__(self, path): # We don't want to create self-destructing paths when using the join # operator return _ConcretePath(str(self), path) def __rtruediv__(self, path): # Same reasoning as truediv return _ConcretePath(path, str(self)) class ArchivePath(InternalDirectory): DEFAULT_PREFIX = 'qiime2-archive-' class ProvenancePath(InternalDirectory): DEFAULT_PREFIX = 'qiime2-provenance-' qiime-2021.8.0/qiime2/core/testing/000077500000000000000000000000001412141660100166375ustar00rootroot00000000000000qiime-2021.8.0/qiime2/core/testing/__init__.py000066400000000000000000000005351412141660100207530ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- qiime-2021.8.0/qiime2/core/testing/citations.bib000066400000000000000000000052451412141660100213200ustar00rootroot00000000000000@article{unger1998does, title={Does knuckle cracking lead to arthritis of the fingers?}, author={Unger, Donald L}, journal={Arthritis \& Rheumatology}, volume={41}, number={5}, pages={949--950}, year={1998}, publisher={Wiley Online Library} } @article{berry1997flying, title={Of flying frogs and levitrons}, author={Berry, Michael Victor and Geim, Andre Konstantin}, journal={European Journal of Physics}, volume={18}, number={4}, pages={307}, year={1997}, publisher={IOP Publishing} } @article{mayer2012walking, title={Walking with coffee: Why does it spill?}, author={Mayer, Hans C and Krechetnikov, Rouslan}, journal={Physical Review E}, volume={85}, number={4}, pages={046117}, year={2012}, publisher={APS} } @article{baerheim1994effect, title={Effect of ale, garlic, and soured cream on the appetite of leeches}, author={Baerheim, Anders and Sandvik, Hogne}, journal={BMJ}, volume={309}, number={6970}, pages={1689}, year={1994}, publisher={British Medical Journal Publishing Group} } @article{witcombe2006sword, title={Sword swallowing and its side effects}, author={Witcombe, Brian and Meyer, Dan}, journal={BMJ}, volume={333}, number={7582}, pages={1285--1287}, year={2006}, publisher={British Medical Journal Publishing Group} } @article{reimers2012response, title={Response behaviors of Svalbard reindeer towards humans and humans disguised as polar bears on Edge{\o}ya}, author={Reimers, Eigil and Eftest{\o}l, Sindre}, journal={Arctic, antarctic, and alpine research}, volume={44}, number={4}, pages={483--489}, year={2012}, publisher={BioOne} } @article{barbeito1967microbiological, title={Microbiological laboratory hazard of bearded men}, author={Barbeito, Manuel S and Mathews, Charles T and Taylor, Larry A}, journal={Applied microbiology}, volume={15}, number={4}, pages={899--906}, year={1967}, publisher={Am Soc Microbiol} } @article{krauth2012depth, title={An in-depth analysis of a piece of shit: distribution of Schistosoma mansoni and hookworm eggs in human stool}, author={Krauth, Stefanie J and Coulibaly, Jean T and Knopp, Stefanie and Traor{\'e}, Mahamadou and N'Goran, Eli{\'e}zer K and Utzinger, J{\"u}rg}, journal={PLoS neglected tropical diseases}, volume={6}, number={12}, pages={e1969}, year={2012}, publisher={Public Library of Science} } @article{silvers1997effects, title={The effects of pre-existing inappropriate highlighting on reading comprehension}, author={Silvers, Vicki L and Kreiner, David S}, journal={Literacy Research and Instruction}, volume={36}, number={3}, pages={217--223}, year={1997}, publisher={Taylor \& Francis} } qiime-2021.8.0/qiime2/core/testing/examples.py000066400000000000000000000172411412141660100210340ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import pandas as pd from qiime2 import Artifact, Metadata from .type import IntSequence1, IntSequence2, Mapping, SingleInt def ints1_factory(): return Artifact.import_data(IntSequence1, [0, 1, 2]) def ints2_factory(): return Artifact.import_data(IntSequence1, [3, 4, 5]) def ints3_factory(): return Artifact.import_data(IntSequence2, [6, 7, 8]) def mapping1_factory(): return Artifact.import_data(Mapping, {'a': 42}) def md1_factory(): return Metadata(pd.DataFrame({'a': ['1', '2', '3']}, index=pd.Index(['0', '1', '2'], name='id'))) def md2_factory(): return Metadata(pd.DataFrame({'b': ['4', '5', '6']}, index=pd.Index(['0', '1', '2'], name='id'))) def single_int1_factory(): return Artifact.import_data(SingleInt, 10) def single_int2_factory(): return Artifact.import_data(SingleInt, 11) def concatenate_ints_simple(use): ints_a = use.init_data('ints_a', ints1_factory) ints_b = use.init_data('ints_b', ints2_factory) ints_c = use.init_data('ints_c', ints3_factory) use.comment('This example demonstrates basic usage.') use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='concatenate_ints'), use.UsageInputs(ints1=ints_a, ints2=ints_b, ints3=ints_c, int1=4, int2=2), use.UsageOutputNames(concatenated_ints='ints_d'), ) def concatenate_ints_complex(use): ints_a = use.init_data('ints_a', ints1_factory) ints_b = use.init_data('ints_b', ints2_factory) ints_c = use.init_data('ints_c', ints3_factory) use.comment('This example demonstrates chained usage (pt 1).') use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='concatenate_ints'), use.UsageInputs(ints1=ints_a, ints2=ints_b, ints3=ints_c, int1=4, int2=2), use.UsageOutputNames(concatenated_ints='ints_d'), ) ints_d = use.get_result('ints_d') use.comment('This example demonstrates chained usage (pt 2).') use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='concatenate_ints'), use.UsageInputs(ints1=ints_d, ints2=ints_b, ints3=ints_c, int1=41, int2=0), use.UsageOutputNames(concatenated_ints='concatenated_ints'), ) def typical_pipeline_simple(use): ints = use.init_data('ints', ints1_factory) mapper = use.init_data('mapper', mapping1_factory) use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='typical_pipeline'), use.UsageInputs(int_sequence=ints, mapping=mapper, do_extra_thing=True), use.UsageOutputNames(out_map='out_map', left='left', right='right', left_viz='left_viz', right_viz='right_viz') ) def typical_pipeline_complex(use): ints1 = use.init_data('ints1', ints1_factory) mapper1 = use.init_data('mapper1', mapping1_factory) use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='typical_pipeline'), use.UsageInputs(int_sequence=ints1, mapping=mapper1, do_extra_thing=True), use.UsageOutputNames(out_map='out_map1', left='left1', right='right1', left_viz='left_viz1', right_viz='right_viz1') ) ints2 = use.get_result('left1') mapper2 = use.get_result('out_map1') use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='typical_pipeline'), use.UsageInputs(int_sequence=ints2, mapping=mapper2, do_extra_thing=False), use.UsageOutputNames(out_map='out_map2', left='left2', right='right2', left_viz='left_viz2', right_viz='right_viz2') ) right2 = use.get_result('right2') right2.assert_has_line_matching( label='a nice label about this assertion', path='ints.txt', expression='1', ) def comments_only(use): use.comment('comment 1') use.comment('comment 2') def comments_only_factory(): def comments_only_closure(use): use.comment('comment 1') use.comment('comment 2') return comments_only_closure def identity_with_metadata_simple(use): ints = use.init_data('ints', ints1_factory) md = use.init_metadata('md', md1_factory) use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='identity_with_metadata'), use.UsageInputs(ints=ints, metadata=md), use.UsageOutputNames(out='out'), ) def identity_with_metadata_merging(use): ints = use.init_data('ints', ints1_factory) md1 = use.init_metadata('md1', md1_factory) md2 = use.init_metadata('md2', md2_factory) md3 = use.merge_metadata('md3', md1, md2) use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='identity_with_metadata'), use.UsageInputs(ints=ints, metadata=md3), use.UsageOutputNames(out='out'), ) def identity_with_metadata_column_get_mdc(use): ints = use.init_data('ints', ints1_factory) md = use.init_metadata('md', md1_factory) mdc = use.get_metadata_column('a', md) use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='identity_with_metadata_column'), use.UsageInputs(ints=ints, metadata=mdc), use.UsageOutputNames(out='out'), ) def variadic_input_simple(use): ints_a = use.init_data('ints_a', ints1_factory) ints_b = use.init_data('ints_b', ints2_factory) ints = use.init_data_collection('ints', list, ints_a, ints_b) single_int1 = use.init_data('single_int1', single_int1_factory) single_int2 = use.init_data('single_int2', single_int2_factory) int_set = use.init_data_collection('int_set', set, single_int1, single_int2) use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='variadic_input_method'), use.UsageInputs(ints=ints, int_set=int_set, nums={7, 8, 9}), use.UsageOutputNames(output='out'), ) def optional_inputs(use): ints_a = use.init_data('ints', ints1_factory) use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='optional_artifacts_method'), use.UsageInputs(ints=ints_a, num1=1), use.UsageOutputNames(output='output'), ) use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='optional_artifacts_method'), use.UsageInputs(ints=ints_a, num1=1, num2=2), use.UsageOutputNames(output='output'), ) use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='optional_artifacts_method'), use.UsageInputs(ints=ints_a, num1=1, num2=None), use.UsageOutputNames(output='ints_b'), ) ints_b = use.get_result('ints_b') use.action( use.UsageAction(plugin_id='dummy_plugin', action_id='optional_artifacts_method'), use.UsageInputs(ints=ints_a, optional1=ints_b, num1=3, num2=4), use.UsageOutputNames(output='output'), ) qiime-2021.8.0/qiime2/core/testing/format.py000066400000000000000000000122461412141660100205060ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from qiime2.plugin import TextFileFormat, ValidationError import qiime2.plugin.model as model class IntSequenceFormat(TextFileFormat): """ A sequence of integers stored on new lines in a file. Since this is a sequence, the integers have an order and repetition of elements is allowed. Sequential values must have an inter-value distance other than 3 to be valid. """ def _validate_n_ints(self, n): with self.open() as fh: last_val = None for idx, line in enumerate(fh, 1): if n is not None and idx >= n: break try: val = int(line.rstrip('\n')) except (TypeError, ValueError): raise ValidationError("Line %d is not an integer." % idx) if last_val is not None and last_val + 3 == val: raise ValidationError("Line %d is 3 more than line %d" % (idx, idx-1)) last_val = val def _validate_(self, level): record_map = {'min': 5, 'max': None} self._validate_n_ints(record_map[level]) class IntSequenceFormatV2(IntSequenceFormat): """ Same as IntSequenceFormat, but has a header "VERSION 2" """ def _validate_(self, level): with self.open() as fh: if fh.readline() != 'VERSION 2\n': raise ValidationError("Missing header: VERSION 2") class MappingFormat(TextFileFormat): """ A mapping of keys to values stored in a TSV file. Since this is a mapping, key-value pairs do not have an order and duplicate keys are disallowed. """ def _validate_(self, level): with self.open() as fh: for line, idx in zip(fh, range(1, 6)): cells = line.rstrip('\n').split('\t') if len(cells) != 2: raise ValidationError("Line %d does not have exactly 2 " "elements seperated by a tab." % idx) class SingleIntFormat(TextFileFormat): """ Exactly one int on a single line in the file. """ def _validate_(self, level): with self.open() as fh: try: int(fh.readline().rstrip('\n')) except (TypeError, ValueError): raise ValidationError("File does not contain an integer") if fh.readline(): raise ValidationError("Too many lines in file.") IntSequenceDirectoryFormat = model.SingleFileDirectoryFormat( 'IntSequenceDirectoryFormat', 'ints.txt', IntSequenceFormat) IntSequenceV2DirectoryFormat = model.SingleFileDirectoryFormat( 'IntSequenceV2DirectoryFormat', 'integers.txt', IntSequenceFormatV2) class IntSequenceMultiFileDirectoryFormat(model.DirectoryFormat): pass # This could have been a `SingleFileDirectoryFormat`, but isn't for testing # purposes class MappingDirectoryFormat(model.DirectoryFormat): mapping = model.File('mapping.tsv', format=MappingFormat) class FourIntsDirectoryFormat(model.DirectoryFormat): """ A sequence of exactly four integers stored across multiple files, some of which are in a nested directory. Each file contains a single integer. Since this is a sequence, the integers have an order (corresponding to filename) and repetition of elements is allowed. """ single_ints = model.FileCollection( r'file[1-2]\.txt|nested/file[3-4]\.txt', format=SingleIntFormat) @single_ints.set_path_maker def single_ints_path_maker(self, num): if not 0 < num < 5: raise ValueError("`num` must be 1-4, not %r." % num) if num > 2: return 'nested/file%d.txt' % num else: return 'file%d.txt' % num class RedundantSingleIntDirectoryFormat(model.DirectoryFormat): """ Two files of SingleIntFormat which are exactly the same. """ int1 = model.File('file1.txt', format=SingleIntFormat) int2 = model.File('file2.txt', format=SingleIntFormat) def _validate_(self, level): if self.int1.view(int) != self.int2.view(int): raise ValidationError("file1.txt does not match file2.txt") class UnimportableFormat(TextFileFormat): """ Unimportable format used for testing. """ UnimportableDirectoryFormat = model.SingleFileDirectoryFormat( 'UnimportableDirectoryFormat', 'ints.txt', UnimportableFormat) class EchoFormat(TextFileFormat): def _validate_(self, level): pass # Anything is a valid echo file EchoDirectoryFormat = model.SingleFileDirectoryFormat( 'EchoDirectoryFormat', 'echo.txt', EchoFormat) class Cephalapod(TextFileFormat): """ Class that inherits from text file format. Used for testing validator sorting. """ CephalapodDirectoryFormat = model.SingleFileDirectoryFormat( 'CephalapodDirectoryFormat', 'squids.tsv', Cephalapod) qiime-2021.8.0/qiime2/core/testing/mapped.py000066400000000000000000000066571412141660100204750ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os from .plugin import dummy_plugin, C1, C2, C3, Foo, Bar, Baz, EchoFormat from qiime2.plugin import ( TypeMap, TypeMatch, Properties, Visualization, Bool, Choices) def constrained_input_visualization(output_dir: str, a: EchoFormat, b: EchoFormat): with open(os.path.join(output_dir, 'index.html'), 'w') as fh: fh.write("

%s

" % a.path.read_text()) fh.write("

%s

" % b.path.read_text()) T, U, V = TypeMap({ (Foo, Foo): Visualization, (Bar, Bar): Visualization, (Baz, Baz): Visualization, (C1[Foo], C1[Foo]): Visualization, (C1[Bar], C1[Bar]): Visualization, (C1[Baz], C1[Baz]): Visualization }) dummy_plugin.visualizers.register_function( function=constrained_input_visualization, inputs={ 'a': T, 'b': U }, parameters={}, name="Constrained Input Visualization", description="Ensure Foo/Bar/Baz match" ) del T, U, V def combinatorically_mapped_method(a: EchoFormat, b: EchoFormat ) -> (EchoFormat, EchoFormat): return a, b T, R = TypeMap({ Foo: Bar, Bar: Baz, Baz: Foo }) X, Y = TypeMap({ C3[Foo | Bar | Baz, Foo | Bar | Baz, Foo]: Foo, C3[Foo | Bar | Baz, Foo | Bar | Baz, Bar]: Bar, C3[Foo | Bar | Baz, Foo | Bar | Baz, Baz]: Baz }) dummy_plugin.methods.register_function( function=combinatorically_mapped_method, inputs={ 'a': C1[T], 'b': X }, parameters={}, outputs=[ ('x', C2[R, R]), ('y', Y) ], name="Combinatorically Mapped Method", description="Test that multiple typemaps can be used" ) del T, R, X, Y def double_bound_variable_method(a: EchoFormat, b: EchoFormat, extra: EchoFormat) -> EchoFormat: return extra T, R = TypeMap({ Foo: Bar, Bar: Baz, Baz: Foo }) dummy_plugin.methods.register_function( function=double_bound_variable_method, inputs={ 'a': T, 'b': T, 'extra': Foo }, parameters={}, outputs=[ ('x', R) ], name="Double Bound Variable Method", description="Test reuse of variables" ) del T, R def bool_flag_swaps_output_method(a: EchoFormat, b: bool) -> EchoFormat: return a P, R = TypeMap({ Choices(True): C1[Foo], Choices(False): Foo }) dummy_plugin.methods.register_function( function=bool_flag_swaps_output_method, inputs={ 'a': Bar }, parameters={ 'b': Bool % P }, outputs=[ ('x', R) ], name='Bool Flag Swaps Output Method', description='Test if a parameter can change output' ) del P, R def predicates_preserved_method(a: EchoFormat) -> EchoFormat: return a P = TypeMatch([Properties('A'), Properties('B'), Properties('C'), Properties('X', 'Y')]) dummy_plugin.methods.register_function( function=predicates_preserved_method, inputs={ 'a': Foo % P }, parameters={}, outputs=[ ('x', Foo % P) ], name='Predicates Preserved Method', description='Test that predicates are preserved' ) del P qiime-2021.8.0/qiime2/core/testing/method.py000066400000000000000000000077601412141660100205030ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import qiime2 # Artifacts and parameters. def concatenate_ints(ints1: list, ints2: list, ints3: list, int1: int, int2: int) -> list: return ints1 + ints2 + ints3 + [int1] + [int2] # Multiple output artifacts. def split_ints(ints: list) -> (list, list): middle = int(len(ints) / 2) left = ints[:middle] right = ints[middle:] return left, right # No parameters, only artifacts. def merge_mappings(mapping1: dict, mapping2: dict) -> dict: merged = mapping1.copy() for key, value in mapping2.items(): if key in merged and merged[key] != value: raise ValueError( "Key %r exists in `mapping1` and `mapping2` with conflicting " "values: %r != %r" % (key, merged[key], value)) merged[key] = value return merged # No input artifacts, only parameters. def params_only_method(name: str, age: int) -> dict: return {name: age} # Unioned primitives def unioned_primitives(foo: int, bar: str = 'auto_bar') -> dict: return {'foo': foo, 'bar': bar} # No input artifacts or parameters. def no_input_method() -> dict: return {'foo': 42} def deprecated_method() -> dict: return {'foo': 43} def long_description_method(mapping1: dict, name: str, age: int) -> dict: return {name: age} def docstring_order_method(req_input: dict, req_param: str, opt_input: dict = None, opt_param: int = None) -> dict: return {req_param: opt_param} def identity_with_metadata(ints: list, metadata: qiime2.Metadata) -> list: assert isinstance(metadata, qiime2.Metadata) return ints # TODO unit tests (test_method.py) for 3 variations of MetadataColumn methods # below def identity_with_metadata_column(ints: list, metadata: qiime2.MetadataColumn) -> list: assert isinstance(metadata, (qiime2.CategoricalMetadataColumn, qiime2.NumericMetadataColumn)) return ints def identity_with_categorical_metadata_column( ints: list, metadata: qiime2.CategoricalMetadataColumn) -> list: assert isinstance(metadata, qiime2.CategoricalMetadataColumn) return ints def identity_with_numeric_metadata_column( ints: list, metadata: qiime2.NumericMetadataColumn) -> list: assert isinstance(metadata, qiime2.NumericMetadataColumn) return ints def identity_with_optional_metadata(ints: list, metadata: qiime2.Metadata = None) -> list: assert isinstance(metadata, (qiime2.Metadata, type(None))) return ints def identity_with_optional_metadata_column( ints: list, metadata: qiime2.MetadataColumn = None) -> list: assert isinstance(metadata, (qiime2.CategoricalMetadataColumn, qiime2.NumericMetadataColumn, type(None))) return ints def optional_artifacts_method(ints: list, num1: int, optional1: list = None, optional2: list = None, num2: int = None) -> list: result = ints + [num1] if optional1 is not None: result += optional1 if optional2 is not None: result += optional2 if num2 is not None: result += [num2] return result def variadic_input_method(ints: list, int_set: int, nums: int, opt_nums: int = None) -> list: results = [] for int_list in ints: results += int_list results += sorted(int_set) results += nums if opt_nums: results += opt_nums return results def type_match_list_and_set(ints: list, strs1: list, strs2: set) -> list: return [0] qiime-2021.8.0/qiime2/core/testing/pipeline.py000066400000000000000000000075321412141660100210250ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from .type import SingleInt, Mapping def parameter_only_pipeline(ctx, int1, int2=2, metadata=None): identity_with_optional_metadata = ctx.get_action( 'dummy_plugin', 'identity_with_optional_metadata') concatenate_ints = ctx.get_action('dummy_plugin', 'concatenate_ints') ints1 = ctx.make_artifact('IntSequence2', [int1, int2, 3]) ints2, = identity_with_optional_metadata(ints1, metadata) ints3, = identity_with_optional_metadata(ints1, metadata) more_ints, = concatenate_ints(ints3, ints2, ints1, int1=int1, int2=int2) return ints1, more_ints def typical_pipeline(ctx, int_sequence, mapping, do_extra_thing, add=1): split_ints = ctx.get_action('dummy_plugin', 'split_ints') most_common_viz = ctx.get_action('dummy_plugin', 'most_common_viz') left, right = split_ints(int_sequence) if do_extra_thing: left = ctx.make_artifact( 'IntSequence1', [i + add for i in left.view(list)]) val, = mapping.view(dict).values() # Some kind of runtime failure if val != '42': raise ValueError("Bad mapping") left_viz, = most_common_viz(left) right_viz, = most_common_viz(right) return mapping, left, right, left_viz, right_viz def optional_artifact_pipeline(ctx, int_sequence, single_int=None): optional_artifact_method = ctx.get_action( 'dummy_plugin', 'optional_artifacts_method') if single_int is None: # not a nested pipeline, just sharing the ctx object single_int = pointless_pipeline(ctx) num1 = single_int.view(int) ints, = optional_artifact_method(int_sequence, num1) return ints def visualizer_only_pipeline(ctx, mapping): no_input_viz = ctx.get_action('dummy_plugin', 'no_input_viz') mapping_viz = ctx.get_action('dummy_plugin', 'mapping_viz') viz1, = no_input_viz() viz2, = mapping_viz(mapping, mapping, 'foo', 'bar') return viz1, viz2 def pipelines_in_pipeline(ctx, int_sequence, mapping): pointless_pipeline = ctx.get_action('dummy_plugin', 'pointless_pipeline') typical_pipeline = ctx.get_action('dummy_plugin', 'typical_pipeline') visualizer_only_pipeline = ctx.get_action( 'dummy_plugin', 'visualizer_only_pipeline') results = [] results += pointless_pipeline() typical_results = typical_pipeline(int_sequence, mapping, True) results += typical_results results += visualizer_only_pipeline(typical_results[0]) return tuple(results) def pointless_pipeline(ctx): # Use a real type expression instead of a string. return ctx.make_artifact(SingleInt, 4) def failing_pipeline(ctx, int_sequence, break_from='arity'): merge_mappings = ctx.get_action('dummy_plugin', 'merge_mappings') list_ = int_sequence.view(list) if list_: integer = list_[0] else: integer = 0 # Made here so that we can make sure it gets cleaned up wrong_output = ctx.make_artifact(SingleInt, integer) if break_from == 'arity': return int_sequence, int_sequence, int_sequence elif break_from == 'return-view': return None elif break_from == 'type': return wrong_output elif break_from == 'method': a = ctx.make_artifact(Mapping, {'foo': 'a'}) b = ctx.make_artifact(Mapping, {'foo': 'b'}) # has the same key merge_mappings(a, b) elif break_from == 'no-plugin': ctx.get_action('not%a$plugin', 'foo') elif break_from == 'no-action': ctx.get_action('dummy_plugin', 'not%a$method') else: raise ValueError('this never works') qiime-2021.8.0/qiime2/core/testing/plugin.py000066400000000000000000000520751412141660100205200ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from importlib import import_module from qiime2.plugin import (Plugin, Bool, Int, Str, Choices, Range, List, Set, Visualization, Metadata, MetadataColumn, Categorical, Numeric, TypeMatch) from .format import ( IntSequenceFormat, IntSequenceFormatV2, IntSequenceMultiFileDirectoryFormat, MappingFormat, SingleIntFormat, IntSequenceDirectoryFormat, IntSequenceV2DirectoryFormat, MappingDirectoryFormat, FourIntsDirectoryFormat, RedundantSingleIntDirectoryFormat, UnimportableFormat, UnimportableDirectoryFormat, EchoFormat, EchoDirectoryFormat, Cephalapod, CephalapodDirectoryFormat, ) from .type import (IntSequence1, IntSequence2, IntSequence3, Mapping, FourInts, SingleInt, Kennel, Dog, Cat, C1, C2, C3, Foo, Bar, Baz, AscIntSequence, Squid, Octopus, Cuttlefish) from .method import (concatenate_ints, split_ints, merge_mappings, identity_with_metadata, identity_with_metadata_column, identity_with_categorical_metadata_column, identity_with_numeric_metadata_column, identity_with_optional_metadata, identity_with_optional_metadata_column, params_only_method, no_input_method, deprecated_method, optional_artifacts_method, long_description_method, docstring_order_method, variadic_input_method, unioned_primitives, type_match_list_and_set) from .visualizer import (most_common_viz, mapping_viz, params_only_viz, no_input_viz) from .pipeline import (parameter_only_pipeline, typical_pipeline, optional_artifact_pipeline, visualizer_only_pipeline, pipelines_in_pipeline, pointless_pipeline, failing_pipeline) from ..cite import Citations from .examples import (concatenate_ints_simple, concatenate_ints_complex, typical_pipeline_simple, typical_pipeline_complex, comments_only, identity_with_metadata_simple, identity_with_metadata_merging, identity_with_metadata_column_get_mdc, variadic_input_simple, optional_inputs, comments_only_factory, ) citations = Citations.load('citations.bib', package='qiime2.core.testing') dummy_plugin = Plugin( name='dummy-plugin', description='Description of dummy plugin.', short_description='Dummy plugin for testing.', version='0.0.0-dev', website='https://github.com/qiime2/qiime2', package='qiime2.core.testing', user_support_text='For help, see https://qiime2.org', citations=[citations['unger1998does'], citations['berry1997flying']] ) import_module('qiime2.core.testing.transformer') import_module('qiime2.core.testing.validator') # Register semantic types dummy_plugin.register_semantic_types(IntSequence1, IntSequence2, IntSequence3, Mapping, FourInts, Kennel, Dog, Cat, SingleInt, C1, C2, C3, Foo, Bar, Baz, AscIntSequence, Squid, Octopus, Cuttlefish) # Register formats dummy_plugin.register_formats( IntSequenceFormatV2, MappingFormat, IntSequenceV2DirectoryFormat, IntSequenceMultiFileDirectoryFormat, MappingDirectoryFormat, EchoDirectoryFormat, EchoFormat, Cephalapod, CephalapodDirectoryFormat) dummy_plugin.register_formats( FourIntsDirectoryFormat, UnimportableDirectoryFormat, UnimportableFormat, citations=[citations['baerheim1994effect']]) dummy_plugin.register_views( int, IntSequenceFormat, IntSequenceDirectoryFormat, SingleIntFormat, RedundantSingleIntDirectoryFormat, citations=[citations['mayer2012walking']]) dummy_plugin.register_semantic_type_to_format( IntSequence1, artifact_format=IntSequenceDirectoryFormat ) dummy_plugin.register_semantic_type_to_format( IntSequence2, artifact_format=IntSequenceV2DirectoryFormat ) dummy_plugin.register_semantic_type_to_format( IntSequence3, artifact_format=IntSequenceMultiFileDirectoryFormat ) dummy_plugin.register_semantic_type_to_format( Mapping, artifact_format=MappingDirectoryFormat ) dummy_plugin.register_semantic_type_to_format( FourInts, artifact_format=FourIntsDirectoryFormat ) dummy_plugin.register_semantic_type_to_format( SingleInt, artifact_format=RedundantSingleIntDirectoryFormat ) dummy_plugin.register_semantic_type_to_format( Kennel[Dog | Cat], artifact_format=MappingDirectoryFormat ) dummy_plugin.register_semantic_type_to_format( C3[C1[Foo | Bar | Baz] | Foo | Bar | Baz, C1[Foo | Bar | Baz] | Foo | Bar | Baz, C1[Foo | Bar | Baz] | Foo | Bar | Baz] | C2[Foo | Bar | Baz, Foo | Bar | Baz] | C1[Foo | Bar | Baz | C2[Foo | Bar | Baz, Foo | Bar | Baz]] | Foo | Bar | Baz, artifact_format=EchoDirectoryFormat) dummy_plugin.register_semantic_type_to_format( AscIntSequence, artifact_format=IntSequenceDirectoryFormat) dummy_plugin.register_semantic_type_to_format( Squid | Octopus | Cuttlefish, artifact_format=CephalapodDirectoryFormat) # TODO add an optional parameter to this method when they are supported dummy_plugin.methods.register_function( function=concatenate_ints, inputs={ 'ints1': IntSequence1 | IntSequence2, 'ints2': IntSequence1, 'ints3': IntSequence2 }, parameters={ 'int1': Int, 'int2': Int }, outputs=[ ('concatenated_ints', IntSequence1) ], name='Concatenate integers', description='This method concatenates integers into' ' a single sequence in the order they are provided.', citations=[citations['baerheim1994effect']], examples={'concatenate_ints_simple': concatenate_ints_simple, 'concatenate_ints_complex': concatenate_ints_complex, 'comments_only': comments_only, # execute factory to make a closure to test pickling 'comments_only_factory': comments_only_factory(), }, ) T = TypeMatch([IntSequence1, IntSequence2]) dummy_plugin.methods.register_function( function=split_ints, inputs={ 'ints': T }, parameters={}, outputs=[ ('left', T), ('right', T) ], name='Split sequence of integers in half', description='This method splits a sequence of integers in half, returning ' 'the two halves (left and right). If the input sequence\'s ' 'length is not evenly divisible by 2, the right half will ' 'have one more element than the left.', citations=[ citations['witcombe2006sword'], citations['reimers2012response']] ) dummy_plugin.methods.register_function( function=merge_mappings, inputs={ 'mapping1': Mapping, 'mapping2': Mapping }, input_descriptions={ 'mapping1': 'Mapping object to be merged' }, parameters={}, outputs=[ ('merged_mapping', Mapping) ], output_descriptions={ 'merged_mapping': 'Resulting merged Mapping object'}, name='Merge mappings', description='This method merges two mappings into a single new mapping. ' 'If a key is shared between mappings and the values differ, ' 'an error will be raised.' ) dummy_plugin.methods.register_function( function=identity_with_metadata, inputs={ 'ints': IntSequence1 | IntSequence2 }, parameters={ 'metadata': Metadata }, outputs=[ ('out', IntSequence1) ], name='Identity', description='This method does nothing, but takes metadata', examples={ 'identity_with_metadata_simple': identity_with_metadata_simple, 'identity_with_metadata_merging': identity_with_metadata_merging}, ) dummy_plugin.methods.register_function( function=long_description_method, inputs={ 'mapping1': Mapping }, input_descriptions={ 'mapping1': ("This is a very long description. If asked about its " "length, I would have to say it is greater than 79 " "characters.") }, parameters={ 'name': Str, 'age': Int }, parameter_descriptions={ 'name': ("This is a very long description. If asked about its length," " I would have to say it is greater than 79 characters.") }, outputs=[ ('out', Mapping) ], output_descriptions={ 'out': ("This is a very long description. If asked about its length," " I would have to say it is greater than 79 characters.") }, name="Long Description", description=("This is a very long description. If asked about its length," " I would have to say it is greater than 79 characters.") ) dummy_plugin.methods.register_function( function=docstring_order_method, inputs={ 'req_input': Mapping, 'opt_input': Mapping }, input_descriptions={ 'req_input': "This should show up first.", 'opt_input': "This should show up third." }, parameters={ 'req_param': Str, 'opt_param': Int }, parameter_descriptions={ 'req_param': "This should show up second.", 'opt_param': "This should show up fourth." }, outputs=[ ('out', Mapping) ], output_descriptions={ 'out': "This should show up last, in it's own section." }, name="Docstring Order", description=("Tests whether inputs and parameters are rendered in " "signature order") ) dummy_plugin.methods.register_function( function=identity_with_metadata_column, inputs={ 'ints': IntSequence1 | IntSequence2 }, parameters={ 'metadata': MetadataColumn[Categorical | Numeric] }, outputs=[ ('out', IntSequence1) ], name='Identity', description='This method does nothing, ' 'but takes a generic metadata column', examples={ 'identity_with_metadata_column_get_mdc': identity_with_metadata_column_get_mdc, }, ) dummy_plugin.methods.register_function( function=identity_with_categorical_metadata_column, inputs={ 'ints': IntSequence1 | IntSequence2 }, parameters={ 'metadata': MetadataColumn[Categorical] }, outputs=[ ('out', IntSequence1) ], name='Identity', description='This method does nothing, but takes a categorical metadata ' 'column' ) dummy_plugin.methods.register_function( function=identity_with_numeric_metadata_column, inputs={ 'ints': IntSequence1 | IntSequence2 }, parameters={ 'metadata': MetadataColumn[Numeric] }, outputs=[ ('out', IntSequence1) ], name='Identity', description='This method does nothing, but takes a numeric metadata column' ) dummy_plugin.methods.register_function( function=identity_with_optional_metadata, inputs={ 'ints': IntSequence1 | IntSequence2 }, parameters={ 'metadata': Metadata }, outputs=[ ('out', IntSequence1) ], name='Identity', description='This method does nothing, but takes optional metadata' ) dummy_plugin.methods.register_function( function=identity_with_optional_metadata_column, inputs={ 'ints': IntSequence1 | IntSequence2 }, parameters={ 'metadata': MetadataColumn[Numeric | Categorical] }, outputs=[ ('out', IntSequence1) ], name='Identity', description='This method does nothing, but takes an optional generic ' 'metadata column' ) dummy_plugin.methods.register_function( function=params_only_method, inputs={}, parameters={ 'name': Str, 'age': Int }, outputs=[ ('out', Mapping) ], name='Parameters only method', description='This method only accepts parameters.', ) dummy_plugin.methods.register_function( function=unioned_primitives, inputs={}, parameters={ 'foo': Int % Range(1, None) | Str % Choices(['auto_foo']), 'bar': Int % Range(1, None) | Str % Choices(['auto_bar']), }, outputs=[ ('out', Mapping) ], name='Unioned primitive parameter', description='This method has a unioned primitive parameter' ) dummy_plugin.methods.register_function( function=no_input_method, inputs={}, parameters={}, outputs=[ ('out', Mapping) ], name='No input method', description='This method does not accept any type of input.' ) dummy_plugin.methods.register_function( function=deprecated_method, inputs={}, parameters={}, outputs=[ ('out', Mapping) ], name='A deprecated method', description='This deprecated method does not accept any type of input.', deprecated=True, ) dummy_plugin.methods.register_function( function=optional_artifacts_method, inputs={ 'ints': IntSequence1, 'optional1': IntSequence1, 'optional2': IntSequence1 | IntSequence2 }, parameters={ 'num1': Int, 'num2': Int }, outputs=[ ('output', IntSequence1) ], name='Optional artifacts method', description='This method declares optional artifacts and concatenates ' 'whatever integers are supplied as input.', examples={'optional_inputs': optional_inputs}, ) dummy_plugin.methods.register_function( function=variadic_input_method, inputs={ 'ints': List[IntSequence1 | IntSequence2], 'int_set': Set[SingleInt] }, parameters={ 'nums': Set[Int], 'opt_nums': List[Int % Range(10, 20)] }, outputs=[ ('output', IntSequence1) ], name='Test variadic inputs', description='This method concatenates all of its variadic inputs', input_descriptions={ 'ints': 'A list of int artifacts', 'int_set': 'A set of int artifacts' }, parameter_descriptions={ 'nums': 'A set of ints', 'opt_nums': 'An optional list of ints' }, output_descriptions={ 'output': 'All of the above mashed together' }, examples={'variadic_input_simple': variadic_input_simple}, ) T = TypeMatch([IntSequence1, IntSequence2]) dummy_plugin.methods.register_function( function=type_match_list_and_set, inputs={ 'ints': T }, parameters={ 'strs1': List[Str], 'strs2': Set[Str] }, outputs=[ ('output', T) ], name='TypeMatch with list and set params', description='Just a method with a TypeMatch and list/set params', input_descriptions={ 'ints': 'An int artifact' }, parameter_descriptions={ 'strs1': 'A list of strings', 'strs2': 'A set of strings' }, output_descriptions={ 'output': '[0]' } ) dummy_plugin.visualizers.register_function( function=params_only_viz, inputs={}, parameters={ 'name': Str, 'age': Int % Range(0, None) }, name='Parameters only viz', description='This visualizer only accepts parameters.' ) dummy_plugin.visualizers.register_function( function=no_input_viz, inputs={}, parameters={}, name='No input viz', description='This visualizer does not accept any type of input.' ) dummy_plugin.visualizers.register_function( function=most_common_viz, inputs={ 'ints': IntSequence1 | IntSequence2 }, parameters={}, name='Visualize most common integers', description='This visualizer produces HTML and TSV outputs containing the ' 'input sequence of integers ordered from most- to ' 'least-frequently occurring, along with their respective ' 'frequencies.', citations=[citations['barbeito1967microbiological']] ) # TODO add optional parameters to this method when they are supported dummy_plugin.visualizers.register_function( function=mapping_viz, inputs={ 'mapping1': Mapping, 'mapping2': Mapping }, parameters={ 'key_label': Str, 'value_label': Str }, name='Visualize two mappings', description='This visualizer produces an HTML visualization of two ' 'key-value mappings, each sorted in alphabetical order by key.' ) dummy_plugin.pipelines.register_function( function=parameter_only_pipeline, inputs={}, parameters={ 'int1': Int, 'int2': Int, 'metadata': Metadata }, outputs=[ ('foo', IntSequence2), ('bar', IntSequence1) ], name='Do multiple things', description='This pipeline only accepts parameters', parameter_descriptions={ 'int1': 'An integer, the first one in fact', 'int2': 'An integer, the second one', 'metadata': 'Very little is done with this' }, output_descriptions={ 'foo': 'Foo - "The Integers of 2"', 'bar': 'Bar - "What a sequences"' }, ) dummy_plugin.pipelines.register_function( function=typical_pipeline, inputs={ 'int_sequence': IntSequence1, 'mapping': Mapping }, parameters={ 'do_extra_thing': Bool, 'add': Int }, outputs=[ ('out_map', Mapping), ('left', IntSequence1), ('right', IntSequence1), ('left_viz', Visualization), ('right_viz', Visualization) ], input_descriptions={ 'int_sequence': 'A sequence of ints', 'mapping': 'A map to a number other than 42 will fail' }, parameter_descriptions={ 'do_extra_thing': 'Increment `left` by `add` if true', 'add': 'Unused if `do_extra_thing` is false' }, output_descriptions={ 'out_map': 'Same as input', 'left': 'Left side of `int_sequence` unless `do_extra_thing`', 'right': 'Right side of `int_sequence`', 'left_viz': '`left` visualized', 'right_viz': '`right` visualized' }, name='A typical pipeline with the potential to raise an error', description='Waste some time shuffling data around for no reason', citations=citations, # ALL of them. examples={'typical_pipeline_simple': typical_pipeline_simple, 'typical_pipeline_complex': typical_pipeline_complex}, ) dummy_plugin.pipelines.register_function( function=optional_artifact_pipeline, inputs={ 'int_sequence': IntSequence1, 'single_int': SingleInt }, parameters={}, outputs=[ ('ints', IntSequence1) ], input_descriptions={ 'int_sequence': 'Some integers', 'single_int': 'An integer' }, output_descriptions={ 'ints': 'More integers' }, name='Do stuff normally, but override this one step sometimes', description='Creates its own single_int, unless provided' ) dummy_plugin.pipelines.register_function( function=visualizer_only_pipeline, inputs={ 'mapping': Mapping }, parameters={}, outputs=[ ('viz1', Visualization), ('viz2', Visualization) ], input_descriptions={ 'mapping': 'A mapping to look at twice' }, output_descriptions={ 'viz1': 'The no input viz', 'viz2': 'Our `mapping` seen through the lense of "foo" *and* "bar"' }, name='Visualize many things', description='Looks at both nothing and a mapping' ) dummy_plugin.pipelines.register_function( function=pipelines_in_pipeline, inputs={ 'int_sequence': IntSequence1, 'mapping': Mapping }, parameters={}, outputs=[ ('int1', SingleInt), ('out_map', Mapping), ('left', IntSequence1), ('right', IntSequence1), ('left_viz', Visualization), ('right_viz', Visualization), ('viz1', Visualization), ('viz2', Visualization) ], name='Do a great many things', description=('Mapping is chained from typical_pipeline into ' 'visualizer_only_pipeline') ) dummy_plugin.pipelines.register_function( function=pointless_pipeline, inputs={}, parameters={}, outputs=[('random_int', SingleInt)], name='Get an integer', description='Integer was chosen to be 4 by a random dice roll' ) dummy_plugin.pipelines.register_function( function=failing_pipeline, inputs={ 'int_sequence': IntSequence1 }, parameters={ 'break_from': Str % Choices( {'arity', 'return-view', 'type', 'method', 'internal', 'no-plugin', 'no-action'}) }, outputs=[('mapping', Mapping)], name='Test different ways of failing', description=('This is useful to make sure all of the intermediate stuff is' ' cleaned up the way it should be.') ) import_module('qiime2.core.testing.mapped') qiime-2021.8.0/qiime2/core/testing/tests/000077500000000000000000000000001412141660100200015ustar00rootroot00000000000000qiime-2021.8.0/qiime2/core/testing/tests/__init__.py000066400000000000000000000005351412141660100221150ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- qiime-2021.8.0/qiime2/core/testing/tests/test_mapped_actions.py000066400000000000000000000165171412141660100244120ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest from qiime2 import Artifact from qiime2.core.testing.util import get_dummy_plugin from ..type import IntSequence1, IntSequence2 class ActionTester(unittest.TestCase): ACTION = 'N/A' def setUp(self): plugin = get_dummy_plugin() self.action = plugin.actions[self.ACTION] def run_action(self, **inputs): results = self.action(**inputs) future = self.action.asynchronous(**inputs) async_results = future.result() for a, b in zip(async_results, results): self.assertEqual(a.type, b.type) return results class TestConstrainedInputVisualization(ActionTester): ACTION = 'constrained_input_visualization' def test_match_foo(self): a = Artifact.import_data('Foo', "element 1", view_type=str) b = Artifact.import_data('Foo', "element 2", view_type=str) viz, = self.run_action(a=a, b=b) contents = (viz._archiver.data_dir / 'index.html').read_text() self.assertIn('element 1', contents) self.assertIn('element 2', contents) def test_match_nested(self): a = Artifact.import_data('C1[Baz]', "element 1", view_type=str) b = Artifact.import_data('C1[Baz]', "element 2", view_type=str) viz, = self.run_action(a=a, b=b) contents = (viz._archiver.data_dir / 'index.html').read_text() self.assertIn('element 1', contents) self.assertIn('element 2', contents) def test_mismatch_foo_bar(self): a = Artifact.import_data('Foo', "element 1", view_type=str) b = Artifact.import_data('Bar', "element 2", view_type=str) with self.assertRaisesRegex(ValueError, 'No solution.*Foo'): viz, = self.run_action(a=a, b=b) def test_mismatch_nested(self): a = Artifact.import_data('C1[Foo]', "element 1", view_type=str) b = Artifact.import_data('Foo', "element 2", view_type=str) with self.assertRaisesRegex(ValueError, 'No solution.*C1'): viz, = self.run_action(a=a, b=b) class TestCombinatoricallyMappedMethod(ActionTester): ACTION = 'combinatorically_mapped_method' def test_match_foo(self): a = Artifact.import_data('C1[Foo]', 'element 1', view_type=str) b = Artifact.import_data('C3[Foo, Foo, Foo]', 'element 2', view_type=str) x, y = self.run_action(a=a, b=b) self.assertEqual(repr(x.type), 'C2[Bar, Bar]') self.assertEqual(repr(y.type), 'Foo') def test_match_bar_foo(self): a = Artifact.import_data('C1[Bar]', 'element 1', view_type=str) b = Artifact.import_data('C3[Foo, Foo, Foo]', 'element 2', view_type=str) x, y = self.run_action(a=a, b=b) self.assertEqual(repr(x.type), 'C2[Baz, Baz]') self.assertEqual(repr(y.type), 'Foo') def test_match_baz_misc(self): a = Artifact.import_data('C1[Baz]', 'element 1', view_type=str) b = Artifact.import_data('C3[Foo, Bar, Baz]', 'element 2', view_type=str) x, y = self.run_action(a=a, b=b) self.assertEqual(repr(x.type), 'C2[Foo, Foo]') self.assertEqual(repr(y.type), 'Baz') def test_mismatch(self): a = Artifact.import_data('Bar', 'element 1', view_type=str) b = Artifact.import_data('C3[Foo, Foo, Foo]', 'element 2', view_type=str) with self.assertRaises(TypeError): self.run_action(a=a, b=b) class TestDoubleBoundVariableMethod(ActionTester): ACTION = 'double_bound_variable_method' def test_predicate_on_second(self): a = Artifact.import_data('Bar', 'element 1', view_type=str) b = Artifact.import_data('Bar % Properties("A")', 'element 2', view_type=str) extra = Artifact.import_data('Foo', 'always foo', view_type=str) x, = self.run_action(a=a, b=b, extra=extra) self.assertEqual(repr(x.type), 'Baz') def test_mismatch(self): a = Artifact.import_data('Foo', 'element 1', view_type=str) b = Artifact.import_data('Bar', 'element 2', view_type=str) extra = Artifact.import_data('Foo', 'always foo', view_type=str) with self.assertRaisesRegex(ValueError, 'match.*same output'): self.run_action(a=a, b=b, extra=extra) class TestBoolFlagSwapsOutputMethod(ActionTester): ACTION = 'bool_flag_swaps_output_method' def test_true(self): a = Artifact.import_data('Bar', 'element', view_type=str) x, = self.run_action(a=a, b=True) self.assertEqual(repr(x.type), 'C1[Foo]') def test_false(self): a = Artifact.import_data('Bar', 'element', view_type=str) x, = self.run_action(a=a, b=False) self.assertEqual(repr(x.type), 'Foo') class TestPredicatesPreservedMethod(ActionTester): ACTION = 'predicates_preserved_method' def test_simple(self): a = Artifact.import_data("Foo % Properties('A')", 'element 1', view_type=str) x, = self.run_action(a=a) self.assertEqual(repr(x.type), "Foo % Properties('A')") def test_mismatch(self): a = Artifact.import_data("Foo % Properties('X')", 'element 1', view_type=str) with self.assertRaises(TypeError): self.run_action(a=a) def test_combinations_preserved(self): a = Artifact.import_data("Foo % Properties('A', 'B')", 'element 1', view_type=str) x, = self.run_action(a=a) self.assertEqual(repr(x.type), "Foo % Properties('A', 'B')") def test_extra_dropped(self): a = Artifact.import_data("Foo % Properties('Extra', 'A', 'B')", 'element 1', view_type=str) x, = self.run_action(a=a) self.assertEqual(repr(x.type), "Foo % Properties('A', 'B')") class TestTypeMatchWithListAndSet(ActionTester): ACTION = 'type_match_list_and_set' def test_intsequence1(self): a = Artifact.import_data('IntSequence1', [1]) x = self.run_action(ints=a, strs1=['a'], strs2={'a'}) self.assertEqual(x.output.type, IntSequence1) def test_intsequence2(self): a = Artifact.import_data('IntSequence2', [1]) x = self.run_action(ints=a, strs1=['a'], strs2={'a'}) self.assertEqual(x.output.type, IntSequence2) class TestUnionedPrimitiveDecode(ActionTester): ACTION = 'unioned_primitives' def test_decode_int(self): exp = dict(foo=1, bar=1) res = self.action.signature.decode_parameters(foo='1', bar='1') self.assertEqual(res, exp) def test_decode_str(self): exp = dict(foo='auto_foo', bar='auto_bar') res = self.action.signature.decode_parameters(**exp) self.assertEqual(res, exp) def test_decode_mix(self): exp = dict(foo=1, bar='auto_bar') res = self.action.signature.decode_parameters(foo='1', bar='auto_bar') self.assertEqual(res, exp) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/testing/transformer.py000066400000000000000000000126371412141660100215640ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import collections from qiime2 import Metadata import pandas as pd from .format import ( FourIntsDirectoryFormat, MappingDirectoryFormat, IntSequenceFormat, IntSequenceFormatV2, IntSequenceDirectoryFormat, IntSequenceV2DirectoryFormat, IntSequenceMultiFileDirectoryFormat, SingleIntFormat, MappingFormat, UnimportableFormat, RedundantSingleIntDirectoryFormat, EchoFormat ) from .plugin import dummy_plugin, citations @dummy_plugin.register_transformer def _2(data: int) -> SingleIntFormat: ff = SingleIntFormat() with ff.open() as fh: fh.write('%d\n' % data) return ff @dummy_plugin.register_transformer def _5(ff: SingleIntFormat) -> int: with ff.open() as fh: return int(fh.read()) @dummy_plugin.register_transformer(citations=[citations['krauth2012depth']]) def _7(data: list) -> IntSequenceFormat: ff = IntSequenceFormat() with ff.open() as fh: for int_ in data: fh.write('%d\n' % int_) return ff @dummy_plugin.register_transformer(citations=citations) def _77(data: list) -> IntSequenceFormatV2: ff = IntSequenceFormatV2() with ff.open() as fh: fh.write('VERSION 2\n') for int_ in data: fh.write('%d\n' % int_) return ff @dummy_plugin.register_transformer def _9(ff: IntSequenceFormat) -> list: with ff.open() as fh: return list(map(int, fh.readlines())) @dummy_plugin.register_transformer def _99(ff: IntSequenceFormatV2) -> list: with ff.open() as fh: fh.readline() # skip header return list(map(int, fh.readlines())) @dummy_plugin.register_transformer def _10(ff: IntSequenceFormat) -> collections.Counter: with ff.open() as fh: return collections.Counter(map(int, fh.readlines())) @dummy_plugin.register_transformer def _1010(ff: IntSequenceFormatV2) -> collections.Counter: with ff.open() as fh: fh.readline() # skip header return collections.Counter(map(int, fh.readlines())) @dummy_plugin.register_transformer def _1000(ff: IntSequenceFormat) -> IntSequenceFormatV2: new_ff = IntSequenceFormatV2() with new_ff.open() as new_fh, ff.open() as fh: new_fh.write("VERSION 2\n") for line in fh: new_fh.write(line) return new_ff # This only exists to test `get_formats` and is functionally useless otherwise @dummy_plugin.register_transformer def _1100(df: IntSequenceMultiFileDirectoryFormat) -> \ IntSequenceDirectoryFormat: return IntSequenceDirectoryFormat() @dummy_plugin.register_transformer def _1001(df: IntSequenceV2DirectoryFormat) -> \ IntSequenceMultiFileDirectoryFormat: return IntSequenceMultiFileDirectoryFormat() @dummy_plugin.register_transformer def _0202(data: int) -> RedundantSingleIntDirectoryFormat: df = RedundantSingleIntDirectoryFormat() df.int1.write_data(data, int) df.int2.write_data(data, int) return df @dummy_plugin.register_transformer def _2020(ff: RedundantSingleIntDirectoryFormat) -> int: return ff.int1.view(int) # int2 must be the same for this format @dummy_plugin.register_transformer def _11(data: dict) -> MappingDirectoryFormat: df = MappingDirectoryFormat() df.mapping.write_data(data, dict) return df @dummy_plugin.register_transformer def _12(data: dict) -> MappingFormat: ff = MappingFormat() with ff.open() as fh: for key, value in data.items(): fh.write('%s\t%s\n' % (key, value)) return ff @dummy_plugin.register_transformer(citations=[citations['silvers1997effects']]) def _13(df: MappingDirectoryFormat) -> dict: # If this had been a `SingleFileDirectoryFormat` then this entire # transformer would have been redundant (the framework could infer it). return df.mapping.view(dict) @dummy_plugin.register_transformer def _14(ff: MappingFormat) -> dict: data = {} with ff.open() as fh: for line in fh: key, value = line.rstrip('\n').split('\t') if key in data: raise ValueError( "mapping.txt file must have unique keys. Key %r was " "observed more than once." % key) data[key] = value return data @dummy_plugin.register_transformer def _15(df: MappingDirectoryFormat) -> Metadata: d = df.mapping.view(dict) return Metadata(pd.DataFrame(d, index=pd.Index(["0"], name='id'))) @dummy_plugin.register_transformer def _3(df: FourIntsDirectoryFormat) -> list: # Note: most uses of `iter_views` will need to look at the first element # of the series of tuples provided by iter_views return [x for _, x in df.single_ints.iter_views(int)] @dummy_plugin.register_transformer def _1(data: list) -> FourIntsDirectoryFormat: df = FourIntsDirectoryFormat() for i, int_ in enumerate(data, 1): df.single_ints.write_data(int_, int, num=i) return df @dummy_plugin.register_transformer def _4(ff: UnimportableFormat) -> int: return 1 @dummy_plugin.register_transformer def _a1(data: str) -> EchoFormat: ff = EchoFormat() with ff.open() as fh: fh.write(data) return ff qiime-2021.8.0/qiime2/core/testing/type.py000066400000000000000000000037651412141660100202050ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import qiime2.plugin as plugin IntSequence1 = plugin.SemanticType('IntSequence1') IntSequence2 = plugin.SemanticType('IntSequence2') IntSequence3 = plugin.SemanticType('IntSequence3') AscIntSequence = plugin.SemanticType('AscIntSequence') Mapping = plugin.SemanticType('Mapping') FourInts = plugin.SemanticType('FourInts') SingleInt = plugin.SemanticType('SingleInt') Kennel = plugin.SemanticType('Kennel', field_names='pet') Dog = plugin.SemanticType('Dog', variant_of=Kennel.field['pet']) Cat = plugin.SemanticType('Cat', variant_of=Kennel.field['pet']) # Kennel[Dog | Cat] C1 = plugin.SemanticType('C1', field_names='first') C2 = plugin.SemanticType('C2', field_names=['first', 'second'], variant_of=C1.field['first'], field_members={'first': [C1], 'second': [C1]}) C3 = plugin.SemanticType('C3', field_names=['first', 'second', 'third'], variant_of=[C1.field['first'], C2.field['first'], C2.field['second']], field_members={'first': [C1, C2], 'second': [C1, C2], 'third': [C1, C2]}) _variants = [ C1.field['first'], C2.field['first'], C3.field['first'], C2.field['second'], C3.field['second'], C3.field['third'] ] # C1[C2[C3[Foo, Bar, Baz], C1[Foo]]] ... etc Foo = plugin.SemanticType('Foo', variant_of=_variants) Bar = plugin.SemanticType('Bar', variant_of=_variants) Baz = plugin.SemanticType('Baz', variant_of=_variants) Squid = plugin.SemanticType('Squid') Octopus = plugin.SemanticType('Octopus') Cuttlefish = plugin.SemanticType('Cuttlefish') qiime-2021.8.0/qiime2/core/testing/util.py000066400000000000000000000102601412141660100201650ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import os.path import zipfile import qiime2.sdk def get_dummy_plugin(): plugin_manager = qiime2.sdk.PluginManager() if 'dummy-plugin' not in plugin_manager.plugins: raise RuntimeError( "When running QIIME 2 unit tests, the QIIMETEST environment " "variable must be defined so that plugins required by unit tests " "are loaded. The value of the QIIMETEST environment variable can " "be anything. Example command: QIIMETEST=1 nosetests") return plugin_manager.plugins['dummy-plugin'] class ArchiveTestingMixin: """Mixin for testing properties of archives created by Archiver.""" def assertArchiveMembers(self, archive_filepath, root_dir, expected): """Assert members are in an archive. Parameters ---------- archive_filepath : str or Path Filepath to archive whose members will be verified against the `expected` members. root_dir : str or Path Root directory of the archive. Will be prepended to the member paths in `expected`. This is useful when the archive's root directory is not known ahead of time (e.g. when it is a random UUID) and the caller is determining the root directory dynamically. expected : set of str Set of expected archive members stored as paths relative to `root_dir`. """ archive_filepath = str(archive_filepath) root_dir = str(root_dir) with zipfile.ZipFile(archive_filepath, mode='r') as zf: observed = set(zf.namelist()) # Path separator '/' is hardcoded because paths in the zipfile will # always use this separator. expected = {root_dir + '/' + member for member in expected} self.assertEqual(observed, expected) def assertExtractedArchiveMembers(self, extract_dir, root_dir, expected): """Assert an archive's members are extracted to a directory. Parameters ---------- extract_dir : str or Path Path to directory the archive was extracted to. root_dir : str or Path Root directory of the archive that was extracted to `extract_dir`. This is useful when the archive's root directory is not known ahead of time (e.g. when it is a random UUID) and the caller is determining the root directory dynamically. expected : set of str Set of expected archive members extracted to `extract_dir`. Stored as paths relative to `root_dir`. """ extract_dir = str(extract_dir) root_dir = str(root_dir) observed = set() for root, _, filenames in os.walk(extract_dir): for filename in filenames: observed.add(os.path.join(root, filename)) expected = {os.path.join(extract_dir, root_dir, member) for member in expected} self.assertEqual(observed, expected) class ReallyEqualMixin: """Mixin for testing implementations of __eq__/__ne__. Based on this public domain code (also explains why the mixin is useful): https://ludios.org/testing-your-eq-ne-cmp/ """ def assertReallyEqual(self, a, b): # assertEqual first, because it will have a good message if the # assertion fails. self.assertEqual(a, b) self.assertEqual(b, a) self.assertTrue(a == b) self.assertTrue(b == a) self.assertFalse(a != b) self.assertFalse(b != a) def assertReallyNotEqual(self, a, b): # assertNotEqual first, because it will have a good message if the # assertion fails. self.assertNotEqual(a, b) self.assertNotEqual(b, a) self.assertFalse(a == b) self.assertFalse(b == a) self.assertTrue(a != b) self.assertTrue(b != a) qiime-2021.8.0/qiime2/core/testing/validator.py000066400000000000000000000030771412141660100212050ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from qiime2 import Metadata from qiime2.plugin import ValidationError from .type import (Kennel, Dog, Cat, AscIntSequence, Squid, Octopus, Cuttlefish) from .format import Cephalapod from .plugin import dummy_plugin @dummy_plugin.register_validator(Kennel[Dog | Cat]) def validator_example_null1(data: dict, level): pass @dummy_plugin.register_validator(Kennel[Dog]) def validator_example_null2(data: Metadata, level): pass @dummy_plugin.register_validator(AscIntSequence) def validate_ascending_seq(data: list, level): # landmine for testing if data == [2021, 8, 24]: raise KeyError prev = float('-inf') for number in data: if not number > prev: raise ValidationError("%s is not greater than %s" % (number, prev)) @dummy_plugin.register_validator(Squid | Cuttlefish) def validator_sort_middle_b(data: Cephalapod, level): pass @dummy_plugin.register_validator(Squid) def validator_sort_last(data: Cephalapod, level): pass @dummy_plugin.register_validator(Squid | Octopus | Cuttlefish) def validator_sort_first(data: Cephalapod, level): pass @dummy_plugin.register_validator(Squid | Octopus) def validator_sort_middle(data: Cephalapod, level): pass qiime-2021.8.0/qiime2/core/testing/visualizer.py000066400000000000000000000066411412141660100214150ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import collections import os import os.path import pandas as pd # Multiple types of visualizations (index.html, index.tsv). def most_common_viz(output_dir: str, ints: collections.Counter) -> None: df = pd.DataFrame(ints.most_common(), columns=["Integer", "Frequency"]) with open(os.path.join(output_dir, 'index.html'), 'w') as fh: fh.write('\n') fh.write('

Most common integers:

\n') fh.write(df.to_html(index=False)) fh.write('') with open(os.path.join(output_dir, 'index.tsv'), 'w') as fh: fh.write(df.to_csv(sep='\t', index=False)) # Multiple html files (index.1.html, index.2.html) def multi_html_viz(output_dir: str, ints: list) -> None: ints = [str(i) for i in ints] with open(os.path.join(output_dir, 'index.1.html'), 'w') as fh: fh.write('\n') fh.write(' '.join(ints)) fh.write('') with open(os.path.join(output_dir, 'index.2.html'), 'w') as fh: fh.write('\n') fh.write(' '.join(reversed(ints))) fh.write('') # No input artifacts, only parameters. def params_only_viz(output_dir: str, name: str = 'Foo Bar', age: int = 42): with open(os.path.join(output_dir, 'index.html'), 'w') as fh: fh.write('\n') fh.write('Name: %s\n' % name) fh.write('Age: %s\n' % age) fh.write('') # No input artifacts or parameters. def no_input_viz(output_dir: str): with open(os.path.join(output_dir, 'index.html'), 'w') as fh: fh.write('\n') fh.write('Hello, World!\n') fh.write('') # Multiple input artifacts and parameters, and a nested directory with required # resources for rendering. def mapping_viz(output_dir: str, mapping1: dict, mapping2: dict, key_label: str, value_label: str) -> None: df1 = _dict_to_dataframe(mapping1, key_label, value_label) df2 = _dict_to_dataframe(mapping2, key_label, value_label) with open(os.path.join(output_dir, 'index.html'), 'w') as fh: fh.write('') fh.write('') fh.write('\n') fh.write('\n') fh.write('

mapping1:

\n') fh.write(df1.to_html(index=False, classes='dummy-class')) fh.write('

mapping2:

\n') fh.write(df2.to_html(index=False, classes='dummy-class')) fh.write('') css_dir = os.path.join(output_dir, 'css') os.mkdir(css_dir) with open(os.path.join(css_dir, 'style.css'), 'w') as fh: fh.write(_css) def _dict_to_dataframe(dict_, key_label, value_label): return pd.DataFrame(sorted(dict_.items()), columns=[key_label, value_label]) # Example table styling taken from http://www.w3schools.com/css/css_table.asp _css = """ .dummy-class { border-collapse: collapse; width: 100%; } .dummy-class th, td { padding: 8px; text-align: left; border-bottom: 1px solid #ddd; } """ qiime-2021.8.0/qiime2/core/tests/000077500000000000000000000000001412141660100163245ustar00rootroot00000000000000qiime-2021.8.0/qiime2/core/tests/__init__.py000066400000000000000000000005351412141660100204400ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- qiime-2021.8.0/qiime2/core/tests/test_path.py000066400000000000000000000065161412141660100207010ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import pathlib import shutil import tempfile import unittest from qiime2.core.path import OwnedPath, OutPath class TestOwnedPath(unittest.TestCase): def setUp(self): self.from_dir = tempfile.mkdtemp() (pathlib.Path(self.from_dir) / 'foo.txt').touch() self.to_dir = tempfile.mkdtemp() # assume to_dir is empty for all tests def test_move_or_copy_owned(self): d = OwnedPath(self.from_dir) # ensure that we are owned d._user_owned = True d._move_or_copy(self.to_dir) # since from_dir is owned, _move_or_copy should copy, not move self.assertTrue(os.path.exists(os.path.join(self.from_dir, 'foo.txt'))) self.assertTrue(os.path.exists(os.path.join(self.to_dir, 'foo.txt'))) shutil.rmtree(self.from_dir) shutil.rmtree(self.to_dir) def test_move_or_copy_not_owned_rename(self): d = OwnedPath(self.from_dir) # ensure that we are not owned d._user_owned = False d._move_or_copy(self.to_dir) # since from_dir is not owned, _move_or_copy should move, not copy self.assertFalse(os.path.exists(os.path.join(self.from_dir, 'foo.txt'))) self.assertTrue(os.path.exists(os.path.join(self.to_dir, 'foo.txt'))) with self.assertRaises(FileNotFoundError): shutil.rmtree(self.from_dir) shutil.rmtree(self.to_dir) @unittest.mock.patch('pathlib.Path.rename', side_effect=FileExistsError) def test_move_or_copy_not_owned_copy(self, _): d = OwnedPath(self.from_dir) # ensure that we are not owned d._user_owned = False d._move_or_copy(self.to_dir) # since from_dir is not owned, but the network fs race condition crops # up, _move_or_copy should copy, not move, but then we still ensure # that the original path has been cleaned up self.assertFalse(os.path.exists(os.path.join(self.from_dir, 'foo.txt'))) self.assertTrue(os.path.exists(os.path.join(self.to_dir, 'foo.txt'))) with self.assertRaises(FileNotFoundError): shutil.rmtree(self.from_dir) shutil.rmtree(self.to_dir) class TestOutPath(unittest.TestCase): def test_new_outpath(self): f = OutPath() self.assertIsInstance(f, OutPath) self.assertTrue(f.is_file()) g = OutPath(dir=True) self.assertIsInstance(g, OutPath) self.assertTrue(g.is_dir()) def test_new_outpath_context_mgr(self): with OutPath() as f: path = str(f) self.assertIsInstance(f, OutPath) self.assertTrue(os.path.isfile(path)) self.assertFalse(os.path.isfile(path)) def test_destructor(self): f = OutPath() path = str(f) self.assertTrue(os.path.isfile(path)) f._destructor() self.assertFalse(os.path.isfile(path)) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/tests/test_util.py000066400000000000000000000341611412141660100207170ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest import tempfile import pathlib import collections import datetime import dateutil.relativedelta as relativedelta import qiime2.core.util as util from qiime2.core.testing.type import Foo, Bar, Baz class TestFindDuplicates(unittest.TestCase): # NOTE: wrapping each input in `iter()` because that is the interface # expected by `find_duplicates`, and avoids the need to test other iterable # types, e.g. list, tuples, generators, etc. def test_empty_iterable(self): obs = util.find_duplicates(iter([])) self.assertEqual(obs, set()) def test_single_value(self): obs = util.find_duplicates(iter(['foo'])) self.assertEqual(obs, set()) def test_multiple_values_no_duplicates(self): obs = util.find_duplicates(iter(['foo', 'bar'])) self.assertEqual(obs, set()) def test_one_duplicate(self): obs = util.find_duplicates(iter(['foo', 'bar', 'foo'])) self.assertEqual(obs, {'foo'}) def test_multiple_duplicates(self): obs = util.find_duplicates( iter(['foo', 'bar', 'foo', 'baz', 'foo', 'bar'])) self.assertEqual(obs, {'foo', 'bar'}) def test_all_duplicates(self): obs = util.find_duplicates( iter(['foo', 'bar', 'baz', 'baz', 'bar', 'foo'])) self.assertEqual(obs, {'foo', 'bar', 'baz'}) def test_different_hashables(self): iterable = iter(['foo', 42, -9.999, 'baz', ('a', 'b'), 42, 'foo', ('a', 'b', 'c'), ('a', 'b')]) obs = util.find_duplicates(iterable) self.assertEqual(obs, {'foo', 42, ('a', 'b')}) class TestDurationTime(unittest.TestCase): def test_time_travel(self): start = datetime.datetime(1987, 10, 27, 1, 21, 2, 50) end = datetime.datetime(1985, 10, 26, 1, 21, 0, 0) reldelta = relativedelta.relativedelta(end, start) self.assertEqual( util.duration_time(reldelta), '-2 years, -1 days, -3 seconds, and 999950 microseconds') def test_no_duration(self): time = datetime.datetime(1985, 10, 26, 1, 21, 0) reldelta = relativedelta.relativedelta(time, time) self.assertEqual(util.duration_time(reldelta), '0 microseconds') def test_singular(self): start = datetime.datetime(1985, 10, 26, 1, 21, 0, 0) end = datetime.datetime(1986, 11, 27, 2, 22, 1, 1) reldelta = relativedelta.relativedelta(end, start) self.assertEqual( util.duration_time(reldelta), '1 year, 1 month, 1 day, 1 hour, 1 minute, 1 second,' ' and 1 microsecond') def test_plural(self): start = datetime.datetime(1985, 10, 26, 1, 21, 0, 0) end = datetime.datetime(1987, 12, 28, 3, 23, 2, 2) reldelta = relativedelta.relativedelta(end, start) self.assertEqual( util.duration_time(reldelta), '2 years, 2 months, 2 days, 2 hours, 2 minutes, 2 seconds,' ' and 2 microseconds') def test_missing(self): start = datetime.datetime(1985, 10, 26, 1, 21, 0, 0) end = datetime.datetime(1987, 10, 27, 1, 21, 2, 50) reldelta = relativedelta.relativedelta(end, start) self.assertEqual( util.duration_time(reldelta), '2 years, 1 day, 2 seconds, and 50 microseconds') def test_unusually_round_number(self): start = datetime.datetime(1985, 10, 26, 1, 21, 0, 0) end = datetime.datetime(1985, 10, 27, 1, 21, 0, 0) reldelta = relativedelta.relativedelta(end, start) self.assertEqual( util.duration_time(reldelta), '1 day') def test_microseconds(self): start = datetime.datetime(1985, 10, 26, 1, 21, 0, 0) end = datetime.datetime(1985, 10, 26, 1, 21, 0, 1955) reldelta = relativedelta.relativedelta(end, start) self.assertEqual( util.duration_time(reldelta), '1955 microseconds') class TestMD5Sum(unittest.TestCase): # All expected results where generated via GNU coreutils md5sum def setUp(self): self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-') self.test_path = pathlib.Path(self.test_dir.name) def tearDown(self): self.test_dir.cleanup() def make_file(self, bytes_): path = self.test_path / 'file' with path.open(mode='wb') as fh: fh.write(bytes_) return path def test_empty_file(self): self.assertEqual(util.md5sum(self.make_file(b'')), 'd41d8cd98f00b204e9800998ecf8427e') def test_single_byte_file(self): self.assertEqual(util.md5sum(self.make_file(b'a')), '0cc175b9c0f1b6a831c399e269772661') def test_large_file(self): path = self.make_file(b'verybigfile' * (1024 * 50)) self.assertEqual(util.md5sum(path), '27d64211ee283283ad866c18afa26611') def test_can_use_string(self): string_path = str(self.make_file(b'Normal text\nand things\n')) self.assertEqual(util.md5sum(string_path), '93b048d0202e4b06b658f3aef1e764d3') class TestMD5SumDirectory(unittest.TestCase): # All expected results where generated via GNU coreutils md5sum def setUp(self): self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-') self.test_path = pathlib.Path(self.test_dir.name) def tearDown(self): self.test_dir.cleanup() def make_file(self, bytes_, relpath): path = self.test_path / relpath with path.open(mode='wb') as fh: fh.write(bytes_) return path def test_empty_directory(self): self.assertEqual(util.md5sum_directory(self.test_path), collections.OrderedDict()) def test_nested_empty_directories(self): (self.test_path / 'foo').mkdir() (self.test_path / 'foo' / 'bar').mkdir() (self.test_path / 'baz').mkdir() self.assertEqual(util.md5sum_directory(self.test_path), collections.OrderedDict()) def test_single_file(self): self.make_file(b'Normal text\nand things\n', 'foobarbaz.txt') self.assertEqual( util.md5sum_directory(self.test_path), collections.OrderedDict([ ('foobarbaz.txt', '93b048d0202e4b06b658f3aef1e764d3') ])) def test_single_file_nested(self): nested_dir = self.test_path / 'bar' nested_dir.mkdir() filepath = (nested_dir / 'foo.baz').relative_to(self.test_path) self.make_file(b'anything at all', filepath) self.assertEqual( util.md5sum_directory(self.test_path), collections.OrderedDict([ ('bar/foo.baz', 'dcc0975b66728be0315abae5968379cb') ])) def test_sorted_decent(self): nested_dir = self.test_path / 'beta' nested_dir.mkdir() filepath = (nested_dir / '10').relative_to(self.test_path) self.make_file(b'10', filepath) filepath = (nested_dir / '1').relative_to(self.test_path) self.make_file(b'1', filepath) filepath = (nested_dir / '2').relative_to(self.test_path) self.make_file(b'2', filepath) nested_dir = self.test_path / 'alpha' nested_dir.mkdir() filepath = (nested_dir / 'foo').relative_to(self.test_path) self.make_file(b'foo', filepath) filepath = (nested_dir / 'bar').relative_to(self.test_path) self.make_file(b'bar', filepath) self.make_file(b'z', 'z') self.assertEqual( list(util.md5sum_directory(self.test_path).items()), [ ('z', 'fbade9e36a3f36d3d676c1b808451dd7'), ('alpha/bar', '37b51d194a7513e45b56f6524f2d51f2'), ('alpha/foo', 'acbd18db4cc2f85cedef654fccc4a4d8'), ('beta/1', 'c4ca4238a0b923820dcc509a6f75849b'), ('beta/10', 'd3d9446802a44259755d38e6d163e820'), ('beta/2', 'c81e728d9d4c2f636f067f89cc14862c'), ]) def test_can_use_string(self): nested_dir = self.test_path / 'bar' nested_dir.mkdir() filepath = (nested_dir / 'foo.baz').relative_to(self.test_path) self.make_file(b'anything at all', filepath) self.assertEqual( util.md5sum_directory(str(self.test_path)), collections.OrderedDict([ ('bar/foo.baz', 'dcc0975b66728be0315abae5968379cb') ])) class TestChecksumFormat(unittest.TestCase): def test_to_simple(self): line = util.to_checksum_format('this/is/a/filepath', 'd9724aeba59d8cea5265f698b2c19684') self.assertEqual( line, 'd9724aeba59d8cea5265f698b2c19684 this/is/a/filepath') def test_from_simple(self): fp, chks = util.from_checksum_format( 'd9724aeba59d8cea5265f698b2c19684 this/is/a/filepath') self.assertEqual(fp, 'this/is/a/filepath') self.assertEqual(chks, 'd9724aeba59d8cea5265f698b2c19684') def test_to_hard(self): # two kinds of backslash n to trip up the escaping: line = util.to_checksum_format('filepath/\n/with/\\newline', '939aaaae6098ebdab049b0f3abe7b68c') # Note raw string self.assertEqual( line, r'\939aaaae6098ebdab049b0f3abe7b68c filepath/\n/with/\\newline') def test_from_hard(self): fp, chks = util.from_checksum_format( r'\939aaaae6098ebdab049b0f3abe7b68c filepath/\n/with/\\newline' + '\n') # newline from a checksum "file" self.assertEqual(fp, 'filepath/\n/with/\\newline') self.assertEqual(chks, '939aaaae6098ebdab049b0f3abe7b68c') def test_filepath_with_leading_backslash(self): line = r'\d41d8cd98f00b204e9800998ecf8427e \\.qza' fp, chks = util.from_checksum_format(line) self.assertEqual(chks, 'd41d8cd98f00b204e9800998ecf8427e') self.assertEqual(fp, r'\.qza') def test_filepath_with_leading_backslashes(self): line = r'\d41d8cd98f00b204e9800998ecf8427e \\\\\\.qza' fp, chks = util.from_checksum_format(line) self.assertEqual(fp, r'\\\.qza') self.assertEqual(chks, 'd41d8cd98f00b204e9800998ecf8427e') def test_impossible_backslash(self): # It may be impossible to generate a single '\' in the md5sum digest, # because each '\' is escaped (as '\\') in the digest. We'll # test for it anyway, for full coverage. fp, _ = util.from_checksum_format( r'fake_checksum \.qza' ) fp2, _ = util.from_checksum_format( r'\fake_checksum \.qza' ) self.assertEqual(fp, r'\.qza') self.assertEqual(fp2, r'\.qza') def test_from_legacy_format(self): fp, chks = util.from_checksum_format( r'0ed29022ace300b4d96847882daaf0ef *this/means/binary/mode') self.assertEqual(fp, 'this/means/binary/mode') self.assertEqual(chks, '0ed29022ace300b4d96847882daaf0ef') def check_roundtrip(self, filepath, checksum): line = util.to_checksum_format(filepath, checksum) new_fp, new_chks = util.from_checksum_format(line) self.assertEqual(new_fp, filepath) self.assertEqual(new_chks, checksum) def test_nonsense(self): self.check_roundtrip( r'^~gpfh)bU)WvN/;3jR6H-*={iEBM`(flY2>_|5mp8{-h>Ou\{{ImLT>h;XuC,.#', '89241859050e5a43ccb5f7aa0bca7a3a') self.check_roundtrip( r"l5AAPGKLP5Mcv0b`@zDR\XTTnF;[2M>O/>,d-^Nti'vpH\{>q)/4&CuU/xQ}z,O", 'c47d43cadb60faf30d9405a3e2592b26') self.check_roundtrip( r'FZ\rywG:7Q%"J@}Rk>\&zbWdS0nhEl_k1y1cMU#Lk_"*#*/uGi>Evl7M1suNNVE', '9c7753f252116473994e8bffba2c620b') class TestSortedPoset(unittest.TestCase): def test_already_sorted_incomparable(self): a = [Foo, Bar, Baz] r = util.sorted_poset(a) # Incomparable elements, so as long as they # are present, any order is valid. self.assertEqual(len(r), 3) self.assertIn(Foo, r) self.assertIn(Bar, r) self.assertIn(Baz, r) def test_already_sorted_all_comparable(self): a = [Foo, Foo | Bar, Foo | Bar | Baz] r = util.sorted_poset(a) self.assertEqual(a, r) def test_already_sorted_all_comparable_reverse(self): a = [Foo, Foo | Bar, Foo | Bar | Baz] r = util.sorted_poset(a, reverse=True) self.assertEqual(list(reversed(a)), r) def test_mixed_elements(self): a = [Foo | Bar, Foo | Baz, Foo] r = util.sorted_poset(a) self.assertEqual(r[0], Foo) # Order of others won't matter def test_mxed_elements_diamond(self): a = [Foo | Bar, Foo, Bar | Baz | Foo, Baz | Foo] r = util.sorted_poset(a) self.assertEqual(r[0], Foo) self.assertEqual(r[-1], Bar | Baz | Foo) def test_multiple_minimums(self): a = [Foo | Bar, Foo, Bar | Baz | Foo, Bar, Baz] r = util.sorted_poset(a) idx_foo = r.index(Foo) idx_bar = r.index(Bar) idx_foobar = r.index(Foo | Bar) self.assertLess(idx_foo, idx_foobar) self.assertLess(idx_bar, idx_foobar) self.assertEqual(r[-1], Bar | Baz | Foo) def test_multiple_equivalents(self): a = [Baz, Foo | Bar, Foo, Bar | Foo, Bar] r = util.sorted_poset(a) idx_foo = r.index(Foo) idx_bar = r.index(Bar) idx_barfoo = r.index(Bar | Foo) idx_foobar = r.index(Foo | Bar) adjacent = -1 <= idx_barfoo - idx_foobar <= 1 self.assertTrue(adjacent) self.assertLess(idx_foo, idx_barfoo) self.assertLess(idx_foo, idx_foobar) self.assertLess(idx_bar, idx_barfoo) self.assertLess(idx_bar, idx_foobar) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/tests/test_validate.py000066400000000000000000000245161412141660100215360ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from qiime2.core.exceptions import ValidationError, ImplementationError import unittest from qiime2.core.validate import ValidationObject from qiime2.sdk import PluginManager from qiime2.plugin.plugin import ValidatorRecord, Plugin from qiime2.core.testing.type import (IntSequence1, AscIntSequence, Kennel, Dog, Squid, Octopus) from qiime2.core.testing.format import IntSequenceFormat, Cephalapod class TestValidationObject(unittest.TestCase): def setUp(self): self.simple_int_seq = IntSequenceFormat() with self.simple_int_seq.open() as fh: fh.write('\n'.join(map(str, range(3)))) self.simple_int_seq.validate(level='max') def test_initialization(self): validator_object = ValidationObject(IntSequence1) self.assertEqual(validator_object.concrete_type, IntSequence1) def test_add_validator(self): def test_validator_method(data: list, level): pass test_record = ValidatorRecord(validator=test_validator_method, view=list, plugin='this_plugin', context=IntSequence1) validator_object = ValidationObject(IntSequence1) validator_object.add_validator(test_record) self.assertEqual(validator_object._validators, [test_record]) def test_add_validation_object(self): first_VO = ValidationObject(IntSequence1) second_VO = ValidationObject(IntSequence1) def first_validator(data: list, level): pass def second_validator(data: list, level): pass first_record = ValidatorRecord(validator=first_validator, view=list, plugin='this_plugin', context=IntSequence1) second_record = ValidatorRecord(validator=second_validator, view=list, plugin='this_plugin', context=IntSequence1) first_VO.add_validator(first_record) second_VO.add_validator(second_record) # Allows us to demonstrate add_validation_object sets _is_sorted to # false first_VO._sort_validators() first_VO.add_validation_object(second_VO) self.assertEqual(first_VO._validators, [first_record, second_record]) self.assertFalse(first_VO._is_sorted) def test_catch_different_concrete_types(self): squid_vo = ValidationObject(Squid) octopus_vo = ValidationObject(Octopus) def squid_validator(data: Cephalapod, level): pass def octopus_validator(data: Cephalapod, level): pass squid_record = ValidatorRecord(validator=squid_validator, view=Cephalapod, plugin='ocean_plugin', context=Squid) octopus_record = ValidatorRecord(validator=octopus_validator, view=Cephalapod, plugin='sea_plugin', context=Octopus) squid_vo.add_validator(squid_record) octopus_vo.add_validator(octopus_record) with self.assertRaisesRegex(TypeError, "Unable to add"): squid_vo.add_validation_object(octopus_vo) def test_public_validators_generation(self): validator_object = ValidationObject(IntSequence1) def first_validator(data: list, level): pass def second_validator(data: list, level): pass first_record = ValidatorRecord(validator=first_validator, view=list, plugin='this_plugin', context=IntSequence1) second_record = ValidatorRecord(validator=second_validator, view=list, plugin='this_plugin', context=IntSequence1) validator_object.add_validator(first_record) validator_object.add_validator(second_record) self.assertEqual(validator_object.validators, [first_record, second_record]) self.assertTrue(validator_object._is_sorted) def test_run_validators(self): validator_object = ValidationObject(IntSequence1) has_run = False def test_validator_method(data: list, level): nonlocal has_run has_run = True self.assertEqual(data, [0, 1, 2]) self.assertEqual(level, 'max') test_record = ValidatorRecord(validator=test_validator_method, view=list, plugin='this_plugin', context=IntSequence1) validator_object.add_validator(test_record) validator_object(self.simple_int_seq, level='max') self.assertTrue(has_run) def test_run_validators_validation_exception(self): validator_object = ValidationObject(AscIntSequence) def test_raising_validation_exception(data: list, level): raise ValidationError("2021-08-24") test_record = ValidatorRecord( validator=test_raising_validation_exception, view=list, plugin='this_plugin', context=AscIntSequence) validator_object.add_validator(test_record) with self.assertRaisesRegex(ValidationError, "2021-08-24"): validator_object(data=[], level=None) def test_run_validators_unknown_exception(self): validator_object = ValidationObject(AscIntSequence) def test_raising_validation_exception(data: list, level): raise KeyError("2021-08-24") test_record = ValidatorRecord( validator=test_raising_validation_exception, view=list, plugin='this_plugin', context=AscIntSequence) validator_object.add_validator(test_record) with self.assertRaisesRegex(ImplementationError, "attempted to validate"): validator_object(data=[], level=None) def test_validator_sorts(self): self.pm = PluginManager() test_object = self.pm.validators[Squid] self.assertFalse(test_object._is_sorted) exp = ['validator_sort_first', 'validator_sort_middle', 'validator_sort_middle_b', 'validator_sort_last'] exp2 = ['validator_sort_first', 'validator_sort_middle_b', 'validator_sort_middle', 'validator_sort_last'] obs = [record.validator.__name__ for record in test_object.validators] self.assertIn(obs, [exp, exp2]) self.assertTrue(test_object._is_sorted) class TestValidatorIntegration(unittest.TestCase): def setUp(self): # setup test plugin self.test_plugin = Plugin(name='validator_test_plugin', version='0.0.1', website='test.com', package='qiime2.core.tests', project_name='validator_test') self.pm = PluginManager() # setup test data self.simple_int_seq = IntSequenceFormat() with self.simple_int_seq.open() as fh: fh.write('\n'.join(map(str, range(3)))) self.simple_int_seq.validate(level='max') def tearDown(self): # This is a deadman switch to ensure that the test_plugin has been # added self.assertIn(self.test_plugin.name, self.pm.plugins) self.pm.forget_singleton() def test_validator_from_each_type_in_expression(self): @self.test_plugin.register_validator(IntSequence1 | AscIntSequence) def blank_validator(data: list, level): pass self.pm.add_plugin(self.test_plugin) def test_no_transformer_available(self): @self.test_plugin.register_validator(IntSequence1 | Kennel[Dog]) def blank_validator(data: list, level): pass with self.assertRaisesRegex( AssertionError, r"Kennel\[Dog\].*blank_validator.*transform.*builtins:list"): self.pm.add_plugin(self.test_plugin) class TestValidatorRegistration(unittest.TestCase): def setUp(self): self.test_plugin = Plugin(name='validator_test_plugin', version='0.0.1', website='test.com', package='qiime2.core.tests', project_name='validator_test') def test_catch_missing_validator_arg(self): run_checker = False with self.assertRaisesRegex(TypeError, "does not contain the" " required arguments"): run_checker = True @self.test_plugin.register_validator(IntSequence1) def validator_missing_level(data: list): pass assert run_checker def test_catch_extra_validator_arg(self): run_checker = False with self.assertRaisesRegex(TypeError, "does not contain the" " required arguments"): run_checker = True @self.test_plugin.register_validator(IntSequence1) def validator_extra_arg(data: list, level, spleen): pass assert run_checker def test_catch_no_data_annotation_in_validator(self): run_checker = False with self.assertRaisesRegex(TypeError, "No expected view type" " provided as annotation for `data`" " variable"): run_checker = True @self.test_plugin.register_validator(IntSequence1) def validator_no_view_annotation(data, level): pass assert run_checker qiime-2021.8.0/qiime2/core/transform.py000066400000000000000000000203251412141660100175510ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import pathlib from qiime2 import sdk from qiime2.plugin import model from qiime2.core import util def identity_transformer(view): return view class ModelType: @staticmethod def from_view_type(view_type): if issubclass(view_type, model.base.FormatBase): if issubclass(view_type, model.SingleFileDirectoryFormatBase): # HACK: this is necessary because we need to be able to "act" # like a FileFormat when looking up transformers, but our # input/output coercion still needs to bridge the # transformation as we do not have transitivity # In other words we have DX and we have transformers of X # In a perfect world we would automatically define DX -> X and # let transitivity handle it, but since that doesn't exist, we # need to treat DX as if it were X and coerce behind the scenes # TODO: redo this when transformers are transitive return SingleFileDirectoryFormatType(view_type) # Normal format type return FormatType(view_type) else: # TODO: supporting stdlib.typing may require an alternate # model type as `isinstance` is a meaningless operation # for them so validation would need to be handled differently return ObjectType(view_type) def __init__(self, view_type): self._pm = sdk.PluginManager() self._view_type = view_type self._view_name = util.get_view_name(self._view_type) self._record = None if self._view_name in self._pm.views: self._record = self._pm.views[self._view_name] def make_transformation(self, other, recorder=None): # record may be None in case of identity transformer transformer, transformer_record = self._get_transformer_to(other) if transformer is None: raise Exception("No transformation from %r to %r" % (self._view_type, other._view_type)) if recorder is not None: recorder(transformer_record, input_name=self._view_name, input_record=self._record, output_name=other._view_name, output_record=other._record) def transformation(view, validate_level='min'): view = self.coerce_view(view) self.validate(view, level=validate_level) new_view = transformer(view) new_view = other.coerce_view(new_view) other.validate(new_view) if transformer is not identity_transformer: other.set_user_owned(new_view, False) return new_view return transformation def _get_transformer_to(self, other): transformer, record = self._lookup_transformer(self._view_type, other._view_type) if transformer is None: return other._get_transformer_from(self) return transformer, record def has_transformation(self, other): """ Checks to see if there exist transformers for other Parameters ---------- other : ModelType subclass The object being checked for transformer Returns ------- bool Does the specified transformer exist for other? """ transformer, _ = self._get_transformer_to(other) return transformer is not None def _get_transformer_from(self, other): return None, None def coerce_view(self, view): return view def _lookup_transformer(self, from_, to_): if from_ == to_: return identity_transformer, None try: record = self._pm.transformers[from_][to_] return record.transformer, record except KeyError: return None, None def set_user_owned(self, view, value): pass class FormatType(ModelType): def coerce_view(self, view): if type(view) is str or isinstance(view, pathlib.Path): return self._view_type(view, mode='r') if isinstance(view, self._view_type): # wrap original path (inheriting the lifetime) and return a # read-only instance return self._view_type(view.path, mode='r') return view def validate(self, view, level='min'): if not isinstance(view, self._view_type): raise TypeError("%r is not an instance of %r." % (view, self._view_type)) # Formats have a validate method, so defer to it view.validate(level) def set_user_owned(self, view, value): view.path._user_owned = value class SingleFileDirectoryFormatType(FormatType): def __init__(self, view_type): # Single file directory formats have only one file named `file` # allowing us construct a model type from the format of `file` self._wrapped_view_type = view_type.file.format super().__init__(view_type) def _get_transformer_to(self, other): # Legend: # - Dx: single directory format of x # - Dy: single directory format of y # - x: input format x # - y: output format y # - ->: implicit transformer # - =>: registered transformer # - :> final transformation # - |: or, used when multiple situation are possible # It looks like all permutations because it is... # Dx :> y | Dy via Dx => y | Dy transformer, record = self._wrap_transformer(self, other) if transformer is not None: return transformer, record # Dx :> Dy via Dx -> x => y | Dy transformer, record = self._wrap_transformer(self, other, wrap_input=True) if transformer is not None: return transformer, record if type(other) is type(self): # Dx :> Dy via Dx -> x => y -> Dy transformer, record = self._wrap_transformer( self, other, wrap_input=True, wrap_output=True) if transformer is not None: return transformer, record # Out of options, try for Dx :> Dy via Dx => y -> Dy return other._get_transformer_from(self) # record is included def _get_transformer_from(self, other): # x | Dx :> Dy via x | Dx => y -> Dy # IMPORTANT: reverse other and self, this method is like __radd__ return self._wrap_transformer(other, self, wrap_output=True) def _wrap_transformer(self, in_, out_, wrap_input=False, wrap_output=False): input = in_._wrapped_view_type if wrap_input else in_._view_type output = out_._wrapped_view_type if wrap_output else out_._view_type transformer, record = self._lookup_transformer(input, output) if transformer is None: return None, None if wrap_input: transformer = in_._wrap_input(transformer) if wrap_output: transformer = out_._wrap_output(transformer) return transformer, record def _wrap_input(self, transformer): def wrapped(view): return transformer(view.file.view(self._wrapped_view_type)) return wrapped def _wrap_output(self, transformer): def wrapped(view): new_view = self._view_type() file_view = transformer(view) if transformer is not identity_transformer: self.set_user_owned(file_view, False) new_view.file.write_data(file_view, self._wrapped_view_type) return new_view return wrapped class ObjectType(ModelType): def validate(self, view, level=None): if not isinstance(view, self._view_type): raise TypeError("%r is not of type %r, cannot transform further." % (view, self._view_type)) qiime-2021.8.0/qiime2/core/type/000077500000000000000000000000001412141660100161435ustar00rootroot00000000000000qiime-2021.8.0/qiime2/core/type/__init__.py000066400000000000000000000031741412141660100202610ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from .collection import List, Set from .semantic import SemanticType, Properties from .primitive import (Str, Int, Float, Metadata, Bool, MetadataColumn, Categorical, Numeric, Range, Start, End, Choices) from .visualization import Visualization from .signature import PipelineSignature, MethodSignature, VisualizerSignature from .meta import TypeMap, TypeMatch from .util import (is_primitive_type, is_semantic_type, is_metadata_type, is_collection_type, is_visualization_type, interrogate_collection_type, parse_primitive, is_union, is_metadata_column_type) __all__ = [ # Type Helpers 'is_semantic_type', 'is_visualization_type', 'is_primitive_type', 'is_metadata_type', 'is_collection_type', 'interrogate_collection_type', 'parse_primitive', 'is_union', 'is_metadata_column_type', # Collection Types 'Set', 'List', # Semantic Types 'SemanticType', 'Properties', # Primitive Types 'Str', 'Int', 'Float', 'Bool', 'Metadata', 'MetadataColumn', 'Categorical', 'Numeric', 'Range', 'Start', 'End', 'Choices', # Visualization Type 'Visualization', # Signatures 'PipelineSignature', 'MethodSignature', 'VisualizerSignature', # Variables 'TypeMap', 'TypeMatch' ] qiime-2021.8.0/qiime2/core/type/collection.py000066400000000000000000000052101412141660100206460ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import json from qiime2.core.type.template import TypeTemplate class _CollectionBase(TypeTemplate): public_proxy = 'encode', 'decode' def __init__(self): # For semantic types self.variant_of = frozenset() def __eq__(self, other): return type(self) is type(other) def get_name(self): return self.__class__.__name__[1:] # drop `_` def get_kind_expr(self, self_expr): if self_expr.fields: return self_expr.fields[0].kind return "" def get_kind(self): raise NotImplementedError def is_variant(self, self_expr, varfield): return False def validate_predicate(self, predicate): raise TypeError("Predicates cannot be applied to %r" % self.get_name()) def is_element_expr(self, self_expr, value): contained_expr = self_expr.fields[0] if isinstance(value, self._view) and len(value) > 0: return all(v in contained_expr for v in value) return False def is_element(self, value): raise NotImplementedError def get_union_membership_expr(self, self_expr): return self.get_name() + '-' + self.get_kind_expr(self_expr) # For primitive types def encode(self, value): return json.dumps(list(value)) def decode(self, string): return self._view(json.loads(string)) class _1DCollectionBase(_CollectionBase): def validate_field(self, name, field): if isinstance(field, _1DCollectionBase): raise TypeError("Cannot nest collection types.") if field.get_name() in {'MetadataColumn', 'Metadata'}: raise TypeError("Cannot use %r with metadata." % self.get_name()) def get_field_names(self): return ['type'] class _Set(_1DCollectionBase): _view = set class _List(_1DCollectionBase): _view = list class _Tuple(_CollectionBase): _view = tuple def get_kind_expr(self, self_expr): return "" def get_field_names(self): return ['*types'] def validate_field_count(self, count): if not count: raise TypeError("Tuple type must contain at least one element.") def validate_field(self, name, field): # Tuples may contain anything, and as many fields as desired pass Set = _Set() List = _List() Tuple = _Tuple() qiime-2021.8.0/qiime2/core/type/grammar.py000066400000000000000000000474351412141660100201600ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import itertools from abc import ABCMeta, abstractmethod from qiime2.core.util import tuplize, ImmutableBase def maximal_antichain(*types): maximal_elements = {} # easy to delete, retains order for t in types: placed = False for e in list(maximal_elements): if e <= t: # Delete first! Duplicate keys would disappear otherwise del maximal_elements[e] maximal_elements[t] = None placed = True if not placed: maximal_elements[t] = None return tuple(maximal_elements) def minimal_antichain(*types): minimal_elements = {} # easy to delete, retains order for t in types: placed = False for e in list(minimal_elements): if t <= e: # Delete first! Duplicate keys would disappear otherwise del minimal_elements[e] minimal_elements[t] = None placed = True if not placed: minimal_elements[t] = None return tuple(minimal_elements) class _ExpBase(ImmutableBase, metaclass=ABCMeta): def __init__(self, template): # Super basic smoke-test assert template is None or template.is_template self.template = template def __getattr__(self, name): if ('template' in self.__dict__ and self.template is not None and name in self.template.public_proxy): return getattr(self.template, name) raise AttributeError("%r object has no attribute %r" % (type(self), name)) # Prevent infinite recursion when pickling due to __getattr__ def __getstate__(self): return self.__dict__ def __setstate__(self, state): self.__dict__ = state @property def name(self): return self.template.get_name_expr(self) @property def kind(self): return self.template.get_kind_expr(self) @abstractmethod def __eq__(self, other): raise NotImplementedError def __ne__(self, other): return not self == other @abstractmethod def __le__(self, other): raise NotImplementedError @abstractmethod def __ge__(self, other): raise NotImplementedError @abstractmethod def __or__(self, other): raise NotImplementedError def __ror__(self, other): return self | other @abstractmethod def __and__(self, other): raise NotImplementedError def __rand__(self, other): return self & other @abstractmethod def equals(self, other): raise NotImplementedError def is_concrete(self): return False def iter_symbols(self): yield self.name class IncompleteExp(_ExpBase): def __init__(self, template): super().__init__(template) if (self.template is None or not list(self.template.get_field_names_expr(self))): raise ValueError("Template %r has no fields, should not be used" " with a IncompleteExp." % (template,)) def __eq__(self, other): if type(self) is not type(other): return NotImplemented return (self.name == other.name and tuple(self.template.get_field_names_expr(self)) == tuple(other.template.get_field_names_expr(self))) def __hash__(self): return (hash(type(self)) ^ hash(self.name) ^ hash(tuple(self.template.get_field_names_expr(self)))) def __repr__(self): fields = ', '.join( '{%s}' % f for f in self.template.get_field_names_expr(self)) return self.name + ('[%s]' % fields) def __le__(self, other): raise TypeError("Cannot compare subtype, %r is missing arguments" " for its fields." % (self,)) def __ge__(self, other): raise TypeError("Cannot compare supertype, %r is missing arguments" " for its fields." % (self,)) def __contains__(self, value): raise TypeError("Cannot check membership of %r, %r is missing" " arguments for its fields." % (value, self)) def __mod__(self, predicate): raise TypeError("Cannot apply predicate %r, %r is missing arguments" " for its fields." % (predicate, self)) def __or__(self, other): raise TypeError("Cannot union with %r, %r is missing arguments" " for its fields." % (other, self)) def __and__(self, other): raise TypeError("Cannot intersect with %r, %r is missing arguments" " for its fields." % (other, self)) def __getitem__(self, fields): fields = tuplize(fields) for field in fields: if not isinstance(field, _AlgebraicExpBase): raise TypeError("Field %r is not complete type expression." % (field,)) self.template.validate_fields_expr(self, fields) return TypeExp(self.template, fields=fields) def equals(self, other): return self == other class _AlgebraicExpBase(_ExpBase): def __le__(self, other): first = self._is_subtype_(other) if first is not NotImplemented: return first second = other._is_supertype_(self) if second is not NotImplemented: return second return False def __ge__(self, other): first = self._is_supertype_(other) if first is not NotImplemented: return first second = other._is_subtype_(self) if second is not NotImplemented: return second return False def __or__(self, other): if not ((self.is_bottom() or other.is_bottom()) or (self.get_union_membership() == other.get_union_membership() and self.get_union_membership() is not None)): raise TypeError("Cannot union %r and %r" % (self, other)) if self >= other: return self if self <= other: return other union = UnionExp((*self.unpack_union(), *other.unpack_union())) return union.normalize() def __and__(self, other): if (not self.can_intersect() or not other.can_intersect() or (self.kind != other.kind and not (self.is_top() or other.is_top()))): raise TypeError("Cannot intersect %r and %r" % (self, other)) # inverse of __or__ if self >= other: return other if self <= other: return self # Distribute over union if isinstance(self, UnionExp) or isinstance(other, UnionExp): m = [] for s, o in itertools.product(self.unpack_union(), other.unpack_union()): m.append(s & o) return UnionExp(m).normalize() elements = list(itertools.chain(self.unpack_intersection(), other.unpack_intersection())) if len(elements) > 1: # Give the expression a chance to collapse, as many intersections # are contradictions collapse = elements[0]._collapse_intersection_(elements[1]) if collapse is not NotImplemented: for e in elements[2:]: collapse = collapse._collapse_intersection_(e) return collapse # Back to the regularly scheduled inverse of __or__ members = minimal_antichain(*self.unpack_intersection(), *other.unpack_intersection()) return IntersectionExp(members) def _collapse_intersection_(self, other): return NotImplemented def equals(self, other): return self <= other <= self def is_concrete(self): return len(list(self.unpack_union())) == 1 def get_union_membership(self): if self.template is not None: return self.template.get_union_membership_expr(self) return True def can_intersect(self): return True # These methods are to be overridden by UnionExp def is_bottom(self): return False def unpack_union(self): yield self # These methods are to be overridden by IntersectionExp def is_top(self): return False def unpack_intersection(self): yield self class TypeExp(_AlgebraicExpBase): def __init__(self, template, fields=(), predicate=None): super().__init__(template) if predicate is not None and predicate.is_top(): predicate = None self.fields = tuple(fields) self.predicate = predicate super()._freeze_() @property def full_predicate(self): if self.predicate is None: return IntersectionExp() return self.predicate def __eq__(self, other): if type(self) is not type(other): return NotImplemented return (self.kind == other.kind and self.name == other.name and self.fields == other.fields and self.full_predicate == other.full_predicate) def __hash__(self): return (hash(type(self)) ^ hash(self.kind) ^ hash(self.name) ^ hash(self.fields) ^ hash(self.predicate)) def __repr__(self): result = self.name if self.fields: result += '[%s]' % ', '.join(repr(f) for f in self.fields) if self.predicate: predicate = repr(self.predicate) if self.predicate.template is None: # is _IdentityExpBase predicate = '(%s)' % predicate result += ' % ' + predicate return result def __getitem__(self, fields): raise TypeError("Cannot apply fields (%r) to %r," " fields already present." % (fields, self)) def __contains__(self, value): return (self.template.is_element_expr(self, value) and value in self.full_predicate) def __iter__(self): yield from {self.duplicate(fields=fields) for fields in itertools.product(*self.fields)} def iter_symbols(self): yield self.name for field in self.fields: yield from field.iter_symbols() def _is_subtype_(self, other): if other.template is None: return NotImplemented if not self.template.is_symbol_subtype_expr(self, other): return False for f1, f2 in itertools.zip_longest(self.fields, other.fields, # more fields = more specific fillvalue=IntersectionExp()): if not (f1 <= f2): return False if not (self.full_predicate <= other.full_predicate): return False return True def _is_supertype_(self, other): return NotImplemented def __mod__(self, predicate): if self.predicate: raise TypeError("%r already has a predicate, will not add %r" % (self, predicate)) if predicate is None or predicate.is_top(): return self return self.duplicate(predicate=predicate) def __rmod__(self, other): raise TypeError("Predicate (%r) must be applied to the right-hand side" " of a type expression." % (other,)) def duplicate(self, fields=(), predicate=None): if fields == (): fields = self.fields else: self.template.validate_fields_expr(self, fields) if predicate is None: predicate = self.predicate elif predicate.is_top(): predicate = None elif predicate.template is not None: self.template.validate_predicate_expr(self, predicate) return self.__class__(self.template, fields=fields, predicate=predicate) def _collapse_intersection_(self, other): if self.name != other.name: return UnionExp() new_fields = tuple( s & o for s, o in itertools.zip_longest(self.fields, other.fields, # same as a type mismatch fillvalue=UnionExp())) if any(f.is_bottom() for f in new_fields): return UnionExp() new_predicate = self.full_predicate & other.full_predicate if new_predicate.is_bottom(): return UnionExp() return self.duplicate(fields=new_fields, predicate=new_predicate) def is_concrete(self): return self._bool_attr_method('is_concrete') def _bool_attr_method(self, method_name): def method(s): return getattr(s, method_name)() if any(not method(f) for f in self.fields): return False if not method(self.full_predicate): return False return True def to_ast(self): ast = { "type": "expression", "builtin": True, "name": self.name, "predicate": self.predicate.to_ast() if self.predicate else None, "fields": [field.to_ast() for field in self.fields] } self.template.update_ast_expr(self, ast) return ast class PredicateExp(_AlgebraicExpBase): def __init__(self, template): super().__init__(template) super()._freeze_() def __eq__(self, other): return self.template == other.template def __hash__(self): return hash(self.template) def __contains__(self, value): return self.template.is_element_expr(self, value) def __repr__(self): return repr(self.template) def _is_subtype_(self, other): if (other.template is not None and self.template.is_symbol_subtype_expr(self, other)): return True return NotImplemented def _is_supertype_(self, other): if (other.template is not None and self.template.is_symbol_supertype_expr(self, other)): return True return NotImplemented def _collapse_intersection_(self, other): first = self.template.collapse_intersection(other.template) if first is None: return UnionExp() elif first is not NotImplemented: return self.__class__(first) second = other.template.collapse_intersection(self.template) if second is None: return UnionExp() elif second is not NotImplemented: return self.__class__(second) return NotImplemented def to_ast(self): ast = { "type": "predicate", "name": self.name, } self.template.update_ast_expr(self, ast) return ast class _IdentityExpBase(_AlgebraicExpBase): """ Base class for IntersectionExp and UnionExp. If there are no members, then they are Top or Bottom types respectively and represent identity values (like 1 for mul and 0 for add) for the type algebra. There is no template object for these expressions. That property will always be `None`. """ _operator = ' ? ' def __init__(self, members=()): super().__init__(template=None) self.members = tuple(members) super()._freeze_() @property def kind(self): if not self.members: return "identity" return self.members[0].kind @property def name(self): return "" def __eq__(self, other): return (type(self) is type(other) and set(self.members) == set(other.members)) def __hash__(self): return hash(type(self)) ^ hash(frozenset(self.members)) def __repr__(self): if not self.members: return self.__class__.__name__ + "()" return self._operator.join(repr(m) for m in self.members) def __iter__(self): for m in self.unpack_union(): yield from m def iter_symbols(self): for m in self.unpack_union(): yield from m.iter_symbols() def get_union_membership(self): if self.members: return self.members[0].get_union_membership() class UnionExp(_IdentityExpBase): _operator = ' | ' # used by _IdentityExpBase.__repr__ def __contains__(self, value): return any(value in s for s in self.members) def _is_subtype_(self, other): if (isinstance(other, self.__class__) and type(other) is not self.__class__): # other is subclass return NotImplemented # if other isn't a union, becomes all(s <= other for s in self.members) return all(any(s <= o for o in other.unpack_union()) for s in self.unpack_union()) def _is_supertype_(self, other): return all(any(s >= o for s in self.unpack_union()) for o in other.unpack_union()) def is_bottom(self): return not self.members def unpack_union(self): yield from self.members def to_ast(self): return { "type": "union", "members": [m.to_ast() for m in self.members] } def normalize(self): elements = self.members groups = {} for e in elements: if type(e) is TypeExp: candidate = e.duplicate(predicate=IntersectionExp()) if candidate in groups: groups[candidate].append(e) else: groups[candidate] = [e] else: # groups should be empty already, but don't even attempt # collapsing if its a union of type expressions and "other" groups = {} break if groups: elements = [] for candidate, group in groups.items(): if len(group) == 1: elements.append(group[0]) else: predicate = UnionExp([t.full_predicate for t in group]) predicate = predicate.normalize() elements.append(candidate.duplicate(predicate=predicate)) if len(elements) < 20: members = maximal_antichain(*elements) else: members = elements if len(members) == 1: return members[0] return UnionExp(members) class IntersectionExp(_IdentityExpBase): _operator = ' & ' # used by _IdentityExpBase.__repr__ def __contains__(self, value): return all(value in s for s in self.members) def _is_subtype_(self, other): if isinstance(other, UnionExp): # Union will treat `self` as an atomic type, comparing # its elements against `self`. This in turn will recurse back to # `self` allowing it to check if it is a subtype of the union # elements. That check will ultimately compare the elements of # `self` against a single element of the union. return NotImplemented return all(any(s <= o for s in self.unpack_intersection()) for o in other.unpack_intersection()) def _is_supertype_(self, other): if isinstance(other, UnionExp): return NotImplemented return all(any(s >= o for o in other.unpack_intersection()) for s in self.unpack_intersection()) def is_top(self): return not self.members def unpack_intersection(self): yield from self.members def to_ast(self): return { "type": "intersection", "members": [m.to_ast() for m in self.members] } qiime-2021.8.0/qiime2/core/type/meta.py000066400000000000000000000244551412141660100174550ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import itertools from types import MappingProxyType from ..util import superscript, tuplize, ImmutableBase from .grammar import UnionExp, TypeExp from .collection import Tuple class TypeVarExp(UnionExp): def __init__(self, members, tmap, input=False, output=False, index=None): self.mapping = tmap self.input = input self.output = output self.index = index super().__init__(members) def __repr__(self): numbers = {} for idx, m in enumerate(self.members, 1): if m in numbers: numbers[m] += superscript(',' + str(idx)) else: numbers[m] = superscript(idx) return " | ".join([repr(k) + v for k, v in numbers.items()]) def uniq_upto_sub(self, a_expr, b_expr): """ Two elements are unique up to a subtype if they are indistinguishable with respect to that subtype. In the case of a type var, that means the same branches must be "available" in the type map. This means that A or B may have additional refinements (or may even be subtypes of each other), so long as that does not change the branch chosen by the type map. """ a_branches = [m for m in self.members if a_expr <= m] b_branches = [m for m in self.members if b_expr <= m] return a_branches == b_branches def __eq__(self, other): return (type(self) is type(other) and self.index == other.index and self.mapping == other.mapping) def __hash__(self): return hash(self.index) ^ hash(self.mapping) def is_concrete(self): return False def can_intersect(self): return False def get_union_membership_expr(self, self_expr): return None def _is_subtype_(self, other): return all(m <= other for m in self.members) def _is_supertype_(self, other): return any(m >= other for m in self.members) def __iter__(self): yield from self.members def unpack_union(self): yield self def to_ast(self): return { "type": "variable", "index": self.index, "group": id(self.mapping), "outputs": self.mapping.input_width(), "mapping": [ ([k.to_ast() for k in key.fields] + [v.to_ast() for v in value.fields]) for key, value in self.mapping.lifted.items()] } class TypeMap(ImmutableBase): def __init__(self, mapping): mapping = {Tuple[tuplize(k)]: Tuple[tuplize(v)] for k, v in mapping.items()} branches = list(mapping) for i, a in enumerate(branches): for j in range(i, len(branches)): b = branches[j] try: intersection = a & b except TypeError: raise ValueError("Cannot place %r and %r in the same " "type variable." % (a, b)) if (intersection.is_bottom() or intersection is a or intersection is b): continue for k in range(i): if intersection <= branches[k]: break else: raise ValueError( "Ambiguous resolution for invocations with type %r." " Could match %r or %r, add a new branch ABOVE these" " two (or modify these branches) to correct this." % (intersection.fields, a.fields, b.fields)) self.__lifted = mapping super()._freeze_() @property def lifted(self): return MappingProxyType(self.__lifted) def __eq__(self, other): return self is other def __hash__(self): return hash(id(self)) def __iter__(self): for idx, members in enumerate( zip(*(k.fields for k in self.lifted.keys()))): yield TypeVarExp(members, self, input=True, index=idx) yield from self.iter_outputs() def solve(self, *inputs): inputs = Tuple[inputs] for branch, outputs in self.lifted.items(): if inputs <= branch: return outputs.fields def input_width(self): return len(next(iter(self.lifted.keys())).fields) def iter_outputs(self, *, _double_as_input=False): start = self.input_width() for idx, members in enumerate( zip(*(v.fields for v in self.lifted.values())), start): yield TypeVarExp(members, self, output=True, index=idx, input=_double_as_input) def _get_intersections(listing): intersections = [] for a, b in itertools.combinations(listing, 2): i = a & b if i.is_bottom() or i is a or i is b: continue intersections.append(i) return intersections def TypeMatch(listing): listing = list(listing) intersections = _get_intersections(listing) to_add = [] while intersections: to_add.extend(intersections) intersections = _get_intersections(intersections) mapping = TypeMap({x: x for x in list(reversed(to_add)) + listing}) # TypeMatch only produces a single variable # iter_outputs is used by match for solving, so the index must match return next(iter(mapping.iter_outputs(_double_as_input=True))) def select_variables(expr): """When called on an expression, will yield selectors to the variable. A selector will either return the variable (or equivalent fragment) in an expression, or will return an entirely new expression with the fragment replaced with the value of `swap`. e.g. >>> select_u, select_t = select_variables(Example[T] % U) >>> t = select_t(Example[T] % U) >>> assert T is t >>> u = select_u(Example[T] % U) >>> assert U is u >>> frag = select_t(Example[Foo] % Bar) >>> assert frag is Foo >>> new_expr = select_t(Example[T] % U, swap=frag) >>> assert new_expr == Example[Foo] % U """ if type(expr) is TypeVarExp: def select(x, swap=None): if swap is not None: return swap return x yield select return if type(expr) is not TypeExp: return if type(expr.full_predicate) is TypeVarExp: def select(x, swap=None): if swap is not None: return x.duplicate(predicate=swap) return x.full_predicate yield select for idx, field in enumerate(expr.fields): for sel in select_variables(field): # Without this closure, the idx in select will be the last # value of the enumerate, same for sel # (Same problem as JS with callbacks inside a loop) def closure(idx, sel): def select(x, swap=None): if swap is not None: new_fields = list(x.fields) new_fields[idx] = sel(x.fields[idx], swap) return x.duplicate(fields=tuple(new_fields)) return sel(x.fields[idx]) return select yield closure(idx, sel) def match(provided, inputs, outputs): provided_binding = {} error_map = {} for key, expr in inputs.items(): for selector in select_variables(expr): var = selector(expr) provided_fragment = selector(provided[key]) try: current_binding = provided_binding[var] except KeyError: provided_binding[var] = provided_fragment error_map[var] = provided[key] else: if not var.uniq_upto_sub(current_binding, provided_fragment): raise ValueError("Received %r and %r, but expected %r" " and %r to match (or to select the same" " output)." % (error_map[var], provided[key], current_binding, provided_fragment)) # provided_binding now maps TypeVarExp instances to a TypeExp instance # which is the relevent fragment from the provided input types grouped_maps = {} for item in provided_binding.items(): var = item[0] if var.mapping not in grouped_maps: grouped_maps[var.mapping] = [item] else: grouped_maps[var.mapping].append(item) # grouped_maps now maps a TypeMap instance to tuples of # (TypeVarExp, TypeExp) which are the items of provided_binding # i.e. all of the bindings are now grouped under their shared type maps output_fragments = {} for mapping, group in grouped_maps.items(): if len(group) != mapping.input_width(): raise ValueError("Missing input variables") inputs = [x[1] for x in sorted(group, key=lambda x: x[0].index)] solved = mapping.solve(*inputs) if solved is None: provided = tuple(error_map[x[0]] for x in sorted(group, key=lambda x: x[0].index)) raise ValueError("No solution for inputs: %r, check the signature " "to see valid combinations." % (provided,)) # type vars share identity by instance of map and index, so we will # be able to see the "same" vars again when looking up the outputs for var, out in zip(mapping.iter_outputs(), solved): output_fragments[var] = out # output_fragments now maps a TypeVarExp to a TypeExp which is the solved # fragment for the given output type variable results = {} for key, expr in outputs.items(): r = expr # output may not have a typevar, so default is the expr for selector in select_variables(expr): var = selector(expr) r = selector(r, swap=output_fragments[var]) results[key] = r # results now maps a key to a full TypeExp as solved by the inputs return results qiime-2021.8.0/qiime2/core/type/parse.py000066400000000000000000000165731412141660100176430ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import ast from . import grammar, meta, collection, primitive, semantic, visualization def string_to_ast(type_expr): try: parsed = ast.parse(type_expr) except SyntaxError: raise ValueError("%r could not be parsed, it may not be a QIIME 2 type" " or it may not be an atomic type. Use" " `ast_to_type` instead." % (type_expr,)) if type(parsed) is not ast.Module: # I don't think this branch *can* be hit raise ValueError("%r is not a type expression." % (type_expr,)) try: expr, = parsed.body except ValueError: raise ValueError("Only one type expression may be parse at a time, got" ": %r" % (type_expr,)) return _expr(expr.value) def _expr(expr): node = type(expr) if node is ast.Name: return _build_atomic(expr.id) if node is ast.Call: args = _parse_args(expr.args) kwargs = _parse_kwargs(expr.keywords) return _build_predicate(expr.func.id, args, kwargs) if node is ast.Subscript: field_expr = expr.slice.value if type(field_expr) is ast.Tuple: field_expr = field_expr.elts else: field_expr = (field_expr,) base = _expr(expr.value) base['fields'] = [_expr(e) for e in field_expr] return base if node is ast.BinOp: op = type(expr.op) left = _expr(expr.left) right = _expr(expr.right) if op is ast.Mod: left['predicate'] = right return left if op is ast.BitOr: return _build_union(left, right) if op is ast.BitAnd: return _build_intersection(left, right) raise ValueError("Unknown expression: %r" % node) def _convert_literals(expr): node = type(expr) if node is ast.List: return [_convert_literals(e) for e in expr.elts] if node is ast.Set: return {_convert_literals(e) for e in expr.elts} if node is ast.Tuple: return tuple(_convert_literals(e) for e in expr.elts) if node is ast.Dict: return {_convert_literals(k): _convert_literals(v) for k, v in zip(expr.keys, expr.values)} if node is ast.Constant: return expr.value if node is ast.Name and expr.id == 'inf': return float('inf') raise ValueError("Unknown literal: %r" % node) def _parse_args(args): return tuple(_convert_literals(e) for e in args) def _parse_kwargs(kwargs): return {e.arg: _convert_literals(e.value) for e in kwargs} def _build_predicate(name, args, kwargs): base = { 'type': 'predicate', 'name': name } if name == 'Properties': return _build_properties(base, args, kwargs) if name == 'Range': return _build_range(base, args, kwargs) if name == 'Choices': return _build_choices(base, args, kwargs) def _normalize_input_collection(args): if len(args) == 1 and isinstance(args[0], (list, set, tuple)): return tuple(args[0]) return args def _build_choices(base, args, kwargs): if 'choices' in kwargs: args = (kwargs['choices'],) args = _normalize_input_collection(args) base['choices'] = list(args) return base def _build_range(base, args, kwargs): inclusive_start = kwargs.get('inclusive_start', True) inclusive_end = kwargs.get('inclusive_end', False) start = None end = None if len(args) == 1: end = args[0] elif len(args) != 0: start, end = args if start == float('-inf'): start = None if end == float('inf'): end = None base['range'] = [start, end] base['inclusive'] = [inclusive_start, inclusive_end] return base def _build_properties(base, args, kwargs): exclude = kwargs.get('exclude', []) if 'include' in kwargs: args = (kwargs['include'],) args = _normalize_input_collection(args) base['include'] = list(args) base['exclude'] = list(exclude) return base def _build_atomic(name): return { 'type': 'expression', 'builtin': name in {'Str', 'Int', 'Float', 'Bool', 'List', 'Set', 'Tuple', 'Visualization', 'Metadata', 'MetadataColumn', 'Numeric', 'Categorical'}, 'name': name, 'predicate': None, 'fields': [] } def _build_union(left, right): return _build_ident(left, right, 'union') def _build_intersection(left, right): return _build_ident(left, right, 'intersection') def _build_ident(left, right, type): members = [] if left['type'] == type: members.extend(left['members']) else: members.append(left) if right['type'] == type: members.extend(right['members']) else: members.append(right) return { 'type': type, 'members': members } def ast_to_type(json_ast, scope=None): if scope is None: scope = {} type_ = json_ast['type'] if type_ == 'expression': predicate = json_ast['predicate'] if predicate is not None: predicate = ast_to_type(predicate, scope=scope) fields = json_ast['fields'] if len(fields) > 0: fields = [ast_to_type(f, scope=scope) for f in fields] name = json_ast['name'] if not json_ast['builtin']: base_template = semantic.SemanticType(name).template elif name == 'Visualization': return visualization.Visualization elif name in {'List', 'Set', 'Tuple'}: base_template = getattr(collection, name).template else: base_template = getattr(primitive, name).template return grammar.TypeExp(base_template, fields=fields, predicate=predicate) if type_ == 'predicate': name = json_ast['name'] if name == 'Choices': return primitive.Choices(json_ast['choices']) if name == 'Range': return primitive.Range(*json_ast['range'], inclusive_start=json_ast['inclusive'][0], inclusive_end=json_ast['inclusive'][1]) if name == 'Properties': return semantic.Properties(json_ast['include'], exclude=json_ast['exclude']) if type_ == 'union': members = [ast_to_type(m, scope=scope) for m in json_ast['members']] return grammar.UnionExp(members) if type_ == 'intersection': members = [ast_to_type(m, scope=scope) for m in json_ast['members']] return grammar.IntersectionExp(members) if type_ == 'variable': var_group = json_ast['group'] if var_group not in scope: mapping = {} out_idx = json_ast['outputs'] for entry in json_ast['mapping']: entry = [ast_to_type(e) for e in entry] mapping[tuple(entry[:out_idx])] = tuple(entry[out_idx:]) scope[var_group] = list(meta.TypeMap(mapping)) return scope[var_group][json_ast['index']] qiime-2021.8.0/qiime2/core/type/primitive.py000066400000000000000000000332571412141660100205370ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import numbers import itertools from qiime2.core.type.template import TypeTemplate, PredicateTemplate import qiime2.metadata as metadata import qiime2.core.util as util _RANGE_DEFAULT_START = float('-inf') _RANGE_DEFAULT_END = float('inf') _RANGE_DEFAULT_INCLUSIVE_START = True _RANGE_DEFAULT_INCLUSIVE_END = False class _PrimitivePredicateBase(PredicateTemplate): def get_kind(self): return 'primitive' def get_name(self): return self.__class__.__name__ class Range(_PrimitivePredicateBase): def __init__(self, *args, inclusive_start=_RANGE_DEFAULT_INCLUSIVE_START, inclusive_end=_RANGE_DEFAULT_INCLUSIVE_END): if len(args) == 2: self.start, self.end = args elif len(args) == 1: self.start = _RANGE_DEFAULT_START self.end, = args elif len(args) == 0: self.start = _RANGE_DEFAULT_START self.end = _RANGE_DEFAULT_END else: raise ValueError("Too many arguments passed, expected 0, 1, or 2.") self.inclusive_start = inclusive_start self.inclusive_end = inclusive_end if self.start is None: self.start = _RANGE_DEFAULT_START if self.end is None: self.end = _RANGE_DEFAULT_END if self.end < self.start: raise ValueError("End of range precedes start.") def __hash__(self): return (hash(type(self)) ^ hash(self.start) ^ hash(self.end) ^ hash(self.inclusive_start) ^ hash(self.inclusive_end)) def __eq__(self, other): return (type(self) is type(other) and self.start == other.start and self.end == other.end and self.inclusive_start == other.inclusive_start and self.inclusive_end == other.inclusive_end) def __repr__(self): args = [] start = self.start if start == float('-inf'): start = None end = self.end if end == float('inf'): end = None args.append(repr(start)) args.append(repr(end)) if self.inclusive_start is not _RANGE_DEFAULT_INCLUSIVE_START: args.append('inclusive_start=%r' % self.inclusive_start) if self.inclusive_end is not _RANGE_DEFAULT_INCLUSIVE_END: args.append('inclusive_end=%r' % self.inclusive_end) return "Range(%s)" % (', '.join(args),) def is_element(self, value): if self.inclusive_start: if value < self.start: return False elif value <= self.start: return False if self.inclusive_end: if value > self.end: return False elif value >= self.end: return False return True def is_symbol_subtype(self, other): if type(self) is not type(other): return False if other.start > self.start: return False elif (other.start == self.start and (not other.inclusive_start) and self.inclusive_start): return False if other.end < self.end: return False elif (other.end == self.end and (not other.inclusive_end) and self.inclusive_end): return False return True def is_symbol_supertype(self, other): if type(self) is not type(other): return False if other.start < self.start: return False elif (other.start == self.start and (not self.inclusive_start) and other.inclusive_start): return False if other.end > self.end: return False elif (other.end == self.end and (not self.inclusive_end) and other.inclusive_end): return False return True def collapse_intersection(self, other): if type(self) is not type(other): return None if self.start < other.start: new_start = other.start new_inclusive_start = other.inclusive_start elif other.start < self.start: new_start = self.start new_inclusive_start = self.inclusive_start else: new_start = self.start new_inclusive_start = ( self.inclusive_start and other.inclusive_start) if self.end > other.end: new_end = other.end new_inclusive_end = other.inclusive_end elif other.end > self.end: new_end = self.end new_inclusive_end = self.inclusive_end else: new_end = self.end new_inclusive_end = self.inclusive_end and other.inclusive_end if new_end < new_start: return None if (new_start == new_end and not (new_inclusive_start and new_inclusive_end)): return None return self.__class__(new_start, new_end, inclusive_start=new_inclusive_start, inclusive_end=new_inclusive_end).template def iter_boundaries(self): if self.start != float('-inf'): yield self.start if self.end != float('inf'): yield self.end def update_ast(self, ast): start = self.start if start == float('-inf'): start = None end = self.end if end == float('inf'): end = None ast['range'] = [start, end] ast['inclusive'] = [self.inclusive_start, self.inclusive_end] def Start(start, inclusive=_RANGE_DEFAULT_INCLUSIVE_START): return Range(start, _RANGE_DEFAULT_END, inclusive_start=inclusive) def End(end, inclusive=_RANGE_DEFAULT_INCLUSIVE_END): return Range(_RANGE_DEFAULT_START, end, inclusive_end=inclusive) class Choices(_PrimitivePredicateBase): def __init__(self, *choices): if not choices: raise ValueError("'Choices' cannot be instantiated with an empty" " set.") # Backwards compatibility with old Choices({1, 2, 3}) syntax if len(choices) == 1: if not isinstance(choices[0], (bool, str)): choices = choices[0] self.choices = choices = tuple(choices) if len(choices) != len(set(choices)): raise ValueError("Duplicates found in choices: %r" % util.find_duplicates(choices)) def __hash__(self): return hash(type(self)) ^ hash(frozenset(self.choices)) def __eq__(self, other): return (type(self) == type(other) and set(self.choices) == set(other.choices)) def __repr__(self): return "%s(%s)" % (self.__class__.__name__, repr(list(self.choices))[1:-1]) def is_element(self, value): return value in self.choices def is_symbol_subtype(self, other): if type(self) is not type(other): return False return set(self.choices) <= set(other.choices) def is_symbol_supertype(self, other): if type(self) is not type(other): return False return set(self.choices) >= set(other.choices) def collapse_intersection(self, other): if type(self) is not type(other): return None new_choices_set = set(self.choices) & set(other.choices) if not new_choices_set: return None # order by appearance: new_choices = [] for c in itertools.chain(self.choices, other.choices): if c in new_choices_set: new_choices.append(c) new_choices_set.remove(c) return self.__class__(new_choices).template def iter_boundaries(self): yield from self.choices def update_ast(self, ast): ast['choices'] = list(self.choices) def unpack_union(self): for c in self.choices: yield self.__class__(c) class _PrimitiveTemplateBase(TypeTemplate): public_proxy = 'encode', 'decode' def __eq__(self, other): return type(self) is type(other) def __hash__(self): return hash(type(self)) def get_name(self): return self.__class__.__name__[1:] # drop `_` def get_kind(self): return 'primitive' def get_field_names(self): return [] def validate_field(self, name, field): raise NotImplementedError def validate_predicate_expr(self, self_expr, predicate_expr): predicate = predicate_expr.template if type(predicate) not in self._valid_predicates: raise TypeError(str(predicate_expr)) for bound in predicate.iter_boundaries(): if not self.is_element_expr(self_expr, bound): raise TypeError(bound) def validate_predicate(self, predicate): raise NotImplementedError class _Int(_PrimitiveTemplateBase): _valid_predicates = {Range} def is_element(self, value): return (value is not True and value is not False and isinstance(value, numbers.Integral)) def is_symbol_subtype(self, other): if other.get_name() == 'Float': return True return super().is_symbol_subtype(other) def decode(self, string): return int(string) def encode(self, value): return str(value) class _Str(_PrimitiveTemplateBase): _valid_predicates = {Choices} def is_element(self, value): return isinstance(value, str) def decode(self, string): return str(string) def encode(self, value): return str(value) class _Float(_PrimitiveTemplateBase): _valid_predicates = {Range} def is_symbol_supertype(self, other): if other.get_name() == 'Int': return True return super().is_symbol_supertype(other) def is_element(self, value): # Works with numpy just fine. return (value is not True and value is not False and isinstance(value, numbers.Real)) def decode(self, string): return float(string) def encode(self, value): return str(value) class _Bool(_PrimitiveTemplateBase): _valid_predicates = {Choices} def is_element(self, value): return value is True or value is False def validate_predicate(self, predicate): if type(predicate) is Choices: if set(predicate.iter_boundaries()) == {True, False}: raise TypeError("Choices should be ommitted when " "Choices(True, False).") def decode(self, string): if string not in ('false', 'true'): raise TypeError("%s is neither 'true' or 'false'" % string) return string == 'true' def encode(self, value): if value: return 'true' else: return 'false' class _Metadata(_PrimitiveTemplateBase): _valid_predicates = set() def is_element(self, value): return isinstance(value, metadata.Metadata) def decode(self, metadata): # This interface should have already retrieved this object. if not self.is_element(metadata): raise TypeError("`Metadata` must be provided by the interface" " directly.") return metadata def encode(self, value): # TODO: Should this be the provenance representation? Does that affect # decode? return value class _MetadataColumn(_PrimitiveTemplateBase): _valid_predicates = set() def is_element_expr(self, self_expr, value): return value in self_expr.fields[0] def is_element(self, value): raise NotImplementedError def get_field_names(self): return ["type"] def validate_field(self, name, field): if field.get_name() not in ("Numeric", "Categorical"): raise TypeError("Unsupported type in field: %r" % (field.get_name(),)) def decode(self, value): # This interface should have already retrieved this object. if not isinstance(value, metadata.MetadataColumn): raise TypeError("`Metadata` must be provided by the interface" " directly.") return value def encode(self, value): # TODO: Should this be the provenance representation? Does that affect # decode? return value class _Categorical(_PrimitiveTemplateBase): _valid_predicates = set() def get_union_membership_expr(self, self_expr): return 'metadata-column' def is_element(self, value): return isinstance(value, metadata.CategoricalMetadataColumn) class _Numeric(_PrimitiveTemplateBase): _valid_predicates = set() def get_union_membership_expr(self, self_expr): return 'metadata-column' def is_element(self, value): return isinstance(value, metadata.NumericMetadataColumn) Int = _Int() Float = _Float() Bool = _Bool() Str = _Str() Metadata = _Metadata() MetadataColumn = _MetadataColumn() Categorical = _Categorical() Numeric = _Numeric() def infer_primitive_type(value): for t in (Int, Float): if value in t: return t % Range(value, value, inclusive_end=True) for t in (Bool, Str): if value in t: return t % Choices(value) for t in (Metadata, MetadataColumn[Categorical], MetadataColumn[Numeric]): if value in t: return t raise ValueError("Unknown primitive type: %r" % (value,)) qiime-2021.8.0/qiime2/core/type/semantic.py000066400000000000000000000264201412141660100203240ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import types import collections.abc import itertools from qiime2.core.type.grammar import IncompleteExp, UnionExp, IntersectionExp from qiime2.core.type.template import TypeTemplate, PredicateTemplate from qiime2.core.type.util import is_semantic_type, is_qiime_type _RESERVED_NAMES = { # Predicates: 'range', 'choice', 'properties', 'arguments', # Primitives: 'integer', 'int', 'string', 'str', 'metadata', 'metadatacolumn', 'categoricalmetadatacolumn', 'numericmetadatacolumn', 'column', 'categoricalcolumn', 'numericcolumn', 'metacol', 'categoricalmetacol', 'numericmetacol', 'metadatacategory', 'float', 'double', 'number', 'set', 'list', 'bag', 'multiset', 'map', 'dict', 'nominal', 'ordinal', 'categorical', 'numeric', 'interval', 'ratio', 'continuous', 'discrete', 'tuple', 'row', 'record', # Type System: 'semantictype', 'propertymap', 'propertiesmap', 'typemap', 'typevariable', 'predicate' } def _validate_name(name): if type(name) is not str: raise TypeError("Names of semantic types must be strings, not %r." % name) if name.lower() in _RESERVED_NAMES: raise ValueError("%r is a reserved name." % name) def SemanticType(name, field_names=None, field_members=None, variant_of=None): """Create a new semantic type. Parameters ---------- name : str The name of the semantic type: this should match the variable to which the semantic type is assigned. field_names : str, iterable of str, optional Name(s) of the fields where member types can be placed. This makes the type a composite type, meaning that fields must be provided to produce realized semantic types. These names will define ad-hoc variant types accessible as `name`.field[`field_names` member]. field_members : mapping, optional A mapping of strings in `field_names` to one or more semantic types which are known to be members of the field (the variant type). variant_of : VariantField, iterable of VariantField, optional Define the semantic type to be a member of one or more variant types allowing it to be placed in the respective fields defined by those variant types. Returns ------- A Semantic Type There are several (private) types which may be returned, but anything returned by this factory will cause `is_semantic_type` to return True. """ _validate_name(name) variant_of = _munge_variant_of(variant_of) field_names = _munge_field_names(field_names) field_members = _munge_field_members(field_names, field_members) return SemanticTemplate(name, field_names, field_members, variant_of) def _munge_variant_of(variant_of): if variant_of is None: variant_of = () elif isinstance(variant_of, VariantField): variant_of = (variant_of,) else: variant_of = tuple(variant_of) for variant in variant_of: if not isinstance(variant, VariantField): raise ValueError("Element %r of %r is not a variant field" " (ExampleType.field['name'])." % (variant, variant_of)) return variant_of def _munge_field_names(field_names): if field_names is None: return () if type(field_names) is str: return (field_names,) field_names = tuple(field_names) for field_name in field_names: if type(field_name) is not str: raise ValueError("Field name %r from %r is not a string." % (field_name, field_names)) if len(set(field_names)) != len(field_names): raise ValueError("Duplicate field names in %r." % field_names) return field_names def _munge_field_members(field_names, field_members): if field_names is None: return {} fixed = {k: () for k in field_names} if field_members is None: return fixed if not isinstance(field_members, collections.abc.Mapping): raise ValueError("") fixed.update(field_members) for key, value in field_members.items(): if key not in field_names: raise ValueError("Field member key: %r is not in `field_names`" " (%r)." % (key, field_names)) if is_qiime_type(value) and is_semantic_type(value): fixed[key] = (value,) else: value = tuple(value) for v in value: if not is_semantic_type(v): raise ValueError("Field member: %r (of field %r) is not a" " semantic type." % (v, key)) fixed[key] = value return fixed class VariantField: def __init__(self, type_name, field_name, field_members): self.type_name = type_name self.field_name = field_name self.field_members = field_members def is_member(self, semantic_type): for field_member in self.field_members: if isinstance(field_member, IncompleteExp): # Pseudo-subtyping like Foo[X] <= Foo[Any]. # (IncompleteExp will never have __le__ because you # are probably doing something wrong with it (this totally # doesn't count!)) if semantic_type.name == field_member.name: return True # ... it doesn't count because this is a way of restricting our # ontology and isn't really crucial. Where it matters would be # in function application where the semantics must be defined # precisely and Foo[Any] is anything but precise. else: if semantic_type <= field_member: return True return False def __repr__(self): return "%s.field[%r]" % (self.type_name, self.field_name) class SemanticTemplate(TypeTemplate): public_proxy = 'field', def __init__(self, name, field_names, field_members, variant_of): self.name = name self.field_names = field_names self.__field = {f: VariantField(name, f, field_members[f]) for f in self.field_names} self.variant_of = variant_of @property def field(self): return types.MappingProxyType(self.__field) def __eq__(self, other): return (type(self) is type(other) and self.name == other.name and self.fields == other.fields and self.variant_of == other.variant_of) def __hash__(self): return (hash(type(self)) ^ hash(self.name) ^ hash(self.fields) ^ hash(self.variant_of)) def get_kind(self): return 'semantic-type' def get_name(self): return self.name def get_field_names(self): return self.field_names def is_element_expr(self, self_expr, value): import qiime2.sdk if not isinstance(value, qiime2.sdk.Artifact): return False return value.type <= self_expr def is_element(self, value): raise NotImplementedError def validate_field(self, name, field): raise NotImplementedError def validate_fields_expr(self, self_expr, fields_expr): self.validate_field_count(len(fields_expr)) for expr, varf in zip(fields_expr, [self.field[n] for n in self.field_names]): if (expr.template is not None and hasattr(expr.template, 'is_variant')): check = expr.template.is_variant else: check = self.is_variant if not check(expr, varf): raise TypeError("%r is not a variant of %r" % (expr, varf)) @classmethod def is_variant(cls, expr, varf): if isinstance(expr, UnionExp): return all(cls.is_variant(e, varf) for e in expr.members) if isinstance(expr, IntersectionExp): return any(cls.is_variant(e, varf) for e in expr.members) return varf.is_member(expr) or varf in expr.template.variant_of def validate_predicate(self, predicate): if not isinstance(predicate, Properties): raise TypeError() def update_ast(self, ast): ast['builtin'] = False class Properties(PredicateTemplate): def __init__(self, *include, exclude=()): if len(include) == 1 and isinstance(include[0], (list, tuple, set, frozenset)): include = tuple(include[0]) if type(exclude) is str: exclude = (exclude,) self.include = tuple(include) self.exclude = tuple(exclude) for prop in itertools.chain(self.include, self.exclude): if type(prop) is not str: raise TypeError("%r in %r is not a string." % (prop, self)) def __hash__(self): return hash(frozenset(self.include)) ^ hash(frozenset(self.exclude)) def __eq__(self, other): return (type(self) is type(other) and set(self.include) == set(other.include) and set(self.exclude) == set(other.exclude)) def __repr__(self): args = [] if self.include: args.append(', '.join(repr(s) for s in self.include)) if self.exclude: args.append("exclude=%r" % list(self.exclude)) return "%s(%s)" % (self.__class__.__name__, ', '.join(args)) def is_symbol_subtype(self, other): if type(self) is not type(other): return False return (set(other.include) <= set(self.include) and set(other.exclude) <= set(self.exclude)) def is_symbol_supertype(self, other): if type(self) is not type(other): return False return (set(other.include) >= set(self.include) and set(other.exclude) >= set(self.exclude)) def collapse_intersection(self, other): if type(self) is not type(other): return None new_include_set = set(self.include) | set(other.include) new_exclude_set = set(self.exclude) | set(other.exclude) new_include = [] new_exclude = [] for inc in itertools.chain(self.include, other.include): if inc in new_include_set: new_include.append(inc) new_include_set.remove(inc) for exc in itertools.chain(self.exclude, other.exclude): if exc in new_exclude_set: new_exclude.append(exc) new_exclude_set.remove(exc) return self.__class__(*new_include, exclude=new_exclude).template def get_kind(self): return 'semantic-type' def get_name(self): return self.__class__.__name__ def is_element(self, expr): return True # attached TypeExp checks this def get_union_membership_expr(self, self_expr): return 'predicate-' + self.get_name() def update_ast(self, ast): ast['include'] = list(self.include) ast['exclude'] = list(self.exclude) qiime-2021.8.0/qiime2/core/type/signature.py000066400000000000000000000472651412141660100205340ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import collections import inspect import copy import itertools import qiime2.sdk from .grammar import TypeExp, UnionExp from .meta import TypeVarExp from .collection import List, Set from .primitive import infer_primitive_type from .visualization import Visualization from . import meta from .util import is_semantic_type, is_primitive_type, parse_primitive from ..util import ImmutableBase class __NoValueMeta(type): def __repr__(self): return "NOVALUE" # This sentinel is a class so that it retains the correct memory address when # pickled class _NOVALUE(metaclass=__NoValueMeta): pass class ParameterSpec(ImmutableBase): NOVALUE = _NOVALUE def __init__(self, qiime_type=NOVALUE, view_type=NOVALUE, default=NOVALUE, description=NOVALUE): self.qiime_type = qiime_type self.view_type = view_type self.default = default self.description = description self._freeze_() def has_qiime_type(self): return self.qiime_type is not self.NOVALUE def has_view_type(self): return self.view_type is not self.NOVALUE def has_default(self): return self.default is not self.NOVALUE def has_description(self): return self.description is not self.NOVALUE def duplicate(self, **kwargs): qiime_type = kwargs.pop('qiime_type', self.qiime_type) view_type = kwargs.pop('view_type', self.view_type) default = kwargs.pop('default', self.default) description = kwargs.pop('description', self.description) if kwargs: raise TypeError("Unknown arguments: %r" % kwargs) return ParameterSpec(qiime_type, view_type, default, description) def __repr__(self): return ("ParameterSpec(qiime_type=%r, view_type=%r, default=%r, " "description=%r)" % (self.qiime_type, self.view_type, self.default, self.description)) def __eq__(self, other): return (self.qiime_type == other.qiime_type and self.view_type == other.view_type and self.default == other.default and self.description == other.description) def __ne__(self, other): return not (self == other) class PipelineSignature: builtin_args = ('ctx',) def __init__(self, callable, inputs, parameters, outputs, input_descriptions=None, parameter_descriptions=None, output_descriptions=None): """ Parameters ---------- callable : callable Callable with view type annotations on parameters and return. inputs : dict Parameter name to semantic type. parameters : dict Parameter name to primitive type. outputs : list of tuple Each tuple contains the name of the output (str) and its QIIME type. input_descriptions : dict, optional Input name to description string. parameter_descriptions : dict, optional Parameter name to description string. output_descriptions : dict, optional Output name to description string. """ inputs, parameters, outputs, signature_order = \ self._parse_signature(callable, inputs, parameters, outputs, input_descriptions, parameter_descriptions, output_descriptions) self._assert_valid_inputs(inputs) self._assert_valid_parameters(parameters) self._assert_valid_outputs(outputs) self._assert_valid_views(inputs, parameters, outputs) self.inputs = inputs self.parameters = parameters self.outputs = outputs self.signature_order = signature_order def _parse_signature(self, callable, inputs, parameters, outputs, input_descriptions=None, parameter_descriptions=None, output_descriptions=None): # Initialize dictionaries if non-existant. if input_descriptions is None: input_descriptions = {} if parameter_descriptions is None: parameter_descriptions = {} if output_descriptions is None: output_descriptions = {} # Copy so we can "exhaust" the collections and check for missing params inputs = copy.copy(inputs) parameters = copy.copy(parameters) input_descriptions = copy.copy(input_descriptions) parameter_descriptions = copy.copy(parameter_descriptions) output_descriptions = copy.copy(output_descriptions) builtin_args = list(self.builtin_args) annotated_inputs = collections.OrderedDict() annotated_parameters = collections.OrderedDict() annotated_outputs = collections.OrderedDict() signature_order = collections.OrderedDict() for name, parameter in inspect.signature(callable).parameters.items(): if (parameter.kind == parameter.VAR_POSITIONAL or parameter.kind == parameter.VAR_KEYWORD): raise TypeError("Variadic definitions are unsupported: %r" % name) if builtin_args: if builtin_args[0] != name: raise TypeError("Missing builtin argument %r, got %r" % (builtin_args[0], name)) builtin_args = builtin_args[1:] continue view_type = ParameterSpec.NOVALUE if parameter.annotation is not parameter.empty: view_type = parameter.annotation default = ParameterSpec.NOVALUE if parameter.default is not parameter.empty: default = parameter.default if name in inputs: description = input_descriptions.pop(name, ParameterSpec.NOVALUE) param_spec = ParameterSpec( qiime_type=inputs.pop(name), view_type=view_type, default=default, description=description) annotated_inputs[name] = param_spec signature_order[name] = param_spec elif name in parameters: description = parameter_descriptions.pop(name, ParameterSpec.NOVALUE) param_spec = ParameterSpec( qiime_type=parameters.pop(name), view_type=view_type, default=default, description=description) annotated_parameters[name] = param_spec signature_order[name] = param_spec elif name not in self.builtin_args: raise TypeError("Parameter in callable without QIIME type:" " %r" % name) # we should have popped both of these empty by this point if inputs or parameters: raise TypeError("Callable does not have parameter(s): %r" % (list(inputs) + list(parameters))) if 'return' in callable.__annotations__: output_views = qiime2.core.util.tuplize( callable.__annotations__['return']) if len(output_views) != len(outputs): raise TypeError("Number of registered outputs (%r) does not" " match annotation (%r)" % (len(outputs), len(output_views))) for (name, qiime_type), view_type in zip(outputs, output_views): description = output_descriptions.pop(name, ParameterSpec.NOVALUE) annotated_outputs[name] = ParameterSpec( qiime_type=qiime_type, view_type=view_type, description=description) else: for name, qiime_type in outputs: description = output_descriptions.pop(name, ParameterSpec.NOVALUE) annotated_outputs[name] = ParameterSpec( qiime_type=qiime_type, description=description) # we should have popped the descriptions empty by this point if input_descriptions or parameter_descriptions or output_descriptions: raise TypeError( "Callable does not have parameter(s)/output(s) found in " "descriptions: %r" % [*input_descriptions, *parameter_descriptions, *output_descriptions]) return (annotated_inputs, annotated_parameters, annotated_outputs, signature_order) def _assert_valid_inputs(self, inputs): for input_name, spec in inputs.items(): if not is_semantic_type(spec.qiime_type): raise TypeError( "Input %r must be a semantic QIIME type, not %r" % (input_name, spec.qiime_type)) if not isinstance(spec.qiime_type, (TypeExp, UnionExp)): raise TypeError( "Input %r must be a complete semantic type expression, " "not %r" % (input_name, spec.qiime_type)) if spec.has_default() and spec.default is not None: raise ValueError( "Input %r has a default value of %r. Only a default " "value of `None` is supported for inputs." % (input_name, spec.default)) for var_selector in meta.select_variables(spec.qiime_type): var = var_selector(spec.qiime_type) if not var.input: raise TypeError("An output variable has been associated" " with an input type: %r" % spec.qiime_type) def _assert_valid_parameters(self, parameters): for param_name, spec in parameters.items(): if not is_primitive_type(spec.qiime_type): raise TypeError( "Parameter %r must be a primitive QIIME type, not %r" % (param_name, spec.qiime_type)) if not isinstance(spec.qiime_type, (TypeExp, UnionExp)): raise TypeError( "Parameter %r must be a complete primitive type " "expression, not %r" % (param_name, spec.qiime_type)) if (spec.has_default() and spec.default is not None and spec.default not in spec.qiime_type): raise TypeError("Default value for parameter %r is not of " "semantic QIIME type %r or `None`." % (param_name, spec.qiime_type)) for var_selector in meta.select_variables(spec.qiime_type): var = var_selector(spec.qiime_type) if not var.input: raise TypeError("An output variable has been associated" " with an input type: %r" % spec.qiime_type) def _assert_valid_outputs(self, outputs): if len(outputs) == 0: raise TypeError("%s requires at least one output" % self.__class__.__name__) for output_name, spec in outputs.items(): if not (is_semantic_type(spec.qiime_type) or spec.qiime_type == Visualization): raise TypeError( "Output %r must be a semantic QIIME type or " "Visualization, not %r" % (output_name, spec.qiime_type)) if not isinstance(spec.qiime_type, (TypeVarExp, TypeExp)): raise TypeError( "Output %r must be a complete type expression, not %r" % (output_name, spec.qiime_type)) for var_selector in meta.select_variables(spec.qiime_type): var = var_selector(spec.qiime_type) if not var.output: raise TypeError("An input variable has been associated" " with an input type: %r") def _assert_valid_views(self, inputs, parameters, outputs): for name, spec in itertools.chain(inputs.items(), parameters.items(), outputs.items()): if spec.has_view_type(): raise TypeError( " Pipelines do not support function annotations (found one" " for parameter: %r)." % name) def decode_parameters(self, **kwargs): params = {} for key, spec in self.parameters.items(): if (spec.has_default() and spec.default is None and kwargs[key] is None): params[key] = None else: params[key] = parse_primitive(spec.qiime_type, kwargs[key]) return params def check_types(self, **kwargs): for name, spec in self.signature_order.items(): parameter = kwargs[name] # A type mismatch is unacceptable unless the value is None # and this parameter's default value is None. if ((parameter not in spec.qiime_type) and not (spec.has_default() and spec.default is None and parameter is None)): if isinstance(parameter, qiime2.sdk.Visualization): raise TypeError( "Parameter %r received a Visualization as an " "argument. Visualizations may not be used as inputs." % name) elif isinstance(parameter, qiime2.sdk.Artifact): raise TypeError( "Parameter %r requires an argument of type %r. An " "argument of type %r was passed." % ( name, spec.qiime_type, parameter.type)) elif isinstance(parameter, qiime2.Metadata): raise TypeError( "Parameter %r received Metadata as an " "argument, which is incompatible with parameter " "type: %r" % (name, spec.qiime_type)) else: # handle primitive types raise TypeError( "Parameter %r received %r as an argument, which is " "incompatible with parameter type: %r" % (name, parameter, spec.qiime_type)) def solve_output(self, **kwargs): solved_outputs = None for _, spec in itertools.chain(self.inputs.items(), self.parameters.items(), self.outputs.items()): if list(meta.select_variables(spec.qiime_type)): break # a variable exists, do the hard work else: # no variables solved_outputs = self.outputs if solved_outputs is None: inputs = {**{k: s.qiime_type for k, s in self.inputs.items()}, **{k: s.qiime_type for k, s in self.parameters.items()}} outputs = {k: s.qiime_type for k, s in self.outputs.items()} input_types = { k: self._infer_type(k, v) for k, v in kwargs.items()} solved = meta.match(input_types, inputs, outputs) solved_outputs = collections.OrderedDict( (k, s.duplicate(qiime_type=solved[k])) for k, s in self.outputs.items()) for output_name, spec in solved_outputs.items(): if not spec.qiime_type.is_concrete(): raise TypeError( "Solved output %r must be a concrete type, not %r" % (output_name, spec.qiime_type)) return solved_outputs def _infer_type(self, key, value): if value is None: if key in self.inputs: return self.inputs[key].qiime_type elif key in self.parameters: return self.parameters[key].qiime_type # Shouldn't happen: raise ValueError("Parameter passed not consistent with signature.") if type(value) is list: inner = UnionExp((self._infer_type(key, v) for v in value)) return List[inner.normalize()] if type(value) is set: inner = UnionExp((self._infer_type(key, v) for v in value)) return Set[inner.normalize()] if isinstance(value, qiime2.sdk.Artifact): return value.type else: return infer_primitive_type(value) def __repr__(self): lines = [] for group in 'inputs', 'parameters', 'outputs': lookup = getattr(self, group) lines.append('%s:' % group) for name, spec in lookup.items(): lines.append(' %s: %r' % (name, spec)) return '\n'.join(lines) def __eq__(self, other): return (type(self) is type(other) and self.inputs == other.inputs and self.parameters == other.parameters and self.outputs == other.outputs and self.signature_order == other.signature_order) def __ne__(self, other): return not (self == other) class MethodSignature(PipelineSignature): builtin_args = () def _assert_valid_outputs(self, outputs): super()._assert_valid_outputs(outputs) # Assert all output types are semantic types. The parent class is less # strict in its output type requirements. for output_name, spec in outputs.items(): if not is_semantic_type(spec.qiime_type): raise TypeError( "Output %r must be a semantic QIIME type, not %r" % (output_name, spec.qiime_type)) def _assert_valid_views(self, inputs, parameters, outputs): for name, spec in itertools.chain(inputs.items(), parameters.items(), outputs.items()): if not spec.has_view_type(): raise TypeError("Method is missing a function annotation for" " parameter: %r" % name) class VisualizerSignature(PipelineSignature): builtin_args = ('output_dir',) def __init__(self, callable, inputs, parameters, input_descriptions=None, parameter_descriptions=None): outputs = [('visualization', Visualization)] output_descriptions = None super().__init__(callable, inputs, parameters, outputs, input_descriptions, parameter_descriptions, output_descriptions) def _assert_valid_outputs(self, outputs): super()._assert_valid_outputs(outputs) output = outputs['visualization'] if output.has_view_type() and output.view_type is not None: raise TypeError( "Visualizer callable cannot return anything. Its return " "annotation must be `None`, not %r. Write output to " "`output_dir`." % output.view_type) def _assert_valid_views(self, inputs, parameters, outputs): for name, spec in itertools.chain(inputs.items(), parameters.items()): if not spec.has_view_type(): raise TypeError("Visualizer is missing a function annotation" " for parameter: %r" % name) qiime-2021.8.0/qiime2/core/type/template.py000066400000000000000000000110771412141660100203360ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from abc import ABCMeta, abstractmethod import itertools import inspect from qiime2.core.type.grammar import (IncompleteExp, TypeExp, PredicateExp, IntersectionExp) class _BaseTemplate(metaclass=ABCMeta): public_proxy = () is_template = True # for smoke-testing @property def __signature__(self): return inspect.signature(self.__init__) @abstractmethod def __eq__(self, other): raise NotImplementedError def __hash__(self, other): return 0 def get_name_expr(self, self_expr): return self.get_name() @abstractmethod def get_name(self): raise NotImplementedError def get_kind_expr(self, self_expr): return self.get_kind() @abstractmethod def get_kind(self): raise NotImplementedError def get_union_membership_expr(self, self_expr): return self.get_kind_expr(self_expr) def is_element_expr(self, self_expr, value): return self.is_element(value) @abstractmethod def is_element(self, value): raise NotImplementedError def collapse_intersection(self, other): return NotImplemented def is_symbol_subtype_expr(self, self_expr, other_expr): return self.is_symbol_subtype(other_expr.template) def is_symbol_subtype(self, other): return self.get_name() == other.get_name() def is_symbol_supertype_expr(self, self_expr, other_expr): return self.is_symbol_supertype(other_expr.template) def is_symbol_supertype(self, other): return self.get_name() == other.get_name() def update_ast_expr(self, self_expr, ast): self.update_ast(ast) def update_ast(self, ast): pass # default is to do nothing class TypeTemplate(_BaseTemplate): def __new__(cls, *args, _pickle=False, **kwargs): self = super().__new__(cls) if _pickle: return self self.__init__(*args, **kwargs) if list(self.get_field_names()): return IncompleteExp(self) else: return TypeExp(self) def __getnewargs_ex__(self): return ((), {'_pickle': True}) def get_field_names_expr(self, expr): return self.get_field_names() @abstractmethod def get_field_names(self): raise NotImplementedError def validate_fields_expr(self, self_expr, fields_expr): self.validate_field_count(len(fields_expr)) for expr, name in itertools.zip_longest( fields_expr, self.get_field_names_expr(self_expr), fillvalue=IntersectionExp()): if expr.template is None: for exp in expr.members: if exp.template is None: for ex in exp.members: self.validate_field(name, ex.template) else: self.validate_field(name, exp.template) else: self.validate_field(name, expr.template) def validate_field_count(self, count): exp = len(self.get_field_names()) if count != exp: raise TypeError("Expected only %r fields, got %r" % (exp, count)) @abstractmethod def validate_field(self, name, field): raise NotImplementedError def validate_predicate_expr(self, self_expr, predicate_expr): if predicate_expr.template is None: for predicate in predicate_expr.members: self.validate_predicate_expr(self_expr, predicate) else: self.validate_predicate(predicate_expr.template) @abstractmethod def validate_predicate(self, predicate): raise NotImplementedError class PredicateTemplate(_BaseTemplate): def __new__(cls, *args, _pickle=False, **kwargs): self = super().__new__(cls) if _pickle: return self self.__init__(*args, **kwargs) return PredicateExp(self) def __getnewargs_ex__(self): return ((), {'_pickle': True}) @abstractmethod def __hash__(self, other): raise NotImplementedError @abstractmethod def is_symbol_subtype(self, other): raise NotImplementedError @abstractmethod def is_symbol_supertype(self, other): raise NotImplementedError qiime-2021.8.0/qiime2/core/type/tests/000077500000000000000000000000001412141660100173055ustar00rootroot00000000000000qiime-2021.8.0/qiime2/core/type/tests/__init__.py000066400000000000000000000005351412141660100214210ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- qiime-2021.8.0/qiime2/core/type/tests/test_collection.py000066400000000000000000000123741412141660100230600ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest from qiime2.core.type import ( is_collection_type, is_primitive_type, is_semantic_type, Set, List, SemanticType, Int, Metadata, MetadataColumn, Categorical, Numeric, Range) class TestIsTypes(unittest.TestCase): def test_list_semantic_type(self): Foo = SemanticType('Foo') self.assertTrue(is_collection_type(List[Foo])) self.assertTrue(is_semantic_type(List[Foo])) self.assertFalse(is_primitive_type(List[Foo])) def test_set_semantic_type(self): Foo = SemanticType('Foo') self.assertTrue(is_collection_type(Set[Foo])) self.assertTrue(is_semantic_type(Set[Foo])) self.assertFalse(is_primitive_type(Set[Foo])) def test_list_primitive_type(self): self.assertTrue(is_collection_type(List[Int % Range(5)])) self.assertTrue(is_primitive_type(List[Int % Range(5)])) self.assertFalse(is_semantic_type(List[Int % Range(5)])) def test_set_primitive_type(self): self.assertTrue(is_collection_type(Set[Int % Range(5)])) self.assertTrue(is_primitive_type(Set[Int % Range(5)])) self.assertFalse(is_semantic_type(Set[Int % Range(5)])) class TestCollectionBase(unittest.TestCase): def test_no_list_metadata(self): with self.assertRaisesRegex(TypeError, 'metadata'): List[Metadata] def test_no_set_metadata(self): with self.assertRaisesRegex(TypeError, 'metadata'): List[Metadata] def test_no_list_metadata_column(self): with self.assertRaisesRegex(TypeError, 'metadata'): List[MetadataColumn[Categorical]] with self.assertRaisesRegex(TypeError, 'metadata'): List[MetadataColumn[Numeric]] def test_no_set_metadata_column(self): with self.assertRaisesRegex(TypeError, 'metadata'): Set[MetadataColumn[Categorical]] with self.assertRaisesRegex(TypeError, 'metadata'): Set[MetadataColumn[Numeric]] def test_no_nesting_list_list(self): with self.assertRaisesRegex(TypeError, 'nest'): List[List[Int]] def test_no_nesting_set_set(self): with self.assertRaisesRegex(TypeError, 'nest'): Set[Set[Int]] def test_no_nesting_mixed(self): with self.assertRaisesRegex(TypeError, 'nest'): List[Set[Int]] class TestCollectionExpression(unittest.TestCase): def test_bad_union(self): with self.assertRaisesRegex(TypeError, 'not union'): List[Int] | Set[Int] def test_union_inside_collection(self): Foo = SemanticType('Foo') Bar = SemanticType('Bar') self.assertTrue(List[Foo] <= List[Foo | Bar]) def test_no_predicate(self): with self.assertRaisesRegex(TypeError, 'cannot be applied'): List[Int] % Range(5) def is_concrete(self): Foo = SemanticType('Foo') self.assertFalse(List[Foo].is_concrete()) self.assertFalse(Set[Int].is_concrete()) def test_to_ast_semantic(self): Foo = SemanticType('Foo') ast = List[Foo].to_ast() self.assertEqual(ast['fields'][0], Foo.to_ast()) def test_to_ast_primitive(self): ast = List[Int % Range(5)].to_ast() self.assertEqual(ast['fields'][0], (Int % Range(5)).to_ast()) def test_contains_list_primitive(self): self.assertTrue([1, 2, 3] in List[Int]) self.assertTrue([-1, 2, 3] in List[Int]) self.assertFalse([-1, 2, 3] in List[Int % Range(0, 5)]) self.assertFalse([1, 1.1, 1.11] in List[Int]) self.assertFalse({1, 2, 3} in List[Int]) self.assertFalse(object() in List[Int]) def test_contains_set_primitive(self): self.assertTrue({1, 2, 3} in Set[Int]) self.assertTrue({-1, 2, 3} in Set[Int]) self.assertFalse({-1, 2, 3} in Set[Int % Range(0, 5)]) self.assertFalse({1, 1.1, 1.11} in Set[Int]) self.assertFalse([1, 2, 3] in Set[Int]) self.assertFalse(object() in Set[Int]) def test_variant_of_field_members(self): Bar = SemanticType('Bar') Foo = SemanticType('Foo', field_names='foo', field_members={'foo': List[Bar]}) with self.assertRaisesRegex(TypeError, 'is not a variant'): Foo[List[Bar]] def test_variant_of_alt(self): Foo = SemanticType('Foo', field_names='foo') Bar = SemanticType('Bar', variant_of=Foo.field['foo']) with self.assertRaisesRegex(TypeError, 'is not a variant'): Foo[Set[Bar]] def test_encode_decode_set(self): value = List[Int].decode("[1, 2, 3]") self.assertEqual(value, [1, 2, 3]) json = List[Int].encode(value) self.assertEqual(json, "[1, 2, 3]") def test_encode_decode_list(self): value = Set[Int].decode("[1, 2, 3]") self.assertEqual(value, {1, 2, 3}) json = Set[Int].encode(value) self.assertEqual(json, "[1, 2, 3]") if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/type/tests/test_grammar.py000066400000000000000000000563471412141660100223630ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import pickle import unittest import collections.abc import qiime2.core.type.grammar as grammar import qiime2.core.type.template as template class _MockBase: public_proxy = 'example', def __init__(self, name, fields=()): self.test_data = {} self.name = name self.fields = fields def track_call(func): def wrapped(self, *args, **kwargs): self.test_data[func.__name__] = True return func(self, *args, **kwargs) return wrapped @track_call def __eq__(self, other): return id(self) == id(other) @track_call def __hash__(self): return hash(id(self)) @track_call def get_field_names(self): return self.fields @track_call def get_name(self): return self.name @track_call def get_kind(self): return "tester" @track_call def validate_union(self, other): pass @track_call def validate_intersection(self, other): pass @track_call def is_element(self, value): return self.name.startswith(value) @track_call def collapse_intersection(self, other): return super().collapse_intersection(other) @track_call def is_symbol_subtype(self, other): return self.name == other.name @track_call def is_symbol_supertype(self, other): return self.name == other.name @track_call def update_ast(self, ast): ast['extra_junk'] = self.name def validate_field(self, name, field): self.test_data['validate_field'] = name if field.name == 'InvalidMember': raise TypeError('InvalidMember cannot be used') @track_call def validate_predicate(self, predicate): pass @track_call def example(self): return ... class MockTemplate(_MockBase, template.TypeTemplate): pass class MockPredicate(_MockBase, template.PredicateTemplate): def __init__(self, name, alphabetize=False): self.alphabetize = alphabetize super().__init__(name) def __repr__(self): return self.name def is_symbol_subtype(self, other): if not self.alphabetize or not other.alphabetize: return super().is_symbol_subtype(other) return self.name <= other.name def is_symbol_supertype(self, other): if not self.alphabetize or not other.alphabetize: return super().is_symbol_supertype(other) return self.name >= other.name def get_kind(self): return "tester-predicate" class TestIncompleteExp(unittest.TestCase): def IncompleteExp(self, name, fields): expr = MockTemplate(name, fields) self.assertIsInstance(expr, grammar.IncompleteExp) return expr def test_construction_sanity(self): expr = MockTemplate('foo') # TypeExpr with self.assertRaisesRegex(ValueError, "no fields"): # template has no fields, so putting it in an IncompleteExp # doesn't make sense expr = grammar.IncompleteExp(expr.template) def test_mod(self): with self.assertRaisesRegex(TypeError, 'predicate'): self.IncompleteExp('foo', ('a',)) % ... def test_or(self): with self.assertRaisesRegex(TypeError, 'union'): self.IncompleteExp('foo', ('a',)) | ... def test_and(self): with self.assertRaisesRegex(TypeError, 'intersect'): self.IncompleteExp('foo', ('a',)) & ... def test_repr(self): self.assertEqual(repr(self.IncompleteExp('Example', ('foo',))), 'Example[{foo}]') self.assertEqual(repr(self.IncompleteExp('Example', ('f', 'b'))), 'Example[{f}, {b}]') def test_le(self): expr_a = self.IncompleteExp('Foo', ('a',)) expr_b = self.IncompleteExp('Bar', ('b',)) with self.assertRaisesRegex(TypeError, 'missing arguments'): expr_a <= expr_b def test_ge(self): expr_a = self.IncompleteExp('Foo', ('a',)) expr_b = self.IncompleteExp('Bar', ('b',)) with self.assertRaisesRegex(TypeError, 'missing arguments'): expr_a >= expr_b def test_in(self): expr_a = self.IncompleteExp('Foo', ('a',)) with self.assertRaisesRegex(TypeError, 'missing arguments'): ... in expr_a def test_field_w_typeexp(self): expr_a = self.IncompleteExp('Foo', ('baz',)) expr_inner = MockTemplate('Bar') result = expr_a[expr_inner] self.assertEqual(repr(result), 'Foo[Bar]') self.assertIsInstance(result, grammar.TypeExp) self.assertEqual(expr_a.template.test_data['validate_field'], 'baz') def test_field_w_incompleteexp(self): expr_a = self.IncompleteExp('Foo', ('a',)) expr_b = self.IncompleteExp('Bar', ('b',)) with self.assertRaisesRegex(TypeError, 'complete type expression'): expr_a[expr_b] def test_field_w_nonsense(self): expr_a = self.IncompleteExp('Foo', ('a',)) with self.assertRaisesRegex(TypeError, 'complete type expression'): expr_a[...] def test_field_wrong_length(self): X = MockTemplate('X') C = self.IncompleteExp('C', ['foo', 'bar']) with self.assertRaisesRegex(TypeError, '1'): C[X] C = self.IncompleteExp('C', ['foo']) with self.assertRaisesRegex(TypeError, '2'): C[X, X] def test_field_nested_expression(self): X = MockTemplate('X') C = self.IncompleteExp('C', ['foo', 'bar']) self.assertEqual(repr(C[X, C[C[X, X], X]]), 'C[X, C[C[X, X], X]]') def test_field_invalid_member(self): C = self.IncompleteExp('C', ['foo']) InvalidMember = MockTemplate('InvalidMember') with self.assertRaisesRegex(TypeError, 'InvalidMember'): C[InvalidMember] def test_field_union(self): X = MockTemplate('X') Y = MockTemplate('Y') Z = MockTemplate('Z') C = self.IncompleteExp('C', ['foo']) result = C[X | Y | Z] self.assertEqual(repr(result), "C[X | Y | Z]") def test_field_invalid_union(self): X = MockTemplate('X') InvalidMember = MockTemplate('InvalidMember') Z = MockTemplate('Z') C = self.IncompleteExp('C', ['foo']) with self.assertRaisesRegex(TypeError, 'InvalidMember'): C[X | InvalidMember | Z] def test_field_insane(self): X = MockTemplate('X') Y = MockTemplate('Y') Z = MockTemplate('Z') InvalidIntersection = grammar.IntersectionExp( members=(MockTemplate('InvalidMember'), Y)) C = self.IncompleteExp('C', ['foo']) with self.assertRaisesRegex(TypeError, 'InvalidMember'): C[X | InvalidIntersection | Z] def test_iter_symbols(self): expr = self.IncompleteExp('Example', ('foo',)) self.assertEqual(list(expr.iter_symbols()), ['Example']) def test_is_concrete(self): expr = self.IncompleteExp('Example', ('foo',)) self.assertFalse(expr.is_concrete()) def test_pickle(self): expr = self.IncompleteExp('Example', ('foo',)) clone = pickle.loads(pickle.dumps(expr)) self.assertEqual(expr, clone) def test_proxy(self): expr = self.IncompleteExp('Example', ('foo',)) self.assertIs(expr.example(), ...) self.assertTrue(expr.template.test_data['example']) def test_eq_nonsense(self): expr_a = self.IncompleteExp('Example', ('foo',)) self.assertEqual(expr_a.__eq__(...), NotImplemented) def test_hash_eq_equals(self): expr_a = self.IncompleteExp('Example', ('foo',)) expr_b = self.IncompleteExp('Example', ('foo',)) self.assertEqual(hash(expr_a), hash(expr_b)) self.assertEqual(expr_a, expr_b) self.assertTrue(expr_a.equals(expr_b)) def test_not_hash_eq_equals_field_mismatch(self): expr_a = self.IncompleteExp('Example', ('foo',)) expr_b = self.IncompleteExp('Example', ('something_else',)) self.assertNotEqual(hash(expr_a), hash(expr_b)) self.assertNotEqual(expr_a, expr_b) self.assertFalse(expr_a.equals(expr_b)) class TestTypeExp(unittest.TestCase): def test_hashable(self): X = MockTemplate('X') Y = MockTemplate('Y', fields=('a',)) Z = MockTemplate('Z') P = MockPredicate('P') self.assertIsInstance(X, collections.abc.Hashable) # There really shouldn't be a collision between these: self.assertNotEqual(hash(X), hash(Z % P)) self.assertEqual(Y[X], Y[X]) self.assertEqual(hash(Y[X]), hash(Y[X])) def test_eq_nonsense(self): X = MockTemplate('X') self.assertIs(X.__eq__(42), NotImplemented) self.assertFalse(X == 42) def test_eq_different_instances(self): X = MockTemplate('X') X_ = MockTemplate('X') self.assertIsNot(X, X_) self.assertEqual(X, X_) def test_field(self): X = MockTemplate('X') with self.assertRaisesRegex(TypeError, 'fields'): X['scikit-bio/assets/.no.gif'] Y = MockTemplate('Y', fields=('foo',))[X] with self.assertRaisesRegex(TypeError, 'fields'): Y[';-)'] def test_repr(self): Face0 = MockTemplate('(o_-)') Face1 = MockTemplate('-_-') Exclaim0 = MockTemplate('!') Exclaim1 = MockTemplate('!', fields=('a',)) Exclaim2 = MockTemplate('!', fields=('a', 'b')) Face2 = MockPredicate('(o_o)') Face3 = grammar.IntersectionExp( (MockPredicate('='), MockPredicate('='))) # repr -> "= & =" Face4 = grammar.UnionExp( (MockPredicate('<'), MockPredicate('<'))) self.assertEqual(repr(Exclaim0), '!') self.assertEqual(repr(Exclaim1[Face1]), '![-_-]') self.assertEqual(repr(Exclaim2[Face1, Exclaim0]), '![-_-, !]') self.assertEqual(repr(Exclaim2[Face1, Exclaim0] % Face2), '![-_-, !] % (o_o)') self.assertEqual(repr(Face0 % Face2), '(o_-) % (o_o)') self.assertEqual(repr(Face0 % Face3), '(o_-) % (= & =)') self.assertEqual(repr(Exclaim2[Face1, Exclaim0] % Face3), '![-_-, !] % (= & =)') self.assertEqual(repr(Exclaim2[Face1, Exclaim0] % Face4), '![-_-, !] % (< | <)') def test_full_predicate(self): expr = MockTemplate('Foo') predicate = MockPredicate('Bar') self.assertIs((expr % predicate).full_predicate, predicate) self.assertTrue(expr.full_predicate.is_top()) def test_in(self): expr = MockTemplate('Foo') self.assertIn('Foo', expr) self.assertTrue(expr.template.test_data['is_element']) expr = MockTemplate('Bar') % MockPredicate('Baz') self.assertIn('Ba', expr) self.assertTrue(expr.template.test_data['is_element']) self.assertTrue(expr.predicate.template.test_data['is_element']) def test_not_in(self): expr = MockTemplate('Foo') self.assertNotIn('Bar', expr) expr = MockTemplate('Bar') % MockPredicate('Baz') self.assertNotIn('Bar', expr) # Bar not a substring of Baz def test_mod(self): Bar = MockTemplate('Bar') Baz = MockPredicate('Baz') noop = Bar % grammar.IntersectionExp() self.assertIs(Bar, noop) with self.assertRaisesRegex(TypeError, 'predicate'): (Bar % Baz) % Baz with self.assertRaisesRegex(TypeError, 'right-hand'): Baz % Bar def test_iter(self): Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') C2 = MockTemplate('C2', fields=('a', 'b')) P = MockPredicate('P') Q = MockPredicate('Q') self.assertEqual( { Foo, C2[Foo, Foo], C2[Foo, C2[Foo % (P & Q), Bar]], C2[Foo, C2[Foo % (P & Q), Foo]], C2[Foo, C2[Bar, Bar]], C2[Foo, C2[Bar, Foo]] }, set( Foo | C2[Foo, Foo | C2[Foo % (P & Q) | Bar, Bar | Foo]] ) ) def test_iter_symbols(self): Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') C2 = MockTemplate('C2', fields=('a', 'b')) P = MockPredicate('P') Q = MockPredicate('Q') self.assertEqual( {'Foo', 'C2', 'Bar'}, set((Foo | C2[Foo, Foo | C2[Foo % (P & Q) | Bar, Bar | Foo]] ).iter_symbols())) def test_is_concrete(self): Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') C2 = MockTemplate('C2', fields=('a', 'b')) P = MockPredicate('P') Q = MockPredicate('Q') self.assertTrue(Foo.is_concrete()) self.assertTrue(C2[Foo, Bar].is_concrete()) self.assertTrue((Foo % P).is_concrete()) self.assertTrue((C2[Foo % P, Bar] % Q).is_concrete()) def test_not_concrete(self): Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') C2 = MockTemplate('C2', fields=('a', 'b')) P = MockPredicate('P') Q = MockPredicate('Q') AnnoyingToMake = grammar.TypeExp( Foo.template, predicate=grammar.UnionExp((P, Q))) self.assertFalse((Foo | Bar).is_concrete()) self.assertFalse(C2[Foo | Bar, Bar].is_concrete()) self.assertFalse(C2[Foo, Bar | Foo].is_concrete()) self.assertFalse(AnnoyingToMake.is_concrete()) def test_to_ast(self): Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') C2 = MockTemplate('C2', fields=('a', 'b')) P = MockPredicate('P') Q = MockPredicate('Q') self.assertEqual( (C2[Foo | Bar % P, Foo % (P & Q)] % (P | Q)).to_ast(), { 'type': 'expression', 'builtin': True, 'name': 'C2', 'predicate': { 'type': 'union', 'members': [ { 'type': 'predicate', 'name': 'P', 'extra_junk': 'P' }, { 'type': 'predicate', 'name': 'Q', 'extra_junk': 'Q' } ] }, 'fields': [ { 'type': 'union', 'members': [ { 'type': 'expression', 'builtin': True, 'name': 'Foo', 'predicate': None, 'fields': [], 'extra_junk': 'Foo' }, { 'type': 'expression', 'builtin': True, 'name': 'Bar', 'predicate': { 'type': 'predicate', 'name': 'P', 'extra_junk': 'P' }, 'fields': [], 'extra_junk': 'Bar' } ] }, { 'type': 'expression', 'builtin': True, 'name': 'Foo', 'predicate': { 'type': 'intersection', 'members': [ { 'type': 'predicate', 'name': 'P', 'extra_junk': 'P' }, { 'type': 'predicate', 'name': 'Q', 'extra_junk': 'Q' } ] }, 'fields': [], 'extra_junk': 'Foo' } ], 'extra_junk': 'C2' }) class TestIntersection(unittest.TestCase): def test_basic(self): P = MockPredicate('P') Q = MockPredicate('Q') result = P & Q self.assertEqual(repr(result), "P & Q") def test_subtype(self): P = MockPredicate('P', alphabetize=True) Q = MockPredicate('Q', alphabetize=True) self.assertIs(Q & P, P) self.assertIs(P & Q, P) def test_identity(self): x = grammar.IntersectionExp() self.assertTrue(x.is_top()) self.assertEqual(repr(x), 'IntersectionExp()') self.assertEqual(x.kind, 'identity') self.assertEqual(x.name, '') def test_in(self): Tree = MockPredicate('Tree') Trick = MockPredicate('Trick') Trek = MockPredicate('Trek') Truck = MockPredicate('Truck') self.assertIn('Tr', Tree & Trick & Trek & Truck) self.assertNotIn('Tre', Tree & Trick & Trek & Truck) self.assertIn('Tre', Tree & Trek) self.assertNotIn('Tree', Tree & Trek) self.assertNotIn('Nope', Tree & Truck) def test_distribution(self): C2 = MockTemplate('C2', fields=('a', 'b')) Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') P = MockPredicate('P') Q = MockPredicate('Q') R = MockPredicate('R') S = MockPredicate('S') self.assertTrue( ((P | Q) & (R | S)).equals(P & R | P & S | Q & R | Q & S)) self.assertTrue( ((P | Q) & (R & S)).equals(P & R & S | Q & R & S)) self.assertEqual(Foo & Bar, grammar.UnionExp()) self.assertEqual(C2[Foo, Bar] & C2[Foo, Foo], grammar.UnionExp()) self.assertTrue((C2[Foo % P, Bar] & C2[Foo % Q, Bar]).equals( C2[Foo % (P & Q), Bar])) class TestUnion(unittest.TestCase): def test_basic(self): P = MockPredicate('P') Q = MockPredicate('Q') result = P | Q self.assertEqual(repr(result), "P | Q") def test_subtype(self): P = MockPredicate('P', alphabetize=True) Q = MockPredicate('Q', alphabetize=True) self.assertIs(Q | P, Q) self.assertIs(P | Q, Q) def test_identity(self): x = grammar.UnionExp() self.assertTrue(x.is_bottom()) self.assertEqual(repr(x), 'UnionExp()') self.assertEqual(x.kind, 'identity') self.assertEqual(x.name, '') def test_in(self): Bat = MockTemplate('Bat') Cat = MockTemplate('Cat') self.assertIn('C', Bat | Cat) self.assertIn('B', Bat | Cat) self.assertNotIn('D', Bat | Cat) def test_distribution(self): Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') P = MockPredicate('P') Q = MockPredicate('Q') R = MockPredicate('R') S = MockPredicate('S') self.assertTrue( ((P | Q) | (R | S)).equals(P | Q | R | S)) self.assertTrue( (P | (Q | R)).equals(P | Q | R)) self.assertEqual( repr(Foo % P | Bar % Q | Foo % R | Bar % S), 'Foo % (P | R) | Bar % (Q | S)') self.assertEqual( repr(grammar.UnionExp( [Foo % P, Bar % Q, Foo % R, Bar % S]).normalize()), 'Foo % (P | R) | Bar % (Q | S)') def test_maximum_antichain(self): P = MockPredicate('P', alphabetize=True) Q = MockPredicate('Q', alphabetize=True) X = MockPredicate('X') Y = MockPredicate('Y') self.assertEqual(repr((P | X) | (Q | Y)), 'X | Q | Y') self.assertTrue(repr(X & Y | (P & X | Q) | P & X & Q & Y), 'X & Y | Q') self.assertTrue(repr(X & Y | P & X | (X | Q)), 'X | Q') class TestSubtyping(unittest.TestCase): def assertStrongSubtype(self, X, Y): self.assertLessEqual(X, Y) # Should be the same in either direction self.assertGreaterEqual(Y, X) # X and Y would be equal otherwise self.assertFalse(X >= Y) def assertNoRelation(self, X, Y): XsubY = X <= Y self.assertEqual(XsubY, Y >= X) YsubX = Y <= X self.assertEqual(YsubX, X >= Y) self.assertFalse(XsubY or YsubX) def test_equal(self): Foo = MockTemplate('Foo') Foo2 = MockTemplate('Foo') self.assertTrue(Foo.equals(Foo2)) self.assertTrue(Foo2.equals(Foo)) def test_symbol_subtype(self): P = MockPredicate('P', alphabetize=True) Q = MockPredicate('Q', alphabetize=True) self.assertStrongSubtype(P, Q) def test_field(self): C2 = MockTemplate('C2', fields=('a', 'b')) Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') Baz = MockTemplate('Baz') self.assertStrongSubtype(C2[Foo, Bar], C2[Foo | Bar, Bar | Baz]) self.assertNoRelation(C2[Baz, Bar], C2[Foo | Bar, Bar | Baz]) self.assertStrongSubtype(C2[Foo, Bar], Bar | C2[Foo, Bar]) self.assertNoRelation(Baz | C2[Foo, Bar], Bar | C2[Foo, Bar]) self.assertStrongSubtype(C2[Foo, Bar | Baz], Bar | C2[Foo, Foo | Bar | Baz]) self.assertNoRelation(C2[Foo | Baz, Bar | Baz], Bar | C2[Foo, Bar]) def test_generic_subtype(self): C1 = MockTemplate('C1', fields=('a',)) Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') self.assertStrongSubtype(C1[Foo] | C1[Bar], C1[Foo | Bar]) self.assertStrongSubtype(C1[C1[Foo] | C1[Bar]], C1[C1[Foo | Bar]]) self.assertStrongSubtype(C1[C1[Foo]] | C1[C1[Bar]], C1[C1[Foo] | C1[Bar]]) self.assertStrongSubtype(C1[C1[Foo]] | C1[C1[Bar]], C1[C1[Foo | Bar]]) def test_predicate_intersection(self): Foo = MockTemplate('Foo') P = MockPredicate('P') Q = MockPredicate('Q') self.assertStrongSubtype(Foo % (P & Q), Foo) self.assertStrongSubtype(Foo % (P & Q), Foo % P) self.assertStrongSubtype(Foo % (P & Q), Foo % Q) def test_union_of_intersections(self): P = MockPredicate('P') Q = MockPredicate('Q') R = MockPredicate('R') self.assertStrongSubtype(P & Q | Q & R, Q | P | R) self.assertStrongSubtype(P & Q | Q & R, Q | P & R) self.assertStrongSubtype(P & Q & R, P & Q | Q & R) self.assertStrongSubtype(P & Q & R, P & Q | Q & R | R & P) def test_type_union(self): Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') Baz = MockTemplate('Baz') self.assertStrongSubtype(Foo | Bar, Foo | Bar | Baz) self.assertNoRelation(Foo | Baz, Baz | Bar) def test_predicate_union(self): P = MockPredicate('P') Q = MockPredicate('Q') R = MockPredicate('R') self.assertStrongSubtype(P | Q, P | Q | R) self.assertNoRelation(P | R, P | Q) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/type/tests/test_meta.py000066400000000000000000000374511412141660100216560ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest import pickle import qiime2.core.type.collection as col import qiime2.core.type.meta as meta from qiime2.core.type.tests.test_grammar import MockTemplate, MockPredicate class TestSelect(unittest.TestCase): def test_select_simple(self): Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') X, Y = meta.TypeMap({ Foo: Bar }) sel, = meta.select_variables(X) self.assertIs(sel(X), X) self.assertIs(sel(Foo), Foo) self.assertIs(sel(X, swap=Foo), Foo) def test_select_inside_field(self): C2 = MockTemplate('C2', fields=('a', 'b')) Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') X, Y = meta.TypeMap({ Foo: Bar }) sel, = meta.select_variables(C2[X, Foo]) self.assertIs(sel(C2[X, Bar]), X) self.assertIs(sel(C2[Bar, Foo]), Bar) self.assertEqual(sel(C2[X, Foo], swap=Foo), C2[Foo, Foo]) def test_select_predicate(self): Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') P = MockPredicate('P') Q = MockPredicate('Q') X, Y = meta.TypeMap({ P & Q: Foo, P: Bar, Q: Foo }) sel, = meta.select_variables(Foo % X) self.assertIs(sel(Foo % X), X) self.assertIs(sel(Foo % P), P) self.assertEqual(sel(Foo % X, swap=Q), Foo % Q) def test_multiple_select(self): C2 = MockTemplate('C2', fields=('a', 'b')) Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') P = MockPredicate('P') Q = MockPredicate('Q') R = MockPredicate('R') X1, Y1 = meta.TypeMap({ Foo: Bar }) X2, Y2 = meta.TypeMap({ P & Q: Foo, P: Bar, Q: Foo }) expr = C2[X1, Foo % X2] % X2 pred_sel, field_sel, field_pred_sel = meta.select_variables(expr) self.assertIs(pred_sel(expr), X2) self.assertIs(pred_sel(C2[Bar, Foo % Q] % P), P) self.assertEqual(pred_sel(expr, swap=R), C2[X1, Foo % X2] % R) self.assertIs(field_sel(expr), X1) self.assertIs(field_sel(C2[Bar, Foo]), Bar) self.assertEqual(field_sel(expr, swap=Foo), C2[Foo, Foo % X2] % X2) self.assertIs(field_pred_sel(expr), X2) self.assertIs(field_pred_sel(C2[Bar, Foo % Q] % P), Q) self.assertEqual(field_pred_sel(expr, swap=R), C2[X1, Foo % R] % X2) class TestTypeMap(unittest.TestCase): def test_missing_branch_requested(self): P = MockPredicate('P') Q = MockPredicate('Q') with self.assertRaisesRegex(ValueError, 'Ambiguous'): meta.TypeMap({P: P, Q: Q}) def test_mismatched_pieces(self): P = MockPredicate('P') Bar = MockTemplate('Bar') with self.assertRaisesRegex(ValueError, 'in the same'): meta.TypeMap({P: P, Bar: Bar}) def test_iter_sorted(self): P = MockPredicate('P', alphabetize=True) Q = MockPredicate('Q', alphabetize=True) Other = MockPredicate('Other') X, Y = meta.TypeMap({ P & Other: Other, P: P, Q & Other: Other, Q: Q, Other: Other }) mapping = X.mapping self.assertEqual( list(mapping.lifted), [col.Tuple[P & Other], col.Tuple[P], col.Tuple[Q & Other], col.Tuple[Q], col.Tuple[Other]]) def test_variables(self): P = MockPredicate('P') Q = MockPredicate('Q') R = MockPredicate('R') S = MockPredicate('S') X, Y = meta.TypeMap({ P & Q: R & S, P: R, Q: S, }) self.assertEqual(X.members, (P & Q, P, Q)) self.assertEqual(Y.members, (R & S, R, S)) self.assertEqual(X.index, 0) self.assertEqual(Y.index, 1) self.assertTrue(X.input) self.assertTrue(Y.output) self.assertFalse(X.output) self.assertFalse(Y.input) # subtyping self.assertFalse(S <= X) self.assertFalse(R <= X) self.assertLessEqual(P, X) self.assertLessEqual(Q, X) self.assertLessEqual(P & Q, X) self.assertLessEqual(P & S | P & R, X) def test_pickle(self): Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') X, Y = meta.TypeMap({ Foo: Bar }) X1, Y1 = pickle.loads(pickle.dumps((X, Y))) # Pickled together self.assertIs(X1.mapping, Y1.mapping) self.assertEqual(X1.index, X.index) self.assertEqual(Y1.index, Y.index) def test_subtype(self): P = MockPredicate('P') Q = MockPredicate('Q') R = MockPredicate('R') S = MockPredicate('S') T, U, Y = meta.TypeMap({ (P & Q, P & Q): R & S, (P & Q, Q): R & S, (P, P): R, (Q, Q): S }) self.assertLessEqual(P, T) self.assertLessEqual(Q, T) self.assertFalse(P | Q <= T) self.assertLessEqual(T, U) class TestTypeMatch(unittest.TestCase): def test_missing_branch_provided(self): P = MockPredicate('P') Q = MockPredicate('Q') T = meta.TypeMatch([P, Q]) self.assertEqual(T.members, (P & Q, P, Q)) def test_variable(self): P = MockPredicate('P') Q = MockPredicate('Q') R = MockPredicate('R') # This strange list is for branch coverage mostly T = meta.TypeMatch([P & Q, P, Q, R]) self.assertTrue(T.input) self.assertTrue(T.output) self.assertEqual(T.index, 1) # it really is supposed to be 1 class TestMatch(unittest.TestCase): def test_single_variable(self): Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') P = MockPredicate('P') X, Y = meta.TypeMap({ Foo % P: Foo, Bar % P: Foo % P, Foo: Bar, Bar: Bar % P }) input_signature = dict(input1=X) output_signature = dict(output1=Y) foop = dict(input1=Foo % P) barp = dict(input1=Bar % P) foo = dict(input1=Foo) bar = dict(input1=Bar) self.assertEqual(meta.match(foop, input_signature, output_signature), dict(output1=Foo)) self.assertEqual(meta.match(barp, input_signature, output_signature), dict(output1=Foo % P)) self.assertEqual(meta.match(foo, input_signature, output_signature), dict(output1=Bar)) self.assertEqual(meta.match(bar, input_signature, output_signature), dict(output1=Bar % P)) def test_nested_match(self): C2 = MockTemplate('C2', fields=('a', 'b')) Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') P = MockPredicate('P') X, Y = meta.TypeMap({ Foo % P: Foo, Bar % P: Foo % P, Foo: Bar, Bar: Bar % P }) input_signature = dict(input1=C2[X, Bar]) output_signature = dict(output1=C2[Bar, Y]) foop = dict(input1=C2[Foo % P, Bar]) barp = dict(input1=C2[Bar % P, Bar]) foo = dict(input1=C2[Foo, Foo]) bar = dict(input1=C2[Bar, Foo]) self.assertEqual(meta.match(foop, input_signature, output_signature), dict(output1=C2[Bar, Foo])) self.assertEqual(meta.match(barp, input_signature, output_signature), dict(output1=C2[Bar, Foo % P])) self.assertEqual(meta.match(foo, input_signature, output_signature), dict(output1=C2[Bar, Bar])) self.assertEqual(meta.match(bar, input_signature, output_signature), dict(output1=C2[Bar, Bar % P])) def test_multiple_variables(self): C2 = MockTemplate('C2', fields=('a', 'b')) Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') P = MockPredicate('P') A, B, C, Y, Z = meta.TypeMap({ (Foo % P, Bar, Bar): (Foo, Foo), (Foo, Bar % P, Foo): (Bar, Foo), (Foo, Foo, Bar): (Foo, Bar), (Bar, Bar % P, Foo): (Bar, Bar) }) input_signature = dict(input1=C2[A, B], input2=C) output_signature = dict(output1=C2[Y, Z]) fbb = dict(input1=C2[Foo % P, Bar], input2=Bar) fbf = dict(input1=C2[Foo, Bar % P], input2=Foo) ffb = dict(input1=C2[Foo, Foo], input2=Bar % P) # subtype on in2! bbf = dict(input1=C2[Bar % P, Bar % P], input2=Foo) # subtype on in1 self.assertEqual(meta.match(fbb, input_signature, output_signature), dict(output1=C2[Foo, Foo])) self.assertEqual(meta.match(fbf, input_signature, output_signature), dict(output1=C2[Bar, Foo])) self.assertEqual(meta.match(ffb, input_signature, output_signature), dict(output1=C2[Foo, Bar])) self.assertEqual(meta.match(bbf, input_signature, output_signature), dict(output1=C2[Bar, Bar])) def test_multiple_mappings(self): C2 = MockTemplate('C2', fields=('a', 'b')) Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') P = MockPredicate('P') X, Y = meta.TypeMap({ Foo % P: Foo, Bar % P: Foo % P, Foo: Bar, Bar: Bar % P }) T, R = meta.TypeMap({ Bar % P: Foo, Foo % P: Foo % P, Bar: Bar, Foo: Bar % P }) input_signature = dict(input1=C2[X, T]) output_signature = dict(output1=C2[R, Y]) foop = dict(input1=C2[Foo % P, Bar]) barp = dict(input1=C2[Bar % P, Bar % P]) foo = dict(input1=C2[Foo, Foo]) bar = dict(input1=C2[Bar, Foo]) self.assertEqual(meta.match(foop, input_signature, output_signature), dict(output1=C2[Bar, Foo])) self.assertEqual(meta.match(barp, input_signature, output_signature), dict(output1=C2[Foo, Foo % P])) self.assertEqual(meta.match(foo, input_signature, output_signature), dict(output1=C2[Bar % P, Bar])) self.assertEqual(meta.match(bar, input_signature, output_signature), dict(output1=C2[Bar % P, Bar % P])) def test_no_solution(self): C2 = MockTemplate('C2', fields=('a', 'b')) Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') P = MockPredicate('P') A, B, C, Y, Z = meta.TypeMap({ (Foo % P, Bar, Bar): (Foo, Foo), (Foo, Bar % P, Foo): (Bar, Foo), (Foo, Foo, Bar): (Foo, Bar), (Bar, Bar % P, Foo): (Bar, Bar) }) input_signature = dict(input1=C2[A, B], input2=C) output_signature = dict(output1=C2[Y, Z]) with self.assertRaisesRegex(ValueError, 'No solution'): meta.match(dict(input1=C2[Foo, Foo], input2=Foo), input_signature, output_signature) def test_inconsistent_binding(self): C2 = MockTemplate('C2', fields=('a', 'b')) Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') P = MockPredicate('P') A, B, C, Y, Z = meta.TypeMap({ (Foo % P, Bar, Bar): (Foo, Foo), (Foo, Bar % P, Foo): (Bar, Foo), (Foo, Foo, Bar): (Foo, Bar), (Bar, Bar % P, Foo): (Bar, Bar) }) input_signature = dict(input1=C2[A, B], input2=C2[C, C]) output_signature = dict(output1=C2[Y, Z]) with self.assertRaisesRegex(ValueError, 'to match'): meta.match(dict(input1=C2[Foo, Bar % P], input2=C2[Foo, Bar]), input_signature, output_signature) def test_consistent_subtype_binding(self): C2 = MockTemplate('C2', fields=('a', 'b')) Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') P = MockPredicate('P') A, B, C, Y, Z = meta.TypeMap({ (Foo % P, Bar, Bar): (Foo, Foo), (Foo, Bar % P, Foo): (Bar, Foo), (Foo, Foo, Bar): (Foo, Bar), (Bar, Bar % P, Foo): (Bar, Bar) }) input_signature = dict(input1=C2[A, B], input2=C2[C, C]) output_signature = dict(output1=C2[Y, Z]) cons = dict(input1=C2[Foo, Bar % P], input2=C2[Foo, Foo % P]) self.assertEqual(meta.match(cons, input_signature, output_signature), dict(output1=C2[Bar, Foo])) def test_missing_variables(self): C2 = MockTemplate('C2', fields=('a', 'b')) Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') P = MockPredicate('P') A, B, C, Y, Z = meta.TypeMap({ (Foo % P, Bar, Bar): (Foo, Foo), (Foo, Bar % P, Foo): (Bar, Foo), (Foo, Foo, Bar): (Foo, Bar), (Bar, Bar % P, Foo): (Bar, Bar) }) input_signature = dict(input1=C2[A, B], input2=Foo) output_signature = dict(output1=C2[Y, Z]) with self.assertRaisesRegex(ValueError, 'Missing'): meta.match(dict(input1=C2[Foo, Foo], input2=Foo), input_signature, output_signature) def test_no_variables(self): Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') P = MockPredicate('P') input_signature = dict(input1=Foo, input2=Bar) output_signature = dict(output1=Bar % P, output2=Foo % P) given = dict(input1=Foo % P, input2=Bar) self.assertEqual(meta.match(given, input_signature, output_signature), output_signature) def test_type_match(self): Foo = MockTemplate('Foo') Bar = MockTemplate('Bar') Baz = MockTemplate('Baz') P = MockPredicate('P') T = meta.TypeMatch([Baz, Foo, Bar]) input_signature = dict(input1=T) output_signature = dict(output1=T) foop = dict(input1=Foo % P) barp = dict(input1=Bar % P) foo = dict(input1=Foo) bar = dict(input1=Bar) self.assertEqual(meta.match(foop, input_signature, output_signature), dict(output1=Foo)) self.assertEqual(meta.match(barp, input_signature, output_signature), dict(output1=Bar)) self.assertEqual(meta.match(foo, input_signature, output_signature), dict(output1=Foo)) self.assertEqual(meta.match(bar, input_signature, output_signature), dict(output1=Bar)) def test_type_match_auto_intersect(self): C1 = MockTemplate('C1', fields=('a',)) Foo = MockTemplate('Foo') P = MockPredicate('P') Q = MockPredicate('Q') R = MockPredicate('R') S = MockPredicate('S') T = meta.TypeMatch([P, Q, R, S]) input_signature = dict(input1=C1[Foo] % T) output_signature = dict(output1=Foo % T) pqrs = dict(input1=C1[Foo] % (P & Q & R & S)) p = dict(input1=C1[Foo] % P) pr = dict(input1=C1[Foo] % (P & R)) qs = dict(input1=C1[Foo] % (Q & S)) self.assertEqual(meta.match(pqrs, input_signature, output_signature), dict(output1=Foo % (P & Q & R & S))) self.assertEqual(meta.match(p, input_signature, output_signature), dict(output1=Foo % P)) self.assertEqual(meta.match(pr, input_signature, output_signature), dict(output1=Foo % (P & R))) self.assertEqual(meta.match(qs, input_signature, output_signature), dict(output1=Foo % (Q & S))) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/type/tests/test_parse.py000066400000000000000000000106751412141660100220410ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest from qiime2.core.type.parse import ast_to_type, string_to_ast from qiime2.core.testing.type import Foo, Bar, C1, C2 from qiime2.plugin import (Int, Float, Str, Bool, Range, Choices, TypeMap, Properties, List, Set, Visualization, Metadata, MetadataColumn, Categorical, Numeric) class TestParsing(unittest.TestCase): def assert_roundtrip(self, type): ast = string_to_ast(repr(type)) type1 = ast_to_type(ast) type2 = ast_to_type(type1.to_ast()) self.assertEqual(type, type1) self.assertEqual(ast, type1.to_ast()) self.assertEqual(type1, type2) def test_simple_semantic_type(self): self.assert_roundtrip(Foo) self.assert_roundtrip(Bar) self.assert_roundtrip(C1[Foo]) def test_union_semantic_type(self): self.assert_roundtrip(Foo | Bar) self.assert_roundtrip(C1[Foo | Bar]) def test_complicated_semantic_type(self): self.assert_roundtrip(C2[C1[Foo % Properties(["A", "B"]) | Bar], Foo % Properties("A") ] % Properties(exclude=["B", "C"])) def test_collection_semantic_type(self): self.assert_roundtrip(List[Foo | Bar]) self.assert_roundtrip(Set[Bar]) def test_visualization(self): self.assert_roundtrip(Visualization) def test_primitive_simple(self): self.assert_roundtrip(Int) self.assert_roundtrip(Float) self.assert_roundtrip(Str) self.assert_roundtrip(Bool) def test_primitive_predicate(self): self.assert_roundtrip(Int % Range(0, 10)) self.assert_roundtrip( Int % (Range(0, 10) | Range(50, 100, inclusive_end=True))) self.assert_roundtrip(Float % Range(None, 10)) self.assert_roundtrip(Float % Range(0, None)) self.assert_roundtrip(Str % Choices("A")) self.assert_roundtrip(Str % Choices(["A"])) self.assert_roundtrip(Str % Choices("A", "B")) self.assert_roundtrip(Str % Choices(["A", "B"])) self.assert_roundtrip(Bool % Choices(True)) self.assert_roundtrip(Bool % Choices(False)) def test_collection_primitive(self): self.assert_roundtrip(Set[Str % Choices('A', 'B', 'C')]) self.assert_roundtrip(List[Int % Range(1, 3, inclusive_end=True) | Str % Choices('A', 'B', 'C')]) def test_metadata_primitive(self): self.assert_roundtrip(Metadata) self.assert_roundtrip(MetadataColumn[Numeric]) self.assert_roundtrip(MetadataColumn[Categorical]) self.assert_roundtrip(MetadataColumn[Numeric | Categorical]) def test_typevars(self): T, U, V, W, X = TypeMap({ (Foo, Bar, Str % Choices('A', 'B')): (C1[Foo], C1[Bar]), (Foo | Bar, Foo, Str): (C1[Bar], C1[Foo]) }) scope = {} T1 = ast_to_type(T.to_ast(), scope=scope) U1 = ast_to_type(U.to_ast(), scope=scope) V1 = ast_to_type(V.to_ast(), scope=scope) W1 = ast_to_type(W.to_ast(), scope=scope) X1 = ast_to_type(X.to_ast(), scope=scope) self.assertEqual(len(scope), 1) self.assertEqual(scope[id(T.mapping)], [T1, U1, V1, W1, X1]) self.assertEqual(T1.mapping.lifted, T.mapping.lifted) self.assertIs(T1.mapping, U1.mapping) self.assertIs(U1.mapping, V1.mapping) self.assertIs(V1.mapping, W1.mapping) self.assertIs(W1.mapping, X1.mapping) def test_syntax_error(self): with self.assertRaisesRegex(ValueError, "could not be parsed"): string_to_ast('$') def test_bad_juju(self): with self.assertRaisesRegex(ValueError, "one type expression"): string_to_ast('import os; os.rmdir("something-important")') def test_more_bad(self): with self.assertRaisesRegex(ValueError, "Unknown expression"): string_to_ast('lambda x: x') def test_weird(self): with self.assertRaisesRegex(ValueError, "Unknown literal"): string_to_ast('FeatureTable(Foo + Bar)') if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/type/tests/test_primitive.py000066400000000000000000000046031412141660100227310ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest import pandas as pd import qiime2.metadata as metadata import qiime2.core.type.primitive as primitive import qiime2.core.type.grammar as grammar class TestIntersectTwoRanges(unittest.TestCase): def assertIntersectEqual(self, a, b, exp): r1 = a & b r2 = b & a self.assertEqual(r1, r2) self.assertEqual(r1, exp) def test_overlap_simple(self): a = primitive.Range(0, 10) b = primitive.Range(3, 7) self.assertIntersectEqual(a, b, b) def test_overlap_inclusive_point(self): a = primitive.Range(0, 5, inclusive_end=True) b = primitive.Range(5, 10) exp = primitive.Range(5, 5, inclusive_start=True, inclusive_end=True) self.assertIntersectEqual(a, b, exp) def test_disjoint_far(self): a = primitive.Range(-10, -5) b = primitive.Range(5, 10) self.assertIntersectEqual(a, b, grammar.UnionExp()) def test_disjoint_exclusive_point(self): a = primitive.Range(0, 5, inclusive_end=False) b = primitive.Range(5, 9, inclusive_start=False) self.assertIntersectEqual(a, b, grammar.UnionExp()) class TestMetadataColumn(unittest.TestCase): def test_decode_categorical_value(self): value = pd.Series({'a': 'a', 'b': 'b', 'c': 'c'}, name='foo') value.index.name = 'id' cat_md = metadata.CategoricalMetadataColumn(value) res = primitive.MetadataColumn[primitive.Categorical].decode(cat_md) self.assertIs(res, cat_md) def test_decode_numeric_value(self): value = pd.Series({'a': 1, 'b': 2, 'c': 3}, name='foo') value.index.name = 'id' num_md = metadata.NumericMetadataColumn(value) res = primitive.MetadataColumn[primitive.Categorical].decode(num_md) self.assertIs(res, num_md) def test_decode_other(self): with self.assertRaisesRegex(TypeError, 'provided.*directly'): primitive.MetadataColumn[primitive.Categorical].decode( "") if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/type/tests/test_semantic.py000066400000000000000000000037761412141660100225360ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest import qiime2.core.type.semantic as semantic import qiime2.core.type.grammar as grammar import qiime2.core.type.primitive as primitive import qiime2.core.type.visualization as visualization class TestIsSemanticType(unittest.TestCase): def test_primitives_not_semantic(self): looped = False for element in dir(primitive): looped = True element = getattr(primitive, element) if isinstance(element, grammar._ExpBase): self.assertFalse(semantic.is_semantic_type(element)) self.assertTrue(looped) def test_visualization_not_semantic(self): self.assertFalse( semantic.is_semantic_type(visualization.Visualization)) def test_type_expr_not_semantic(self): TypeExpr = grammar.TypeExp(None) self.assertFalse(semantic.is_semantic_type(TypeExpr)) def test_simple_semantic_type(self): A = semantic.SemanticType('A') X = semantic.SemanticType('X') Foo = semantic.SemanticType('Foo', field_names=['a', 'b']) self.assertTrue(semantic.is_semantic_type(A)) self.assertTrue(semantic.is_semantic_type(X)) self.assertTrue(semantic.is_semantic_type(Foo)) def test_composite_semantic_type(self): Foo = semantic.SemanticType('Foo', field_names=['a', 'b']) A = semantic.SemanticType('A', variant_of=Foo.field['a']) B = semantic.SemanticType('B', variant_of=Foo.field['b']) self.assertTrue(semantic.is_semantic_type(A)) self.assertTrue(semantic.is_semantic_type(B)) self.assertTrue(semantic.is_semantic_type(Foo[A, B])) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/type/tests/test_util.py000066400000000000000000000777731412141660100217200ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest from qiime2.core.type import ( parse_primitive, Int, Float, Bool, Str, List, Set, Metadata, MetadataColumn) class TestParsePrimitiveNonCollectionsSimple(unittest.TestCase): def test_metadata_expr(self): with self.assertRaisesRegex(ValueError, 'Metadata may not be parsed'): parse_primitive(Metadata, '42') def test_metadata_column_expr(self): with self.assertRaisesRegex(ValueError, 'MetadataColumn.* may not be parsed'): parse_primitive(MetadataColumn, '42') def test_int_type_int_value(self): obs = parse_primitive(Int, '42') self.assertEqual(obs, 42) self.assertIsInstance(obs, int) def test_float_type_int_value(self): obs = parse_primitive(Float, '42') self.assertEqual(obs, 42.0) self.assertIsInstance(obs, float) def test_bool_type_int_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(Bool, '42') def test_str_type_int_value(self): obs = parse_primitive(Str, '42') self.assertEqual(obs, '42') self.assertIsInstance(obs, str) def test_int_type_float_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(Int, '42.0') def test_float_type_float_value(self): obs = parse_primitive(Float, '42.0') self.assertEqual(obs, 42.0) self.assertIsInstance(obs, float) def test_bool_type_float_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(Bool, '42.0') def test_str_type_float_value(self): obs = parse_primitive(Str, '42.0') self.assertEqual(obs, '42.0') self.assertIsInstance(obs, str) def test_int_type_bool_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(Int, 'True') def test_float_type_bool_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(Float, 'True') def test_bool_type_bool_value(self): obs = parse_primitive(Bool, 'True') self.assertEqual(obs, True) self.assertIsInstance(obs, bool) def test_str_type_bool_value(self): obs = parse_primitive(Str, 'True') self.assertEqual(obs, 'True') self.assertIsInstance(obs, str) def test_int_type_str_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(Int, 'peanut') def test_float_type_str_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(Float, 'peanut') def test_bool_type_str_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(Bool, 'peanut') def test_str_type_str_value(self): obs = parse_primitive(Str, 'peanut') self.assertEqual(obs, 'peanut') self.assertIsInstance(obs, str) class TestParsePrimitiveNonCollectionNonStringInputs(unittest.TestCase): def test_int_type_int_value(self): obs = parse_primitive(Int, 1) self.assertEqual(obs, 1) self.assertIsInstance(obs, int) def test_float_type_float_value(self): obs = parse_primitive(Float, 3.3) self.assertEqual(obs, 3.3) self.assertIsInstance(obs, float) def test_bool_type_bool_value(self): obs = parse_primitive(Bool, True) self.assertEqual(obs, True) self.assertIsInstance(obs, bool) def test_str_type_str_value(self): obs = parse_primitive(Str, 'peanut') self.assertEqual(obs, 'peanut') self.assertIsInstance(obs, str) def test_int_type_bool_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(Int, True) class TestParsePrimitiveNonCollectionsSimpleUnions(unittest.TestCase): def setUp(self): super().setUp() self.exprs = [ Int | Bool, Int | Str, Float | Bool, Float | Str, Bool | Str, ] def test_int_union_float_expr_int_value(self): # Int | Float == Float obs = parse_primitive(Int | Float, '42') self.assertEqual(obs, 42.0) self.assertIsInstance(obs, float) def test_int_union_float_expr_float_value(self): # Int | Float == Float obs = parse_primitive(Int | Float, '42.0') self.assertEqual(obs, 42.0) self.assertIsInstance(obs, float) def test_int_union_float_expr_bool_value(self): # Int | Float == Float with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(Int | Float, 'True') def test_int_union_float_expr_str_value(self): # Int | Float == Float with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(Int | Float, 'peanut') def test_simple_unions_with_int_value(self): for expr in self.exprs: with self.subTest(expr=expr): obs = parse_primitive(expr, '42') self.assertEqual(obs, 42) self.assertIsInstance(obs, int) def test_simple_unions_with_float_value(self): for expr in self.exprs: with self.subTest(expr=expr): obs = parse_primitive(expr, '42.1') self.assertEqual(obs, 42.1) self.assertIsInstance(obs, float) def test_simple_unions_with_bool_value(self): for expr in self.exprs: with self.subTest(expr=expr): obs = parse_primitive(expr, 'True') self.assertEqual(obs, True) self.assertIsInstance(obs, bool) def test_simple_unions_with_str_value(self): for expr in self.exprs: with self.subTest(expr=expr): obs = parse_primitive(expr, 'peanut') self.assertEqual(obs, 'peanut') self.assertIsInstance(obs, str) class TestParsePrimitiveCollectionsSimple(unittest.TestCase): def test_list_of_int(self): obs = parse_primitive(List[Int], ('1', '2', '3')) self.assertEqual(obs, [1, 2, 3]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], int) def test_list_of_int_bad_value_variant_a(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Int], ('True', '2', '3')) def test_list_of_int_bad_value_variant_b(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Int], ('1', '2', 'False')) def test_set_of_int(self): obs = parse_primitive(Set[Int], ('1', '2', '3')) self.assertEqual(obs, {1, 2, 3}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), int) def test_list_of_float(self): obs = parse_primitive(List[Float], ('1.0', '2.0', '3.0')) self.assertEqual(obs, [1.0, 2.0, 3.0]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], float) def test_set_of_float(self): obs = parse_primitive(Set[Float], ('1.0', '2.0', '3.0')) self.assertEqual(obs, {1.0, 2.0, 3.0}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), float) def test_list_of_bool(self): obs = parse_primitive(List[Bool], ('True', 'False', 'True')) self.assertEqual(obs, [True, False, True]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], bool) def test_set_of_bool(self): obs = parse_primitive(Set[Bool], ('True', 'False')) self.assertEqual(obs, {True, False}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), bool) def test_list_of_str(self): obs = parse_primitive(List[Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, ['peanut', 'the', 'dog']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) def test_set_of_str(self): obs = parse_primitive(Set[Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, {'peanut', 'the', 'dog'}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), str) # The next tests _aren't_ monomorphic, because unions of Int and Float # always yield a Float (List[Int] | List[Float] == List[Float]). def test_list_int_or_float_with_int_value(self): obs = parse_primitive(List[Int] | List[Float], ('1', '2', '3')) self.assertEqual(obs, [1.0, 2.0, 3.0]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], float) def test_set_int_or_float_with_int_value(self): obs = parse_primitive(Set[Int] | Set[Float], ('1', '2', '3')) self.assertEqual(obs, {1.0, 2.0, 3.0}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), float) def test_list_int_or_float_with_float_value(self): obs = parse_primitive(List[Int] | List[Float], ('1.1', '2.2', '3.3')) self.assertEqual(obs, [1.1, 2.2, 3.3]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], float) def test_set_int_or_float_with_float_value(self): obs = parse_primitive(Set[Int] | Set[Float], ('1.1', '2.2', '3.3')) self.assertEqual(obs, {1.1, 2.2, 3.3}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), float) def test_list_int_or_float_int_value(self): obs = parse_primitive(List[Int | Float], ('1', '2', '3')) self.assertEqual(obs, [1.0, 2.0, 3.0]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], float) def test_set_int_or_float_int_value(self): obs = parse_primitive(Set[Int | Float], ('1', '2', '3')) self.assertEqual(obs, {1.0, 2.0, 3.0}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), float) class TestParsePrimitiveCollectionsMonomorphic(unittest.TestCase): def test_list_int_or_bool_with_int_value(self): obs = parse_primitive(List[Int] | List[Bool], ('1', '2', '3')) self.assertEqual(obs, [1, 2, 3]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], int) def test_list_int_or_bool_with_bool_value(self): obs = parse_primitive(List[Int] | List[Bool], ('True', 'False', 'True')) self.assertEqual(obs, [True, False, True]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], bool) def test_list_int_or_bool_with_mixed_value_variant_a(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Int] | List[Bool], ('True', '2', '3')) def test_list_int_or_bool_with_mixed_value_variant_b(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Int] | List[Bool], ('1', '2', 'True')) def test_list_int_or_bool_with_mixed_value_variant_c(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Int] | List[Bool], ('False', '2', 'True')) def test_set_int_or_bool_with_int_value(self): obs = parse_primitive(Set[Int] | Set[Bool], ('1', '2', '3')) self.assertEqual(obs, {1, 2, 3}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), int) def test_set_int_or_bool_with_bool_value(self): obs = parse_primitive(Set[Int] | Set[Bool], ('True', 'False')) self.assertEqual(obs, {True, False}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), bool) def test_list_int_or_str_with_int_value(self): obs = parse_primitive(List[Int] | List[Str], ('1', '2', '3')) self.assertEqual(obs, [1, 2, 3]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], int) def test_list_int_or_str_with_str_value(self): obs = parse_primitive(List[Int] | List[Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, ['peanut', 'the', 'dog']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) def test_list_int_or_str_with_mixed_value_variant_a(self): obs = parse_primitive(List[Int] | List[Str], ('1', 'the', 'dog')) self.assertEqual(obs, ['1', 'the', 'dog']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) self.assertIsInstance(obs[1], str) def test_list_int_or_str_with_mixed_value_variant_b(self): obs = parse_primitive(List[Int] | List[Str], ('peanut', 'the', '1')) self.assertEqual(obs, ['peanut', 'the', '1']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) self.assertIsInstance(obs[2], str) def test_set_int_or_str_with_int_value(self): obs = parse_primitive(Set[Int] | Set[Str], ('1', '2', '3')) self.assertEqual(obs, {1, 2, 3}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), int) def test_set_int_or_str_with_str_value(self): obs = parse_primitive(Set[Int] | Set[Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, {'peanut', 'the', 'dog'}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), str) def test_list_float_or_bool_with_float_value(self): obs = parse_primitive(List[Float] | List[Bool], ('1.1', '2.2', '3.3')) self.assertEqual(obs, [1.1, 2.2, 3.3]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], float) def test_list_float_or_bool_with_bool_value(self): obs = parse_primitive(List[Float] | List[Bool], ('True', 'False', 'True')) self.assertEqual(obs, [True, False, True]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], bool) def test_list_float_or_bool_with_mixed_value_variant_a(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Float] | List[Bool], ('1.1', 'False', 'True')) def test_list_float_or_bool_with_mixed_value_variant_b(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Float] | List[Bool], ('True', 'False', '3.3')) def test_set_float_or_bool_with_float_value(self): obs = parse_primitive(Set[Float] | Set[Bool], ('1.1', '2.2', '3.3')) self.assertEqual(obs, {1.1, 2.2, 3.3}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), float) def test_set_float_or_bool_with_bool_value(self): obs = parse_primitive(Set[Float] | Set[Bool], ('True', 'False', 'True')) self.assertEqual(obs, {True, False}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), bool) def test_list_float_or_str_with_float_value(self): obs = parse_primitive(List[Float] | List[Str], ('1.1', '2.2', '3.3')) self.assertEqual(obs, [1.1, 2.2, 3.3]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], float) def test_list_float_or_str_with_str_value(self): obs = parse_primitive(List[Float] | List[Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, ['peanut', 'the', 'dog']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) def test_list_float_or_str_with_mixed_value_variant_a(self): obs = parse_primitive(List[Float] | List[Str], ('1.1', 'the', 'dog')) self.assertEqual(obs, ['1.1', 'the', 'dog']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) def test_list_float_or_str_with_mixed_value_variant_b(self): obs = parse_primitive(List[Float] | List[Str], ('peanut', 'the', '3.3')) self.assertEqual(obs, ['peanut', 'the', '3.3']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[-1], str) def test_set_float_or_str_with_float_value(self): obs = parse_primitive(Set[Float] | Set[Str], ('1.1', '2.2', '3.3')) self.assertEqual(obs, {1.1, 2.2, 3.3}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), float) def test_set_float_or_str_with_str_value(self): obs = parse_primitive(Set[Float] | Set[Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, {'peanut', 'the', 'dog'}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), str) def test_list_bool_or_str_with_bool_value(self): obs = parse_primitive(List[Bool] | List[Str], ('True', 'False', 'True')) self.assertEqual(obs, [True, False, True]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], bool) def test_list_bool_or_str_with_str_value(self): obs = parse_primitive(List[Bool] | List[Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, ['peanut', 'the', 'dog']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) def test_list_bool_or_str_with_mixed_value_variant_a(self): obs = parse_primitive(List[Bool] | List[Str], ('True', 'the', 'dog')) self.assertEqual(obs, ['True', 'the', 'dog']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) def test_list_bool_or_str_with_mixed_value_variant_b(self): obs = parse_primitive(List[Bool] | List[Str], ('peanut', 'the', 'True')) self.assertEqual(obs, ['peanut', 'the', 'True']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[-1], str) def test_set_bool_or_str_with_bool_value(self): obs = parse_primitive(Set[Bool] | Set[Str], ('True', 'False', 'True')) self.assertEqual(obs, {True, False}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), bool) def test_set_bool_or_str_with_str_value(self): obs = parse_primitive(Set[Bool] | Set[Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, {'peanut', 'the', 'dog'}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), str) def test_list_bool_or_str_with_mixed_value(self): obs = parse_primitive(List[Bool] | List[Str], ('peanut', 'the', 'True')) self.assertEqual(obs, ['peanut', 'the', 'True']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) self.assertIsInstance(obs[-1], str) class TestParsePrimitiveCollectionsComposite(unittest.TestCase): def test_list_int_or_bool_with_int_value(self): obs = parse_primitive(List[Int | Bool], ('1', '2', '3')) self.assertEqual(obs, [1, 2, 3]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], int) def test_list_int_or_bool_with_float_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Int | Bool], ('1.1', '2.2', '3.3')) def test_list_int_or_bool_with_bool_value(self): obs = parse_primitive(List[Int | Bool], ('True', 'False', 'True')) self.assertEqual(obs, [True, False, True]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], bool) def test_list_int_or_bool_with_str_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Int | Bool], ('peanut', 'the', 'dog')) def test_list_int_or_bool_with_mixed_value(self): obs = parse_primitive(List[Int | Bool], ('1', 'False', '2', 'True')) self.assertEqual(obs, [1, False, 2, True]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], int) self.assertIsInstance(obs[1], bool) def test_list_int_or_bool_with_mixed_value_variant_a(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Int | Bool], ('peanut', 'False', '2', 'True')) def test_list_int_or_bool_with_mixed_value_variant_b(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Int | Bool], ('1', 'False', '2', 'peanut')) def test_list_int_or_bool_with_bad_mix_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Int | Bool], ('1', 'True', 'dog')) def test_set_int_or_bool_with_int_value(self): obs = parse_primitive(Set[Int | Bool], ('1', '2', '3')) self.assertEqual(obs, {1, 2, 3}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), int) def test_set_int_or_bool_with_bool_value(self): obs = parse_primitive(Set[Int | Bool], ('True', 'False', 'True')) self.assertEqual(obs, {True, False}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), bool) def test_set_int_or_bool_with_mixed_value(self): obs = parse_primitive(Set[Int | Bool], ('1', 'False', '2', 'True')) self.assertEqual(obs, {1, False, 2, True}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), bool) self.assertIsInstance(obs.pop(), int) def test_list_int_or_str_with_int_value(self): obs = parse_primitive(List[Int | Str], ('1', '2', '3')) self.assertEqual(obs, [1, 2, 3]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], int) def test_list_int_or_str_with_str_value(self): obs = parse_primitive(List[Int | Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, ['peanut', 'the', 'dog']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) def test_list_int_or_str_with_mixed_value_variant_a(self): obs = parse_primitive(List[Int | Str], ('1', 'the', 'dog')) self.assertEqual(obs, [1, 'the', 'dog']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], int) self.assertIsInstance(obs[1], str) def test_list_int_or_str_with_mixed_value_variant_b(self): obs = parse_primitive(List[Int | Str], ('peanut', 'the', '1')) self.assertEqual(obs, ['peanut', 'the', 1]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) self.assertIsInstance(obs[2], int) def test_set_int_or_str_with_int_value(self): obs = parse_primitive(Set[Int | Str], ('1', '2', '3')) self.assertEqual(obs, {1, 2, 3}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), int) def test_set_int_or_str_with_str_value(self): obs = parse_primitive(Set[Int | Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, {'peanut', 'the', 'dog'}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), str) def test_set_int_or_str_with_mixed_value(self): obs = parse_primitive(Set[Int | Str], ('1', 'the', '2', 'dog')) self.assertEqual(obs, {1, 'the', 2, 'dog'}) self.assertIsInstance(obs, set) def test_list_float_or_bool_with_float_value(self): obs = parse_primitive(List[Float | Bool], ('1.1', '2.2', '3.3')) self.assertEqual(obs, [1.1, 2.2, 3.3]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], float) def test_list_float_or_bool_with_bool_value(self): obs = parse_primitive(List[Float | Bool], ('True', 'False', 'True')) self.assertEqual(obs, [True, False, True]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], bool) def test_list_float_or_bool_with_mixed_value_variant_a(self): obs = parse_primitive(List[Float | Bool], ('True', '2.2', '3.3')) self.assertEqual(obs, [True, 2.2, 3.3]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], bool) self.assertIsInstance(obs[1], float) def test_list_float_or_bool_with_mixed_value_variant_b(self): obs = parse_primitive(List[Float | Bool], ('1.1', '2.2', 'False')) self.assertEqual(obs, [1.1, 2.2, False]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], float) self.assertIsInstance(obs[-1], bool) def test_list_float_or_bool_with_bad_mix_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Float | Bool], ('1.1', '2.2', 'peanut')) def test_set_float_or_bool_with_float_value(self): obs = parse_primitive(Set[Float | Bool], ('1.1', '2.2', '3.3')) self.assertEqual(obs, {1.1, 2.2, 3.3}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), float) def test_set_float_or_bool_with_bool_value(self): obs = parse_primitive(Set[Float | Bool], ('True', 'False', 'True')) self.assertEqual(obs, {True, False}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), bool) def test_list_float_or_str_with_float_value(self): obs = parse_primitive(List[Float | Str], ('1.1', '2.2', '3.3')) self.assertEqual(obs, [1.1, 2.2, 3.3]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], float) def test_list_float_or_str_with_str_value(self): obs = parse_primitive(List[Float | Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, ['peanut', 'the', 'dog']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) def test_list_float_or_str_with_mixed_value_variant_a(self): obs = parse_primitive(List[Float | Str], ('peanut', '2.2', '3.3')) self.assertEqual(obs, ['peanut', 2.2, 3.3]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) self.assertIsInstance(obs[1], float) def test_list_float_or_str_with_mixed_value_variant_b(self): obs = parse_primitive(List[Float | Str], ('1.1', '2.2', 'dog')) self.assertEqual(obs, [1.1, 2.2, 'dog']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], float) self.assertIsInstance(obs[-1], str) def test_set_float_or_str_with_float_value(self): obs = parse_primitive(Set[Float | Str], ('1.1', '2.2', '3.3')) self.assertEqual(obs, {1.1, 2.2, 3.3}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), float) def test_set_float_or_str_with_str_value(self): obs = parse_primitive(Set[Float | Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, {'peanut', 'the', 'dog'}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), str) def test_list_bool_or_str_with_bool_value(self): obs = parse_primitive(List[Bool | Str], ('True', 'False', 'True')) self.assertEqual(obs, [True, False, True]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], bool) def test_list_bool_or_str_with_str_value(self): obs = parse_primitive(List[Bool | Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, ['peanut', 'the', 'dog']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) def test_list_bool_or_str_with_mixed_value_variant_a(self): obs = parse_primitive(List[Bool | Str], ('True', 'the', 'dog')) self.assertEqual(obs, [True, 'the', 'dog']) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], bool) self.assertIsInstance(obs[-1], str) def test_list_bool_or_str_with_mixed_value_variant_b(self): obs = parse_primitive(List[Bool | Str], ('peanut', 'the', 'True')) self.assertEqual(obs, ['peanut', 'the', True]) self.assertIsInstance(obs, list) self.assertIsInstance(obs[0], str) self.assertIsInstance(obs[-1], bool) def test_set_bool_or_str_with_bool_value(self): obs = parse_primitive(Set[Bool | Str], ('True', 'False', 'True')) self.assertEqual(obs, {True, False}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), bool) def test_set_bool_or_str_with_str_value(self): obs = parse_primitive(Set[Bool | Str], ('peanut', 'the', 'dog')) self.assertEqual(obs, {'peanut', 'the', 'dog'}) self.assertIsInstance(obs, set) self.assertIsInstance(obs.pop(), str) class TestParsePrimitiveCollectionsComplex(unittest.TestCase): def test_list_int_bool_or_list_float_with_bool_int_value(self): obs = parse_primitive(List[Int | Bool] | List[Float], ('1', '2', 'True', 'False')) self.assertEqual(obs, [1, 2, True, False]) def test_list_int_bool_or_list_float_with_float_value(self): obs = parse_primitive(List[Int | Bool] | List[Float], ('1.1', '2.2', '3.3', '4.4')) self.assertEqual(obs, [1.1, 2.2, 3.3, 4.4]) def test_list_int_bool_or_list_float_with_bad_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Int | Bool] | List[Float], ('1', '2.2', 'True', 'False')) def test_list_int_str_or_list_float_with_str_int_value(self): obs = parse_primitive(List[Int | Str] | List[Float], ('1', '2', 'peanut', 'the')) self.assertEqual(obs, [1, 2, 'peanut', 'the']) def test_list_int_str_or_list_float_with_float_value(self): obs = parse_primitive(List[Int | Str] | List[Float], ('1.1', '2.2', '3.3', '4.4')) self.assertEqual(obs, [1.1, 2.2, 3.3, 4.4]) def test_list_int_str_or_list_float_str_with_float_value(self): obs = parse_primitive(List[Int | Str] | List[Float | Str], ('1.1', '2.2', '3.3', '4.4')) self.assertEqual(obs, [1.1, 2.2, 3.3, 4.4]) def test_list_int_str_or_list_float_str_bool_with_float_value(self): obs = parse_primitive(List[Int | Str] | List[Float | Str | Bool], ('1.1', '2.2', '3.3', '4.4')) self.assertEqual(obs, [1.1, 2.2, 3.3, 4.4]) def test_list_int_str_or_list_float_str_bool_with_float_str_value(self): obs = parse_primitive(List[Int | Str] | List[Float | Str | Bool], ('1.1', '2.2', 'the', 'peanut')) self.assertEqual(obs, [1.1, 2.2, 'the', 'peanut']) def test_list_int_str_or_list_float_str_bool_with_float_bool_value(self): obs = parse_primitive(List[Int | Str] | List[Float | Str | Bool], ('1.1', '2.2', 'True', 'False')) self.assertEqual(obs, [1.1, 2.2, True, False]) def test_list_int_str_or_list_float_with_mixed_value(self): obs = parse_primitive(List[Int | Str] | List[Float], ('1.1', '2', 'True', 'peanut')) self.assertEqual(obs, ['1.1', 2, 'True', 'peanut']) def test_list_float_bool_or_list_str_with_float_bool_value(self): obs = parse_primitive(List[Float | Bool] | List[Int], ('1', '2', 'True', 'False')) self.assertEqual(obs, [1, 2, True, False]) def test_list_float_bool_or_list_str_with_int_value(self): obs = parse_primitive(List[Float | Bool] | List[Int], ('1', '2', '3', '4')) self.assertEqual(obs, [1, 2, 3, 4]) def test_list_float_bool_or_list_str_with_bad_value(self): with self.assertRaisesRegex(ValueError, 'Could not coerce'): parse_primitive(List[Float | Bool] | List[Int], ('1', '2.2', 'True', 'peanut')) def test_set_int_bool_or_list_float_with_bool_int_value(self): obs = parse_primitive(Set[Int | Bool] | Set[Float], ('1', '2', 'True', 'False')) self.assertEqual(obs, {1, 2, True, False}) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/core/type/util.py000066400000000000000000000160471412141660100175020ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import collections from qiime2.core.util import tuplize from qiime2.core.type.collection import List, Set from qiime2.core.type.primitive import Int, Float, Bool, Str from qiime2.core.type.grammar import UnionExp, _ExpBase, IntersectionExp from qiime2.core.type.parse import ast_to_type def _strip_predicates(expr): if isinstance(expr, UnionExp): return UnionExp(_strip_predicates(m) for m in expr.members).normalize() if hasattr(expr, 'fields'): new_fields = tuple(_strip_predicates(f) for f in expr.fields) return expr.duplicate(fields=new_fields, predicate=IntersectionExp()) def val_to_bool(value): if type(value) is bool: return value elif str(value).lower() == 'true': return True elif str(value).lower() == 'false': return False else: raise ValueError('Could not cast to bool') def val_to_int(v): if type(v) is int: return v elif type(v) is str: return int(v) else: raise ValueError('Could not cast to int') VariadicRecord = collections.namedtuple('VariadicRecord', ['pytype', 'q2type']) _VARIADIC = { 'List': VariadicRecord(pytype=list, q2type=List), 'Set': VariadicRecord(pytype=set, q2type=Set), } CoercionRecord = collections.namedtuple('CoercionRecord', ['func', 'pytype']) # Beware visitor, order matters in this here mapper _COERCION_MAPPER = { Int: CoercionRecord(pytype=int, func=val_to_int), Float: CoercionRecord(pytype=float, func=float), Bool: CoercionRecord(pytype=bool, func=val_to_bool), Str: CoercionRecord(pytype=str, func=str), } _COERCE_ERROR = ValueError( 'Could not coerce value based on expression provided.') CollectionStyle = collections.namedtuple( 'CollectionStyle', ['style', 'members', 'view', 'expr', 'base']) def _norm_input(t): if type(t) is dict: return ast_to_type(t) elif not isinstance(t, _ExpBase): raise TypeError("%r is not a QIIME 2 type" % (t,)) return t def is_qiime_type(t): try: _norm_input(t) except Exception: return False else: return True def is_primitive_type(t): expr = _norm_input(t) return hasattr(expr, 'kind') and expr.kind == 'primitive' def is_metadata_type(t): expr = _norm_input(t) return is_primitive_type(t) and expr.name.startswith('Metadata') def is_metadata_column_type(t): expr = _norm_input(t) return is_primitive_type(t) and expr.name.endswith('MetadataColumn') def is_semantic_type(t): expr = _norm_input(t) return hasattr(expr, 'kind') and expr.kind == 'semantic-type' def is_visualization_type(t): expr = _norm_input(t) return hasattr(expr, 'kind') and expr.kind == 'visualization' def is_union(t): expr = _norm_input(t) return isinstance(expr, UnionExp) def is_collection_type(t): expr = _norm_input(t) if expr.name in _VARIADIC: return True if is_union(expr): for m in expr.members: if m.name in _VARIADIC: return True return False def interrogate_collection_type(t): expr = _norm_input(t) style = None # simple, monomorphic, composite, complex members = None # T , [T1, T2] , [T1, T2], [[T1], [T2, T3]] view = None # set, list base = None if expr.name in _VARIADIC: view, base = _VARIADIC[expr.name] field, = expr.fields if isinstance(field, UnionExp): style = 'composite' members = list(field.members) else: style = 'simple' members = field elif isinstance(expr, UnionExp): if expr.members[0].name in _VARIADIC: members = [] for member in expr.members: field, = member.fields if isinstance(field, UnionExp): style = 'complex' members.append(list(field.members)) else: members.append([field]) if style != 'complex': style = 'monomorphic' # use last iteration view, base = _VARIADIC[member.name] if style == 'monomorphic': members = [m[0] for m in members] return CollectionStyle(style=style, members=members, view=view, expr=expr, base=base) def _ordered_coercion(types): types = tuple(types) return tuple(k for k in _COERCION_MAPPER.keys() if k in types) def _interrogate_types(allowed, value): ordered_allowed = _ordered_coercion(allowed) for coerce_type in (_COERCION_MAPPER[x].func for x in ordered_allowed): try: return coerce_type(value) except ValueError: pass raise _COERCE_ERROR def parse_primitive(t, value): expr = _norm_input(t) result = [] allowed = None homogeneous = True if is_metadata_type(expr): raise ValueError('%r may not be parsed with this util.' % (expr,)) expr = _strip_predicates(expr) collection_style = interrogate_collection_type(expr) if collection_style.style in ('simple', 'monomorphic', 'composite'): allowed = collection_style.members if collection_style.style == 'composite': homogeneous = False elif collection_style.style == 'complex': # Sort here so that we can start with any simple lists in the memberset for subexpr in sorted(collection_style.members, key=len): expr = collection_style.base[UnionExp(subexpr)] try: return parse_primitive(expr, value) except ValueError: pass raise _COERCE_ERROR elif collection_style.style is None: value = tuplize(value) if expr in (Int, Float, Bool, Str): # No sense in walking over all options when we know # what it should be allowed = expr else: allowed = _COERCION_MAPPER.keys() else: pass assert allowed is not None for v in value: result.append(_interrogate_types(allowed, v)) # Some exprs require homogeneous values, make it so if homogeneous: all_matching = False for member in allowed: if all(type(x) == _COERCION_MAPPER[member].pytype for x in result): all_matching = True break if not all_matching and collection_style.style == 'monomorphic': for subexpr in allowed: expr = collection_style.base[subexpr] try: return parse_primitive(expr, value) except ValueError: pass raise _COERCE_ERROR if collection_style.view is None: return result[0] else: return collection_style.view(result) qiime-2021.8.0/qiime2/core/type/visualization.py000066400000000000000000000020231412141660100214130ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from qiime2.core.type.template import TypeTemplate class _Visualization(TypeTemplate): def get_kind(self): return "visualization" def __eq__(self, other): return type(self) is type(other) def get_field_names(self): return [] def get_name(self): return "Visualization" def is_element(self, value): import qiime2.sdk return isinstance(value, qiime2.sdk.Visualization) def validate_field(self, name, field): raise TypeError def get_union_membership_expr(self, self_expr): return None def validate_predicate(self, predicate, expr): raise TypeError Visualization = _Visualization() qiime-2021.8.0/qiime2/core/util.py000066400000000000000000000166521412141660100165230ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import contextlib import warnings import hashlib import os import io import collections import decorator def get_view_name(view): from .format import FormatBase if not isinstance(view, type): view = view.__class__ if issubclass(view, FormatBase): # Not qualname because we don't have a notion of "nested" formats return view.__name__ return ':'.join([view.__module__, view.__qualname__]) def tuplize(x): if type(x) is not tuple: return (x,) return x def overrides(cls): def decorator(func): if not hasattr(cls, func.__name__): raise AssertionError("%r does not override %r" % (func, cls.__name__)) return func return decorator def superscript(number): table = { '0': chr(8304), '1': chr(185), '2': chr(178), '3': chr(179), **{str(i): chr(x) for i, x in enumerate(range(8308, 8314), 4)}, 'a': chr(7491), 'e': chr(7497), 'f': chr(7584), 'i': chr(8305), 'n': chr(8319), '-': chr(8315), '.': chr(39), ',': chr(39) } return ''.join([table[d] for d in str(number)]) def find_duplicates(iterable): """Find duplicate values in an iterable. Parameters ---------- iterable : iterable Iterable to search for duplicates. Returns ------- set Values that are duplicated in `iterable`. Notes ----- Values in `iterable` must be hashable. """ # Modified from https://stackoverflow.com/a/9835819/3776794 to return # duplicates instead of remove duplicates from an iterable. seen = set() duplicates = set() for value in iterable: if value in seen: duplicates.add(value) else: seen.add(value) return duplicates # Concept from: http://stackoverflow.com/a/11157649/579416 def duration_time(relative_delta): attrs = ['years', 'months', 'days', 'hours', 'minutes', 'seconds', 'microseconds'] results = [] for attr in attrs: value = getattr(relative_delta, attr) if value != 0: if value == 1: # Remove plural 's' attr = attr[:-1] results.append("%d %s" % (value, attr)) if results: text = results[-1] if results[:-1]: text = ', and '.join([', '.join(results[:-1]), text]) return text else: # Great Scott! No time has passed! return '0 %s' % attrs[-1] def md5sum(filepath): md5 = hashlib.md5() with open(str(filepath), mode='rb') as fh: for chunk in iter(lambda: fh.read(io.DEFAULT_BUFFER_SIZE), b""): md5.update(chunk) return md5.hexdigest() def md5sum_directory(directory): directory = str(directory) sums = collections.OrderedDict() for root, dirs, files in os.walk(directory, topdown=True): dirs[:] = sorted([d for d in dirs if not d[0] == '.']) for file in sorted(files): if file[0] == '.': continue path = os.path.join(root, file) sums[os.path.relpath(path, start=directory)] = md5sum(path) return sums def to_checksum_format(filepath, checksum): # see https://www.gnu.org # /software/coreutils/manual/html_node/md5sum-invocation.html if '\\' in filepath or '\n' in filepath: filepath = filepath.replace('\\', '\\\\').replace('\n', '\\n') checksum = '\\' + checksum return '%s %s' % (checksum, filepath) def from_checksum_format(line): line = line.rstrip('\n') parts = line.split(' ', 1) if len(parts) < 2: parts = line.split(' *', 1) checksum, filepath = parts if checksum[0] == '\\': chars = '' escape = False # Gross, but regular `.replace` will overlap with itself and # negative lookbehind in regex is *probably* harder than scanning for char in filepath: # 1) Escape next character if not escape and char == '\\': escape = True continue # 2) Handle escape sequence if escape: try: chars += {'\\': '\\', 'n': '\n'}[char] except KeyError: chars += '\\' + char # Wasn't an escape after all escape = False continue # 3) Nothing interesting chars += char checksum = checksum[1:] filepath = chars return filepath, checksum @contextlib.contextmanager def warning(): def _warnformat(msg, category, filename, lineno, file=None, line=None): return '%s:%s: %s: %s\n' % (filename, lineno, category.__name__, msg) default_warn_format = warnings.formatwarning try: warnings.formatwarning = _warnformat warnings.filterwarnings('always') yield warnings.warn finally: warnings.formatwarning = default_warn_format # Descriptor protocol for creating an attribute that is bound to an # (arbitrarily nested) attribute accessible to the instance at runtime. class LateBindingAttribute: def __init__(self, attribute): self._attribute = attribute def __get__(self, obj, cls=None): attrs = self._attribute.split('.') curr_attr = obj for attr in attrs: curr_attr = getattr(curr_attr, attr) return staticmethod(curr_attr).__get__(obj, cls) # Removes the first parameter from a callable's signature. class DropFirstParameter(decorator.FunctionMaker): @classmethod def from_function(cls, function): return cls.create(function, "return None", {}) def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.signature = self._remove_first_arg(self.signature) self.shortsignature = self._remove_first_arg(self.shortsignature) def _remove_first_arg(self, string): return ",".join(string.split(',')[1:])[1:] def _immutable_error(obj, *args): raise TypeError('%s is immutable.' % obj.__class__.__name__) class ImmutableBase: def _freeze_(self): """Disables __setattr__ when called. It is idempotent.""" self._frozen = True # The particular value doesn't matter __delattr__ = __setitem__ = __delitem__ = _immutable_error def __setattr__(self, *args): # This doesn't stop silly things like # object.__setattr__(obj, ...), but that's a pretty rude thing # to do anyways. We are just trying to avoid accidental mutation. if hasattr(self, '_frozen'): _immutable_error(self) super().__setattr__(*args) def sorted_poset(iterable, *, key=None, reverse=False): values = list(iterable) elements = values if key is not None: elements = [key(x) for x in values] result = [] sorted_elements = [] for value, element in zip(values, elements): idx = 0 for idx, placed in enumerate(sorted_elements, 1): if element <= placed: idx -= 1 break result.insert(idx, value) sorted_elements.insert(idx, element) if reverse: result = list(reversed(result)) return result qiime-2021.8.0/qiime2/core/validate.py000066400000000000000000000142001412141660100173220ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from qiime2.core.exceptions import ValidationError, ImplementationError from qiime2.core.transform import ModelType from qiime2.core.util import sorted_poset class ValidationObject: r""" Store, sort and run all semantic validators for a for a single, complete semantic type(a `concrete type`). Attributes ---------- concrete_type: SemanticType The semantic type for which the validators are valid for. """ def __init__(self, concrete_type): r""" Create a new ValidationObject to add ValidatorRecords to. Parameters ---------- concrete_type: semantic type The single, complete semantic type that the validators are to be associated with. """ # Private Attributes # ------------------ # _validators: list # A list of ValidatorRecords # _is_sorted: Bool # Tracks whether or not `_validators` has been sorted or not. self._validators = [] self.concrete_type = concrete_type self._is_sorted = False def add_validator(self, validator_record): r""" Adds new validator record to plugin. Parameters ---------- validator_record: ValidatorRecord ValidatorRecord is a collections.namedtuple found in `qiime2/plugin/plugin.py`. Notes ----- Used by Plugin to add a `ValidatorRecord` for a new validator to a plugin. Usually called through the `register_validator` decorator. """ self._validators.append(validator_record) self._is_sorted = False def add_validation_object(self, *others): r""" Incorporates another validation object of the same concrete type. Parameters ---------- *others: Any number of validation objects of the same concrete type. Notes ----- Used to combine validation objects from different plugins. This is done non-heirarchically by `PluginManager` by creating a new, blank object for each `concrete_type` that it encounters, then adds the objects from each plugin. """ for other in others: if self.concrete_type != other.concrete_type: raise TypeError('Unable to add ValidationObject of' ' `concrete_type: %s to ValidationObject of' ' `concrete_type: %s`' % (other.concrete_type, self.concrete_type)) self._validators += other._validators self._is_sorted = False @property def validators(self) -> list: r""" Public access method for the validators stored in ValidationObject. Returns ------- list A sorted list of validator records. """ if not self._is_sorted: self._sort_validators() return self._validators def _sort_validators(self): r""" Sorts validators Notes ----- A partial order sort of the validators. The runtime for this sort is :math:`\theta(n^2)`. This is not a concern, as the number of validators present for any particular type is expected to remain trivially low. The validators are sorted from general to specific. """ self._validators = sorted_poset( iterable=self._validators, key=lambda record: record.context, reverse=True) self._is_sorted = True def __call__(self, data, level): r""" Validates that provided data meets the conditions of a semantic type. Parameters ---------- data: A view of the data to be validated. level: {'min', 'max'} specifies the level validation occurs at. Notes ----- Use of `level` is required but the behaviour is defined in the individual validators. """ from_mt = ModelType.from_view_type(type(data)) for record in self.validators: to_mt = ModelType.from_view_type(record.view) transformation = from_mt.make_transformation(to_mt) data = transformation(data) try: record.validator(data=data, level=level) except ValidationError: raise except Exception as e: raise ImplementationError("An unexpected error occured when %r" " from %r attempted to validate %r" % (record.validator.__name__, record.plugin, data)) from e def assert_transformation_available(self, data): r""" Checks that required transformations exist. Parameters ---------- data: view view type of input data. Raises ------ AssertionError If no transformation exists from the data view to the view expected by a particular validator. Notes ----- Called by `qiime2.sdk.PluginManager._consistency_check` to ensure the transformers required to run the validators are defined. """ mt = ModelType.from_view_type(data) for record in self._validators: mt_other = ModelType.from_view_type(record.view) if not mt.has_transformation(mt_other): raise AssertionError( 'Could not validate %s using %r because there was no' ' transformation from %r to %r' % (self.concrete_type, record.validator.__name__, mt._view_name, mt_other._view_name) ) qiime-2021.8.0/qiime2/jupyter/000077500000000000000000000000001412141660100157345ustar00rootroot00000000000000qiime-2021.8.0/qiime2/jupyter/__init__.py000066400000000000000000000007511412141660100200500ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from .hooks import load_jupyter_server_extension from .template import make_html __all__ = ['make_html', 'load_jupyter_server_extension'] qiime-2021.8.0/qiime2/jupyter/handlers.py000066400000000000000000000046251412141660100201150ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import pathlib import tornado.web as web from notebook.base.handlers import IPythonHandler import qiime2.core.archive.archiver as archiver class _ArchiveCheck(archiver._Archive): """This is only what is needed to verify a path is an archive""" # TODO: make this part of the archiver API at some point def open(self, relpath): abspath = os.path.join(str(self.path), str(self.uuid), relpath) return open(abspath, 'r') def relative_iterdir(self, relpath='.'): for p in pathlib.Path(self.path).iterdir(): yield str(p.relative_to(self.path)) class QIIME2RedirectHandler(IPythonHandler): """Add a location to location_store for later retrieval""" def initialize(self, result_store): self.result_store = result_store def get(self): location = self.get_query_argument('location') if not os.path.exists(location): # Client DOM should explain that the user should re-run the cell self.send_error(409) # Conflict return # is it actually a QIIME 2 result, or a random part of the filesystem archive = _ArchiveCheck(pathlib.Path(location)) self.result_store[archive.uuid] = os.path.join( location, str(archive.uuid), 'data') self.redirect('view/%s/' % archive.uuid) class QIIME2ResultHandler(web.StaticFileHandler): def initialize(self, path, default_filename): super().initialize(path, default_filename) self.result_store = path # path is actually result_store @classmethod def get_absolute_path(cls, root, path): uuid, path = path.split('/', 1) root = root[uuid] # This is janky, but validate_absolute_path is the only thing # that will use this data, so it can know to unpack the tuple again return (super().get_absolute_path(root, path), uuid) def validate_absolute_path(self, root, abspath_uuid): absolute_path, uuid = abspath_uuid root = self.result_store[uuid] return super().validate_absolute_path(root, absolute_path) qiime-2021.8.0/qiime2/jupyter/hooks.py000066400000000000000000000017401412141660100174330ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- def load_jupyter_server_extension(nb_server): from .handlers import QIIME2RedirectHandler, QIIME2ResultHandler from notebook.utils import url_path_join result_store = {} app = nb_server.web_app def route(path): return url_path_join(app.settings['base_url'], 'qiime2', path) app.add_handlers(r'.*', [ (route(r'redirect'), QIIME2RedirectHandler, {'result_store': result_store}), (route(r'view/(.*)'), QIIME2ResultHandler, # This *is* odd, but it's because we are tricking StaticFileHandler {'path': result_store, 'default_filename': 'index.html'}) ]) qiime-2021.8.0/qiime2/jupyter/template.py000066400000000000000000000050461412141660100201260ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import urllib.parse def make_html(location): url = "/qiime2/redirect?location={location}".format( location=urllib.parse.quote(location)) # This is dark magic. An image has an onload handler, which let's me # grab the parent dom in an anonymous way without needing to scope the # output cells of Jupyter with some kind of random ID. # Using transparent pixel from: https://stackoverflow.com/a/14115340/579416 return ('
'.format( anon_func=_anonymous_function, url=url)) # 404 - the extension isn't installed # 428 - the result went out of scope, re-run cell # 302->200 - set up the iframe for that location _anonymous_function = '''\ function(div, url){ if (typeof require !== 'undefined') { var baseURL = require.toUrl('').split('/').slice(0, -2).join('/'); } else { var baseURL = JSON.parse( document.getElementById('jupyter-config-data').innerHTML ).baseUrl.slice(0, -1); } url = baseURL + url; fetch(url).then(function(res) { if (res.status === 404) { div.innerHTML = 'Install QIIME 2 Jupyter extension with:
' + 'jupyter serverextension enable --py qiime2' + ' --sys-prefix
then restart your server.' + '

(Interactive output not available on ' + 'static notebook viewer services like nbviewer.)'; } else if (res.status === 409) { div.innerHTML = 'Visualization no longer in scope. Re-run this cell' + ' to see the visualization.'; } else if (res.ok) { url = res.url; div.innerHTML = '
Open in a: new window' } else { div.innerHTML = 'Something has gone wrong. Check notebook server for' + ' errors.'; } }); }''' qiime-2021.8.0/qiime2/metadata/000077500000000000000000000000001412141660100160125ustar00rootroot00000000000000qiime-2021.8.0/qiime2/metadata/__init__.py000066400000000000000000000011731412141660100201250ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from .metadata import (Metadata, MetadataColumn, NumericMetadataColumn, CategoricalMetadataColumn) from .io import MetadataFileError __all__ = ['Metadata', 'MetadataColumn', 'NumericMetadataColumn', 'CategoricalMetadataColumn', 'MetadataFileError'] qiime-2021.8.0/qiime2/metadata/base.py000066400000000000000000000040111412141660100172720ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- SUPPORTED_COLUMN_TYPES = {'categorical', 'numeric'} SUPPORTED_ID_HEADERS = { 'case_insensitive': { 'id', 'sampleid', 'sample id', 'sample-id', 'featureid', 'feature id', 'feature-id' }, # For backwards-compatibility with existing formats. 'exact_match': { # QIIME 1 mapping files. "#Sample ID" was never supported, but # we're including it here for symmetry with the other supported # headers that allow a space between words. '#SampleID', '#Sample ID', # biom-format: observation metadata and "classic" (TSV) OTU tables. '#OTUID', '#OTU ID', # Qiita sample/prep information files. 'sample_name' } } FORMATTED_ID_HEADERS = "Case-insensitive: %s\n\nCase-sensitive: %s" % ( ', '.join(repr(e) for e in sorted( SUPPORTED_ID_HEADERS['case_insensitive'])), ', '.join(repr(e) for e in sorted( SUPPORTED_ID_HEADERS['exact_match'])) ) def is_id_header(name): """Determine if a name is a valid ID column header. This function may be used to determine if a value in a metadata file is a valid ID column header, or if a pandas ``Index.name`` matches the ID header requirements. The "ID header" corresponds to the ``Metadata.id_header`` and ``MetadataColumn.id_header`` properties. Parameters ---------- name : string or None Name to check against ID header requirements. Returns ------- bool ``True`` if `name` is a valid ID column header, ``False`` otherwise. """ return name and (name in SUPPORTED_ID_HEADERS['exact_match'] or name.lower() in SUPPORTED_ID_HEADERS['case_insensitive']) qiime-2021.8.0/qiime2/metadata/io.py000066400000000000000000000421461412141660100170020ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import csv import itertools import os.path import re import numpy as np import pandas as pd from qiime2.core.util import find_duplicates from .base import SUPPORTED_COLUMN_TYPES, FORMATTED_ID_HEADERS, is_id_header from .metadata import Metadata, MetadataColumn class MetadataFileError(Exception): _suffix = ( "There may be more errors present in the metadata file. To get a full " "report, sample/feature metadata files can be validated with Keemei: " "https://keemei.qiime2.org\n\nFind details on QIIME 2 metadata " "requirements here: https://docs.qiime2.org/%s/tutorials/metadata/") def __init__(self, message, include_suffix=True): # Lazy import because `qiime2.__release__` is available at runtime but # not at import time (otherwise the release value could be interpolated # into `_suffix` in the class definition above). import qiime2 if include_suffix: message = message + '\n\n' + self._suffix % qiime2.__release__ super().__init__(message) class MetadataReader: def __init__(self, filepath): if not os.path.isfile(filepath): raise MetadataFileError( "Metadata file path doesn't exist, or the path points to " "something other than a file. Please check that the path " "exists, has read permissions, and points to a regular file " "(not a directory): %s" % filepath) self._filepath = filepath # Used by `read()` to store an iterator yielding rows with # leading/trailing whitespace stripped from their cells (this is a # preprocessing step that should happen with *every* row). The iterator # protocol is the only guaranteed API on this object. self._reader = None def read(self, into, column_types=None): if column_types is None: column_types = {} try: # Newline settings based on recommendation from csv docs: # https://docs.python.org/3/library/csv.html#id3 # Ignore BOM on read (but do not write BOM) with open(self._filepath, 'r', newline='', encoding='utf-8-sig') as fh: tsv_reader = csv.reader(fh, dialect='excel-tab', strict=True) self._reader = (self._strip_cell_whitespace(row) for row in tsv_reader) header = self._read_header() directives = self._read_directives(header) ids, data = self._read_data(header) except UnicodeDecodeError as e: if ('0xff in position 0' in str(e) or '0xfe in position 0' in str(e)): raise MetadataFileError( "Metadata file must be encoded as UTF-8 or ASCII, found " "UTF-16. If this file is from Microsoft Excel, save " "as a plain text file, not 'UTF-16 Unicode'") raise MetadataFileError( "Metadata file must be encoded as UTF-8 or ASCII. The " "following error occurred when decoding the file:\n\n%s" % e) finally: self._reader = None index = pd.Index(ids, name=header[0], dtype=object) df = pd.DataFrame(data, columns=header[1:], index=index, dtype=object) for name, type in column_types.items(): if name not in df.columns: raise MetadataFileError( "Column name %r specified in `column_types` is not a " "column in the metadata file." % name) if type not in SUPPORTED_COLUMN_TYPES: fmt_column_types = ', '.join( repr(e) for e in sorted(SUPPORTED_COLUMN_TYPES)) raise MetadataFileError( "Column name %r specified in `column_types` has an " "unrecognized column type %r. Supported column types: %s" % (name, type, fmt_column_types)) resolved_column_types = directives.get('types', {}) resolved_column_types.update(column_types) try: # Cast each column to the appropriate dtype based on column type. df = df.apply(self._cast_column, axis='index', column_types=resolved_column_types) except MetadataFileError as e: # HACK: If an exception is raised within `DataFrame.apply`, pandas # adds an extra tuple element to `e.args`, making the original # error message difficult to read because a tuple is repr'd instead # of a string. To work around this, we catch and reraise a # MetadataFileError with the original error message. We use # `include_suffix=False` to avoid adding another suffix to the # error message we're reraising. msg = e.args[0] raise MetadataFileError(msg, include_suffix=False) try: return into(df) except Exception as e: raise MetadataFileError( "There was an issue with loading the metadata file:\n\n%s" % e) def _read_header(self): header = None for row in self._reader: if self._is_header(row): header = row break elif self._is_comment(row): continue elif self._is_empty(row): continue elif self._is_directive(row): raise MetadataFileError( "Found directive %r while searching for header. " "Directives may only appear immediately after the header." % row[0]) else: raise MetadataFileError( "Found unrecognized ID column name %r while searching for " "header. The first column name in the header defines the " "ID column, and must be one of these values:\n\n%s\n\n" "NOTE: Metadata files must contain tab-separated values." % (row[0], FORMATTED_ID_HEADERS)) if header is None: raise MetadataFileError( "Failed to locate header. The metadata file may be empty, or " "consists only of comments or empty rows.") # Trim trailing empty cells from header. data_extent = None for idx, cell in enumerate(header): if cell != '': data_extent = idx header = header[:data_extent+1] # Basic validation to 1) fail early before processing entire file; and # 2) make some basic guarantees about the header for things in this # class that use the header as part of reading the file. column_names = set(header) if '' in column_names: raise MetadataFileError( "Found at least one column without a name in the header. Each " "column must be named.") elif len(header) != len(column_names): duplicates = find_duplicates(header) raise MetadataFileError( "Column names must be unique. The following column names are " "duplicated: %s" % (', '.join(repr(e) for e in sorted(duplicates)))) # Skip the first element of the header because we know it is a valid ID # header. The other column names are validated to ensure they *aren't* # valid ID headers. for column_name in header[1:]: if is_id_header(column_name): raise MetadataFileError( "Metadata column name %r conflicts with a name reserved " "for the ID column header. Reserved ID column headers:" "\n\n%s" % (column_name, FORMATTED_ID_HEADERS)) return header def _read_directives(self, header): directives = {} for row in self._reader: if not self._is_directive(row): self._reader = itertools.chain([row], self._reader) break if not self._is_column_types_directive(row): raise MetadataFileError( "Unrecognized directive %r. Only the #q2:types " "directive is supported at this time." % row[0]) if 'types' in directives: raise MetadataFileError( "Found duplicate directive %r. Each directive may " "only be specified a single time." % row[0]) row = self._match_header_len(row, header) column_types = {} for column_name, column_type in zip(header[1:], row[1:]): if column_type: type_nocase = column_type.lower() if type_nocase in SUPPORTED_COLUMN_TYPES: column_types[column_name] = type_nocase else: fmt_column_types = ', '.join( repr(e) for e in sorted(SUPPORTED_COLUMN_TYPES)) raise MetadataFileError( "Column %r has an unrecognized column type %r " "specified in its #q2:types directive. " "Supported column types (case-insensitive): %s" % (column_name, column_type, fmt_column_types)) directives['types'] = column_types return directives def _read_data(self, header): ids = [] data = [] for row in self._reader: if self._is_comment(row): continue elif self._is_empty(row): continue elif self._is_directive(row): raise MetadataFileError( "Found directive %r outside of the directives section of " "the file. Directives may only appear immediately after " "the header." % row[0]) elif self._is_header(row): raise MetadataFileError( "Metadata ID %r conflicts with a name reserved for the ID " "column header. Reserved ID column headers:\n\n%s" % (row[0], FORMATTED_ID_HEADERS)) row = self._match_header_len(row, header) ids.append(row[0]) data.append(row[1:]) return ids, data def _strip_cell_whitespace(self, row): return [cell.strip() for cell in row] def _match_header_len(self, row, header): row_len = len(row) header_len = len(header) if row_len < header_len: # Pad row with empty cells to match header length. row = row + [''] * (header_len - row_len) elif row_len > header_len: trailing_row = row[header_len:] if not self._is_empty(trailing_row): raise MetadataFileError( "Metadata row contains more cells than are declared by " "the header. The row has %d cells, while the header " "declares %d cells." % (row_len, header_len)) row = row[:header_len] return row def _is_empty(self, row): # `all` returns True for an empty iterable, so this check works for a # row of zero elements (corresponds to a blank line in the file). return all((cell == '' for cell in row)) def _is_comment(self, row): return ( len(row) > 0 and row[0].startswith('#') and not self._is_directive(row) and not self._is_header(row) ) def _is_header(self, row): if len(row) == 0: return False return is_id_header(row[0]) def _is_directive(self, row): return len(row) > 0 and row[0].startswith('#q2:') def _is_column_types_directive(self, row): return len(row) > 0 and row[0] == '#q2:types' def _cast_column(self, series, column_types): if series.name in column_types: if column_types[series.name] == 'numeric': return self._to_numeric(series) else: # 'categorical' return self._to_categorical(series) else: # Infer type try: return self._to_numeric(series) except MetadataFileError: return self._to_categorical(series) def _to_categorical(self, series): # Replace empty strings with `None` to force the series to remain # dtype=object (this only matters if the series consists solely of # missing data). Replacing with np.nan and casting to dtype=object # won't retain the correct dtype in the resulting dataframe # (`DataFrame.apply` seems to force series consisting solely of np.nan # to dtype=float64, even if dtype=object is specified. # # To replace a value with `None`, the following invocation of # `Series.replace` must be used because `None` is a sentinel: # https://stackoverflow.com/a/17097397/3776794 return series.replace([''], [None]) def _to_numeric(self, series): series = series.replace('', np.nan) is_numeric = series.apply(self._is_numeric) if is_numeric.all(): return pd.to_numeric(series, errors='raise') else: non_numerics = series[~is_numeric].unique() raise MetadataFileError( "Cannot convert metadata column %r to numeric. The following " "values could not be interpreted as numeric: %s" % (series.name, ', '.join(repr(e) for e in sorted(non_numerics)))) def _is_numeric(self, value): return (isinstance(value, float) or len(_numeric_regex.findall(value)) == 1) class MetadataWriter: def __init__(self, metadata): self._metadata = metadata def write(self, filepath): # Newline settings based on recommendation from csv docs: # https://docs.python.org/3/library/csv.html#id3 # Do NOT write a BOM, hence utf-8 not utf-8-sig with open(filepath, 'w', newline='', encoding='utf-8') as fh: tsv_writer = csv.writer(fh, dialect='excel-tab', strict=True) md = self._metadata header = [md.id_header] types_directive = ['#q2:types'] if isinstance(md, Metadata): for name, props in md.columns.items(): header.append(name) types_directive.append(props.type) elif isinstance(md, MetadataColumn): header.append(md.name) types_directive.append(md.type) else: raise NotImplementedError tsv_writer.writerow(header) tsv_writer.writerow(types_directive) df = md.to_dataframe() df.fillna('', inplace=True) df = df.applymap(self._format) tsv_writer.writerows(df.itertuples(index=True)) def _format(self, value): if isinstance(value, str): return value elif isinstance(value, float): # Use fixed precision or scientific notation as necessary (both are # roundtrippable in the metadata file format), with up to 15 digits # *total* precision (i.e. before and after the decimal point), # rounding if necessary. Trailing zeros or decimal points will not # be included in the formatted string (e.g. 42.0 will be formatted # as "42"). A precision of 15 digits is used because that is within # the 64-bit floating point spec (things get weird after that). # # Using repr() and str() each have their own predefined precision # which varies across Python versions. Using the string formatting # presentation types (e.g. %g, %f) without specifying a precision # will usually default to 6 digits past the decimal point, which # seems a little low. # # References: # # - https://stackoverflow.com/a/2440786/3776794 # - https://stackoverflow.com/a/2440708/3776794 # - https://docs.python.org/3/library/string.html# # format-specification-mini-language # - https://stackoverflow.com/a/20586479/3776794 # - https://drj11.wordpress.com/2007/07/03/python-poor-printing- # of-floating-point/ return '{0:.15g}'.format(value) else: raise NotImplementedError # Credit: https://stackoverflow.com/a/4703508/3776794 _numeric_pattern = r""" ^[-+]? # optional sign (?: (?: \d* \. \d+ ) # .1 .12 .123 etc 9.1 etc 98.1 etc | (?: \d+ \.? ) # 1. 12. 123. etc 1 12 123 etc ) # followed by optional exponent part if desired (?: [Ee] [+-]? \d+ ) ?$ """ _numeric_regex = re.compile(_numeric_pattern, re.VERBOSE) qiime-2021.8.0/qiime2/metadata/metadata.py000066400000000000000000001157261412141660100201600ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import abc import collections import itertools import sqlite3 import types import warnings import pandas as pd import numpy as np import qiime2 from qiime2.core.util import find_duplicates from .base import SUPPORTED_COLUMN_TYPES, FORMATTED_ID_HEADERS, is_id_header class _MetadataBase: """Base class for functionality shared between Metadata and MetadataColumn. Parameters ---------- index : pandas.Index IDs associated with the metadata. """ @property def id_header(self): """Name identifying the IDs associated with the metadata. This property is read-only. Returns ------- str Name of IDs associated with the metadata. """ return self._id_header @property def ids(self): """IDs associated with the metadata. This property is read-only. Returns ------- tuple of str Metadata IDs. """ return self._ids @property def id_count(self): """Number of metadata IDs. This property is read-only. Returns ------- int Number of metadata IDs. """ return len(self._ids) @property def artifacts(self): """Artifacts that are the source of the metadata. This property is read-only. Returns ------- tuple of qiime2.Artifact Source artifacts of the metadata. """ return tuple(self._artifacts) def __init__(self, index): if index.empty: raise ValueError( "%s must contain at least one ID." % self.__class__.__name__) id_header = index.name self._assert_valid_id_header(id_header) self._id_header = id_header self._validate_index(index, axis='id') self._ids = tuple(index) self._artifacts = [] def __eq__(self, other): return ( isinstance(other, self.__class__) and self._id_header == other._id_header and self._artifacts == other._artifacts ) def __ne__(self, other): return not (self == other) def _add_artifacts(self, artifacts): deduped = set(self._artifacts) for artifact in artifacts: if not isinstance(artifact, qiime2.Artifact): raise TypeError( "Expected Artifact object, received %r" % artifact) if artifact in deduped: raise ValueError( "Duplicate source artifacts are not supported on %s " "objects. The following artifact is a duplicate of " "another source artifact: %r" % (self.__class__.__name__, artifact)) deduped.add(artifact) self._artifacts.extend(artifacts) # Static helpers below for code reuse in Metadata and MetadataColumn @classmethod def _assert_valid_id_header(cls, name): if not is_id_header(name): raise ValueError( "pandas index name (`Index.name`) must be one of the " "following values, not %r:\n\n%s" % (name, FORMATTED_ID_HEADERS)) @classmethod def _validate_index(cls, index, *, axis): if axis == 'id': label = 'ID' elif axis == 'column': label = 'column name' else: raise NotImplementedError for value in index: if not isinstance(value, str): raise TypeError( "Detected non-string metadata %s of type %r: %r" % (label, type(value), value)) if not value: raise ValueError( "Detected empty metadata %s. %ss must consist of at least " "one character." % (label, label)) if axis == 'id' and value.startswith('#'): raise ValueError( "Detected metadata %s that begins with a pound sign " "(#): %r" % (label, value)) if is_id_header(value): raise ValueError( "Detected metadata %s %r that conflicts with a name " "reserved for the ID header. Reserved ID headers:\n\n%s" % (label, value, FORMATTED_ID_HEADERS)) if len(index) != len(set(index)): duplicates = find_duplicates(index) raise ValueError( "Metadata %ss must be unique. The following %ss are " "duplicated: %s" % (label, label, ', '.join(repr(e) for e in sorted(duplicates)))) @classmethod def _filter_ids_helper(cls, df_or_series, ids, ids_to_keep): # `ids_to_keep` can be any iterable, so turn it into a list so that it # can be iterated over multiple times below (and length-checked). ids_to_keep = list(ids_to_keep) if len(ids_to_keep) == 0: raise ValueError("`ids_to_keep` must contain at least one ID.") duplicates = find_duplicates(ids_to_keep) if duplicates: raise ValueError( "`ids_to_keep` must contain unique IDs. The following IDs are " "duplicated: %s" % (', '.join(repr(e) for e in sorted(duplicates)))) ids_to_keep = set(ids_to_keep) missing_ids = ids_to_keep - ids if missing_ids: raise ValueError( "The following IDs are not present in the metadata: %s" % (', '.join(repr(e) for e in sorted(missing_ids)))) # While preserving order, get rid of any IDs not contained in # `ids_to_keep`. ids_to_discard = ids - ids_to_keep return df_or_series.drop(labels=ids_to_discard, axis='index', inplace=False, errors='raise') def save(self, filepath, ext=None): """Save a TSV metadata file. The TSV metadata file format is described at https://docs.qiime2.org in the Metadata Tutorial. The file will always include the ``#q2:types`` directive in order to make the file roundtrippable without relying on column type inference. Parameters ---------- filepath : str Path to save TSV metadata file at. ext : str Preferred file extension (.tsv, .txt, etc). Will be left blank if no extension is included. Including a period in the extension is optional, and any additional periods delimiting the filepath and the extension will be reduced to a single period. Returns ------- str Filepath and extension (if provided) that the file was saved to. See Also -------- Metadata.load """ from .io import MetadataWriter if ext is None: ext = '' else: ext = '.' + ext.lstrip('.') filepath = filepath.rstrip('.') if not filepath.endswith(ext): filepath += ext MetadataWriter(self).write(filepath) return filepath # Other properties such as units can be included here in the future! ColumnProperties = collections.namedtuple('ColumnProperties', ['type']) class Metadata(_MetadataBase): """Store metadata associated with identifiers in a study. Metadata is tabular in nature, mapping study identifiers (e.g. sample or feature IDs) to columns of metadata associated with each ID. For more details about metadata in QIIME 2, including the TSV metadata file format, see the Metadata Tutorial at https://docs.qiime2.org. The following text focuses on design and considerations when working with ``Metadata`` objects at the API level. A ``Metadata`` object is composed of zero or more ``MetadataColumn`` objects. A ``Metadata`` object always contains at least one ID, regardless of the number of columns. Each column in the ``Metadata`` object has an associated column type representing either *categorical* or *numeric* data. Each metadata column is represented by an object corresponding to the column's type: ``CategoricalMetadataColumn`` or ``NumericMetadataColumn``, respectively. A ``Metadata`` object is closely linked to its corresponding TSV metadata file format described at https://docs.qiime2.org. Therefore, certain requirements present in the file format are also enforced on the in-memory object in order to make serialized ``Metadata`` objects roundtrippable when loaded from disk again. For example, IDs cannot begin with a pound character (``#``) because those IDs would be interpreted as comment rows when written to disk as TSV. See the metadata file format spec for more details about data formatting requirements. In addition to being loaded from or saved to disk, a ``Metadata`` object can be constructed from a ``pandas.DataFrame`` object. See the *Parameters* section below for details on how to construct ``Metadata`` objects from dataframes. ``Metadata`` objects have various methods to access, filter, and merge data. A dataframe can be retrieved from the ``Metadata`` object for further data manipulation using the pandas API. Individual ``MetadataColumn`` objects can be retrieved to gain access to APIs applicable to a single metadata column. Parameters ---------- dataframe : pandas.DataFrame Dataframe containing metadata. The dataframe's index defines the IDs, and the index name (``Index.name``) must match one of the required ID headers described in the metadata file format spec. Each column in the dataframe defines a metadata column, and the metadata column's type (i.e. *categorical* or *numeric*) is determined based on the column's dtype. If a column has ``dtype=object``, it may contain strings or pandas missing values (e.g. ``np.nan``, ``None``). Columns matching this requirement are assumed to be *categorical*. If a column in the dataframe has ``dtype=float`` or ``dtype=int``, it may contain floating point numbers or integers, as well as pandas missing values (e.g. ``np.nan``). Columns matching this requirement are assumed to be *numeric*. Regardless of column type (categorical vs numeric), the dataframe stored within the ``Metadata`` object will have any missing values normalized to ``np.nan``. Columns with ``dtype=int`` will be cast to ``dtype=float``. To obtain a dataframe from the ``Metadata`` object containing these normalized data types and values, use ``Metadata.to_dataframe()``. """ @classmethod def load(cls, filepath, column_types=None): """Load a TSV metadata file. The TSV metadata file format is described at https://docs.qiime2.org in the Metadata Tutorial. Parameters ---------- filepath : str Path to TSV metadata file to be loaded. column_types : dict, optional Override metadata column types specified or inferred in the file. This is a dict mapping column names (str) to column types (str). Valid column types are 'categorical' and 'numeric'. Column names may be omitted from this dict to use the column types read from the file. Returns ------- Metadata Metadata object loaded from `filepath`. Raises ------ MetadataFileError If the metadata file is invalid in any way (e.g. doesn't meet the file format's requirements). See Also -------- save """ from .io import MetadataReader return MetadataReader(filepath).read(into=cls, column_types=column_types) @property def columns(self): """Ordered mapping of column names to ColumnProperties. The mapping that is returned is read-only. This property is also read-only. Returns ------- types.MappingProxyType Ordered mapping of column names to ColumnProperties. """ # Read-only proxy to the OrderedDict mapping column names to # ColumnProperties. return types.MappingProxyType(self._columns) @property def column_count(self): """Number of metadata columns. This property is read-only. Returns ------- int Number of metadata columns. Notes ----- Zero metadata columns are allowed. See Also -------- id_count """ return len(self._columns) def __init__(self, dataframe): if not isinstance(dataframe, pd.DataFrame): raise TypeError( "%s constructor requires a pandas.DataFrame object, not " "%r" % (self.__class__.__name__, type(dataframe))) super().__init__(dataframe.index) self._dataframe, self._columns = self._normalize_dataframe(dataframe) self._validate_index(self._dataframe.columns, axis='column') def _normalize_dataframe(self, dataframe): norm_df = dataframe.copy() # Do not attempt to strip empty metadata if not norm_df.columns.empty: norm_df.columns = norm_df.columns.str.strip() norm_df.index = norm_df.index.str.strip() columns = collections.OrderedDict() for column_name, series in norm_df.items(): metadata_column = self._metadata_column_factory(series) norm_df[column_name] = metadata_column.to_series() properties = ColumnProperties(type=metadata_column.type) columns[column_name] = properties return norm_df, columns def _metadata_column_factory(self, series): dtype = series.dtype if NumericMetadataColumn._is_supported_dtype(dtype): column = NumericMetadataColumn(series) elif CategoricalMetadataColumn._is_supported_dtype(dtype): column = CategoricalMetadataColumn(series) else: raise TypeError( "Metadata column %r has an unsupported pandas dtype of %s. " "Supported dtypes: float, int, object" % (series.name, dtype)) column._add_artifacts(self.artifacts) return column def __repr__(self): """String summary of the metadata and its columns.""" lines = [] # Header lines.append(self.__class__.__name__) lines.append('-' * len(self.__class__.__name__)) # Dimensions lines.append('%d ID%s x %d column%s' % ( self.id_count, '' if self.id_count == 1 else 's', self.column_count, '' if self.column_count == 1 else 's', )) # Column properties if self.column_count != 0: max_name_len = max((len(name) for name in self.columns)) for name, props in self.columns.items(): padding = ' ' * ((max_name_len - len(name)) + 1) lines.append('%s:%s%r' % (name, padding, props)) # Epilogue lines.append('') lines.append('Call to_dataframe() for a tabular representation.') return '\n'.join(lines) def __eq__(self, other): """Determine if this metadata is equal to another. ``Metadata`` objects are equal if their IDs, columns (including column names, types, and ordering), ID headers, source artifacts, and metadata values are equal. Parameters ---------- other : Metadata Metadata to test for equality. Returns ------- bool Indicates whether this ``Metadata`` object is equal to `other`. See Also -------- __ne__ """ return ( super().__eq__(other) and self._columns == other._columns and self._dataframe.equals(other._dataframe) ) def __ne__(self, other): """Determine if this metadata is not equal to another. ``Metadata`` objects are not equal if their IDs, columns (including column names, types, or ordering), ID headers, source artifacts, or metadata values are not equal. Parameters ---------- other : Metadata Metadata to test for inequality. Returns ------- bool Indicates whether this ``Metadata`` object is not equal to `other`. See Also -------- __eq__ """ return not (self == other) def to_dataframe(self): """Create a pandas dataframe from the metadata. The dataframe's index name (``Index.name``) will match this metadata object's ``id_header``, and the index will contain this metadata object's IDs. The dataframe's column names will match the column names in this metadata. Categorical columns will be stored as ``dtype=object`` (containing strings), and numeric columns will be stored as ``dtype=float``. Returns ------- pandas.DataFrame Dataframe constructed from the metadata. """ return self._dataframe.copy() def get_column(self, name): """Retrieve metadata column based on column name. Parameters ---------- name : str Name of the metadata column to retrieve. Returns ------- MetadataColumn Requested metadata column (``CategoricalMetadataColumn`` or ``NumericMetadataColumn``). See Also -------- get_ids """ try: series = self._dataframe[name] except KeyError: raise ValueError( '%r is not a column in the metadata. Available columns: ' '%s' % (name, ', '.join(repr(c) for c in self.columns))) return self._metadata_column_factory(series) def get_ids(self, where=None): """Retrieve IDs matching search criteria. Parameters ---------- where : str, optional SQLite WHERE clause specifying criteria IDs must meet to be included in the results. All IDs are included by default. Returns ------- set IDs matching search criteria specified in `where`. See Also -------- ids filter_ids get_column Notes ----- The ID header (``Metadata.id_header``) may be used in the `where` clause to query the table's ID column. """ if where is None: return set(self._ids) conn = sqlite3.connect(':memory:') conn.row_factory = lambda cursor, row: row[0] # https://github.com/pandas-dev/pandas/blob/ # 7c7bd569ce8e0f117c618d068e3d2798134dbc73/pandas/io/sql.py#L1306 with warnings.catch_warnings(): warnings.filterwarnings( 'ignore', 'The spaces in these column names will not.*') self._dataframe.to_sql('metadata', conn, index=True, index_label=self.id_header) c = conn.cursor() # In general we wouldn't want to format our query in this way because # it leaves us open to sql injection, but it seems acceptable here for # a few reasons: # 1) This is a throw-away database which we're just creating to have # access to the query language, so any malicious behavior wouldn't # impact any data that isn't temporary # 2) The substitution syntax recommended in the docs doesn't allow # us to specify complex `where` statements, which is what we need to # do here. For example, we need to specify things like: # WHERE Subject='subject-1' AND SampleType='gut' # but their qmark/named-style syntaxes only supports substition of # variables, such as: # WHERE Subject=? # 3) sqlite3.Cursor.execute will only execute a single statement so # inserting multiple statements # (e.g., "Subject='subject-1'; DROP...") will result in an # OperationalError being raised. query = ('SELECT "{0}" FROM metadata WHERE {1} GROUP BY "{0}" ' 'ORDER BY "{0}";'.format(self.id_header, where)) try: c.execute(query) except sqlite3.OperationalError as e: conn.close() raise ValueError("Selection of IDs failed with query:\n %s\n\n" "If one of the metadata column names specified " "in the `where` statement is on this list " "of reserved keywords " "(http://www.sqlite.org/lang_keywords.html), " "please ensure it is quoted appropriately in the " "`where` statement." % query) from e ids = set(c.fetchall()) conn.close() return ids def merge(self, *others): """Merge this ``Metadata`` object with other ``Metadata`` objects. Returns a new ``Metadata`` object containing the merged contents of this ``Metadata`` object and `others`. The merge is not in-place and will always return a **new** merged ``Metadata`` object. The merge will include only those IDs that are shared across **all** ``Metadata`` objects being merged (i.e. the merge is an *inner join*). Each metadata column being merged must have a unique name; merging metadata with overlapping column names will result in an error. Parameters ---------- others : tuple One or more ``Metadata`` objects to merge with this ``Metadata`` object. Returns ------- Metadata New object containing merged metadata. The merged IDs will be in the same relative order as the IDs in this ``Metadata`` object after performing the inner join. The merged column order will match the column order of ``Metadata`` objects being merged from left to right. Raises ------ ValueError If zero ``Metadata`` objects are provided in `others` (there is nothing to merge in this case). Notes ----- The merged ``Metadata`` object will always have its ``id_header`` property set to ``'id'``, regardless of the ``id_header`` values on the ``Metadata`` objects being merged. The merged ``Metadata`` object tracks all source artifacts that it was built from to preserve provenance (i.e. the ``.artifacts`` property on all ``Metadata`` objects is merged). """ if len(others) < 1: raise ValueError( "At least one Metadata object must be provided to merge into " "this Metadata object (otherwise there is nothing to merge).") dfs = [] columns = [] artifacts = [] for md in itertools.chain([self], others): df = md._dataframe dfs.append(df) columns.extend(df.columns.tolist()) artifacts.extend(md.artifacts) columns = pd.Index(columns) if columns.has_duplicates: raise ValueError( "Cannot merge metadata with overlapping columns. The " "following columns overlap: %s" % ', '.join([repr(e) for e in columns[columns.duplicated()].unique()])) merged_df = dfs[0].join(dfs[1:], how='inner') # Not using DataFrame.empty because empty columns are allowed in # Metadata. if merged_df.index.empty: raise ValueError( "Cannot merge because there are no IDs shared across metadata " "objects.") merged_df.index.name = 'id' merged_md = self.__class__(merged_df) merged_md._add_artifacts(artifacts) return merged_md def filter_ids(self, ids_to_keep): """Filter metadata by IDs. Parameters ---------- ids_to_keep : iterable of str IDs that should be retained in the filtered ``Metadata`` object. If any IDs in `ids_to_keep` are not contained in this ``Metadata`` object, a ``ValueError`` will be raised. The filtered ``Metadata`` object will retain the same relative ordering of IDs in this ``Metadata`` object. Thus, the ordering of IDs in `ids_to_keep` does not determine the ordering of IDs in the filtered ``Metadata`` object. Returns ------- Metadata The metadata filtered by IDs. See Also -------- get_ids filter_columns """ filtered_df = self._filter_ids_helper(self._dataframe, self.get_ids(), ids_to_keep) filtered_md = self.__class__(filtered_df) filtered_md._add_artifacts(self.artifacts) return filtered_md def filter_columns(self, *, column_type=None, drop_all_unique=False, drop_zero_variance=False, drop_all_missing=False): """Filter metadata by columns. Parameters ---------- column_type : str, optional If supplied, will retain only columns of this type. The currently supported column types are 'numeric' and 'categorical'. drop_all_unique : bool, optional If ``True``, columns that contain a unique value for every ID will be dropped. Missing data (``np.nan``) are ignored when determining unique values. If a column consists solely of missing data, it will be dropped. drop_zero_variance : bool, optional If ``True``, columns that contain the same value for every ID will be dropped. Missing data (``np.nan``) are ignored when determining variance. If a column consists solely of missing data, it will be dropped. drop_all_missing : bool, optional If ``True``, columns that have a missing value (``np.nan``) for every ID will be dropped. Returns ------- Metadata The metadata filtered by columns. See Also -------- filter_ids """ if (column_type is not None and column_type not in SUPPORTED_COLUMN_TYPES): raise ValueError( "Unknown column type %r. Supported column types: %s" % (column_type, ', '.join(sorted(SUPPORTED_COLUMN_TYPES)))) # Build up a set of columns to drop. Short-circuit as soon as we know a # given column can be dropped (no need to apply further filters to it). columns_to_drop = set() for column, props in self.columns.items(): if column_type is not None and props.type != column_type: columns_to_drop.add(column) continue series = self._dataframe[column] if drop_all_unique or drop_zero_variance: # Ignore nans in the unique count, and compare to the number of # non-nan values in the series. num_unique = series.nunique(dropna=True) if drop_all_unique and num_unique == series.count(): columns_to_drop.add(column) continue # If num_unique == 0, the series was empty (all nans). If # num_unique == 1, the series contained only a single unique # value (ignoring nans). if drop_zero_variance and num_unique < 2: columns_to_drop.add(column) continue if drop_all_missing and series.isnull().all(): columns_to_drop.add(column) continue filtered_df = self._dataframe.drop(columns_to_drop, axis=1, inplace=False) filtered_md = self.__class__(filtered_df) filtered_md._add_artifacts(self.artifacts) return filtered_md class MetadataColumn(_MetadataBase, metaclass=abc.ABCMeta): """Abstract base class representing a single metadata column. Concrete subclasses represent specific metadata column types, e.g. ``CategoricalMetadataColumn`` and ``NumericMetadataColumn``. See the ``Metadata`` class docstring for details about ``Metadata`` and ``MetadataColumn`` objects, including a description of column types. The main difference in constructing ``MetadataColumn`` vs ``Metadata`` objects is that ``MetadataColumn`` objects are constructed from a ``pandas.Series`` object instead of a ``pandas.DataFrame``. Otherwise, the same restrictions, considerations, and data normalization are applied as with ``Metadata`` objects. """ # Abstract, must be defined by subclasses. type = None @classmethod @abc.abstractmethod def _is_supported_dtype(cls, dtype): """ Contract: Return ``True`` if the series `dtype` is supported by this object and can be handled appropriately by ``_normalize_``. Return ``False`` otherwise. """ raise NotImplementedError @classmethod @abc.abstractmethod def _normalize_(cls, series): """ Contract: Return a copy of `series` that has been converted to the appropriate internal dtype and has any other necessary normalization or validation applied (e.g. missing value representations, disallowing certain values, etc). Raise an error with a detailed error message if the operation cannot be completed. """ raise NotImplementedError @property def name(self): """Metadata column name. This property is read-only. Returns ------- str Metadata column name. """ return self._series.name def __init__(self, series): if not isinstance(series, pd.Series): raise TypeError( "%s constructor requires a pandas.Series object, not %r" % (self.__class__.__name__, type(series))) super().__init__(series.index) if not self._is_supported_dtype(series.dtype): raise TypeError( "%s %r does not support a pandas.Series object with dtype %s" % (self.__class__.__name__, series.name, series.dtype)) self._series = self._normalize_(series) self._validate_index([self._series.name], axis='column') def __repr__(self): """String summary of the metadata column.""" return '<%s name=%r id_count=%d>' % (self.__class__.__name__, self.name, self.id_count) def __eq__(self, other): """Determine if this metadata column is equal to another. ``MetadataColumn`` objects are equal if their IDs, column names, column types, ID headers, source artifacts, and metadata values are equal. Parameters ---------- other : MetadataColumn Metadata column to test for equality. Returns ------- bool Indicates whether this ``MetadataColumn`` object is equal to `other`. See Also -------- __ne__ """ return ( super().__eq__(other) and self.name == other.name and self._series.equals(other._series) ) def __ne__(self, other): """Determine if this metadata column is not equal to another. ``MetadataColumn`` objects are not equal if their IDs, column names, column types, ID headers, source artifacts, or metadata values are not equal. Parameters ---------- other : MetadataColumn Metadata column to test for inequality. Returns ------- bool Indicates whether this ``MetadataColumn`` object is not equal to `other`. See Also -------- __eq__ """ return not (self == other) def to_series(self): """Create a pandas series from the metadata column. The series index name (``Index.name``) will match this metadata column's ``id_header``, and the index will contain this metadata column's IDs. The series name will match this metadata column's name. Returns ------- pandas.Series Series constructed from the metadata column. See Also -------- to_dataframe """ return self._series.copy() def to_dataframe(self): """Create a pandas dataframe from the metadata column. The dataframe will contain exactly one column. The dataframe's index name (``Index.name``) will match this metadata column's ``id_header``, and the index will contain this metadata column's IDs. The dataframe's column name will match this metadata column's name. Returns ------- pandas.DataFrame Dataframe constructed from the metadata column. See Also -------- to_series """ return self._series.to_frame() def get_value(self, id): """Retrieve metadata column value associated with an ID. Parameters ---------- id : str ID corresponding to the metadata column value to retrieve. Returns ------- object Value associated with the provided `id`. """ if id not in self._series.index: raise ValueError("ID %r is not present in %r" % (id, self)) return self._series.loc[id] def has_missing_values(self): """Determine if the metadata column has one or more missing values. Returns ------- bool ``True`` if the metadata column has one or more missing values (``np.nan``), ``False`` otherwise. See Also -------- drop_missing_values get_ids """ return len(self.get_ids(where_values_missing=True)) > 0 def drop_missing_values(self): """Filter out missing values from the metadata column. Returns ------- MetadataColumn Metadata column with missing values removed. See Also -------- has_missing_values get_ids """ missing = self.get_ids(where_values_missing=True) present = self.get_ids() - missing return self.filter_ids(present) def get_ids(self, where_values_missing=False): """Retrieve IDs matching search criteria. Parameters ---------- where_values_missing : bool, optional If ``True``, only return IDs that are associated with missing values (``np.nan``). If ``False`` (the default), return all IDs in the metadata column. Returns ------- set IDs matching search criteria. See Also -------- ids filter_ids has_missing_values drop_missing_values """ if where_values_missing: ids = self._series[self._series.isnull()].index else: ids = self._ids return set(ids) def filter_ids(self, ids_to_keep): """Filter metadata column by IDs. Parameters ---------- ids_to_keep : iterable of str IDs that should be retained in the filtered ``MetadataColumn`` object. If any IDs in `ids_to_keep` are not contained in this ``MetadataColumn`` object, a ``ValueError`` will be raised. The filtered ``MetadataColumn`` object will retain the same relative ordering of IDs in this ``MetadataColumn`` object. Thus, the ordering of IDs in `ids_to_keep` does not determine the ordering of IDs in the filtered ``MetadataColumn`` object. Returns ------- MetadataColumn The metadata column filtered by IDs. See Also -------- get_ids """ filtered_series = self._filter_ids_helper(self._series, self.get_ids(), ids_to_keep) filtered_mdc = self.__class__(filtered_series) filtered_mdc._add_artifacts(self.artifacts) return filtered_mdc class CategoricalMetadataColumn(MetadataColumn): """A single metadata column containing categorical data. See the ``Metadata`` class docstring for details about ``Metadata`` and ``MetadataColumn`` objects, including a description of column types and supported data formats. """ type = 'categorical' @classmethod def _is_supported_dtype(cls, dtype): return dtype == 'object' @classmethod def _normalize_(cls, series): def normalize(value): if isinstance(value, str): value = value.strip() if value == '': raise ValueError( "%s does not support empty strings as values. Use an " "appropriate pandas missing value type " "(e.g. `numpy.nan`) or supply a non-empty string as " "the value in column %r." % (cls.__name__, series.name)) else: return value elif pd.isnull(value): # permits np.nan, Python float nan, None return np.nan else: raise TypeError( "%s only supports strings or missing values. Found value " "%r of type %r in column %r." % (cls.__name__, value, type(value), series.name)) norm_series = series.apply(normalize, convert_dtype=False) norm_series.index = norm_series.index.str.strip() norm_series.name = norm_series.name.strip() return norm_series class NumericMetadataColumn(MetadataColumn): """A single metadata column containing numeric data. See the ``Metadata`` class docstring for details about ``Metadata`` and ``MetadataColumn`` objects, including a description of column types and supported data formats. """ type = 'numeric' @classmethod def _is_supported_dtype(cls, dtype): return dtype == 'float' or dtype == 'int' @classmethod def _normalize_(cls, series): series = series.astype(float, copy=True, errors='raise') if np.isinf(series).any(): raise ValueError( "%s does not support positive or negative infinity as a " "floating point value in column %r." % (cls.__name__, series.name)) return series qiime-2021.8.0/qiime2/metadata/tests/000077500000000000000000000000001412141660100171545ustar00rootroot00000000000000qiime-2021.8.0/qiime2/metadata/tests/__init__.py000066400000000000000000000005351412141660100212700ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- qiime-2021.8.0/qiime2/metadata/tests/data/000077500000000000000000000000001412141660100200655ustar00rootroot00000000000000qiime-2021.8.0/qiime2/metadata/tests/data/invalid/000077500000000000000000000000001412141660100215135ustar00rootroot00000000000000qiime-2021.8.0/qiime2/metadata/tests/data/invalid/column-name-conflicts-with-id-header.tsv000066400000000000000000000001001412141660100312260ustar00rootroot00000000000000sampleid col1 featureid col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/comments-and-empty-rows-only.tsv000066400000000000000000000000561412141660100277420ustar00rootroot00000000000000# # Hello, World! # Foo, # Bar, # Baz qiime-2021.8.0/qiime2/metadata/tests/data/invalid/data-longer-than-header.tsv000066400000000000000000000001041412141660100266170ustar00rootroot00000000000000sampleid col1 col2 col3 id1 1 a foo id2 2 b bar overflow id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/directive-after-directives-section.tsv000066400000000000000000000002231412141660100311240ustar00rootroot00000000000000id col1 col2 col3 # directives must appear *immediately* below header #q2:types numeric categorical categorical id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/directive-before-header.tsv000066400000000000000000000001371412141660100267160ustar00rootroot00000000000000#q2:types numeric categorical categorical id col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/directive-longer-than-header.tsv000066400000000000000000000001551412141660100276720ustar00rootroot00000000000000sampleid col1 col2 col3 #q2:types numeric categorical categorical numeric id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/duplicate-column-names-with-whitespace.tsv000066400000000000000000000000771412141660100317260ustar00rootroot00000000000000id " col1 " col2 col1 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/duplicate-column-names.tsv000066400000000000000000000000651412141660100266200ustar00rootroot00000000000000id col1 col2 col1 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/duplicate-directives.tsv000066400000000000000000000002151412141660100263600ustar00rootroot00000000000000id col1 col2 col3 #q2:types numeric categorical categorical #q2:types categorical categorical categorical id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/duplicate-ids-with-whitespace.tsv000066400000000000000000000000741412141660100301040ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar "id1 " 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/duplicate-ids.tsv000066400000000000000000000000651412141660100250010ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar id1 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/empty-column-name.tsv000066400000000000000000000000611412141660100256150ustar00rootroot00000000000000id col1 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/empty-file000066400000000000000000000000001412141660100234770ustar00rootroot00000000000000qiime-2021.8.0/qiime2/metadata/tests/data/invalid/empty-id.tsv000066400000000000000000000000621412141660100237770ustar00rootroot00000000000000ID col1 col2 col3 id1 1 a foo 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/header-only-with-comments-and-empty-rows.tsv000066400000000000000000000000761412141660100321430ustar00rootroot00000000000000# Hello, World! id col1 col2 col3 # Foo, # Bar, # Baz qiime-2021.8.0/qiime2/metadata/tests/data/invalid/header-only.tsv000066400000000000000000000000221412141660100244520ustar00rootroot00000000000000id col1 col2 col3 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/id-conflicts-with-id-header.tsv000066400000000000000000000000721412141660100274170ustar00rootroot00000000000000sampleid col1 col2 col3 id1 1 a foo id 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/invalid-header.tsv000066400000000000000000000001041412141660100251200ustar00rootroot00000000000000invalid_id_header col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/non-utf-8.tsv000066400000000000000000000001541412141660100240040ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/qiime1-empty.tsv000066400000000000000000000001301412141660100245640ustar00rootroot00000000000000#SampleID col1 col2 col3 # A QIIME 1 mapping file can have comments # below the header. qiime-2021.8.0/qiime2/metadata/tests/data/invalid/simple-utf-16be.txt000066400000000000000000000001541412141660100250740ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/simple-utf-16le.txt000066400000000000000000000001541412141660100251060ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/types-directive-non-numeric.tsv000066400000000000000000000007451412141660100276270ustar00rootroot00000000000000# All sorts of strings that shouldn't be interpreted as numbers in `col2`! # Note that the first few values in `col2` *can* be interpreted as numbers, # just to have a mixed set of numeric and non-numeric values. id col1 col2 #q2:types numeric numeric id1 1 42 id2 1 -42.50 id3 1 id4 1 a id5 1 foo id6 1 1,000 id7 1 1.000.0 id8 1 $42 id9 1 nan id10 1 NaN id11 1 NA id12 1 inf id13 1 +inf id14 1 -inf id15 1 Infinity id16 1 1_000_000 id17 1 0xAF id18 1 1e3e4 id19 1 e3 id20 1 sample-1 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/unrecognized-column-type.tsv000066400000000000000000000001271412141660100272170ustar00rootroot00000000000000id col1 col2 col3 #q2:types numeric foo categorical id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/unrecognized-directive.tsv000066400000000000000000000001111412141660100267120ustar00rootroot00000000000000id col1 col2 col3 #q2:foo bar baz bar id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/whitespace-only-column-name.tsv000066400000000000000000000000701412141660100275720ustar00rootroot00000000000000id col1 " " col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/invalid/whitespace-only-id.tsv000066400000000000000000000000701412141660100257530ustar00rootroot00000000000000ID col1 col2 col3 id1 1 a foo " " 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/000077500000000000000000000000001412141660100211645ustar00rootroot00000000000000qiime-2021.8.0/qiime2/metadata/tests/data/valid/BOM-simple.txt000066400000000000000000000000701412141660100236260ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/all-cells-padded.tsv000066400000000000000000000000361412141660100250100ustar00rootroot00000000000000id col1 col2 col3 id1 id2 id3 qiime-2021.8.0/qiime2/metadata/tests/data/valid/biom-observation-metadata.tsv000066400000000000000000000001561412141660100267610ustar00rootroot00000000000000#OTUID taxonomy confidence # optional comments OTU_1 k__Bacteria;p__Firmicutes 0.890 OTU_2 k__Bacteria 0.9999 qiime-2021.8.0/qiime2/metadata/tests/data/valid/case-insensitive-types-directive.tsv000066400000000000000000000001361412141660100303110ustar00rootroot00000000000000id col1 col2 col3 #q2:types CATEGORICAL CategoricaL NuMeRiC id1 1 a -5 id2 2 b 0.0 id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/column-order.tsv000066400000000000000000000000541412141660100243270ustar00rootroot00000000000000id z y x id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/comments.tsv000066400000000000000000000010461412141660100235500ustar00rootroot00000000000000# pre-header # comment id col1 col2 col3 # post-header # comment id1 1 a foo id2 2 b bar # intra-data comment with another # sign # ## # comment with leading whitespace is still a comment. # comment with tab characters is also a comment! "# if the first cell is quoted, the parsing rules first process and strip double quotes, then check if the first cell begins with a pound sign" " # same rule applies if the de-quoted cell has leading whitespace (leading/trailing whitespace is *always* ignored)" id3 3 c 42 # trailing # comment qiime-2021.8.0/qiime2/metadata/tests/data/valid/complete-types-directive.tsv000066400000000000000000000001431412141660100266460ustar00rootroot00000000000000id col1 col2 col3 #q2:types categorical categorical categorical id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/empty-rows.tsv000066400000000000000000000002571412141660100240540ustar00rootroot00000000000000 id col1 col2 col3 id1 1 a foo id2 2 b bar " " " " " " " " " " id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/empty-types-directive.tsv000066400000000000000000000000771412141660100262020ustar00rootroot00000000000000id col1 col2 col3 #q2:types id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/jagged-trailing-columns.tsv000066400000000000000000000000701412141660100264250ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/jagged.tsv000066400000000000000000000000551412141660100231430ustar00rootroot00000000000000id col1 col2 col3 id1 1 a id2 2 b bar id3 3 qiime-2021.8.0/qiime2/metadata/tests/data/valid/leading-trailing-whitespace.tsv000066400000000000000000000006271412141660100272730ustar00rootroot00000000000000 # Leading/trailing whitespace is ignored in *any* type of cell, including # comments, empty rows, headers, directives, and data cells. # Double-quotes are always processed prior to stripping leading/trailing # whitespace within the cell. id "col1 " " col2" col3 #q2:types " numeric " categorical " categorical " id1 " 1 " a foo " " " id2 " 2 b "bar " id3 3 "c " 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/mac-line-endings.tsv000066400000000000000000000000651412141660100250350ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/minimal.tsv000066400000000000000000000000051412141660100233430ustar00rootroot00000000000000id a qiime-2021.8.0/qiime2/metadata/tests/data/valid/missing-data.tsv000066400000000000000000000010051412141660100242760ustar00rootroot00000000000000# Missing data can be represented with empty cells or whitespace-only cells. # Test that values used to represent missing data in other programs # (e.g. pandas) are not treated as missing (e.g. "NA", "N/A"). Also test # columns that consist solely of missing data. By default, an empty column will # be treated as numeric data (column "NA" in this example). "col4" overrides # this behavior to make its empty column categorical. id col1 NA col3 col4 #q2:types categorical None 1 null nan N/A NA " " NA qiime-2021.8.0/qiime2/metadata/tests/data/valid/no-columns.tsv000066400000000000000000000000151412141660100240100ustar00rootroot00000000000000id a b my-id qiime-2021.8.0/qiime2/metadata/tests/data/valid/no-id-or-column-name-type-cast.tsv000066400000000000000000000001101412141660100274560ustar00rootroot00000000000000id 42.0 1000 -4.2 0.000001 2 b 2.5 0.004000 1 b 4.2 0.000000 3 c -9.999 qiime-2021.8.0/qiime2/metadata/tests/data/valid/no-newline-at-eof.tsv000066400000000000000000000000641412141660100251460ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42qiime-2021.8.0/qiime2/metadata/tests/data/valid/non-standard-characters.tsv000066400000000000000000000013731412141660100264330ustar00rootroot00000000000000# See the corresponding unit test case for the goals of this file. The file # tests the following cases for IDs, column names, and cells. Many of the # choices are based on use-cases/bugs reported on the forum, Slack, etc. # # - Unicode characters # - Parentheses, underscores, less than (<), and greater than (>) # - Single and double quotes. Double quotes must be escaped according to the # Excel TSV dialect's double quote escaping rules. # - Escaped newlines (\n), carriage returns (\r), tabs (\t), and spaces # - Inline comment characters aren't treated as comments id ↩c@l1™ col(#2) #col'3 """""" "col 5" ©id##1 ƒoo (foo) #f o #o "fo o" ((id))2 ''2'' b#r "ba r" 'id_3<>' "b""ar" "c d" "4 2" """id#4""" b__a_z <42> >42 "i d 5" baz 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/numeric-column.tsv000066400000000000000000000001571412141660100246620ustar00rootroot00000000000000id col1 id1 0 id2 2.0 id3 0.00030 id4 -4.2 id5 1e-4 id6 1e4 id7 +1.5E+2 id8 id9 1. id10 .5 id11 1e-08 id12 -0 qiime-2021.8.0/qiime2/metadata/tests/data/valid/partial-types-directive.tsv000066400000000000000000000001131412141660100264670ustar00rootroot00000000000000id col1 col2 col3 #q2:types categorical id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/qiime1.tsv000066400000000000000000000001731412141660100231100ustar00rootroot00000000000000#SampleID col1 col2 col3 # A QIIME 1 mapping file can have comments # below the header. id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/qiita-preparation-information.tsv000066400000000000000000000001541412141660100276760ustar00rootroot00000000000000sample_name BARCODE EXPERIMENT_DESIGN_DESCRIPTION id.1 ACGT longitudinal study id.2 TGCA longitudinal study qiime-2021.8.0/qiime2/metadata/tests/data/valid/qiita-sample-information.tsv000066400000000000000000000001321412141660100266270ustar00rootroot00000000000000sample_name DESCRIPTION TITLE id.1 description 1 A Title id.2 description 2 Another Title qiime-2021.8.0/qiime2/metadata/tests/data/valid/recommended-ids.tsv000066400000000000000000000000731412141660100247610ustar00rootroot00000000000000id col1 c6ca034a-223f-40b4-a0e0-45942912a5ea foo My.ID bar qiime-2021.8.0/qiime2/metadata/tests/data/valid/rows-shorter-than-header.tsv000066400000000000000000000000441412141660100265540ustar00rootroot00000000000000id col1 col2 col3 id1 1 a id2 2 id3 qiime-2021.8.0/qiime2/metadata/tests/data/valid/simple-with-directive.tsv000066400000000000000000000001371412141660100261410ustar00rootroot00000000000000id col1 col2 col3 #q2:types numeric categorical categorical id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/simple.tsv000066400000000000000000000000651412141660100232140ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/simple.txt000066400000000000000000000000651412141660100232170ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/single-column.tsv000066400000000000000000000000321412141660100244710ustar00rootroot00000000000000id col1 id1 1 id2 2 id3 3 qiime-2021.8.0/qiime2/metadata/tests/data/valid/single-id.tsv000066400000000000000000000000361412141660100235740ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo qiime-2021.8.0/qiime2/metadata/tests/data/valid/trailing-columns.tsv000066400000000000000000000000751412141660100252130ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/unix-line-endings.tsv000066400000000000000000000000651412141660100252600ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/data/valid/windows-line-endings.tsv000066400000000000000000000000711412141660100257640ustar00rootroot00000000000000id col1 col2 col3 id1 1 a foo id2 2 b bar id3 3 c 42 qiime-2021.8.0/qiime2/metadata/tests/test_io.py000066400000000000000000001330331412141660100211770ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import collections import os.path import pkg_resources import tempfile import unittest import numpy as np import pandas as pd from qiime2.metadata import (Metadata, CategoricalMetadataColumn, NumericMetadataColumn, MetadataFileError) def get_data_path(filename): return pkg_resources.resource_filename('qiime2.metadata.tests', 'data/%s' % filename) # NOTE: many of the test files in the `data` directory intentionally have # leading/trailing whitespace characters on some lines, as well as mixed usage # of spaces, tabs, carriage returns, and newlines. When editing these files, # please make sure your code editor doesn't strip these leading/trailing # whitespace characters (e.g. Atom does this by default), nor automatically # modify the files in some other way such as converting Windows-style CRLF # line terminators to Unix-style newlines. # # When committing changes to the files, carefully review the diff to make sure # unintended changes weren't introduced. class TestLoadErrors(unittest.TestCase): def test_path_does_not_exist(self): with self.assertRaisesRegex(MetadataFileError, "Metadata file path doesn't exist"): Metadata.load( '/qiime2/unit/tests/hopefully/this/path/does/not/exist') def test_path_is_directory(self): fp = get_data_path('valid') with self.assertRaisesRegex(MetadataFileError, "path points to something other than a " "file"): Metadata.load(fp) def test_non_utf_8_file(self): fp = get_data_path('invalid/non-utf-8.tsv') with self.assertRaisesRegex(MetadataFileError, 'encoded as UTF-8 or ASCII'): Metadata.load(fp) def test_utf_16_le_file(self): fp = get_data_path('invalid/simple-utf-16le.txt') with self.assertRaisesRegex(MetadataFileError, 'UTF-16 Unicode'): Metadata.load(fp) def test_utf_16_be_file(self): fp = get_data_path('invalid/simple-utf-16be.txt') with self.assertRaisesRegex(MetadataFileError, 'UTF-16 Unicode'): Metadata.load(fp) def test_empty_file(self): fp = get_data_path('invalid/empty-file') with self.assertRaisesRegex(MetadataFileError, 'locate header.*file may be empty'): Metadata.load(fp) def test_comments_and_empty_rows_only(self): fp = get_data_path('invalid/comments-and-empty-rows-only.tsv') with self.assertRaisesRegex(MetadataFileError, 'locate header.*only of comments or empty ' 'rows'): Metadata.load(fp) def test_header_only(self): fp = get_data_path('invalid/header-only.tsv') with self.assertRaisesRegex(MetadataFileError, 'at least one ID'): Metadata.load(fp) def test_header_only_with_comments_and_empty_rows(self): fp = get_data_path( 'invalid/header-only-with-comments-and-empty-rows.tsv') with self.assertRaisesRegex(MetadataFileError, 'at least one ID'): Metadata.load(fp) def test_qiime1_empty_mapping_file(self): fp = get_data_path('invalid/qiime1-empty.tsv') with self.assertRaisesRegex(MetadataFileError, 'at least one ID'): Metadata.load(fp) def test_invalid_header(self): fp = get_data_path('invalid/invalid-header.tsv') with self.assertRaisesRegex(MetadataFileError, 'unrecognized ID column name.*' 'invalid_id_header'): Metadata.load(fp) def test_empty_id(self): fp = get_data_path('invalid/empty-id.tsv') with self.assertRaisesRegex(MetadataFileError, 'empty metadata ID'): Metadata.load(fp) def test_whitespace_only_id(self): fp = get_data_path('invalid/whitespace-only-id.tsv') with self.assertRaisesRegex(MetadataFileError, 'empty metadata ID'): Metadata.load(fp) def test_empty_column_name(self): fp = get_data_path('invalid/empty-column-name.tsv') with self.assertRaisesRegex(MetadataFileError, 'column without a name'): Metadata.load(fp) def test_whitespace_only_column_name(self): fp = get_data_path('invalid/whitespace-only-column-name.tsv') with self.assertRaisesRegex(MetadataFileError, 'column without a name'): Metadata.load(fp) def test_duplicate_ids(self): fp = get_data_path('invalid/duplicate-ids.tsv') with self.assertRaisesRegex(MetadataFileError, 'IDs must be unique.*id1'): Metadata.load(fp) def test_duplicate_ids_with_whitespace(self): fp = get_data_path('invalid/duplicate-ids-with-whitespace.tsv') with self.assertRaisesRegex(MetadataFileError, 'IDs must be unique.*id1'): Metadata.load(fp) def test_duplicate_column_names(self): fp = get_data_path('invalid/duplicate-column-names.tsv') with self.assertRaisesRegex(MetadataFileError, 'Column names must be unique.*col1'): Metadata.load(fp) def test_duplicate_column_names_with_whitespace(self): fp = get_data_path( 'invalid/duplicate-column-names-with-whitespace.tsv') with self.assertRaisesRegex(MetadataFileError, 'Column names must be unique.*col1'): Metadata.load(fp) def test_id_conflicts_with_id_header(self): fp = get_data_path('invalid/id-conflicts-with-id-header.tsv') with self.assertRaisesRegex(MetadataFileError, "ID 'id' conflicts.*ID column header"): Metadata.load(fp) def test_column_name_conflicts_with_id_header(self): fp = get_data_path( 'invalid/column-name-conflicts-with-id-header.tsv') with self.assertRaisesRegex(MetadataFileError, "column name 'featureid' conflicts.*ID " "column header"): Metadata.load(fp) def test_column_types_unrecognized_column_name(self): fp = get_data_path('valid/simple.tsv') with self.assertRaisesRegex(MetadataFileError, 'not_a_column.*column_types.*not a column ' 'in the metadata file'): Metadata.load(fp, column_types={'not_a_column': 'numeric'}) def test_column_types_unrecognized_column_type(self): fp = get_data_path('valid/simple.tsv') with self.assertRaisesRegex(MetadataFileError, 'col2.*column_types.*unrecognized column ' 'type.*CATEGORICAL'): Metadata.load(fp, column_types={'col1': 'numeric', 'col2': 'CATEGORICAL'}) def test_column_types_not_convertible_to_numeric(self): fp = get_data_path('valid/simple.tsv') with self.assertRaisesRegex(MetadataFileError, "column 'col3' to numeric.*could not be " "interpreted as numeric: 'bar', 'foo'"): Metadata.load(fp, column_types={'col1': 'numeric', 'col2': 'categorical', 'col3': 'numeric'}) def test_column_types_override_directive_not_convertible_to_numeric(self): fp = get_data_path('valid/simple-with-directive.tsv') with self.assertRaisesRegex(MetadataFileError, "column 'col3' to numeric.*could not be " "interpreted as numeric: 'bar', 'foo'"): Metadata.load(fp, column_types={'col3': 'numeric'}) def test_directive_before_header(self): fp = get_data_path('invalid/directive-before-header.tsv') with self.assertRaisesRegex(MetadataFileError, 'directive.*#q2:types.*searching for ' 'header'): Metadata.load(fp) def test_unrecognized_directive(self): fp = get_data_path('invalid/unrecognized-directive.tsv') with self.assertRaisesRegex(MetadataFileError, 'Unrecognized directive.*#q2:foo.*' '#q2:types directive is supported'): Metadata.load(fp) def test_duplicate_directives(self): fp = get_data_path('invalid/duplicate-directives.tsv') with self.assertRaisesRegex(MetadataFileError, 'duplicate directive.*#q2:types'): Metadata.load(fp) def test_unrecognized_column_type_in_directive(self): fp = get_data_path('invalid/unrecognized-column-type.tsv') with self.assertRaisesRegex(MetadataFileError, 'col2.*unrecognized column type.*foo.*' '#q2:types directive'): Metadata.load(fp) def test_column_types_directive_not_convertible_to_numeric(self): fp = get_data_path('invalid/types-directive-non-numeric.tsv') # This error message regex is intentionally verbose because we want to # assert that many different types of non-numeric strings aren't # interpreted as numbers. The error message displays a sorted list of # all values that couldn't be converted to numbers, making it possible # to test a variety of non-numeric strings in a single test case. msg = (r"column 'col2' to numeric.*could not be interpreted as " r"numeric: '\$42', '\+inf', '-inf', '0xAF', '1,000', " r"'1\.000\.0', '1_000_000', '1e3e4', 'Infinity', 'NA', 'NaN', " "'a', 'e3', 'foo', 'inf', 'nan', 'sample-1'") with self.assertRaisesRegex(MetadataFileError, msg): Metadata.load(fp) def test_directive_after_directives_section(self): fp = get_data_path( 'invalid/directive-after-directives-section.tsv') with self.assertRaisesRegex(MetadataFileError, '#q2:types.*outside of the directives ' 'section'): Metadata.load(fp) def test_directive_longer_than_header(self): fp = get_data_path('invalid/directive-longer-than-header.tsv') with self.assertRaisesRegex(MetadataFileError, 'row has 5 cells.*header declares 4 ' 'cells'): Metadata.load(fp) def test_data_longer_than_header(self): fp = get_data_path('invalid/data-longer-than-header.tsv') with self.assertRaisesRegex(MetadataFileError, 'row has 5 cells.*header declares 4 ' 'cells'): Metadata.load(fp) class TestLoadSuccess(unittest.TestCase): def setUp(self): self.temp_dir_obj = tempfile.TemporaryDirectory( prefix='qiime2-metadata-tests-temp-') self.temp_dir = self.temp_dir_obj.name # This Metadata object is compared against observed Metadata objects in # many of the tests, so just define it once here. self.simple_md = Metadata( pd.DataFrame({'col1': [1.0, 2.0, 3.0], 'col2': ['a', 'b', 'c'], 'col3': ['foo', 'bar', '42']}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) # Basic sanity check to make sure the columns are ordered and typed as # expected. It'd be unfortunate to compare observed results to expected # results that aren't representing what we think they are! obs_columns = [(name, props.type) for name, props in self.simple_md.columns.items()] exp_columns = [('col1', 'numeric'), ('col2', 'categorical'), ('col3', 'categorical')] self.assertEqual(obs_columns, exp_columns) def tearDown(self): self.temp_dir_obj.cleanup() def test_simple(self): # Simple metadata file without comments, empty rows, jaggedness, # missing data, odd IDs or column names, directives, etc. The file has # multiple column types (numeric, categorical, and something that has # mixed numbers and strings, which must be interpreted as categorical). fp = get_data_path('valid/simple.tsv') obs_md = Metadata.load(fp) self.assertEqual(obs_md, self.simple_md) def test_bom_simple_txt(self): # This is the encoding that notepad.exe will use most commonly fp = get_data_path('valid/BOM-simple.txt') obs_md = Metadata.load(fp) self.assertEqual(obs_md, self.simple_md) def test_different_file_extension(self): fp = get_data_path('valid/simple.txt') obs_md = Metadata.load(fp) self.assertEqual(obs_md, self.simple_md) def test_no_newline_at_eof(self): fp = get_data_path('valid/no-newline-at-eof.tsv') obs_md = Metadata.load(fp) self.assertEqual(obs_md, self.simple_md) def test_unix_line_endings(self): fp = get_data_path('valid/unix-line-endings.tsv') obs_md = Metadata.load(fp) self.assertEqual(obs_md, self.simple_md) def test_windows_line_endings(self): fp = get_data_path('valid/windows-line-endings.tsv') obs_md = Metadata.load(fp) self.assertEqual(obs_md, self.simple_md) def test_mac_line_endings(self): fp = get_data_path('valid/mac-line-endings.tsv') obs_md = Metadata.load(fp) self.assertEqual(obs_md, self.simple_md) def test_no_source_artifacts(self): fp = get_data_path('valid/simple.tsv') metadata = Metadata.load(fp) self.assertEqual(metadata.artifacts, ()) def test_retains_column_order(self): # Explicitly test that the file's column order is retained in the # Metadata object. Many of the test cases use files with column names # in alphabetical order (e.g. "col1", "col2", "col3"), which matches # how pandas orders columns in a DataFrame when supplied with a dict # (many of the test cases use this feature of the DataFrame # constructor when constructing the expected DataFrame). fp = get_data_path('valid/column-order.tsv') obs_md = Metadata.load(fp) # Supply DataFrame constructor with explicit column ordering instead of # a dict. exp_index = pd.Index(['id1', 'id2', 'id3'], name='id') exp_columns = ['z', 'y', 'x'] exp_data = [ [1.0, 'a', 'foo'], [2.0, 'b', 'bar'], [3.0, 'c', '42'] ] exp_df = pd.DataFrame(exp_data, index=exp_index, columns=exp_columns) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_leading_trailing_whitespace(self): fp = get_data_path('valid/leading-trailing-whitespace.tsv') obs_md = Metadata.load(fp) self.assertEqual(obs_md, self.simple_md) def test_comments(self): fp = get_data_path('valid/comments.tsv') obs_md = Metadata.load(fp) self.assertEqual(obs_md, self.simple_md) def test_empty_rows(self): fp = get_data_path('valid/empty-rows.tsv') obs_md = Metadata.load(fp) self.assertEqual(obs_md, self.simple_md) def test_qiime1_mapping_file(self): fp = get_data_path('valid/qiime1.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['id1', 'id2', 'id3'], name='#SampleID') exp_df = pd.DataFrame({'col1': [1.0, 2.0, 3.0], 'col2': ['a', 'b', 'c'], 'col3': ['foo', 'bar', '42']}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_qiita_sample_information_file(self): fp = get_data_path('valid/qiita-sample-information.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['id.1', 'id.2'], name='sample_name') exp_df = pd.DataFrame({ 'DESCRIPTION': ['description 1', 'description 2'], 'TITLE': ['A Title', 'Another Title']}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_qiita_preparation_information_file(self): fp = get_data_path('valid/qiita-preparation-information.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['id.1', 'id.2'], name='sample_name') exp_df = pd.DataFrame({ 'BARCODE': ['ACGT', 'TGCA'], 'EXPERIMENT_DESIGN_DESCRIPTION': ['longitudinal study', 'longitudinal study']}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_biom_observation_metadata_file(self): fp = get_data_path('valid/biom-observation-metadata.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['OTU_1', 'OTU_2'], name='#OTUID') exp_df = pd.DataFrame([['k__Bacteria;p__Firmicutes', 0.890], ['k__Bacteria', 0.9999]], columns=['taxonomy', 'confidence'], index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_supported_id_headers(self): case_insensitive = { 'id', 'sampleid', 'sample id', 'sample-id', 'featureid', 'feature id', 'feature-id' } exact_match = { '#SampleID', '#Sample ID', '#OTUID', '#OTU ID', 'sample_name' } # Build a set of supported headers, including exact matches and headers # with different casing. headers = set() for header in case_insensitive: headers.add(header) headers.add(header.upper()) headers.add(header.title()) for header in exact_match: headers.add(header) fp = os.path.join(self.temp_dir, 'metadata.tsv') count = 0 for header in headers: with open(fp, 'w') as fh: fh.write('%s\tcolumn\nid1\tfoo\nid2\tbar\n' % header) obs_md = Metadata.load(fp) exp_index = pd.Index(['id1', 'id2'], name=header) exp_df = pd.DataFrame({'column': ['foo', 'bar']}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) count += 1 # Since this test case is a little complicated, make sure that the # expected number of comparisons are happening. self.assertEqual(count, 26) def test_recommended_ids(self): fp = get_data_path('valid/recommended-ids.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['c6ca034a-223f-40b4-a0e0-45942912a5ea', 'My.ID'], name='id') exp_df = pd.DataFrame({'col1': ['foo', 'bar']}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_non_standard_characters(self): # Test that non-standard characters in IDs, column names, and cells are # handled correctly. The test case isn't exhaustive (e.g. it doesn't # test every Unicode character; that would be a nice additional test # case to have in the future). Instead, this test aims to be more of an # integration test for the robustness of the reader to non-standard # data. Many of the characters and their placement within the data file # are based on use-cases/bugs reported on the forum, Slack, etc. The # data file has comments explaining these test case choices in more # detail. fp = get_data_path('valid/non-standard-characters.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['©id##1', '((id))2', "'id_3<>'", '"id#4"', 'i d\r\t\n5'], name='id') exp_columns = ['↩c@l1™', 'col(#2)', "#col'3", '""', 'col\t \r\n5'] exp_data = [ ['ƒoo', '(foo)', '#f o #o', 'fo\ro', np.nan], ["''2''", 'b#r', 'ba\nr', np.nan, np.nan], ['b"ar', 'c\td', '4\r\n2', np.nan, np.nan], ['b__a_z', '<42>', '>42', np.nan, np.nan], ['baz', np.nan, '42'] ] exp_df = pd.DataFrame(exp_data, index=exp_index, columns=exp_columns) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_missing_data(self): fp = get_data_path('valid/missing-data.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['None', 'nan', 'NA'], name='id') exp_df = pd.DataFrame(collections.OrderedDict([ ('col1', [1.0, np.nan, np.nan]), ('NA', [np.nan, np.nan, np.nan]), ('col3', ['null', 'N/A', 'NA']), ('col4', np.array([np.nan, np.nan, np.nan], dtype=object))]), index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) # Test that column types are correct (mainly for the two empty columns; # one should be numeric, the other categorical). obs_columns = [(name, props.type) for name, props in obs_md.columns.items()] exp_columns = [('col1', 'numeric'), ('NA', 'numeric'), ('col3', 'categorical'), ('col4', 'categorical')] self.assertEqual(obs_columns, exp_columns) def test_minimal_file(self): # Simplest possible metadata file consists of one ID and zero columns. fp = get_data_path('valid/minimal.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['a'], name='id') exp_df = pd.DataFrame({}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_single_id(self): fp = get_data_path('valid/single-id.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['id1'], name='id') exp_df = pd.DataFrame({'col1': [1.0], 'col2': ['a'], 'col3': ['foo']}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_no_columns(self): fp = get_data_path('valid/no-columns.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['a', 'b', 'my-id'], name='id') exp_df = pd.DataFrame({}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_single_column(self): fp = get_data_path('valid/single-column.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['id1', 'id2', 'id3'], name='id') exp_df = pd.DataFrame({'col1': [1.0, 2.0, 3.0]}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_trailing_columns(self): fp = get_data_path('valid/trailing-columns.tsv') obs_md = Metadata.load(fp) self.assertEqual(obs_md, self.simple_md) def test_jagged_trailing_columns(self): # Test case based on https://github.com/qiime2/qiime2/issues/335 fp = get_data_path('valid/jagged-trailing-columns.tsv') obs_md = Metadata.load(fp) self.assertEqual(obs_md, self.simple_md) def test_padding_rows_shorter_than_header(self): fp = get_data_path('valid/rows-shorter-than-header.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['id1', 'id2', 'id3'], name='id') exp_df = pd.DataFrame({'col1': [1.0, 2.0, np.nan], 'col2': ['a', np.nan, np.nan], 'col3': [np.nan, np.nan, np.nan]}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_all_cells_padded(self): fp = get_data_path('valid/all-cells-padded.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['id1', 'id2', 'id3'], name='id') exp_df = pd.DataFrame({'col1': [np.nan, np.nan, np.nan], 'col2': [np.nan, np.nan, np.nan], 'col3': [np.nan, np.nan, np.nan]}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_does_not_cast_ids_or_column_names(self): fp = get_data_path('valid/no-id-or-column-name-type-cast.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['0.000001', '0.004000', '0.000000'], dtype=object, name='id') exp_columns = ['42.0', '1000', '-4.2'] exp_data = [ [2.0, 'b', 2.5], [1.0, 'b', 4.2], [3.0, 'c', -9.999] ] exp_df = pd.DataFrame(exp_data, index=exp_index, columns=exp_columns) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_numeric_column(self): fp = get_data_path('valid/numeric-column.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['id1', 'id2', 'id3', 'id4', 'id5', 'id6', 'id7', 'id8', 'id9', 'id10', 'id11', 'id12'], name='id') exp_df = pd.DataFrame({'col1': [0.0, 2.0, 0.0003, -4.2, 1e-4, 1e4, 1.5e2, np.nan, 1.0, 0.5, 1e-8, -0.0]}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_numeric_column_as_categorical(self): fp = get_data_path('valid/numeric-column.tsv') obs_md = Metadata.load(fp, column_types={'col1': 'categorical'}) exp_index = pd.Index(['id1', 'id2', 'id3', 'id4', 'id5', 'id6', 'id7', 'id8', 'id9', 'id10', 'id11', 'id12'], name='id') exp_df = pd.DataFrame({'col1': ['0', '2.0', '0.00030', '-4.2', '1e-4', '1e4', '+1.5E+2', np.nan, '1.', '.5', '1e-08', '-0']}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_with_complete_types_directive(self): fp = get_data_path('valid/complete-types-directive.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['id1', 'id2', 'id3'], name='id') exp_df = pd.DataFrame({'col1': ['1', '2', '3'], 'col2': ['a', 'b', 'c'], 'col3': ['foo', 'bar', '42']}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_with_partial_types_directive(self): fp = get_data_path('valid/partial-types-directive.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['id1', 'id2', 'id3'], name='id') exp_df = pd.DataFrame({'col1': ['1', '2', '3'], 'col2': ['a', 'b', 'c'], 'col3': ['foo', 'bar', '42']}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_with_empty_types_directive(self): fp = get_data_path('valid/empty-types-directive.tsv') obs_md = Metadata.load(fp) self.assertEqual(obs_md, self.simple_md) def test_with_case_insensitive_types_directive(self): fp = get_data_path('valid/case-insensitive-types-directive.tsv') obs_md = Metadata.load(fp) exp_index = pd.Index(['id1', 'id2', 'id3'], name='id') exp_df = pd.DataFrame({'col1': ['1', '2', '3'], 'col2': ['a', 'b', 'c'], 'col3': [-5.0, 0.0, 42.0]}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_column_types_without_directive(self): fp = get_data_path('valid/simple.tsv') obs_md = Metadata.load(fp, column_types={'col1': 'categorical'}) exp_index = pd.Index(['id1', 'id2', 'id3'], name='id') exp_df = pd.DataFrame({'col1': ['1', '2', '3'], 'col2': ['a', 'b', 'c'], 'col3': ['foo', 'bar', '42']}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) def test_column_types_override_directive(self): fp = get_data_path('valid/simple-with-directive.tsv') obs_md = Metadata.load(fp, column_types={'col1': 'categorical', 'col2': 'categorical'}) exp_index = pd.Index(['id1', 'id2', 'id3'], name='id') exp_df = pd.DataFrame({'col1': ['1', '2', '3'], 'col2': ['a', 'b', 'c'], 'col3': ['foo', 'bar', '42']}, index=exp_index) exp_md = Metadata(exp_df) self.assertEqual(obs_md, exp_md) class TestSave(unittest.TestCase): def setUp(self): self.temp_dir_obj = tempfile.TemporaryDirectory( prefix='qiime2-metadata-tests-temp-') self.temp_dir = self.temp_dir_obj.name self.filepath = os.path.join(self.temp_dir, 'metadata.tsv') def tearDown(self): self.temp_dir_obj.cleanup() def test_simple(self): md = Metadata(pd.DataFrame( {'col1': [1.0, 2.0, 3.0], 'col2': ['a', 'b', 'c'], 'col3': ['foo', 'bar', '42']}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) md.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "id\tcol1\tcol2\tcol3\n" "#q2:types\tnumeric\tcategorical\tcategorical\n" "id1\t1\ta\tfoo\n" "id2\t2\tb\tbar\n" "id3\t3\tc\t42\n" ) self.assertEqual(obs, exp) def test_save_metadata_auto_extension(self): md = Metadata(pd.DataFrame( {'col1': [1.0, 2.0, 3.0], 'col2': ['a', 'b', 'c'], 'col3': ['foo', 'bar', '42']}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) # Filename & extension endswith is matching (non-default). fp = os.path.join(self.temp_dir, 'metadatatsv') obs_md = md.save(fp, '.tsv') obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadatatsv.tsv') # No period in filename; no extension included. fp = os.path.join(self.temp_dir, 'metadata') obs_md = md.save(fp) obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadata') # No period in filename; no period in extension. fp = os.path.join(self.temp_dir, 'metadata') obs_md = md.save(fp, 'tsv') obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadata.tsv') # No period in filename; multiple periods in extension. fp = os.path.join(self.temp_dir, 'metadata') obs_md = md.save(fp, '..tsv') obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadata.tsv') # Single period in filename; no period in extension. fp = os.path.join(self.temp_dir, 'metadata.') obs_md = md.save(fp, 'tsv') obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadata.tsv') # Single period in filename; single period in extension. fp = os.path.join(self.temp_dir, 'metadata.') obs_md = md.save(fp, '.tsv') obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadata.tsv') # Single period in filename; multiple periods in extension. fp = os.path.join(self.temp_dir, 'metadata.') obs_md = md.save(fp, '..tsv') obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadata.tsv') # Multiple periods in filename; single period in extension. fp = os.path.join(self.temp_dir, 'metadata..') obs_md = md.save(fp, '.tsv') obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadata.tsv') # Multiple periods in filename; multiple periods in extension. fp = os.path.join(self.temp_dir, 'metadata..') obs_md = md.save(fp, '..tsv') obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadata.tsv') # No extension in filename; no extension input. fp = os.path.join(self.temp_dir, 'metadata') obs_md = md.save(fp) obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadata') # No extension in filename; extension input. fp = os.path.join(self.temp_dir, 'metadata') obs_md = md.save(fp, '.tsv') obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadata.tsv') # Extension in filename; no extension input. fp = os.path.join(self.temp_dir, 'metadata.tsv') obs_md = md.save(fp) obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadata.tsv') # Extension in filename; extension input (non-matching). fp = os.path.join(self.temp_dir, 'metadata.tsv') obs_md = md.save(fp, '.txt') obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadata.tsv.txt') # Extension in filename; extension input (matching). fp = os.path.join(self.temp_dir, 'metadata.tsv') obs_md = md.save(fp, '.tsv') obs_filename = os.path.basename(obs_md) self.assertEqual(obs_filename, 'metadata.tsv') def test_no_bom(self): md = Metadata(pd.DataFrame( {'col1': [1.0, 2.0, 3.0], 'col2': ['a', 'b', 'c'], 'col3': ['foo', 'bar', '42']}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) md.save(self.filepath) with open(self.filepath, 'rb') as fh: obs = fh.read(2) self.assertEqual(obs, b'id') def test_different_file_extension(self): md = Metadata(pd.DataFrame( {'col1': [1.0, 2.0, 3.0], 'col2': ['a', 'b', 'c'], 'col3': ['foo', 'bar', '42']}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) filepath = os.path.join(self.temp_dir, 'metadata.txt') md.save(filepath) with open(filepath, 'r') as fh: obs = fh.read() exp = ( "id\tcol1\tcol2\tcol3\n" "#q2:types\tnumeric\tcategorical\tcategorical\n" "id1\t1\ta\tfoo\n" "id2\t2\tb\tbar\n" "id3\t3\tc\t42\n" ) self.assertEqual(obs, exp) def test_some_missing_data(self): md = Metadata( pd.DataFrame({'col1': [42.0, np.nan, -3.5], 'col2': ['a', np.nan, np.nan]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) md.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "id\tcol1\tcol2\n" "#q2:types\tnumeric\tcategorical\n" "id1\t42\ta\n" "id2\t\t\n" "id3\t-3.5\t\n" ) self.assertEqual(obs, exp) def test_all_missing_data(self): # nan-only columns that are numeric or categorical. md = Metadata( pd.DataFrame({'col1': [np.nan, np.nan, np.nan], 'col2': np.array([np.nan, np.nan, np.nan], dtype=object)}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) md.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "id\tcol1\tcol2\n" "#q2:types\tnumeric\tcategorical\n" "id1\t\t\n" "id2\t\t\n" "id3\t\t\n" ) self.assertEqual(obs, exp) def test_unsorted_column_order(self): index = pd.Index(['id1', 'id2', 'id3'], name='id') columns = ['z', 'b', 'y'] data = [ [1.0, 'a', 'foo'], [2.0, 'b', 'bar'], [3.0, 'c', '42'] ] md = Metadata(pd.DataFrame(data, index=index, columns=columns)) md.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "id\tz\tb\ty\n" "#q2:types\tnumeric\tcategorical\tcategorical\n" "id1\t1\ta\tfoo\n" "id2\t2\tb\tbar\n" "id3\t3\tc\t42\n" ) self.assertEqual(obs, exp) def test_alternate_id_header(self): md = Metadata(pd.DataFrame( {'col1': [1.0, 2.0, 3.0], 'col2': ['a', 'b', 'c'], 'col3': ['foo', 'bar', '42']}, index=pd.Index(['id1', 'id2', 'id3'], name='#SampleID'))) md.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "#SampleID\tcol1\tcol2\tcol3\n" "#q2:types\tnumeric\tcategorical\tcategorical\n" "id1\t1\ta\tfoo\n" "id2\t2\tb\tbar\n" "id3\t3\tc\t42\n" ) self.assertEqual(obs, exp) def test_various_numbers(self): numbers = [ 0.0, -0.0, np.nan, 1.0, 42.0, -33.0, 1e-10, 1.5e15, 0.0003, -4.234, # This last number should be rounded because it exceeds 15 digits # of precision. 12.34567891234567 ] index = pd.Index(['id1', 'id2', 'id3', 'id4', 'id5', 'id6', 'id7', 'id8', 'id9', 'id10', 'id11'], name='ID') md = Metadata(pd.DataFrame({'numbers': numbers}, index=index)) md.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "ID\tnumbers\n" "#q2:types\tnumeric\n" "id1\t0\n" "id2\t-0\n" "id3\t\n" "id4\t1\n" "id5\t42\n" "id6\t-33\n" "id7\t1e-10\n" "id8\t1.5e+15\n" "id9\t0.0003\n" "id10\t-4.234\n" "id11\t12.3456789123457\n" ) self.assertEqual(obs, exp) def test_minimal(self): md = Metadata(pd.DataFrame({}, index=pd.Index(['my-id'], name='id'))) md.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "id\n" "#q2:types\n" "my-id\n" ) self.assertEqual(obs, exp) def test_single_id(self): md = Metadata(pd.DataFrame( {'col1': ['foo'], 'col2': [4.002]}, index=pd.Index(['my-id'], name='featureid'))) md.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "featureid\tcol1\tcol2\n" "#q2:types\tcategorical\tnumeric\n" "my-id\tfoo\t4.002\n" ) self.assertEqual(obs, exp) def test_no_columns(self): md = Metadata(pd.DataFrame( {}, index=pd.Index(['foo', 'bar', 'baz'], name='id'))) md.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "id\n" "#q2:types\n" "foo\n" "bar\n" "baz\n" ) self.assertEqual(obs, exp) def test_single_column(self): md = Metadata(pd.DataFrame( {'col1': ['42', '4.3', '4.4000']}, index=pd.Index(['foo', 'bar', 'baz'], name='id'))) md.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "id\tcol1\n" "#q2:types\tcategorical\n" "foo\t42\n" "bar\t4.3\n" "baz\t4.4000\n" ) self.assertEqual(obs, exp) def test_ids_and_column_names_as_numeric_strings(self): index = pd.Index(['0.000001', '0.004000', '0.000000'], dtype=object, name='id') columns = ['42.0', '1000', '-4.2'] data = [ [2.0, 'b', 2.5], [1.0, 'b', 4.2], [3.0, 'c', -9.999] ] df = pd.DataFrame(data, index=index, columns=columns) md = Metadata(df) md.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "id\t42.0\t1000\t-4.2\n" "#q2:types\tnumeric\tcategorical\tnumeric\n" "0.000001\t2\tb\t2.5\n" "0.004000\t1\tb\t4.2\n" "0.000000\t3\tc\t-9.999\n" ) self.assertEqual(obs, exp) # A couple of basic tests for CategoricalMetadataColumn and # NumericMetadataColumn below. Those classes simply transform themselves # into single-column Metadata objects within `MetadataColumn.save()` and # use the same writer code from there on. def test_categorical_metadata_column(self): mdc = CategoricalMetadataColumn(pd.Series( ['foo', 'bar', '42.50'], name='categorical-column', index=pd.Index(['id1', 'id2', 'id3'], name='id'))) mdc.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "id\tcategorical-column\n" "#q2:types\tcategorical\n" "id1\tfoo\n" "id2\tbar\n" "id3\t42.50\n" ) self.assertEqual(obs, exp) def test_numeric_metadata_column(self): mdc = NumericMetadataColumn(pd.Series( [1e-15, 42.50, -999.0], name='numeric-column', index=pd.Index(['id1', 'id2', 'id3'], name='#OTU ID'))) mdc.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "#OTU ID\tnumeric-column\n" "#q2:types\tnumeric\n" "id1\t1e-15\n" "id2\t42.5\n" "id3\t-999\n" ) self.assertEqual(obs, exp) # TODO this class spot-checks some of the more "difficult" valid files to make # sure they can be read, written to disk, and read again in a lossless way. # A more complete strategy (with fewer test cases) would be performing a # roundtrip test on every valid file under the `data` directory (e.g. with a # `glob` and for loop). class TestRoundtrip(unittest.TestCase): def setUp(self): self.temp_dir_obj = tempfile.TemporaryDirectory( prefix='qiime2-metadata-tests-temp-') self.temp_dir = self.temp_dir_obj.name self.filepath = os.path.join(self.temp_dir, 'metadata.tsv') def tearDown(self): self.temp_dir_obj.cleanup() def test_simple(self): fp = get_data_path('valid/simple.tsv') md1 = Metadata.load(fp) md1.save(self.filepath) md2 = Metadata.load(self.filepath) self.assertEqual(md1, md2) def test_non_standard_characters(self): fp = get_data_path('valid/non-standard-characters.tsv') md1 = Metadata.load(fp) md1.save(self.filepath) md2 = Metadata.load(self.filepath) self.assertEqual(md1, md2) def test_missing_data(self): fp = get_data_path('valid/missing-data.tsv') md1 = Metadata.load(fp) md1.save(self.filepath) md2 = Metadata.load(self.filepath) self.assertEqual(md1, md2) def test_minimal_file(self): fp = get_data_path('valid/minimal.tsv') md1 = Metadata.load(fp) md1.save(self.filepath) md2 = Metadata.load(self.filepath) self.assertEqual(md1, md2) def test_numeric_column(self): fp = get_data_path('valid/numeric-column.tsv') md1 = Metadata.load(fp) md1.save(self.filepath) md2 = Metadata.load(self.filepath) self.assertEqual(md1, md2) def test_all_cells_padded(self): fp = get_data_path('valid/all-cells-padded.tsv') md1 = Metadata.load(fp) md1.save(self.filepath) md2 = Metadata.load(self.filepath) self.assertEqual(md1, md2) def test_categorical_metadata_column(self): fp = get_data_path('valid/simple.tsv') md1 = Metadata.load(fp) mdc1 = md1.get_column('col2') self.assertIsInstance(mdc1, CategoricalMetadataColumn) mdc1.save(self.filepath) md2 = Metadata.load(self.filepath) mdc2 = md2.get_column('col2') self.assertIsInstance(mdc1, CategoricalMetadataColumn) self.assertEqual(mdc1, mdc2) def test_numeric_metadata_column(self): fp = get_data_path('valid/simple.tsv') md1 = Metadata.load(fp) mdc1 = md1.get_column('col1') self.assertIsInstance(mdc1, NumericMetadataColumn) mdc1.save(self.filepath) md2 = Metadata.load(self.filepath) mdc2 = md2.get_column('col1') self.assertIsInstance(mdc1, NumericMetadataColumn) self.assertEqual(mdc1, mdc2) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/metadata/tests/test_metadata.py000066400000000000000000001643641412141660100223630ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import collections import unittest import warnings import pandas as pd import numpy as np from qiime2 import Artifact from qiime2.metadata import (Metadata, CategoricalMetadataColumn, NumericMetadataColumn) from qiime2.core.testing.util import get_dummy_plugin, ReallyEqualMixin class TestInvalidMetadataConstruction(unittest.TestCase): def test_non_dataframe(self): with self.assertRaisesRegex( TypeError, 'Metadata constructor.*DataFrame.*not.*Series'): Metadata(pd.Series([1, 2, 3], name='col', index=pd.Index(['a', 'b', 'c'], name='id'))) def test_no_ids(self): with self.assertRaisesRegex(ValueError, 'Metadata.*at least one ID'): Metadata(pd.DataFrame({}, index=pd.Index([], name='id'))) with self.assertRaisesRegex(ValueError, 'Metadata.*at least one ID'): Metadata(pd.DataFrame({'column': []}, index=pd.Index([], name='id'))) def test_invalid_id_header(self): # default index name with self.assertRaisesRegex(ValueError, r'Index\.name.*None'): Metadata(pd.DataFrame( {'col': [1, 2, 3]}, index=pd.Index(['a', 'b', 'c']))) with self.assertRaisesRegex(ValueError, r'Index\.name.*my-id-header'): Metadata(pd.DataFrame( {'col': [1, 2, 3]}, index=pd.Index(['a', 'b', 'c'], name='my-id-header'))) def test_non_str_id(self): with self.assertRaisesRegex( TypeError, 'non-string metadata ID.*type.*float.*nan'): Metadata(pd.DataFrame( {'col': [1, 2, 3]}, index=pd.Index(['a', np.nan, 'c'], name='id'))) def test_non_str_column_name(self): with self.assertRaisesRegex( TypeError, 'non-string metadata column name.*type.*' 'float.*nan'): Metadata(pd.DataFrame( {'col': [1, 2, 3], np.nan: [4, 5, 6]}, index=pd.Index(['a', 'b', 'c'], name='id'))) def test_empty_id(self): with self.assertRaisesRegex( ValueError, 'empty metadata ID.*at least one character'): Metadata(pd.DataFrame( {'col': [1, 2, 3]}, index=pd.Index(['a', '', 'c'], name='id'))) def test_empty_column_name(self): with self.assertRaisesRegex( ValueError, 'empty metadata column name.*' 'at least one character'): Metadata(pd.DataFrame( {'col': [1, 2, 3], '': [4, 5, 6]}, index=pd.Index(['a', 'b', 'c'], name='id'))) def test_pound_sign_id(self): with self.assertRaisesRegex( ValueError, "metadata ID.*begins with a pound sign.*'#b'"): Metadata(pd.DataFrame( {'col': [1, 2, 3]}, index=pd.Index(['a', '#b', 'c'], name='id'))) def test_id_conflicts_with_id_header(self): with self.assertRaisesRegex( ValueError, "metadata ID 'sample-id'.*conflicts.*reserved.*" "ID header"): Metadata(pd.DataFrame( {'col': [1, 2, 3]}, index=pd.Index(['a', 'sample-id', 'c'], name='id'))) def test_column_name_conflicts_with_id_header(self): with self.assertRaisesRegex( ValueError, "metadata column name 'featureid'.*conflicts.*" "reserved.*ID header"): Metadata(pd.DataFrame( {'col': [1, 2, 3], 'featureid': [4, 5, 6]}, index=pd.Index(['a', 'b', 'c'], name='id'))) def test_duplicate_ids(self): with self.assertRaisesRegex(ValueError, "Metadata IDs.*unique.*'a'"): Metadata(pd.DataFrame( {'col': [1, 2, 3]}, index=pd.Index(['a', 'b', 'a'], name='id'))) def test_duplicate_column_names(self): data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] with self.assertRaisesRegex(ValueError, "Metadata column names.*unique.*'col1'"): Metadata(pd.DataFrame(data, columns=['col1', 'col2', 'col1'], index=pd.Index(['a', 'b', 'c'], name='id'))) def test_unsupported_column_dtype(self): with self.assertRaisesRegex( TypeError, "Metadata column 'col2'.*unsupported.*dtype.*bool"): Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': [True, False, True]}, index=pd.Index(['a', 'b', 'c'], name='id'))) def test_categorical_column_unsupported_type(self): with self.assertRaisesRegex( TypeError, "CategoricalMetadataColumn.*strings or missing " r"values.*42\.5.*float.*'col2'"): Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', 'bar', 42.5]}, index=pd.Index(['a', 'b', 'c'], name='id'))) def test_categorical_column_empty_str(self): with self.assertRaisesRegex( ValueError, "CategoricalMetadataColumn.*empty strings.*" "column 'col2'"): Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', '', 'bar']}, index=pd.Index(['a', 'b', 'c'], name='id'))) def test_numeric_column_infinity(self): with self.assertRaisesRegex( ValueError, "NumericMetadataColumn.*positive or negative " "infinity.*column 'col2'"): Metadata(pd.DataFrame( {'col1': ['foo', 'bar', 'baz'], 'col2': [42, float('+inf'), 4.3]}, index=pd.Index(['a', 'b', 'c'], name='id'))) class TestMetadataConstructionAndProperties(unittest.TestCase): def assertEqualColumns(self, obs_columns, exp): obs = [(name, props.type) for name, props in obs_columns.items()] self.assertEqual(obs, exp) def test_minimal(self): md = Metadata(pd.DataFrame({}, index=pd.Index(['a'], name='id'))) self.assertEqual(md.id_count, 1) self.assertEqual(md.column_count, 0) self.assertEqual(md.id_header, 'id') self.assertEqual(md.ids, ('a',)) self.assertEqualColumns(md.columns, []) def test_single_id(self): index = pd.Index(['id1'], name='id') df = pd.DataFrame({'col1': [1.0], 'col2': ['a'], 'col3': ['foo']}, index=index) md = Metadata(df) self.assertEqual(md.id_count, 1) self.assertEqual(md.column_count, 3) self.assertEqual(md.id_header, 'id') self.assertEqual(md.ids, ('id1',)) self.assertEqualColumns(md.columns, [('col1', 'numeric'), ('col2', 'categorical'), ('col3', 'categorical')]) def test_no_columns(self): index = pd.Index(['id1', 'id2', 'foo'], name='id') df = pd.DataFrame({}, index=index) md = Metadata(df) self.assertEqual(md.id_count, 3) self.assertEqual(md.column_count, 0) self.assertEqual(md.id_header, 'id') self.assertEqual(md.ids, ('id1', 'id2', 'foo')) self.assertEqualColumns(md.columns, []) def test_single_column(self): index = pd.Index(['id1', 'a', 'my-id'], name='id') df = pd.DataFrame({'column': ['foo', 'bar', 'baz']}, index=index) md = Metadata(df) self.assertEqual(md.id_count, 3) self.assertEqual(md.column_count, 1) self.assertEqual(md.id_header, 'id') self.assertEqual(md.ids, ('id1', 'a', 'my-id')) self.assertEqualColumns(md.columns, [('column', 'categorical')]) def test_retains_column_order(self): # Supply DataFrame constructor with explicit column ordering instead of # a dict. index = pd.Index(['id1', 'id2', 'id3'], name='id') columns = ['z', 'a', 'ch'] data = [ [1.0, 'a', 'foo'], [2.0, 'b', 'bar'], [3.0, 'c', '42'] ] df = pd.DataFrame(data, index=index, columns=columns) md = Metadata(df) self.assertEqual(md.id_count, 3) self.assertEqual(md.column_count, 3) self.assertEqual(md.id_header, 'id') self.assertEqual(md.ids, ('id1', 'id2', 'id3')) self.assertEqualColumns(md.columns, [('z', 'numeric'), ('a', 'categorical'), ('ch', 'categorical')]) def test_supported_id_headers(self): case_insensitive = { 'id', 'sampleid', 'sample id', 'sample-id', 'featureid', 'feature id', 'feature-id' } exact_match = { '#SampleID', '#Sample ID', '#OTUID', '#OTU ID', 'sample_name' } # Build a set of supported headers, including exact matches and headers # with different casing. headers = set() for header in case_insensitive: headers.add(header) headers.add(header.upper()) headers.add(header.title()) for header in exact_match: headers.add(header) count = 0 for header in headers: index = pd.Index(['id1', 'id2'], name=header) df = pd.DataFrame({'column': ['foo', 'bar']}, index=index) md = Metadata(df) self.assertEqual(md.id_header, header) count += 1 # Since this test case is a little complicated, make sure that the # expected number of comparisons are happening. self.assertEqual(count, 26) def test_recommended_ids(self): index = pd.Index(['c6ca034a-223f-40b4-a0e0-45942912a5ea', 'My.ID'], name='id') df = pd.DataFrame({'col1': ['foo', 'bar']}, index=index) md = Metadata(df) self.assertEqual(md.id_count, 2) self.assertEqual(md.column_count, 1) self.assertEqual(md.id_header, 'id') self.assertEqual(md.ids, ('c6ca034a-223f-40b4-a0e0-45942912a5ea', 'My.ID')) self.assertEqualColumns(md.columns, [('col1', 'categorical')]) def test_non_standard_characters(self): index = pd.Index(['©id##1', '((id))2', "'id_3<>'", '"id#4"', 'i d\r\t\n5'], name='id') columns = ['↩c@l1™', 'col(#2)', "#col'3", '""', 'col\t \r\n5'] data = [ ['ƒoo', '(foo)', '#f o #o', 'fo\ro', np.nan], ["''2''", 'b#r', 'ba\nr', np.nan, np.nan], ['b"ar', 'c\td', '4\r\n2', np.nan, np.nan], ['b__a_z', '<42>', '>42', np.nan, np.nan], ['baz', np.nan, '42'] ] df = pd.DataFrame(data, index=index, columns=columns) md = Metadata(df) self.assertEqual(md.id_count, 5) self.assertEqual(md.column_count, 5) self.assertEqual(md.id_header, 'id') self.assertEqual( md.ids, ('©id##1', '((id))2', "'id_3<>'", '"id#4"', 'i d\r\t\n5')) self.assertEqualColumns(md.columns, [('↩c@l1™', 'categorical'), ('col(#2)', 'categorical'), ("#col'3", 'categorical'), ('""', 'categorical'), ('col\t \r\n5', 'numeric')]) def test_missing_data(self): index = pd.Index(['None', 'nan', 'NA', 'foo'], name='id') df = pd.DataFrame(collections.OrderedDict([ ('col1', [1.0, np.nan, np.nan, np.nan]), ('NA', [np.nan, np.nan, np.nan, np.nan]), ('col3', ['null', 'N/A', np.nan, 'NA']), ('col4', np.array([np.nan, np.nan, np.nan, np.nan], dtype=object))]), index=index) md = Metadata(df) self.assertEqual(md.id_count, 4) self.assertEqual(md.column_count, 4) self.assertEqual(md.id_header, 'id') self.assertEqual(md.ids, ('None', 'nan', 'NA', 'foo')) self.assertEqualColumns(md.columns, [('col1', 'numeric'), ('NA', 'numeric'), ('col3', 'categorical'), ('col4', 'categorical')]) def test_does_not_cast_ids_or_column_names(self): index = pd.Index(['0.000001', '0.004000', '0.000000'], dtype=object, name='id') columns = ['42.0', '1000', '-4.2'] data = [ [2.0, 'b', 2.5], [1.0, 'b', 4.2], [3.0, 'c', -9.999] ] df = pd.DataFrame(data, index=index, columns=columns) md = Metadata(df) self.assertEqual(md.id_count, 3) self.assertEqual(md.column_count, 3) self.assertEqual(md.id_header, 'id') self.assertEqual(md.ids, ('0.000001', '0.004000', '0.000000')) self.assertEqualColumns(md.columns, [('42.0', 'numeric'), ('1000', 'categorical'), ('-4.2', 'numeric')]) def test_mixed_column_types(self): md = Metadata( pd.DataFrame({'col0': [1.0, 2.0, 3.0], 'col1': ['a', 'b', 'c'], 'col2': ['foo', 'bar', '42'], 'col3': ['1.0', '2.5', '-4.002'], 'col4': [1, 2, 3], 'col5': [1, 2, 3.5], 'col6': [1e-4, -0.0002, np.nan], 'col7': ['cat', np.nan, 'dog'], 'col8': ['a', 'a', 'a'], 'col9': [0, 0, 0]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) self.assertEqual(md.id_count, 3) self.assertEqual(md.column_count, 10) self.assertEqual(md.id_header, 'id') self.assertEqual(md.ids, ('id1', 'id2', 'id3')) self.assertEqualColumns(md.columns, [('col0', 'numeric'), ('col1', 'categorical'), ('col2', 'categorical'), ('col3', 'categorical'), ('col4', 'numeric'), ('col5', 'numeric'), ('col6', 'numeric'), ('col7', 'categorical'), ('col8', 'categorical'), ('col9', 'numeric')]) def test_case_insensitive_duplicate_ids(self): index = pd.Index(['a', 'b', 'A'], name='id') df = pd.DataFrame({'column': ['1', '2', '3']}, index=index) metadata = Metadata(df) self.assertEqual(metadata.ids, ('a', 'b', 'A')) def test_case_insensitive_duplicate_column_names(self): index = pd.Index(['a', 'b', 'c'], name='id') df = pd.DataFrame({'column': ['1', '2', '3'], 'Column': ['4', '5', '6']}, index=index) metadata = Metadata(df) self.assertEqual(set(metadata.columns), {'column', 'Column'}) def test_categorical_column_leading_trailing_whitespace_value(self): md1 = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', ' bar ', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) md2 = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', 'bar', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) self.assertEqual(md1, md2) def test_leading_trailing_whitespace_id(self): md1 = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': [4, 5, 6]}, index=pd.Index(['a', ' b ', 'c'], name='id'))) md2 = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': [4, 5, 6]}, index=pd.Index(['a', 'b', 'c'], name='id'))) self.assertEqual(md1, md2) def test_leading_trailing_whitespace_column_name(self): md1 = Metadata(pd.DataFrame( {'col1': [1, 2, 3], ' col2 ': [4, 5, 6]}, index=pd.Index(['a', 'b', 'c'], name='id'))) md2 = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': [4, 5, 6]}, index=pd.Index(['a', 'b', 'c'], name='id'))) self.assertEqual(md1, md2) class TestSourceArtifacts(unittest.TestCase): def setUp(self): self.md = Metadata(pd.DataFrame( {'col': [1, 2, 3]}, index=pd.Index(['a', 'b', 'c'], name='id'))) def test_no_source_artifacts(self): self.assertEqual(self.md.artifacts, ()) def test_add_zero_artifacts(self): self.md._add_artifacts([]) self.assertEqual(self.md.artifacts, ()) def test_add_artifacts(self): # First two artifacts have the same data but different UUIDs. artifact1 = Artifact.import_data('Mapping', {'a': '1', 'b': '3'}) self.md._add_artifacts([artifact1]) artifact2 = Artifact.import_data('Mapping', {'a': '1', 'b': '3'}) artifact3 = Artifact.import_data('IntSequence1', [1, 2, 3, 4]) self.md._add_artifacts([artifact2, artifact3]) self.assertEqual(self.md.artifacts, (artifact1, artifact2, artifact3)) def test_add_non_artifact(self): artifact = Artifact.import_data('Mapping', {'a': '1', 'b': '3'}) with self.assertRaisesRegex(TypeError, "Artifact object.*42"): self.md._add_artifacts([artifact, 42]) # Test that the object hasn't been mutated. self.assertEqual(self.md.artifacts, ()) def test_add_duplicate_artifact(self): artifact1 = Artifact.import_data('Mapping', {'a': '1', 'b': '3'}) artifact2 = Artifact.import_data('IntSequence1', [1, 2, 3, 4]) self.md._add_artifacts([artifact1, artifact2]) with self.assertRaisesRegex( ValueError, "Duplicate source artifacts.*artifact: Mapping"): self.md._add_artifacts([artifact1]) # Test that the object hasn't been mutated. self.assertEqual(self.md.artifacts, (artifact1, artifact2)) class TestRepr(unittest.TestCase): def test_singular(self): md = Metadata(pd.DataFrame({'col1': [42]}, index=pd.Index(['a'], name='id'))) obs = repr(md) self.assertIn('Metadata', obs) self.assertIn('1 ID x 1 column', obs) self.assertIn("col1: ColumnProperties(type='numeric')", obs) def test_plural(self): md = Metadata(pd.DataFrame({'col1': [42, 42], 'col2': ['foo', 'bar']}, index=pd.Index(['a', 'b'], name='id'))) obs = repr(md) self.assertIn('Metadata', obs) self.assertIn('2 IDs x 2 columns', obs) self.assertIn("col1: ColumnProperties(type='numeric')", obs) self.assertIn("col2: ColumnProperties(type='categorical')", obs) def test_column_name_padding(self): data = [[0, 42, 'foo']] index = pd.Index(['my-id'], name='id') columns = ['col1', 'longer-column-name', 'c'] md = Metadata(pd.DataFrame(data, index=index, columns=columns)) obs = repr(md) self.assertIn('Metadata', obs) self.assertIn('1 ID x 3 columns', obs) self.assertIn( "col1: ColumnProperties(type='numeric')", obs) self.assertIn( "longer-column-name: ColumnProperties(type='numeric')", obs) self.assertIn( "c: ColumnProperties(type='categorical')", obs) class TestEqualityOperators(unittest.TestCase, ReallyEqualMixin): def setUp(self): get_dummy_plugin() def test_type_mismatch(self): md = Metadata( pd.DataFrame({'col1': [1.0, 2.0, 3.0], 'col2': ['a', 'b', 'c'], 'col3': ['foo', 'bar', '42']}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) mdc = md.get_column('col1') self.assertIsInstance(md, Metadata) self.assertIsInstance(mdc, NumericMetadataColumn) self.assertReallyNotEqual(md, mdc) def test_id_header_mismatch(self): data = {'col1': ['foo', 'bar'], 'col2': [42, 43]} md1 = Metadata(pd.DataFrame( data, index=pd.Index(['id1', 'id2'], name='id'))) md2 = Metadata(pd.DataFrame( data, index=pd.Index(['id1', 'id2'], name='ID'))) self.assertReallyNotEqual(md1, md2) def test_source_mismatch(self): # Metadata created from an artifact vs not shouldn't compare equal, # even if the data is the same. artifact = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) md_from_artifact = artifact.view(Metadata) md_no_artifact = Metadata(md_from_artifact.to_dataframe()) pd.testing.assert_frame_equal(md_from_artifact.to_dataframe(), md_no_artifact.to_dataframe()) self.assertReallyNotEqual(md_from_artifact, md_no_artifact) def test_artifact_mismatch(self): # Metadata created from different artifacts shouldn't compare equal, # even if the data is the same. artifact1 = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) artifact2 = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) md1 = artifact1.view(Metadata) md2 = artifact2.view(Metadata) pd.testing.assert_frame_equal(md1.to_dataframe(), md2.to_dataframe()) self.assertReallyNotEqual(md1, md2) def test_id_mismatch(self): md1 = Metadata(pd.DataFrame({'a': '1', 'b': '2'}, index=pd.Index(['0'], name='id'))) md2 = Metadata(pd.DataFrame({'a': '1', 'b': '2'}, index=pd.Index(['1'], name='id'))) self.assertReallyNotEqual(md1, md2) def test_column_name_mismatch(self): md1 = Metadata(pd.DataFrame({'a': '1', 'b': '2'}, index=pd.Index(['0'], name='id'))) md2 = Metadata(pd.DataFrame({'a': '1', 'c': '2'}, index=pd.Index(['0'], name='id'))) self.assertReallyNotEqual(md1, md2) def test_column_type_mismatch(self): md1 = Metadata(pd.DataFrame({'col1': ['42', '43']}, index=pd.Index(['id1', 'id2'], name='id'))) md2 = Metadata(pd.DataFrame({'col1': [42, 43]}, index=pd.Index(['id1', 'id2'], name='id'))) self.assertReallyNotEqual(md1, md2) def test_column_order_mismatch(self): index = pd.Index(['id1', 'id2'], name='id') md1 = Metadata(pd.DataFrame([[42, 'foo'], [43, 'bar']], index=index, columns=['z', 'a'])) md2 = Metadata(pd.DataFrame([['foo', 42], ['bar', 43]], index=index, columns=['a', 'z'])) self.assertReallyNotEqual(md1, md2) def test_data_mismatch(self): md1 = Metadata(pd.DataFrame({'a': '1', 'b': '3'}, index=pd.Index(['0'], name='id'))) md2 = Metadata(pd.DataFrame({'a': '1', 'b': '2'}, index=pd.Index(['0'], name='id'))) self.assertReallyNotEqual(md1, md2) def test_equality_without_artifact(self): md1 = Metadata(pd.DataFrame({'a': '1', 'b': '3'}, index=pd.Index(['0'], name='id'))) md2 = Metadata(pd.DataFrame({'a': '1', 'b': '3'}, index=pd.Index(['0'], name='id'))) self.assertReallyEqual(md1, md2) def test_equality_with_artifact(self): artifact = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) md1 = artifact.view(Metadata) md2 = artifact.view(Metadata) self.assertReallyEqual(md1, md2) def test_equality_with_missing_data(self): md1 = Metadata(pd.DataFrame( {'col1': [1, np.nan, 4.2], 'col2': [np.nan, 'foo', np.nan]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) md2 = Metadata(pd.DataFrame( {'col1': [1, np.nan, 4.2], 'col2': [np.nan, 'foo', np.nan]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) self.assertReallyEqual(md1, md2) class TestToDataframe(unittest.TestCase): def test_minimal(self): df = pd.DataFrame({}, index=pd.Index(['id1'], name='id')) md = Metadata(df) obs = md.to_dataframe() pd.testing.assert_frame_equal(obs, df) def test_id_header_preserved(self): df = pd.DataFrame({'col1': [42, 2.5], 'col2': ['foo', 'bar']}, index=pd.Index(['id1', 'id2'], name='#SampleID')) md = Metadata(df) obs = md.to_dataframe() pd.testing.assert_frame_equal(obs, df) self.assertEqual(obs.index.name, '#SampleID') def test_dataframe_copy(self): df = pd.DataFrame({'col1': [42, 2.5], 'col2': ['foo', 'bar']}, index=pd.Index(['id1', 'id2'], name='id')) md = Metadata(df) obs = md.to_dataframe() pd.testing.assert_frame_equal(obs, df) self.assertIsNot(obs, df) def test_retains_column_order(self): index = pd.Index(['id1', 'id2'], name='id') columns = ['z', 'a', 'ch'] data = [ [1.0, 'a', 'foo'], [2.0, 'b', 'bar'] ] df = pd.DataFrame(data, index=index, columns=columns) md = Metadata(df) obs = md.to_dataframe() pd.testing.assert_frame_equal(obs, df) self.assertEqual(obs.columns.tolist(), ['z', 'a', 'ch']) def test_missing_data(self): # Different missing data representations should be normalized to np.nan index = pd.Index(['None', 'nan', 'NA', 'id1'], name='id') df = pd.DataFrame(collections.OrderedDict([ ('col1', [42.5, np.nan, float('nan'), 3]), ('NA', [np.nan, 'foo', float('nan'), None]), ('col3', ['null', 'N/A', np.nan, 'NA']), ('col4', np.array([np.nan, np.nan, np.nan, np.nan], dtype=object))]), index=index) md = Metadata(df) obs = md.to_dataframe() exp = pd.DataFrame(collections.OrderedDict([ ('col1', [42.5, np.nan, np.nan, 3.0]), ('NA', [np.nan, 'foo', np.nan, np.nan]), ('col3', ['null', 'N/A', np.nan, 'NA']), ('col4', np.array([np.nan, np.nan, np.nan, np.nan], dtype=object))]), index=index) pd.testing.assert_frame_equal(obs, exp) self.assertEqual(obs.dtypes.to_dict(), {'col1': np.float64, 'NA': object, 'col3': object, 'col4': object}) self.assertTrue(np.isnan(obs['col1']['NA'])) self.assertTrue(np.isnan(obs['NA']['NA'])) self.assertTrue(np.isnan(obs['NA']['id1'])) def test_dtype_int_normalized_to_dtype_float(self): index = pd.Index(['id1', 'id2', 'id3'], name='id') df = pd.DataFrame({'col1': [42, -43, 0], 'col2': [42.0, -43.0, 0.0], 'col3': [42, np.nan, 0]}, index=index) self.assertEqual(df.dtypes.to_dict(), {'col1': np.int64, 'col2': np.float64, 'col3': np.float64}) md = Metadata(df) obs = md.to_dataframe() exp = pd.DataFrame({'col1': [42.0, -43.0, 0.0], 'col2': [42.0, -43.0, 0.0], 'col3': [42.0, np.nan, 0.0]}, index=index) pd.testing.assert_frame_equal(obs, exp) self.assertEqual(obs.dtypes.to_dict(), {'col1': np.float64, 'col2': np.float64, 'col3': np.float64}) class TestGetColumn(unittest.TestCase): def setUp(self): get_dummy_plugin() def test_column_name_not_found(self): df = pd.DataFrame({'col1': [42, 2.5], 'col2': ['foo', 'bar']}, index=pd.Index(['id1', 'id2'], name='id')) md = Metadata(df) with self.assertRaisesRegex(ValueError, "'col3'.*not a column.*'col1', 'col2'"): md.get_column('col3') def test_artifacts_are_propagated(self): A = Artifact.import_data('Mapping', {'a': '1', 'b': '3'}) md = A.view(Metadata) obs = md.get_column('b') exp = CategoricalMetadataColumn( pd.Series(['3'], name='b', index=pd.Index(['0'], name='id'))) exp._add_artifacts([A]) self.assertEqual(obs, exp) self.assertEqual(obs.artifacts, (A,)) def test_categorical_column(self): df = pd.DataFrame({'col1': [42, 2.5], 'col2': ['foo', 'bar']}, index=pd.Index(['id1', 'id2'], name='id')) md = Metadata(df) obs = md.get_column('col2') exp = CategoricalMetadataColumn( pd.Series(['foo', 'bar'], name='col2', index=pd.Index(['id1', 'id2'], name='id'))) self.assertEqual(obs, exp) def test_numeric_column(self): df = pd.DataFrame({'col1': [42, 2.5], 'col2': ['foo', 'bar']}, index=pd.Index(['id1', 'id2'], name='id')) md = Metadata(df) obs = md.get_column('col1') exp = NumericMetadataColumn( pd.Series([42, 2.5], name='col1', index=pd.Index(['id1', 'id2'], name='id'))) self.assertEqual(obs, exp) def test_id_header_preserved(self): df = pd.DataFrame({'col1': [42, 2.5], 'col2': ['foo', 'bar']}, index=pd.Index(['a', 'b'], name='#OTU ID')) md = Metadata(df) obs = md.get_column('col1') exp = NumericMetadataColumn( pd.Series([42, 2.5], name='col1', index=pd.Index(['a', 'b'], name='#OTU ID'))) self.assertEqual(obs, exp) self.assertEqual(obs.id_header, '#OTU ID') class TestGetIDs(unittest.TestCase): def test_default(self): df = pd.DataFrame({'Subject': ['subject-1', 'subject-1', 'subject-2'], 'SampleType': ['gut', 'tongue', 'gut']}, index=pd.Index(['S1', 'S2', 'S3'], name='id')) metadata = Metadata(df) actual = metadata.get_ids() expected = {'S1', 'S2', 'S3'} self.assertEqual(actual, expected) def test_incomplete_where(self): df = pd.DataFrame({'Subject': ['subject-1', 'subject-1', 'subject-2'], 'SampleType': ['gut', 'tongue', 'gut']}, index=pd.Index(['S1', 'S2', 'S3'], name='sampleid')) metadata = Metadata(df) where = "Subject='subject-1' AND SampleType=" with self.assertRaises(ValueError): metadata.get_ids(where) where = "Subject=" with self.assertRaises(ValueError): metadata.get_ids(where) def test_invalid_where(self): df = pd.DataFrame({'Subject': ['subject-1', 'subject-1', 'subject-2'], 'SampleType': ['gut', 'tongue', 'gut']}, index=pd.Index(['S1', 'S2', 'S3'], name='sampleid')) metadata = Metadata(df) where = "not-a-column-name='subject-1'" with self.assertRaises(ValueError): metadata.get_ids(where) def test_empty_result(self): df = pd.DataFrame({'Subject': ['subject-1', 'subject-1', 'subject-2'], 'SampleType': ['gut', 'tongue', 'gut']}, index=pd.Index(['S1', 'S2', 'S3'], name='id')) metadata = Metadata(df) where = "Subject='subject-3'" actual = metadata.get_ids(where) expected = set() self.assertEqual(actual, expected) def test_simple_expression(self): df = pd.DataFrame({'Subject': ['subject-1', 'subject-1', 'subject-2'], 'SampleType': ['gut', 'tongue', 'gut']}, index=pd.Index(['S1', 'S2', 'S3'], name='id')) metadata = Metadata(df) where = "Subject='subject-1'" actual = metadata.get_ids(where) expected = {'S1', 'S2'} self.assertEqual(actual, expected) where = "Subject='subject-2'" actual = metadata.get_ids(where) expected = {'S3'} self.assertEqual(actual, expected) where = "Subject='subject-3'" actual = metadata.get_ids(where) expected = set() self.assertEqual(actual, expected) where = "SampleType='gut'" actual = metadata.get_ids(where) expected = {'S1', 'S3'} self.assertEqual(actual, expected) where = "SampleType='tongue'" actual = metadata.get_ids(where) expected = {'S2'} self.assertEqual(actual, expected) def test_more_complex_expressions(self): df = pd.DataFrame({'Subject': ['subject-1', 'subject-1', 'subject-2'], 'SampleType': ['gut', 'tongue', 'gut']}, index=pd.Index(['S1', 'S2', 'S3'], name='id')) metadata = Metadata(df) where = "Subject='subject-1' OR Subject='subject-2'" actual = metadata.get_ids(where) expected = {'S1', 'S2', 'S3'} self.assertEqual(actual, expected) where = "Subject='subject-1' AND Subject='subject-2'" actual = metadata.get_ids(where) expected = set() self.assertEqual(actual, expected) where = "Subject='subject-1' AND SampleType='gut'" actual = metadata.get_ids(where) expected = {'S1'} self.assertEqual(actual, expected) def test_query_by_id(self): df = pd.DataFrame({'Subject': ['subject-1', 'subject-1', 'subject-2'], 'SampleType': ['gut', 'tongue', 'gut']}, index=pd.Index(['S1', 'S2', 'S3'], name='id')) metadata = Metadata(df) actual = metadata.get_ids(where="id='S2' OR id='S1'") expected = {'S1', 'S2'} self.assertEqual(actual, expected) def test_query_by_alternate_id_header(self): metadata = Metadata(pd.DataFrame( {}, index=pd.Index(['id1', 'id2', 'id3'], name='#OTU ID'))) obs = metadata.get_ids(where="\"#OTU ID\" IN ('id2', 'id3')") exp = {'id2', 'id3'} self.assertEqual(obs, exp) def test_no_columns(self): metadata = Metadata( pd.DataFrame({}, index=pd.Index(['a', 'b', 'my-id'], name='id'))) obs = metadata.get_ids() exp = {'a', 'b', 'my-id'} self.assertEqual(obs, exp) def test_query_mixed_column_types(self): df = pd.DataFrame({'Name': ['Foo', 'Bar', 'Baz', 'Baaz'], # numbers that would sort incorrectly as strings 'Age': [9, 10, 11, 101], 'Age_Str': ['9', '10', '11', '101'], 'Weight': [80.5, 85.3, np.nan, 120.0]}, index=pd.Index(['S1', 'S2', 'S3', 'S4'], name='id')) metadata = Metadata(df) # string pattern matching obs = metadata.get_ids(where="Name LIKE 'Ba_'") exp = {'S2', 'S3'} self.assertEqual(obs, exp) # string comparison obs = metadata.get_ids(where="Age_Str >= 11") exp = {'S1', 'S3'} self.assertEqual(obs, exp) # numeric comparison obs = metadata.get_ids(where="Age >= 11") exp = {'S3', 'S4'} self.assertEqual(obs, exp) # numeric comparison with missing data obs = metadata.get_ids(where="Weight < 100") exp = {'S1', 'S2'} self.assertEqual(obs, exp) def test_column_with_space_in_name(self): df = pd.DataFrame({'Subject': ['subject-1', 'subject-1', 'subject-2'], 'Sample Type': ['gut', 'tongue', 'gut']}, index=pd.Index(['S1', 'S2', 'S3'], name='id')) metadata = Metadata(df) with warnings.catch_warnings(record=True) as w: warnings.simplefilter('always') metadata.get_ids() # The list of captured warnings should be empty self.assertFalse(w) class TestMerge(unittest.TestCase): def setUp(self): get_dummy_plugin() def test_merging_nothing(self): md = Metadata(pd.DataFrame( {'a': [1, 2, 3], 'b': [4, 5, 6]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) with self.assertRaisesRegex(ValueError, 'At least one Metadata.*nothing to merge'): md.merge() def test_merging_two(self): md1 = Metadata(pd.DataFrame( {'a': [1, 2, 3], 'b': [4, 5, 6]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) md2 = Metadata(pd.DataFrame( {'c': [7, 8, 9], 'd': [10, 11, 12]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) obs = md1.merge(md2) exp = Metadata(pd.DataFrame( {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9], 'd': [10, 11, 12]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) self.assertEqual(obs, exp) def test_merging_three(self): md1 = Metadata(pd.DataFrame( {'a': [1, 2, 3], 'b': [4, 5, 6]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) md2 = Metadata(pd.DataFrame( {'c': [7, 8, 9], 'd': [10, 11, 12]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) md3 = Metadata(pd.DataFrame( {'e': [13, 14, 15], 'f': [16, 17, 18]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) obs = md1.merge(md2, md3) exp = Metadata(pd.DataFrame( {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9], 'd': [10, 11, 12], 'e': [13, 14, 15], 'f': [16, 17, 18]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) self.assertEqual(obs, exp) def test_merging_unaligned_indices(self): md1 = Metadata(pd.DataFrame( {'a': [1, 2, 3], 'b': [4, 5, 6]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) md2 = Metadata(pd.DataFrame( {'c': [9, 8, 7], 'd': [12, 11, 10]}, index=pd.Index(['id3', 'id2', 'id1'], name='id'))) md3 = Metadata(pd.DataFrame( {'e': [13, 15, 14], 'f': [16, 18, 17]}, index=pd.Index(['id1', 'id3', 'id2'], name='id'))) obs = md1.merge(md2, md3) exp = Metadata(pd.DataFrame( {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9], 'd': [10, 11, 12], 'e': [13, 14, 15], 'f': [16, 17, 18]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) self.assertEqual(obs, exp) def test_inner_join(self): md1 = Metadata(pd.DataFrame( {'a': [1, 2, 3], 'b': [4, 5, 6]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) md2 = Metadata(pd.DataFrame( {'c': [7, 8, 9], 'd': [10, 11, 12]}, index=pd.Index(['id2', 'X', 'Y'], name='id'))) md3 = Metadata(pd.DataFrame( {'e': [13, 14, 15], 'f': [16, 17, 18]}, index=pd.Index(['X', 'id3', 'id2'], name='id'))) # Single shared ID. obs = md1.merge(md2, md3) exp = Metadata(pd.DataFrame( {'a': [2], 'b': [5], 'c': [7], 'd': [10], 'e': [15], 'f': [18]}, index=pd.Index(['id2'], name='id'))) self.assertEqual(obs, exp) # Multiple shared IDs. obs = md1.merge(md3) exp = Metadata(pd.DataFrame( {'a': [2, 3], 'b': [5, 6], 'e': [15, 14], 'f': [18, 17]}, index=pd.Index(['id2', 'id3'], name='id'))) self.assertEqual(obs, exp) def test_index_and_column_merge_order(self): md1 = Metadata(pd.DataFrame( [[1], [2], [3], [4]], index=pd.Index(['id1', 'id2', 'id3', 'id4'], name='id'), columns=['a'])) md2 = Metadata(pd.DataFrame( [[5], [6], [7]], index=pd.Index(['id4', 'id3', 'id1'], name='id'), columns=['b'])) md3 = Metadata(pd.DataFrame( [[8], [9], [10]], index=pd.Index(['id1', 'id4', 'id3'], name='id'), columns=['c'])) obs = md1.merge(md2, md3) exp = Metadata(pd.DataFrame( [[1, 7, 8], [3, 6, 10], [4, 5, 9]], index=pd.Index(['id1', 'id3', 'id4'], name='id'), columns=['a', 'b', 'c'])) self.assertEqual(obs, exp) # Merging in different order produces different ID/column order. obs = md2.merge(md1, md3) exp = Metadata(pd.DataFrame( [[5, 4, 9], [6, 3, 10], [7, 1, 8]], index=pd.Index(['id4', 'id3', 'id1'], name='id'), columns=['b', 'a', 'c'])) self.assertEqual(obs, exp) def test_id_column_only(self): md1 = Metadata(pd.DataFrame({}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) md2 = Metadata(pd.DataFrame({}, index=pd.Index(['id2', 'X', 'id1'], name='id'))) md3 = Metadata(pd.DataFrame({}, index=pd.Index(['id1', 'id3', 'id2'], name='id'))) obs = md1.merge(md2, md3) exp = Metadata( pd.DataFrame({}, index=pd.Index(['id1', 'id2'], name='id'))) self.assertEqual(obs, exp) def test_merged_id_column_name(self): md1 = Metadata(pd.DataFrame( {'a': [1, 2]}, index=pd.Index(['id1', 'id2'], name='sample ID'))) md2 = Metadata(pd.DataFrame( {'b': [3, 4]}, index=pd.Index(['id1', 'id2'], name='feature ID'))) obs = md1.merge(md2) exp = Metadata(pd.DataFrame( {'a': [1, 2], 'b': [3, 4]}, index=pd.Index(['id1', 'id2'], name='id'))) self.assertEqual(obs, exp) def test_merging_preserves_column_types(self): # Test that column types remain the same even if a categorical column # *could* be reinterpreted as numeric after the merge. md1 = Metadata(pd.DataFrame( {'a': [1, 2, 3], 'b': [np.nan, np.nan, np.nan]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) md2 = Metadata(pd.DataFrame( {'c': ['1', 'foo', '3'], 'd': np.array([np.nan, np.nan, np.nan], dtype=object)}, index=pd.Index(['id1', 'id4', 'id3'], name='id'))) obs = md1.merge(md2) exp = Metadata(pd.DataFrame( {'a': [1, 3], 'b': [np.nan, np.nan], 'c': ['1', '3'], 'd': np.array([np.nan, np.nan], dtype=object)}, index=pd.Index(['id1', 'id3'], name='id'))) self.assertEqual(obs, exp) self.assertEqual(obs.columns['a'].type, 'numeric') self.assertEqual(obs.columns['b'].type, 'numeric') self.assertEqual(obs.columns['c'].type, 'categorical') self.assertEqual(obs.columns['d'].type, 'categorical') def test_no_artifacts(self): md1 = Metadata(pd.DataFrame( {'a': [1, 2]}, index=pd.Index(['id1', 'id2'], name='id'))) md2 = Metadata(pd.DataFrame( {'b': [3, 4]}, index=pd.Index(['id1', 'id2'], name='id'))) metadata = md1.merge(md2) self.assertEqual(metadata.artifacts, ()) def test_with_artifacts(self): artifact1 = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) artifact2 = Artifact.import_data('Mapping', {'d': '4'}) md_from_artifact1 = artifact1.view(Metadata) md_from_artifact2 = artifact2.view(Metadata) md_no_artifact = Metadata(pd.DataFrame( {'c': ['3', '42']}, index=pd.Index(['0', '1'], name='id'))) # Merge three metadata objects -- the first has an artifact, the second # does not, and the third has an artifact. obs_md = md_from_artifact1.merge(md_no_artifact, md_from_artifact2) exp_df = pd.DataFrame( {'a': '1', 'b': '2', 'c': '3', 'd': '4'}, index=pd.Index(['0'], name='id')) exp_md = Metadata(exp_df) exp_md._add_artifacts((artifact1, artifact2)) self.assertEqual(obs_md, exp_md) self.assertEqual(obs_md.artifacts, (artifact1, artifact2)) def test_disjoint_indices(self): md1 = Metadata(pd.DataFrame( {'a': [1, 2, 3], 'b': [4, 5, 6]}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) md2 = Metadata(pd.DataFrame( {'c': [7, 8, 9], 'd': [10, 11, 12]}, index=pd.Index(['X', 'Y', 'Z'], name='id'))) with self.assertRaisesRegex(ValueError, 'no IDs shared'): md1.merge(md2) def test_duplicate_columns(self): md1 = Metadata(pd.DataFrame( {'a': [1, 2], 'b': [3, 4]}, index=pd.Index(['id1', 'id2'], name='id'))) md2 = Metadata(pd.DataFrame( {'c': [5, 6], 'b': [7, 8]}, index=pd.Index(['id1', 'id2'], name='id'))) with self.assertRaisesRegex(ValueError, "columns overlap: 'b'"): md1.merge(md2) def test_duplicate_columns_self_merge(self): md = Metadata(pd.DataFrame( {'a': [1, 2], 'b': [3, 4]}, index=pd.Index(['id1', 'id2'], name='id'))) with self.assertRaisesRegex(ValueError, "columns overlap: 'a', 'b'"): md.merge(md) class TestFilterIDs(unittest.TestCase): def setUp(self): get_dummy_plugin() def test_supports_iterable(self): md = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', 'bar', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) obs = md.filter_ids(iter({'a', 'c'})) exp = Metadata(pd.DataFrame( {'col1': [1, 3], 'col2': ['foo', 'baz']}, index=pd.Index(['a', 'c'], name='id'))) self.assertEqual(obs, exp) def test_keep_all(self): md = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', 'bar', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) obs = md.filter_ids({'a', 'b', 'c'}) self.assertEqual(obs, md) self.assertIsNot(obs, md) def test_keep_multiple(self): md = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', 'bar', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) obs = md.filter_ids({'a', 'c'}) exp = Metadata(pd.DataFrame( {'col1': [1, 3], 'col2': ['foo', 'baz']}, index=pd.Index(['a', 'c'], name='id'))) self.assertEqual(obs, exp) def test_keep_one(self): md = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', 'bar', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) obs = md.filter_ids({'b'}) exp = Metadata(pd.DataFrame( {'col1': [2], 'col2': ['bar']}, index=pd.Index(['b'], name='id'))) self.assertEqual(obs, exp) def test_filtering_preserves_column_types(self): # Test that column types remain the same even if a categorical column # *could* be reinterpreted as numeric after the filter. md = Metadata(pd.DataFrame( {'a': [1, 2, 3], 'b': [np.nan, np.nan, np.nan], 'c': ['1', 'foo', '3'], 'd': np.array([np.nan, np.nan, np.nan], dtype=object)}, index=pd.Index(['id1', 'id2', 'id3'], name='id'))) obs = md.filter_ids({'id1', 'id3'}) exp = Metadata(pd.DataFrame( {'a': [1, 3], 'b': [np.nan, np.nan], 'c': ['1', '3'], 'd': np.array([np.nan, np.nan], dtype=object)}, index=pd.Index(['id1', 'id3'], name='id'))) self.assertEqual(obs, exp) self.assertEqual(obs.columns['a'].type, 'numeric') self.assertEqual(obs.columns['b'].type, 'numeric') self.assertEqual(obs.columns['c'].type, 'categorical') self.assertEqual(obs.columns['d'].type, 'categorical') def test_alternate_id_header(self): md = Metadata(pd.DataFrame( {'col1': [1, 2, 3, 4], 'col2': ['foo', 'bar', 'baz', 'bazz']}, index=pd.Index(['a', 'b', 'c', 'd'], name='#Sample ID'))) obs = md.filter_ids({'b', 'd'}) exp = Metadata(pd.DataFrame( {'col1': [2, 4], 'col2': ['bar', 'bazz']}, index=pd.Index(['b', 'd'], name='#Sample ID'))) self.assertEqual(obs, exp) def test_retains_column_order(self): data = [[1, 'foo', 'cat'], [2, 'bar', 'dog'], [3, 'baz', 'bat']] md = Metadata(pd.DataFrame( data, columns=['z', 'a', 'ch'], index=pd.Index(['a', 'b', 'c'], name='id'))) obs = md.filter_ids({'b', 'c'}) exp_data = [[2, 'bar', 'dog'], [3, 'baz', 'bat']] exp = Metadata(pd.DataFrame( exp_data, columns=['z', 'a', 'ch'], index=pd.Index(['b', 'c'], name='id'))) self.assertEqual(obs, exp) def test_no_artifacts(self): md = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', 'bar', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) self.assertEqual(md.artifacts, ()) filtered = md.filter_ids({'b'}) self.assertEqual(filtered.artifacts, ()) def test_with_artifacts(self): artifact1 = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) artifact2 = Artifact.import_data('Mapping', {'d': '4'}) md = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', 'bar', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) md._add_artifacts([artifact1, artifact2]) obs = md.filter_ids({'a', 'c'}) exp = Metadata(pd.DataFrame( {'col1': [1, 3], 'col2': ['foo', 'baz']}, index=pd.Index(['a', 'c'], name='id'))) exp._add_artifacts([artifact1, artifact2]) self.assertEqual(obs, exp) self.assertEqual(obs.artifacts, (artifact1, artifact2)) def test_empty_ids_to_keep(self): md = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', 'bar', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) with self.assertRaisesRegex(ValueError, 'ids_to_keep.*at least one ID'): md.filter_ids({}) def test_duplicate_ids_to_keep(self): md = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', 'bar', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) with self.assertRaisesRegex(ValueError, "ids_to_keep.*unique IDs.*'b'"): md.filter_ids(['b', 'c', 'b']) def test_missing_ids_to_keep(self): md = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', 'bar', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) with self.assertRaisesRegex(ValueError, "IDs.*not present.*'d', 'id1'"): md.filter_ids({'b', 'id1', 'c', 'd'}) class TestFilterColumns(unittest.TestCase): def setUp(self): get_dummy_plugin() # This object can be reused in many of the tests because its columns # match various filtering criteria, allowing test cases to test # individual parameters or combinations of parameters. self.metadata = Metadata(pd.DataFrame( {'cat': ['foo', 'bar', np.nan, 'foo'], 'num': [42, np.nan, -5.5, 42], 'uniq-cat': ['foo', np.nan, 'bar', np.nan], 'uniq-num': [np.nan, 9.9, np.nan, 42], 'zvar-cat': ['foo', np.nan, 'foo', 'foo'], 'zvar-num': [9.9, 9.9, np.nan, 9.9], 'empty-cat': np.array([np.nan, np.nan, np.nan, np.nan], dtype=object), 'empty-num': [np.nan, np.nan, np.nan, np.nan]}, index=pd.Index(['a', 'b', 'c', 'd'], name='id'))) # Basic sanity check to ensure column types are what we expect them to # be. obs = {n: p.type for n, p in self.metadata.columns.items()} exp = {'cat': 'categorical', 'num': 'numeric', 'uniq-cat': 'categorical', 'uniq-num': 'numeric', 'zvar-cat': 'categorical', 'zvar-num': 'numeric', 'empty-cat': 'categorical', 'empty-num': 'numeric'} self.assertEqual(obs, exp) def test_unknown_column_type(self): with self.assertRaisesRegex( ValueError, "Unknown column type 'foo'.*categorical, numeric"): self.metadata.filter_columns(column_type='foo') def test_no_filters(self): obs = self.metadata.filter_columns() self.assertEqual(obs, self.metadata) self.assertIsNot(obs, self.metadata) def test_all_filters_no_columns(self): md = Metadata(pd.DataFrame( {}, index=pd.Index(['a', 'b', 'c'], name='id'))) obs = md.filter_columns( column_type='categorical', drop_all_unique=True, drop_zero_variance=True, drop_all_missing=True) self.assertEqual(obs, md) self.assertIsNot(obs, md) obs = md.filter_columns( column_type='numeric', drop_all_unique=True, drop_zero_variance=True, drop_all_missing=True) self.assertEqual(obs, md) self.assertIsNot(obs, md) def test_all_filters(self): obs = self.metadata.filter_columns( column_type='categorical', drop_all_unique=True, drop_zero_variance=True, drop_all_missing=True) self.assertEqual(set(obs.columns), {'cat'}) obs = self.metadata.filter_columns( column_type='numeric', drop_all_unique=True, drop_zero_variance=True, drop_all_missing=True) self.assertEqual(set(obs.columns), {'num'}) def test_all_columns_filtered(self): categorical = self.metadata.filter_columns(column_type='categorical') obs = categorical.filter_columns(column_type='numeric') exp = Metadata(pd.DataFrame( {}, index=pd.Index(['a', 'b', 'c', 'd'], name='id'))) self.assertEqual(obs, exp) def test_filter_to_categorical(self): obs = self.metadata.filter_columns(column_type='categorical') self.assertEqual(set(obs.columns), {'cat', 'uniq-cat', 'zvar-cat', 'empty-cat'}) def test_filter_to_numeric(self): obs = self.metadata.filter_columns(column_type='numeric') self.assertEqual(set(obs.columns), {'num', 'uniq-num', 'zvar-num', 'empty-num'}) def test_drop_all_unique(self): obs = self.metadata.filter_columns(drop_all_unique=True) self.assertEqual(set(obs.columns), {'cat', 'num', 'zvar-cat', 'zvar-num'}) def test_drop_zero_variance(self): obs = self.metadata.filter_columns(drop_zero_variance=True) self.assertEqual(set(obs.columns), {'cat', 'num', 'uniq-cat', 'uniq-num'}) def test_drop_all_missing(self): obs = self.metadata.filter_columns(drop_all_missing=True) self.assertEqual( set(obs.columns), {'cat', 'num', 'uniq-cat', 'uniq-num', 'zvar-cat', 'zvar-num'}) def test_drop_all_unique_with_single_id(self): md = Metadata(pd.DataFrame( {'cat': ['foo'], 'num': [-4.2], 'empty-cat': np.array([np.nan], dtype=object), 'empty-num': [np.nan]}, index=pd.Index(['id1'], name='id'))) obs = md.filter_columns(drop_all_unique=True) exp = Metadata(pd.DataFrame({}, index=pd.Index(['id1'], name='id'))) self.assertEqual(obs, exp) def test_drop_zero_variance_with_single_id(self): md = Metadata(pd.DataFrame( {'cat': ['foo'], 'num': [-4.2], 'empty-cat': np.array([np.nan], dtype=object), 'empty-num': [np.nan]}, index=pd.Index(['id1'], name='id'))) obs = md.filter_columns(drop_zero_variance=True) exp = Metadata(pd.DataFrame({}, index=pd.Index(['id1'], name='id'))) self.assertEqual(obs, exp) def test_retains_column_order(self): data = [[42, 'foo', 2.5], [42, 'bar', 0.5], [11, 'foo', 0.0]] md = Metadata(pd.DataFrame( data, columns=['z', 'a', 'ch'], index=pd.Index(['id1', 'id2', 'id3'], name='id'))) obs = md.filter_columns(column_type='numeric') exp_data = [[42, 2.5], [42, 0.5], [11, 0.0]] exp = Metadata(pd.DataFrame( exp_data, columns=['z', 'ch'], index=pd.Index(['id1', 'id2', 'id3'], name='id'))) self.assertEqual(obs, exp) def test_alternate_id_header(self): md = Metadata(pd.DataFrame( {'col1': ['foo', 'bar'], 'col2': [-4.2, -4.2], 'col3': ['bar', 'baz']}, index=pd.Index(['id1', 'id2'], name='feature-id'))) obs = md.filter_columns(drop_zero_variance=True) exp = Metadata(pd.DataFrame( {'col1': ['foo', 'bar'], 'col3': ['bar', 'baz']}, index=pd.Index(['id1', 'id2'], name='feature-id'))) self.assertEqual(obs, exp) def test_no_artifacts(self): md = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', 'bar', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) self.assertEqual(md.artifacts, ()) filtered = md.filter_columns(column_type='categorical') self.assertEqual(filtered.artifacts, ()) def test_with_artifacts(self): artifact1 = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) artifact2 = Artifact.import_data('Mapping', {'d': '4'}) md = Metadata(pd.DataFrame( {'col1': [1, 2, 3], 'col2': ['foo', 'bar', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) md._add_artifacts([artifact1, artifact2]) obs = md.filter_columns(column_type='categorical') exp = Metadata(pd.DataFrame( {'col2': ['foo', 'bar', 'baz']}, index=pd.Index(['a', 'b', 'c'], name='id'))) exp._add_artifacts([artifact1, artifact2]) self.assertEqual(obs, exp) self.assertEqual(obs.artifacts, (artifact1, artifact2)) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/metadata/tests/test_metadata_column.py000066400000000000000000001076131412141660100237320ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os.path import tempfile import unittest import pandas as pd import numpy as np from qiime2 import Artifact from qiime2.metadata import (MetadataColumn, CategoricalMetadataColumn, NumericMetadataColumn) from qiime2.core.testing.util import get_dummy_plugin, ReallyEqualMixin # Dummy class for testing MetadataColumn ABC class DummyMetadataColumn(MetadataColumn): type = 'dummy' @classmethod def _is_supported_dtype(cls, dtype): return dtype == 'float' or dtype == 'int' @classmethod def _normalize_(cls, series): return series.astype(float, copy=True, errors='raise') class TestInvalidMetadataColumnConstruction(unittest.TestCase): def test_non_series(self): with self.assertRaisesRegex( TypeError, 'DummyMetadataColumn constructor.*Series.*not.*' 'DataFrame'): DummyMetadataColumn(pd.DataFrame( {'col1': [1, 2, 3]}, index=pd.Index(['a', 'b', 'c'], name='id'))) def test_no_ids(self): with self.assertRaisesRegex(ValueError, 'DummyMetadataColumn.*at least one ID'): DummyMetadataColumn(pd.Series([], name='col', index=pd.Index([], name='id'), dtype=object)) def test_invalid_id_header(self): # default index name with self.assertRaisesRegex(ValueError, r'Index\.name.*None'): DummyMetadataColumn(pd.Series([1, 2, 3], name='col', index=pd.Index(['a', 'b', 'c'], dtype=object))) with self.assertRaisesRegex(ValueError, r'Index\.name.*my-id-header'): DummyMetadataColumn(pd.Series( [1, 2, 3], name='col', index=pd.Index(['a', 'b', 'c'], name='my-id-header'))) def test_non_str_id(self): with self.assertRaisesRegex( TypeError, 'non-string metadata ID.*type.*float.*nan'): DummyMetadataColumn(pd.Series( [1, 2, 3], name='col', index=pd.Index(['a', np.nan, 'c'], name='id'))) def test_non_str_column_name(self): # default series name with self.assertRaisesRegex( TypeError, 'non-string metadata column name.*type.*' 'NoneType.*None'): DummyMetadataColumn(pd.Series( [1, 2, 3], index=pd.Index(['a', 'b', 'c'], name='id'))) with self.assertRaisesRegex( TypeError, 'non-string metadata column name.*type.*' 'float.*nan'): DummyMetadataColumn(pd.Series( [1, 2, 3], name=np.nan, index=pd.Index(['a', 'b', 'c'], name='id'))) def test_empty_id(self): with self.assertRaisesRegex( ValueError, 'empty metadata ID.*at least one character'): DummyMetadataColumn(pd.Series( [1, 2, 3], name='col', index=pd.Index(['a', '', 'c'], name='id'))) def test_empty_column_name(self): with self.assertRaisesRegex( ValueError, 'empty metadata column name.*' 'at least one character'): DummyMetadataColumn(pd.Series( [1, 2, 3], name='', index=pd.Index(['a', 'b', 'c'], name='id'))) def test_pound_sign_id(self): with self.assertRaisesRegex( ValueError, "metadata ID.*begins with a pound sign.*'#b'"): DummyMetadataColumn(pd.Series( [1, 2, 3], name='col', index=pd.Index(['a', '#b', 'c'], name='id'))) def test_id_conflicts_with_id_header(self): with self.assertRaisesRegex( ValueError, "metadata ID 'sample-id'.*conflicts.*reserved.*" "ID header"): DummyMetadataColumn(pd.Series( [1, 2, 3], name='col', index=pd.Index(['a', 'sample-id', 'c'], name='id'))) def test_column_name_conflicts_with_id_header(self): with self.assertRaisesRegex( ValueError, "metadata column name 'featureid'.*conflicts.*" "reserved.*ID header"): DummyMetadataColumn(pd.Series( [1, 2, 3], name='featureid', index=pd.Index(['a', 'b', 'c'], name='id'))) def test_duplicate_ids(self): with self.assertRaisesRegex(ValueError, "Metadata IDs.*unique.*'a'"): DummyMetadataColumn(pd.Series( [1, 2, 3], name='col', index=pd.Index(['a', 'b', 'a'], name='id'))) def test_unsupported_column_dtype(self): with self.assertRaisesRegex( TypeError, "DummyMetadataColumn 'col1' does not support.*" "Series.*dtype.*bool"): DummyMetadataColumn(pd.Series( [True, False, True], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) class TestMetadataColumnConstructionAndProperties(unittest.TestCase): def test_single_id(self): index = pd.Index(['id1'], name='id') series = pd.Series([42], name='col1', index=index) mdc = DummyMetadataColumn(series) self.assertEqual(mdc.id_count, 1) self.assertEqual(mdc.id_header, 'id') self.assertEqual(mdc.ids, ('id1',)) self.assertEqual(mdc.name, 'col1') def test_multiple_ids(self): index = pd.Index(['id1', 'a', 'my-id'], name='id') series = pd.Series([42, 4.2, -4.2], name='column', index=index) mdc = DummyMetadataColumn(series) self.assertEqual(mdc.id_count, 3) self.assertEqual(mdc.id_header, 'id') self.assertEqual(mdc.ids, ('id1', 'a', 'my-id')) self.assertEqual(mdc.name, 'column') def test_supported_id_headers(self): case_insensitive = { 'id', 'sampleid', 'sample id', 'sample-id', 'featureid', 'feature id', 'feature-id' } exact_match = { '#SampleID', '#Sample ID', '#OTUID', '#OTU ID', 'sample_name' } # Build a set of supported headers, including exact matches and headers # with different casing. headers = set() for header in case_insensitive: headers.add(header) headers.add(header.upper()) headers.add(header.title()) for header in exact_match: headers.add(header) count = 0 for header in headers: index = pd.Index(['id1', 'id2'], name=header) series = pd.Series([0, 123], name='column', index=index) mdc = DummyMetadataColumn(series) self.assertEqual(mdc.id_header, header) count += 1 # Since this test case is a little complicated, make sure that the # expected number of comparisons are happening. self.assertEqual(count, 26) def test_recommended_ids(self): index = pd.Index(['c6ca034a-223f-40b4-a0e0-45942912a5ea', 'My.ID'], name='id') series = pd.Series([-1, -2], name='col1', index=index) mdc = DummyMetadataColumn(series) self.assertEqual(mdc.id_count, 2) self.assertEqual(mdc.id_header, 'id') self.assertEqual(mdc.ids, ('c6ca034a-223f-40b4-a0e0-45942912a5ea', 'My.ID')) self.assertEqual(mdc.name, 'col1') def test_non_standard_characters(self): index = pd.Index(['©id##1', '((id))2', "'id_3<>'", '"id#4"', 'i d\r\t\n5'], name='id') series = pd.Series([0, 1, 2, 3, 4], name='↩c@l1™', index=index) mdc = DummyMetadataColumn(series) self.assertEqual(mdc.id_count, 5) self.assertEqual(mdc.id_header, 'id') self.assertEqual( mdc.ids, ('©id##1', '((id))2', "'id_3<>'", '"id#4"', 'i d\r\t\n5')) self.assertEqual(mdc.name, '↩c@l1™') def test_missing_data(self): index = pd.Index(['None', 'nan', 'NA'], name='id') series = pd.Series([np.nan, np.nan, np.nan], name='NA', index=index) mdc = DummyMetadataColumn(series) self.assertEqual(mdc.id_count, 3) self.assertEqual(mdc.id_header, 'id') self.assertEqual(mdc.ids, ('None', 'nan', 'NA')) self.assertEqual(mdc.name, 'NA') def test_does_not_cast_ids_or_column_name(self): index = pd.Index(['0.000001', '0.004000', '0.000000'], dtype=object, name='id') series = pd.Series([2.0, 1.0, 3.0], name='42.0', index=index) mdc = DummyMetadataColumn(series) self.assertEqual(mdc.id_count, 3) self.assertEqual(mdc.id_header, 'id') self.assertEqual(mdc.ids, ('0.000001', '0.004000', '0.000000')) self.assertEqual(mdc.name, '42.0') def test_case_insensitive_duplicate_ids(self): index = pd.Index(['a', 'b', 'A'], name='id') series = pd.Series([1, 2, 3], name='column', index=index) mdc = DummyMetadataColumn(series) self.assertEqual(mdc.ids, ('a', 'b', 'A')) class TestSourceArtifacts(unittest.TestCase): def setUp(self): self.mdc = DummyMetadataColumn(pd.Series( [1, 2, 3], name='col', index=pd.Index(['a', 'b', 'c'], name='id'))) def test_no_source_artifacts(self): self.assertEqual(self.mdc.artifacts, ()) def test_add_zero_artifacts(self): self.mdc._add_artifacts([]) self.assertEqual(self.mdc.artifacts, ()) def test_add_artifacts(self): # First two artifacts have the same data but different UUIDs. artifact1 = Artifact.import_data('Mapping', {'a': '1', 'b': '3'}) self.mdc._add_artifacts([artifact1]) artifact2 = Artifact.import_data('Mapping', {'a': '1', 'b': '3'}) artifact3 = Artifact.import_data('IntSequence1', [1, 2, 3, 4]) self.mdc._add_artifacts([artifact2, artifact3]) self.assertEqual(self.mdc.artifacts, (artifact1, artifact2, artifact3)) def test_add_non_artifact(self): artifact = Artifact.import_data('Mapping', {'a': '1', 'b': '3'}) with self.assertRaisesRegex(TypeError, "Artifact object.*42"): self.mdc._add_artifacts([artifact, 42]) # Test that the object hasn't been mutated. self.assertEqual(self.mdc.artifacts, ()) def test_add_duplicate_artifact(self): artifact1 = Artifact.import_data('Mapping', {'a': '1', 'b': '3'}) artifact2 = Artifact.import_data('IntSequence1', [1, 2, 3, 4]) self.mdc._add_artifacts([artifact1, artifact2]) with self.assertRaisesRegex( ValueError, "Duplicate source artifacts.*DummyMetadataColumn.*" "artifact: Mapping"): self.mdc._add_artifacts([artifact1]) # Test that the object hasn't been mutated. self.assertEqual(self.mdc.artifacts, (artifact1, artifact2)) class TestRepr(unittest.TestCase): def test_single_id(self): mdc = DummyMetadataColumn(pd.Series( [42], name='foo', index=pd.Index(['id1'], name='id'))) obs = repr(mdc) self.assertEqual(obs, "") def test_multiple_ids(self): mdc = DummyMetadataColumn(pd.Series( [42, 43, 44], name='my column', index=pd.Index(['id1', 'id2', 'id3'], name='id'))) obs = repr(mdc) self.assertEqual( obs, "") class TestEqualityOperators(unittest.TestCase, ReallyEqualMixin): def setUp(self): get_dummy_plugin() def test_type_mismatch(self): dummy = DummyMetadataColumn(pd.Series( [1.0, 2.0, 3.0], name='col1', index=pd.Index(['id1', 'id2', 'id3'], name='id'))) numeric = NumericMetadataColumn(pd.Series( [1.0, 2.0, 3.0], name='col1', index=pd.Index(['id1', 'id2', 'id3'], name='id'))) categorical = CategoricalMetadataColumn(pd.Series( ['a', 'b', 'c'], name='col1', index=pd.Index(['id1', 'id2', 'id3'], name='id'))) self.assertReallyNotEqual(dummy, numeric) self.assertReallyNotEqual(dummy, categorical) def test_id_header_mismatch(self): mdc1 = DummyMetadataColumn(pd.Series( [42, 43], name='col1', index=pd.Index(['id1', 'id2'], name='id'))) mdc2 = DummyMetadataColumn(pd.Series( [42, 43], name='col1', index=pd.Index(['id1', 'id2'], name='ID'))) self.assertReallyNotEqual(mdc1, mdc2) def test_artifacts_mismatch(self): artifact1 = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) artifact2 = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) series = pd.Series([42, 43], name='col1', index=pd.Index(['id1', 'id2'], name='id')) # No artifacts mdc1 = DummyMetadataColumn(series) # Has an artifact mdc2 = DummyMetadataColumn(series) mdc2._add_artifacts([artifact1]) # Has a different artifact mdc3 = DummyMetadataColumn(series) mdc3._add_artifacts([artifact2]) self.assertReallyNotEqual(mdc1, mdc2) self.assertReallyNotEqual(mdc2, mdc3) def test_id_mismatch(self): mdc1 = DummyMetadataColumn(pd.Series( [42, 43], name='col1', index=pd.Index(['id1', 'id2'], name='id'))) mdc2 = DummyMetadataColumn(pd.Series( [42, 43], name='col1', index=pd.Index(['id1', 'id3'], name='id'))) self.assertReallyNotEqual(mdc1, mdc2) def test_column_name_mismatch(self): mdc1 = DummyMetadataColumn(pd.Series( [42, 43], name='col1', index=pd.Index(['id1', 'id2'], name='id'))) mdc2 = DummyMetadataColumn(pd.Series( [42, 43], name='col2', index=pd.Index(['id1', 'id2'], name='id'))) self.assertReallyNotEqual(mdc1, mdc2) def test_data_mismatch(self): mdc1 = DummyMetadataColumn(pd.Series( [42, 43], name='col1', index=pd.Index(['id1', 'id2'], name='id'))) mdc2 = DummyMetadataColumn(pd.Series( [42, 42], name='col1', index=pd.Index(['id1', 'id2'], name='id'))) self.assertReallyNotEqual(mdc1, mdc2) def test_equality_without_artifact(self): mdc1 = DummyMetadataColumn(pd.Series( [42, 43], name='col1', index=pd.Index(['id1', 'id2'], name='id'))) mdc2 = DummyMetadataColumn(pd.Series( [42, 43], name='col1', index=pd.Index(['id1', 'id2'], name='id'))) self.assertReallyEqual(mdc1, mdc2) def test_equality_with_artifact(self): artifact = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) mdc1 = DummyMetadataColumn(pd.Series( [42, 43], name='col1', index=pd.Index(['id1', 'id2'], name='id'))) mdc1._add_artifacts([artifact]) mdc2 = DummyMetadataColumn(pd.Series( [42, 43], name='col1', index=pd.Index(['id1', 'id2'], name='id'))) mdc2._add_artifacts([artifact]) self.assertReallyEqual(mdc1, mdc2) def test_equality_with_missing_data(self): mdc1 = DummyMetadataColumn(pd.Series( [42, np.nan, 43, np.nan], name='col1', index=pd.Index(['id1', 'id2', 'id3', 'id4'], name='id'))) mdc2 = DummyMetadataColumn(pd.Series( [42, np.nan, 43, np.nan], name='col1', index=pd.Index(['id1', 'id2', 'id3', 'id4'], name='id'))) self.assertReallyEqual(mdc1, mdc2) # Extensive tests of the MetadataWriter are performed in test_io.py. This test # is a sanity check that a new MetadataColumn subclass (DummyMetadataColumn) # can be written to disk with its column type preserved. This test would have # caught a bug in the original implementation of MetadataColumn.save(), which # converted itself into a Metadata object, losing the "dummy" column type and # replacing it with "numeric". In order for a MetadataColumn to turn itself # into a Metadata object in a lossless/safe way, the Metadata constructor needs # a `column_types` parameter to preserve column types. class TestSave(unittest.TestCase): def setUp(self): self.temp_dir_obj = tempfile.TemporaryDirectory( prefix='qiime2-metadata-tests-temp-') self.temp_dir = self.temp_dir_obj.name self.filepath = os.path.join(self.temp_dir, 'metadata.tsv') def tearDown(self): self.temp_dir_obj.cleanup() def test_basic(self): mdc = DummyMetadataColumn(pd.Series( [42, 42.5, -999.123], name='dummy-column', index=pd.Index(['id1', 'id2', 'id3'], name='id'))) mdc.save(self.filepath) with open(self.filepath, 'r') as fh: obs = fh.read() exp = ( "id\tdummy-column\n" "#q2:types\tdummy\n" "id1\t42\n" "id2\t42.5\n" "id3\t-999.123\n" ) self.assertEqual(obs, exp) class TestToSeries(unittest.TestCase): def test_single_id(self): series = pd.Series([0.0], name='col', index=pd.Index(['id1'], name='id')) mdc = DummyMetadataColumn(series) obs = mdc.to_series() pd.testing.assert_series_equal(obs, series) def test_multiple_ids(self): series = pd.Series([-1.5, np.nan, 42], name='col', index=pd.Index(['id1', 'id2', 'id3'], name='id')) mdc = DummyMetadataColumn(series) obs = mdc.to_series() pd.testing.assert_series_equal(obs, series) def test_id_header_preserved(self): series = pd.Series( [-1.5, 0.0, 42], name='col', index=pd.Index(['id1', 'id2', 'id3'], name='#OTU ID')) mdc = DummyMetadataColumn(series) obs = mdc.to_series() pd.testing.assert_series_equal(obs, series) self.assertEqual(obs.index.name, '#OTU ID') def test_series_copy(self): series = pd.Series([1, 2.5, 3], name='col', index=pd.Index(['id1', 'id2', 'id3'], name='id')) mdc = DummyMetadataColumn(series) obs = mdc.to_series() pd.testing.assert_series_equal(obs, series) self.assertIsNot(obs, series) class TestToDataframe(unittest.TestCase): def test_single_id(self): series = pd.Series([0.0], name='col', index=pd.Index(['id1'], name='id')) mdc = DummyMetadataColumn(series) obs = mdc.to_dataframe() exp = pd.DataFrame({'col': [0.0]}, index=pd.Index(['id1'], name='id')) pd.testing.assert_frame_equal(obs, exp) def test_multiple_ids(self): series = pd.Series([0.0, 4.2, np.nan], name='my column', index=pd.Index(['a', 'b', 'c'], name='id')) mdc = DummyMetadataColumn(series) obs = mdc.to_dataframe() exp = pd.DataFrame({'my column': [0.0, 4.2, np.nan]}, index=pd.Index(['a', 'b', 'c'], name='id')) pd.testing.assert_frame_equal(obs, exp) def test_id_header_preserved(self): series = pd.Series([0.0, 4.2, 123], name='my column', index=pd.Index(['a', 'b', 'c'], name='#Sample ID')) mdc = DummyMetadataColumn(series) obs = mdc.to_dataframe() exp = pd.DataFrame({'my column': [0.0, 4.2, 123]}, index=pd.Index(['a', 'b', 'c'], name='#Sample ID')) pd.testing.assert_frame_equal(obs, exp) self.assertEqual(obs.index.name, '#Sample ID') class TestGetValue(unittest.TestCase): def test_id_not_found(self): series = pd.Series([1, 2, 3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id')) mdc = DummyMetadataColumn(series) with self.assertRaisesRegex( ValueError, "'d' is not present.*DummyMetadataColumn.*'col1'"): mdc.get_value('d') def test_get_value(self): series = pd.Series([1, 2, np.nan], name='col1', index=pd.Index(['a', 'b', 'c'], name='id')) mdc = DummyMetadataColumn(series) obs = mdc.get_value('a') self.assertEqual(obs, 1.0) obs = mdc.get_value('b') self.assertEqual(obs, 2.0) obs = mdc.get_value('c') self.assertTrue(np.isnan(obs)) class TestHasMissingValues(unittest.TestCase): def test_no_missing_values(self): series = pd.Series([0.0, 2.2, 3.3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id')) mdc = DummyMetadataColumn(series) obs = mdc.has_missing_values() self.assertEqual(obs, False) def test_with_missing_values(self): series = pd.Series([0.0, np.nan, 3.3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id')) mdc = DummyMetadataColumn(series) obs = mdc.has_missing_values() self.assertEqual(obs, True) class TestDropMissingValues(unittest.TestCase): def test_no_missing_values(self): series = pd.Series([0.0, 2.2, 3.3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id')) mdc = DummyMetadataColumn(series) obs = mdc.drop_missing_values() self.assertEqual(obs, mdc) self.assertIsNot(obs, mdc) def test_with_missing_values(self): series = pd.Series( [0.0, np.nan, 3.3, np.nan, np.nan, 4.4], name='col1', index=pd.Index(['a', 'b', 'c', 'd', 'e', 'f'], name='sampleid')) mdc = DummyMetadataColumn(series) obs = mdc.drop_missing_values() exp = DummyMetadataColumn(pd.Series( [0.0, 3.3, 4.4], name='col1', index=pd.Index(['a', 'c', 'f'], name='sampleid'))) self.assertEqual(obs, exp) def test_artifacts_are_propagated(self): artifact = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) series = pd.Series( [0.0, np.nan, 3.3, np.nan, np.nan, 4.4], name='col1', index=pd.Index(['a', 'b', 'c', 'd', 'e', 'f'], name='sampleid')) mdc = DummyMetadataColumn(series) mdc._add_artifacts([artifact]) obs = mdc.drop_missing_values() exp = DummyMetadataColumn(pd.Series( [0.0, 3.3, 4.4], name='col1', index=pd.Index(['a', 'c', 'f'], name='sampleid'))) exp._add_artifacts([artifact]) self.assertEqual(obs, exp) self.assertEqual(obs.artifacts, (artifact,)) class TestGetIDs(unittest.TestCase): def test_single_id(self): series = pd.Series([1.234], name='col1', index=pd.Index(['my id'], name='id')) mdc = DummyMetadataColumn(series) obs = mdc.get_ids() self.assertEqual(obs, {'my id'}) def test_multiple_ids(self): series = pd.Series( [1.234, np.nan, 5.67, np.nan, 8.9], name='col1', index=pd.Index(['id1', 'id2', 'id3', 'id4', 'id5'], name='id')) mdc = DummyMetadataColumn(series) obs = mdc.get_ids() self.assertEqual(obs, {'id1', 'id2', 'id3', 'id4', 'id5'}) def test_where_values_missing(self): series = pd.Series( [1.234, np.nan, 5.67, np.nan, 8.9], name='col1', index=pd.Index(['id1', 'id2', 'id3', 'id4', 'id5'], name='id')) mdc = DummyMetadataColumn(series) obs = mdc.get_ids(where_values_missing=True) self.assertEqual(obs, {'id2', 'id4'}) def test_where_values_missing_all_missing(self): series = pd.Series( [np.nan, np.nan, np.nan], name='col1', index=pd.Index(['id1', 'id2', 'id3'], name='id')) mdc = DummyMetadataColumn(series) obs = mdc.get_ids(where_values_missing=True) self.assertEqual(obs, {'id1', 'id2', 'id3'}) class TestFilterIDs(unittest.TestCase): def setUp(self): get_dummy_plugin() def test_supports_iterable(self): mdc = DummyMetadataColumn(pd.Series( [1, 2, 3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) obs = mdc.filter_ids(iter({'a', 'c'})) exp = DummyMetadataColumn(pd.Series( [1, 3], name='col1', index=pd.Index(['a', 'c'], name='id'))) self.assertEqual(obs, exp) def test_keep_all(self): mdc = DummyMetadataColumn(pd.Series( [1, 2, 3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) obs = mdc.filter_ids({'a', 'b', 'c'}) self.assertEqual(obs, mdc) self.assertIsNot(obs, mdc) def test_keep_multiple(self): mdc = DummyMetadataColumn(pd.Series( [1, 2, 3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) obs = mdc.filter_ids({'a', 'c'}) exp = DummyMetadataColumn(pd.Series( [1, 3], name='col1', index=pd.Index(['a', 'c'], name='id'))) self.assertEqual(obs, exp) def test_keep_one(self): mdc = DummyMetadataColumn(pd.Series( [1, 2, 3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) obs = mdc.filter_ids({'b'}) exp = DummyMetadataColumn(pd.Series( [2], name='col1', index=pd.Index(['b'], name='id'))) self.assertEqual(obs, exp) def test_alternate_id_header(self): mdc = DummyMetadataColumn(pd.Series( [1, 2, 3], name='col1', index=pd.Index(['a', 'b', 'c'], name='#OTU ID'))) obs = mdc.filter_ids({'b', 'c'}) exp = DummyMetadataColumn(pd.Series( [2, 3], name='col1', index=pd.Index(['b', 'c'], name='#OTU ID'))) self.assertEqual(obs, exp) def test_no_artifacts(self): mdc = DummyMetadataColumn(pd.Series( [1, 2, 3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) self.assertEqual(mdc.artifacts, ()) filtered = mdc.filter_ids({'b'}) self.assertEqual(filtered.artifacts, ()) def test_with_artifacts(self): artifact1 = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) artifact2 = Artifact.import_data('Mapping', {'d': '4'}) mdc = DummyMetadataColumn(pd.Series( [1, 2, 3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) mdc._add_artifacts([artifact1, artifact2]) obs = mdc.filter_ids({'a', 'c'}) exp = DummyMetadataColumn(pd.Series( [1, 3], name='col1', index=pd.Index(['a', 'c'], name='id'))) exp._add_artifacts([artifact1, artifact2]) self.assertEqual(obs, exp) self.assertEqual(obs.artifacts, (artifact1, artifact2)) def test_empty_ids_to_keep(self): mdc = DummyMetadataColumn(pd.Series( [1, 2, 3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) with self.assertRaisesRegex(ValueError, 'ids_to_keep.*at least one ID'): mdc.filter_ids({}) def test_duplicate_ids_to_keep(self): mdc = DummyMetadataColumn(pd.Series( [1, 2, 3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) with self.assertRaisesRegex(ValueError, "ids_to_keep.*unique IDs.*'b'"): mdc.filter_ids(['b', 'c', 'b']) def test_missing_ids_to_keep(self): mdc = DummyMetadataColumn(pd.Series( [1, 2, 3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) with self.assertRaisesRegex(ValueError, "IDs.*not present.*'d', 'id1'"): mdc.filter_ids({'b', 'id1', 'c', 'd'}) # The tests for CategoricalMetadataColumn and NumericMetadataColumn only test # behavior specific to these subclasses. More extensive tests of these objects # are performed above by testing the MetadataColumn ABC in a generic way. class TestCategoricalMetadataColumn(unittest.TestCase): def test_unsupported_dtype(self): with self.assertRaisesRegex( TypeError, "CategoricalMetadataColumn 'col1' does not support" ".*Series.*dtype.*float64"): CategoricalMetadataColumn(pd.Series( [42.5, 42.6, 42.7], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) def test_unsupported_type_value(self): with self.assertRaisesRegex( TypeError, "CategoricalMetadataColumn.*strings or missing " r"values.*42\.5.*float.*'col1'"): CategoricalMetadataColumn(pd.Series( ['foo', 'bar', 42.5], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) def test_empty_str_value(self): with self.assertRaisesRegex( ValueError, "CategoricalMetadataColumn.*empty strings.*" "column 'col1'"): CategoricalMetadataColumn(pd.Series( ['foo', '', 'bar'], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) def test_type_property(self): self.assertEqual(CategoricalMetadataColumn.type, 'categorical') def test_supported_dtype(self): series = pd.Series( ['foo', np.nan, 'bar', 'foo'], name='my column', index=pd.Index(['a', 'b', 'c', 'd'], name='id')) mdc = CategoricalMetadataColumn(series) self.assertEqual(mdc.id_count, 4) self.assertEqual(mdc.id_header, 'id') self.assertEqual(mdc.ids, ('a', 'b', 'c', 'd')) self.assertEqual(mdc.name, 'my column') obs_series = mdc.to_series() pd.testing.assert_series_equal(obs_series, series) self.assertEqual(obs_series.dtype, object) def test_numeric_strings_preserved_as_strings(self): series = pd.Series( ['1', np.nan, '2.5', '3.0'], name='my column', index=pd.Index(['a', 'b', 'c', 'd'], name='id')) mdc = CategoricalMetadataColumn(series) self.assertEqual(mdc.id_count, 4) self.assertEqual(mdc.id_header, 'id') self.assertEqual(mdc.ids, ('a', 'b', 'c', 'd')) self.assertEqual(mdc.name, 'my column') obs_series = mdc.to_series() pd.testing.assert_series_equal(obs_series, series) self.assertEqual(obs_series.dtype, object) def test_missing_data_normalized(self): # Different missing data representations should be normalized to np.nan mdc = CategoricalMetadataColumn(pd.Series( [np.nan, 'foo', float('nan'), None], name='col1', index=pd.Index(['a', 'b', 'c', 'd'], name='id'))) obs = mdc.to_series() exp = pd.Series( [np.nan, 'foo', np.nan, np.nan], name='col1', index=pd.Index(['a', 'b', 'c', 'd'], name='id')) pd.testing.assert_series_equal(obs, exp) self.assertEqual(obs.dtype, object) self.assertTrue(np.isnan(obs['a'])) self.assertTrue(np.isnan(obs['c'])) self.assertTrue(np.isnan(obs['d'])) def test_all_missing_data(self): mdc = CategoricalMetadataColumn(pd.Series( np.array([np.nan, np.nan, np.nan], dtype=object), name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) obs = mdc.to_series() exp = pd.Series( np.array([np.nan, np.nan, np.nan], dtype=object), name='col1', index=pd.Index(['a', 'b', 'c'], name='id')) pd.testing.assert_series_equal(obs, exp) self.assertEqual(obs.dtype, object) def test_leading_trailing_whitespace_value(self): col1 = CategoricalMetadataColumn(pd.Series( ['foo', ' bar ', 'baz'], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) col2 = CategoricalMetadataColumn(pd.Series( ['foo', 'bar', 'baz'], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) self.assertEqual(col1, col2) def test_leading_trailing_whitespace_id(self): col1 = CategoricalMetadataColumn(pd.Series( ['foo', ' bar ', 'baz'], name='col', index=pd.Index(['a', ' b ', 'c'], name='id'))) col2 = CategoricalMetadataColumn(pd.Series( ['foo', ' bar ', 'baz'], name='col', index=pd.Index(['a', 'b', 'c'], name='id'))) self.assertEqual(col1, col2) def test_leading_trailing_whitespace_column_name(self): col1 = CategoricalMetadataColumn(pd.Series( ['foo', ' bar ', 'baz'], name=' col ', index=pd.Index(['a', 'b', 'c'], name='id'))) col2 = CategoricalMetadataColumn(pd.Series( ['foo', ' bar ', 'baz'], name='col', index=pd.Index(['a', 'b', 'c'], name='id'))) self.assertEqual(col1, col2) class TestNumericMetadataColumn(unittest.TestCase): def test_unsupported_dtype(self): with self.assertRaisesRegex( TypeError, "NumericMetadataColumn 'col1' does not support" ".*Series.*dtype.*bool"): NumericMetadataColumn(pd.Series( [True, False, True], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) def test_infinity_value(self): with self.assertRaisesRegex( ValueError, "NumericMetadataColumn.*positive or negative " "infinity.*column 'col1'"): NumericMetadataColumn(pd.Series( [42, float('+inf'), 4.3], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) def test_type_property(self): self.assertEqual(NumericMetadataColumn.type, 'numeric') def test_supported_dtype_float(self): series = pd.Series( [1.23, np.nan, 4.56, -7.891], name='my column', index=pd.Index(['a', 'b', 'c', 'd'], name='id')) mdc = NumericMetadataColumn(series) self.assertEqual(mdc.id_count, 4) self.assertEqual(mdc.id_header, 'id') self.assertEqual(mdc.ids, ('a', 'b', 'c', 'd')) self.assertEqual(mdc.name, 'my column') obs_series = mdc.to_series() pd.testing.assert_series_equal(obs_series, series) self.assertEqual(obs_series.dtype, np.float64) def test_supported_dtype_int(self): series = pd.Series( [0, 1, 42, -2], name='my column', index=pd.Index(['a', 'b', 'c', 'd'], name='id')) mdc = NumericMetadataColumn(series) self.assertEqual(mdc.id_count, 4) self.assertEqual(mdc.id_header, 'id') self.assertEqual(mdc.ids, ('a', 'b', 'c', 'd')) self.assertEqual(mdc.name, 'my column') obs_series = mdc.to_series() exp_series = pd.Series( [0.0, 1.0, 42.0, -2.0], name='my column', index=pd.Index(['a', 'b', 'c', 'd'], name='id')) pd.testing.assert_series_equal(obs_series, exp_series) self.assertEqual(obs_series.dtype, np.float64) def test_missing_data_normalized(self): # Different missing data representations should be normalized to np.nan mdc = NumericMetadataColumn(pd.Series( [np.nan, 4.2, float('nan'), -5.678], name='col1', index=pd.Index(['a', 'b', 'c', 'd'], name='id'))) obs = mdc.to_series() exp = pd.Series( [np.nan, 4.2, np.nan, -5.678], name='col1', index=pd.Index(['a', 'b', 'c', 'd'], name='id')) pd.testing.assert_series_equal(obs, exp) self.assertEqual(obs.dtype, np.float64) self.assertTrue(np.isnan(obs['a'])) self.assertTrue(np.isnan(obs['c'])) def test_all_missing_data(self): mdc = NumericMetadataColumn(pd.Series( [np.nan, np.nan, np.nan], name='col1', index=pd.Index(['a', 'b', 'c'], name='id'))) obs = mdc.to_series() exp = pd.Series( [np.nan, np.nan, np.nan], name='col1', index=pd.Index(['a', 'b', 'c'], name='id')) pd.testing.assert_series_equal(obs, exp) self.assertEqual(obs.dtype, np.float64) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/plugin/000077500000000000000000000000001412141660100155305ustar00rootroot00000000000000qiime-2021.8.0/qiime2/plugin/__init__.py000066400000000000000000000023111412141660100176360ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from .model import (TextFileFormat, BinaryFileFormat, DirectoryFormat, ValidationError) from .plugin import Plugin from qiime2.core.cite import Citations, CitationRecord from qiime2.core.type import (SemanticType, Int, Str, Float, Metadata, MetadataColumn, Categorical, Numeric, Properties, Range, Start, End, Choices, Bool, Set, List, Visualization, TypeMap, TypeMatch) __all__ = ['TextFileFormat', 'BinaryFileFormat', 'DirectoryFormat', 'Plugin', 'SemanticType', 'Set', 'List', 'Bool', 'Int', 'Str', 'Float', 'Metadata', 'MetadataColumn', 'Categorical', 'Numeric', 'Properties', 'Range', 'Start', 'End', 'Choices', 'Visualization', 'TypeMap', 'TypeMatch', 'ValidationError', 'Citations', 'CitationRecord'] qiime-2021.8.0/qiime2/plugin/model/000077500000000000000000000000001412141660100166305ustar00rootroot00000000000000qiime-2021.8.0/qiime2/plugin/model/__init__.py000066400000000000000000000014121412141660100207370ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from .directory_format import ( DirectoryFormat, File, FileCollection, SingleFileDirectoryFormat, SingleFileDirectoryFormatBase) from .file_format import TextFileFormat, BinaryFileFormat from .base import ValidationError __all__ = ['DirectoryFormat', 'File', 'FileCollection', 'TextFileFormat', 'BinaryFileFormat', 'SingleFileDirectoryFormat', 'SingleFileDirectoryFormatBase', 'ValidationError'] qiime-2021.8.0/qiime2/plugin/model/base.py000066400000000000000000000014421412141660100201150ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from qiime2.core.format import FormatBase from qiime2.core.exceptions import ValidationError __all__ = ['FormatBase', 'ValidationError', '_check_validation_level'] # TODO: once sniff is dropped, move this up into FormatBase as validate method def _check_validation_level(level): if level not in ('min', 'max'): raise ValueError('Invalid validation level requested (%s), must ' 'be \'min\' or \'max\'.' % level) qiime-2021.8.0/qiime2/plugin/model/directory_format.py000066400000000000000000000166411412141660100225660ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import re import sys import pathlib from qiime2.core import transform from .base import FormatBase, ValidationError, _check_validation_level class PathMakerDescriptor: def __init__(self, file): self.file = file def __get__(self, obj, cls=None): if obj is None: raise Exception() return getattr(obj, self.file.name).path_maker class File: def __init__(self, pathspec, *, format=None): if format is None: raise TypeError("Must provide a format.") self.pathspec = pathspec self.format = format def __get__(self, obj, cls=None): if obj is None: return self return BoundFile(self.name, self.pathspec, self.format, obj) class FileCollection(File): def __init__(self, pathspec, *, format=None): super().__init__(pathspec, format=format) self._path_maker = None def set_path_maker(self, function): self._path_maker = function return PathMakerDescriptor(self) def __get__(self, obj, cls=None): if obj is None: return self if self._path_maker is None: raise NotImplementedError() return BoundFileCollection(self.name, self.pathspec, self.format, obj, path_maker=self._path_maker) class BoundFile: @property def mode(self): return self._directory_format._mode def __init__(self, name, pathspec, format, directory_format): self.name = name self.pathspec = pathspec self.format = format self._directory_format = directory_format self._path_maker = lambda s: pathspec def view(self, view_type): from_type = transform.ModelType.from_view_type(self.format) to_type = transform.ModelType.from_view_type(view_type) transformation = from_type.make_transformation(to_type) return transformation(self.path_maker()) def write_data(self, view, view_type, **kwargs): # TODO: make `view_type` optional like in `Artifact.import_data` if self.mode != 'w': raise TypeError("Cannot use `set`/`add` when mode=%r" % self.mode) from_type = transform.ModelType.from_view_type(view_type) to_type = transform.ModelType.from_view_type(self.format) transformation = from_type.make_transformation(to_type) result = transformation(view) result.path._move_or_copy(self.path_maker(**kwargs)) def _validate_members(self, collected_paths, level): found_members = False root = pathlib.Path(self._directory_format.path) for path in collected_paths: if re.fullmatch(self.pathspec, str(path.relative_to(root))): if collected_paths[path]: # Not a ValidationError, this just shouldn't happen. raise ValueError("%r was already validated by another" " field, the pathspecs (regexes) must" " overlap." % path) collected_paths[path] = True found_members = True self.format(path, mode='r').validate(level) if not found_members: raise ValidationError( "Missing one or more files for %s: %r" % (self._directory_format.__class__.__name__, self.pathspec)) @property def path_maker(self): def bound_path_maker(**kwargs): # Must wrap in a naive Path, otherwise an OutPath would be summoned # into this world, and would destroy everything in its path. path = (pathlib.Path(self._directory_format.path) / self._path_maker(self._directory_format, **kwargs)) # NOTE: path makers are bound to the directory format, so must be # provided as the first argument which will look like `self` to # the plugin-dev. path.parent.mkdir(parents=True, exist_ok=True) return path return bound_path_maker class BoundFileCollection(BoundFile): def __init__(self, name, pathspec, format, directory_format, path_maker): super().__init__(name, pathspec, format, directory_format) self._path_maker = path_maker def view(self, view_type): raise NotImplementedError("Use `iter_views` instead.") def iter_views(self, view_type): # Don't want an OutPath, just a Path root = pathlib.Path(self._directory_format.path) paths = [fp for fp in sorted(root.glob('**/*')) if re.match(self.pathspec, str(fp.relative_to(root)))] from_type = transform.ModelType.from_view_type(self.format) to_type = transform.ModelType.from_view_type(view_type) transformation = from_type.make_transformation(to_type) for fp in paths: # TODO: include capture? yield fp.relative_to(root), transformation(fp) class _DirectoryMeta(type): def __init__(self, name, bases, dct): super().__init__(name, bases, dct) if hasattr(self, '_fields'): fields = self._fields.copy() else: fields = [] for key, value in dct.items(): if isinstance(value, File): # TODO: validate that the paths described by `value` are unique # within a DirectoryFormat value.name = key fields.append(key) self._fields = fields class DirectoryFormat(FormatBase, metaclass=_DirectoryMeta): def validate(self, level='max'): _check_validation_level(level) if not self.path.is_dir(): raise ValidationError("%s is not a directory." % self.path) collected_paths = {p: None for p in self.path.glob('**/*') if not p.name.startswith('.') and p.is_file()} for field in self._fields: getattr(self, field)._validate_members(collected_paths, level) for path, value in collected_paths.items(): if value: continue if value is None: raise ValidationError("Unrecognized file (%s) for %s." % (path, self.__class__.__name__)) if hasattr(self, '_validate_'): try: self._validate_(level) except ValidationError as e: raise ValidationError( "%s is not a(n) %s:\n\n%s" % (self.path, self.__class__.__name__, str(e)) ) from e class SingleFileDirectoryFormatBase(DirectoryFormat): pass def SingleFileDirectoryFormat(name, pathspec, format): # TODO: do the same hack namedtuple does so we don't mangle globals # (arguably the code is going to be broken if defined dynamically anyways, # but better to find that out later than writing in the module namespace # even if it isn't called module-level [which is must be!]) df = type(name, (SingleFileDirectoryFormatBase,), {'file': File(pathspec, format=format)}) df.__module__ = sys._getframe(1).f_globals.get('__name__', '__main__') return df qiime-2021.8.0/qiime2/plugin/model/file_format.py000066400000000000000000000042711412141660100214750ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import abc from qiime2.core import transform from .base import FormatBase, ValidationError, _check_validation_level class _FileFormat(FormatBase, metaclass=abc.ABCMeta): def validate(self, level='max'): _check_validation_level(level) if not self.path.is_file(): raise ValidationError("%s is not a file." % self.path) if hasattr(self, '_validate_'): try: self._validate_(level) except ValidationError as e: raise ValidationError( "%s is not a(n) %s file:\n\n%s" % (self.path, self.__class__.__name__, str(e)) ) from e # TODO: remove this branch elif hasattr(self, 'sniff'): if not self.sniff(): raise ValidationError("%s is not a(n) %s file" % (self.path, self.__class__.__name__)) # TODO: define an abc.abstractmethod for `validate` when sniff is # removed instead of this else: raise NotImplementedError("%r does not implement validate." % type(self)) def view(self, view_type): from_type = transform.ModelType.from_view_type(self.__class__) to_type = transform.ModelType.from_view_type(view_type) transformation = from_type.make_transformation(to_type) return transformation(self) class TextFileFormat(_FileFormat): def open(self): mode = 'r' if self._mode == 'r' else 'r+' # ignore BOM only when reading, do not emit BOM on write encoding = 'utf-8-sig' if mode == 'r' else 'utf-8' return self.path.open(mode=mode, encoding=encoding) class BinaryFileFormat(_FileFormat): def open(self): mode = 'rb' if self._mode == 'r' else 'r+b' return self.path.open(mode=mode) qiime-2021.8.0/qiime2/plugin/model/tests/000077500000000000000000000000001412141660100177725ustar00rootroot00000000000000qiime-2021.8.0/qiime2/plugin/model/tests/__init__.py000066400000000000000000000005351412141660100221060ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- qiime-2021.8.0/qiime2/plugin/model/tests/test_file_format.py000066400000000000000000000050301412141660100236700ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import unittest import tempfile import qiime2.plugin.model as model from qiime2.core.testing.plugin import SingleIntFormat class TestTextFileFormat(unittest.TestCase): PAYLOAD = "Somewhere over the rainbow." def setUp(self): self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-') def tearDown(self): self.test_dir.cleanup() def test_open_read_good(self): path = os.path.join(self.test_dir.name, 'file') with open(path, 'w', encoding='utf-8') as fh: fh.write(self.PAYLOAD) ff = model.TextFileFormat(path, mode='r') with ff.open() as fh: self.assertEqual(self.PAYLOAD, fh.read()) def test_open_read_ignore_bom(self): path = os.path.join(self.test_dir.name, 'file') with open(path, 'w', encoding='utf-8-sig') as fh: fh.write(self.PAYLOAD) ff = model.TextFileFormat(path, mode='r') with ff.open() as fh: self.assertEqual(self.PAYLOAD, fh.read()) def test_open_write_good(self): ff = model.TextFileFormat() with ff.open() as fh: fh.write(self.PAYLOAD) with open(str(ff), mode='r', encoding='utf-8') as fh: self.assertEqual(self.PAYLOAD, fh.read()) def test_open_write_no_bom(self): ff = model.TextFileFormat() with ff.open() as fh: fh.write(self.PAYLOAD) with open(str(ff), mode='rb') as fh: self.assertEqual(b'S', fh.read(1)) class TestFileFormat(unittest.TestCase): def setUp(self): self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-') path = os.path.join(self.test_dir.name, 'int') with open(path, 'w') as fh: fh.write('1') self.format = SingleIntFormat(path, mode='r') def tearDown(self): self.test_dir.cleanup() def test_view_expected(self): number = self.format.view(int) self.assertEqual(1, number) def test_view_invalid_type(self): with self.assertRaisesRegex( Exception, "No transformation.*SingleIntFormat.*float"): self.format.view(float) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/plugin/plugin.py000066400000000000000000000322031412141660100174000ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import collections import inspect import types import qiime2.sdk import qiime2.core.type.grammar as grammar from qiime2.core.validate import ValidationObject from qiime2.plugin.model import DirectoryFormat from qiime2.plugin.model.base import FormatBase from qiime2.core.type import is_semantic_type from qiime2.core.util import get_view_name TransformerRecord = collections.namedtuple( 'TransformerRecord', ['transformer', 'plugin', 'citations']) SemanticTypeRecord = collections.namedtuple( 'SemanticTypeRecord', ['semantic_type', 'plugin']) SemanticTypeFragmentRecord = collections.namedtuple( 'SemanticTypeFragmentRecord', ['fragment', 'plugin']) FormatRecord = collections.namedtuple('FormatRecord', ['format', 'plugin']) ViewRecord = collections.namedtuple( 'ViewRecord', ['name', 'view', 'plugin', 'citations']) TypeFormatRecord = collections.namedtuple( 'TypeFormatRecord', ['type_expression', 'format', 'plugin']) ValidatorRecord = collections.namedtuple( 'ValidatorRecord', ['validator', 'view', 'plugin', 'context']) class Plugin: def __init__(self, name, version, website, package=None, project_name=None, citation_text=None, user_support_text=None, short_description=None, description=None, citations=None): self.id = name.replace('-', '_') self.name = name self.version = version self.website = website # Filled in by the PluginManager if not provided. self.package = package self.project_name = project_name if user_support_text is None: self.user_support_text = ('Please post to the QIIME 2 forum for ' 'help with this plugin: https://forum.' 'qiime2.org') else: self.user_support_text = user_support_text if short_description is None: self.short_description = '' else: self.short_description = short_description if description is None: self.description = ('No description available. ' 'See plugin website: %s' % self.website) else: self.description = description if citations is None: self.citations = () else: self.citations = tuple(citations) self.methods = PluginMethods(self) self.visualizers = PluginVisualizers(self) self.pipelines = PluginPipelines(self) self.formats = {} self.views = {} self.type_fragments = {} self.transformers = {} self.type_formats = [] self.validators = {} def freeze(self): pass @property def actions(self): # TODO this doesn't handle method/visualizer name collisions. The # auto-generated `qiime2.plugins..actions` API has the # same problem. This should be solved at method/visualizer registration # time, which will solve the problem for both APIs. actions = {} actions.update(self.methods) actions.update(self.visualizers) actions.update(self.pipelines) return types.MappingProxyType(actions) @property def types(self): types = {} for record in self.type_formats: for type_ in record.type_expression: types[str(type_)] = \ SemanticTypeRecord(semantic_type=type_, plugin=self) return types def register_formats(self, *formats, citations=None): for format in formats: if not issubclass(format, FormatBase): raise TypeError("%r is not a valid format." % format) self.register_views(*formats, citations=citations) def register_views(self, *views, citations=None): if citations is None: citations = () else: citations = tuple(citations) for view in views: if not isinstance(view, type): raise TypeError("%r should be a class." % view) is_format = False if issubclass(view, FormatBase): is_format = True name = get_view_name(view) if name in self.views: raise NameError("View %r is already registered by this " "plugin." % name) self.views[name] = ViewRecord( name=name, view=view, plugin=self, citations=citations) if is_format: self.formats[name] = FormatRecord(format=view, plugin=self) def register_validator(self, semantic_expression): if not is_semantic_type(semantic_expression): raise TypeError('%s is not a Semantic Type' % semantic_expression) def decorator(validator): validator_signature = inspect.getfullargspec(validator) if 'data' not in validator_signature.annotations: raise TypeError('No expected view type provided as annotation' ' for `data` variable in %r.' % (validator.__name__)) if not ['data', 'level'] == validator_signature.args: raise TypeError('The function signature: %r does not contain' ' the required arguments and only the required' ' arguments: %r' % ( validator_signature.args, ['data', 'level'])) for semantic_type in semantic_expression: if semantic_type not in self.validators: self.validators[semantic_type] = \ ValidationObject(semantic_type) self.validators[semantic_type].add_validator( ValidatorRecord( validator=validator, view=validator.__annotations__['data'], plugin=self, context=semantic_expression)) return validator return decorator def register_transformer(self, _fn=None, *, citations=None): """ A transformer has the type Callable[[type], type] """ # `_fn` allows us to figure out if we are called with or without # arguments in order to support both: # ``` # @plugin.register_transformer # def _(x: A) -> B: # ... # ``` # and # ``` # @plugin.register_transformer(restrict=True) # def _(x: A) -> B: # ... # ``` if citations is None: citations = () else: citations = tuple(citations) def decorator(transformer): annotations = transformer.__annotations__.copy() if len(annotations) != 2: raise TypeError("A transformer must only have a single input" " and output annotation.") try: output = annotations.pop('return') except KeyError: raise TypeError("A transformer must provide a return type.") if type(output) is tuple: raise TypeError("A transformer can only return a single type," " not %r." % (output,)) input = list(annotations.values())[0] if (input, output) in self.transformers: raise TypeError("Duplicate transformer (%r) from %r to %r." % (transformer, input, output)) if input == output: raise TypeError("Plugins should not register identity" " transformations (%r, %r to %r)." % (transformer, input, output)) self.transformers[input, output] = TransformerRecord( transformer=transformer, plugin=self, citations=citations) return transformer if _fn is None: return decorator else: # Apply the decorator as we were applied with a single function return decorator(_fn) def register_semantic_types(self, *type_fragments): for type_fragment in type_fragments: if not is_semantic_type(type_fragment): raise TypeError("%r is not a semantic type." % type_fragment) if not (isinstance(type_fragment, grammar.IncompleteExp) or (type_fragment.is_concrete() and not type_fragment.fields)): raise ValueError("%r is not a semantic type symbol." % type_fragment) if type_fragment.name in self.type_fragments: raise ValueError("Duplicate semantic type symbol %r." % type_fragment) self.type_fragments[type_fragment.name] = \ SemanticTypeFragmentRecord( fragment=type_fragment, plugin=self) def register_semantic_type_to_format(self, semantic_type, artifact_format): if not issubclass(artifact_format, DirectoryFormat): raise TypeError("%r is not a directory format." % artifact_format) if not is_semantic_type(semantic_type): raise TypeError("%r is not a semantic type." % semantic_type) if not is_semantic_type(semantic_type): raise ValueError("%r is not a semantic type expression." % semantic_type) for t in semantic_type: if t.predicate is not None: raise ValueError("%r has a predicate, differentiating format" " on predicate is not supported.") self.type_formats.append(TypeFormatRecord( type_expression=semantic_type, format=artifact_format, plugin=self)) class PluginActions(dict): def __init__(self, plugin): self._plugin_id = plugin.id super().__init__() class PluginMethods(PluginActions): def register_function(self, function, inputs, parameters, outputs, name, description, input_descriptions=None, parameter_descriptions=None, output_descriptions=None, citations=None, deprecated=False, examples=None): if citations is None: citations = () else: citations = tuple(citations) if examples is None: examples = {} method = qiime2.sdk.Method._init(function, inputs, parameters, outputs, self._plugin_id, name, description, input_descriptions, parameter_descriptions, output_descriptions, citations, deprecated, examples) self[method.id] = method class PluginVisualizers(PluginActions): def register_function(self, function, inputs, parameters, name, description, input_descriptions=None, parameter_descriptions=None, citations=None, deprecated=False, examples=None): if citations is None: citations = () else: citations = tuple(citations) if examples is None: examples = {} visualizer = qiime2.sdk.Visualizer._init(function, inputs, parameters, self._plugin_id, name, description, input_descriptions, parameter_descriptions, citations, deprecated, examples) self[visualizer.id] = visualizer class PluginPipelines(PluginActions): def register_function(self, function, inputs, parameters, outputs, name, description, input_descriptions=None, parameter_descriptions=None, output_descriptions=None, citations=None, deprecated=False, examples=None): if citations is None: citations = () else: citations = tuple(citations) if examples is None: examples = {} pipeline = qiime2.sdk.Pipeline._init(function, inputs, parameters, outputs, self._plugin_id, name, description, input_descriptions, parameter_descriptions, output_descriptions, citations, deprecated, examples) self[pipeline.id] = pipeline qiime-2021.8.0/qiime2/plugin/testing.py000066400000000000000000000213331412141660100175610ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import pkg_resources import tempfile import unittest import shutil import pathlib import qiime2 from qiime2.sdk import usage from qiime2.plugin.model.base import FormatBase # TODO Split out into more specific subclasses if necessary. class TestPluginBase(unittest.TestCase): """Test harness for simplifying testing QIIME 2 plugins. ``TestPluginBase`` extends ``unittest.TestCase``, with a few extra helpers and assertions. Attributes ---------- package : str The name of the plugin package to be tested. test_dir_prefix : str The prefix for the temporary testing dir created by the harness. """ package = None test_dir_prefix = 'qiime2-plugin' def setUp(self): """Test runner setup hook. If overriding this hook in a test, call ``__super__`` to invoke this method in the overridden hook, otherwise the harness might not work as expected. """ try: package = self.package.split('.')[0] except AttributeError: self.fail('Test class must have a package property.') # plugins are keyed by their names, so a search inside the plugin # object is required to match to the correct plugin plugin = None for name, plugin_ in qiime2.sdk.PluginManager().plugins.items(): if plugin_.package == package: plugin = plugin_ if plugin is not None: self.plugin = plugin else: self.fail('%s is not a registered QIIME 2 plugin.' % package) # TODO use qiime2 temp dir when ported to framework, and when the # configurable temp dir exists self.temp_dir = tempfile.TemporaryDirectory( prefix='%s-test-temp-' % self.test_dir_prefix) def tearDown(self): """Test runner teardown hook. If overriding this hook in a test, call ``__super__`` to invoke this method in the overridden hook, otherwise the harness might not work as expected. """ self.temp_dir.cleanup() def get_data_path(self, filename): """Convenience method for getting a data asset while testing. Test data stored in the ``data/`` dir local to the running test can be accessed via this method. Parameters ---------- filename : str The name of the file to look up. Returns ------- filepath : str The materialized filepath to the requested test data. """ return pkg_resources.resource_filename(self.package, 'data/%s' % filename) def get_transformer(self, from_type, to_type): """Convenience method for getting a registered transformer. This helper deliberately side-steps the framework's validation machinery, so that it is possible for plugin developers to test failing conditions. Parameters ---------- from_type : A View Type The :term:`View` type of the source data. to_type : A View Type The :term:`View` type to transform to. Returns ------- transformer : A Transformer Function The registered tranformer from ``from_type`` to ``to_type``. """ try: transformer_record = self.plugin.transformers[from_type, to_type] except KeyError: self.fail( "Could not find registered transformer from %r to %r." % (from_type, to_type)) return transformer_record.transformer def assertRegisteredSemanticType(self, semantic_type): """Test assertion for ensuring a plugin's semantic type is registered. Fails if the semantic type requested is not found in the Plugin Manager. Parameters ---------- semantic_type : A Semantic Type The :term:`Semantic Type` to test the presence of. """ try: record = self.plugin.type_fragments[semantic_type.name] except KeyError: self.fail( "Semantic type %r is not registered on the plugin." % semantic_type) self.assertEqual(record.fragment, semantic_type) def assertSemanticTypeRegisteredToFormat(self, semantic_type, exp_format): """Test assertion for ensuring a semantic type is registered to a format. Fails if the semantic type requested is not registered to the format specified with ``exp_format``. Also fails if the semantic type isn't registered to **any** format. Parameters ---------- semantic_type : A Semantic Type The :term:`Semantic Type` to check for. exp_format : A Format The :term:`Format` to check that the Semantic Type is registed on. """ obs_format = None for type_format_record in self.plugin.type_formats: if type_format_record.type_expression == semantic_type: obs_format = type_format_record.format break self.assertIsNotNone( obs_format, "Semantic type %r is not registered to a format." % semantic_type) self.assertEqual( obs_format, exp_format, "Expected semantic type %r to be registered to format %r, not %r." % (semantic_type, exp_format, obs_format)) def transform_format(self, source_format, target, filename=None, filenames=None): """Helper utility for loading data and transforming it. Combines several other utilities in this class, will load files from ``data/``, as ``source_format``, then transform to the ``target`` view. This helper deliberately side-steps the framework's validation machinery, so that it is possible for plugin developers to test failing conditions. Parameters ---------- source_format : A Format The :term:`Format` to load the data as. target : A View Type The :term:`View Type ` to transform the data to. filename : str The name of the file to load from ``data``. Use this for formats that use a single file in their format definition. Mutually exclusive with the ``filenames`` parameter. filenames : list[str] The names of the files to load from ``data``. Use this for formats that use multiple files in their format definition. Mutually exclusive with the ``filename`` parameter. Returns ------- input : A Format The data loaded from ``data`` as the specified ``source_format``. obs : A View Type The loaded data, transformed to the specified ``target`` view type. """ # Guard any non-QIIME2 Format sources from being tested if not issubclass(source_format, FormatBase): raise ValueError("`source_format` must be a subclass of " "FormatBase.") # Guard against invalid filename(s) usage if filename is not None and filenames is not None: raise ValueError("Cannot use both `filename` and `filenames` at " "the same time.") # Handle format initialization source_path = None if filename: source_path = self.get_data_path(filename) elif filenames: source_path = self.temp_dir.name for filename in filenames: filepath = self.get_data_path(filename) shutil.copy(filepath, source_path) input = source_format(source_path, mode='r') transformer = self.get_transformer(source_format, target) obs = transformer(input) if issubclass(target, FormatBase): self.assertIsInstance(obs, (type(pathlib.Path()), str, target)) else: self.assertIsInstance(obs, target) return input, obs def execute_examples(self): if self.plugin is None: raise ValueError('Attempted to run `execute_examples` without ' 'configuring test harness.') for _, action in self.plugin.actions.items(): for name, example_f in action.examples.items(): with self.subTest(example=name): use = usage.ExecutionUsage() example_f(use) qiime-2021.8.0/qiime2/plugin/tests/000077500000000000000000000000001412141660100166725ustar00rootroot00000000000000qiime-2021.8.0/qiime2/plugin/tests/__init__.py000066400000000000000000000005351412141660100210060ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- qiime-2021.8.0/qiime2/plugin/tests/test_plugin.py000066400000000000000000000171031412141660100216030ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import types import unittest import qiime2.plugin import qiime2.sdk from qiime2.core.testing.type import (IntSequence1, IntSequence2, Mapping, FourInts, Kennel, Dog, Cat, SingleInt) from qiime2.core.testing.util import get_dummy_plugin class TestPlugin(unittest.TestCase): def setUp(self): self.plugin = get_dummy_plugin() def test_name(self): self.assertEqual(self.plugin.name, 'dummy-plugin') def test_version(self): self.assertEqual(self.plugin.version, '0.0.0-dev') def test_website(self): self.assertEqual(self.plugin.website, 'https://github.com/qiime2/qiime2') def test_package(self): self.assertEqual(self.plugin.package, 'qiime2.core.testing') def test_citations(self): self.assertEqual(self.plugin.citations[0].type, 'article') def test_user_support_text(self): self.assertEqual(self.plugin.user_support_text, 'For help, see https://qiime2.org') def test_short_description_text(self): self.assertEqual(self.plugin.short_description, 'Dummy plugin for testing.') def test_description_text(self): self.assertEqual(self.plugin.description, 'Description of dummy plugin.') def test_citations_default(self): plugin = qiime2.plugin.Plugin( name='local-dummy-plugin', version='0.0.0-dev', website='https://github.com/qiime2/qiime2', package='qiime2.core.testing') self.assertEqual(plugin.citations, ()) def test_user_support_text_default(self): plugin = qiime2.plugin.Plugin( name='local-dummy-plugin', version='0.0.0-dev', website='https://github.com/qiime2/qiime2', package='qiime2.core.testing') self.assertTrue(plugin.user_support_text.startswith('Please post')) self.assertTrue(plugin.user_support_text.endswith( 'https://forum.qiime2.org')) def test_actions(self): actions = self.plugin.actions self.assertIsInstance(actions, types.MappingProxyType) self.assertEqual(actions.keys(), {'merge_mappings', 'concatenate_ints', 'split_ints', 'most_common_viz', 'mapping_viz', 'identity_with_metadata', 'identity_with_metadata_column', 'identity_with_categorical_metadata_column', 'identity_with_numeric_metadata_column', 'identity_with_optional_metadata', 'identity_with_optional_metadata_column', 'params_only_method', 'no_input_method', 'optional_artifacts_method', 'variadic_input_method', 'params_only_viz', 'no_input_viz', 'long_description_method', 'parameter_only_pipeline', 'typical_pipeline', 'optional_artifact_pipeline', 'pointless_pipeline', 'visualizer_only_pipeline', 'pipelines_in_pipeline', 'failing_pipeline', 'docstring_order_method', 'constrained_input_visualization', 'combinatorically_mapped_method', 'double_bound_variable_method', 'bool_flag_swaps_output_method', 'predicates_preserved_method', 'deprecated_method', 'unioned_primitives', 'type_match_list_and_set', }) for action in actions.values(): self.assertIsInstance(action, qiime2.sdk.Action) # Read-only dict. with self.assertRaises(TypeError): actions["i-shouldn't-do-this"] = "my-action" with self.assertRaises(TypeError): actions["merge_mappings"] = "my-action" def test_methods(self): methods = self.plugin.methods self.assertEqual(methods.keys(), {'merge_mappings', 'concatenate_ints', 'split_ints', 'identity_with_metadata', 'identity_with_metadata_column', 'identity_with_categorical_metadata_column', 'identity_with_numeric_metadata_column', 'identity_with_optional_metadata', 'identity_with_optional_metadata_column', 'params_only_method', 'no_input_method', 'optional_artifacts_method', 'long_description_method', 'variadic_input_method', 'docstring_order_method', 'combinatorically_mapped_method', 'double_bound_variable_method', 'bool_flag_swaps_output_method', 'predicates_preserved_method', 'deprecated_method', 'unioned_primitives', 'type_match_list_and_set', }) for method in methods.values(): self.assertIsInstance(method, qiime2.sdk.Method) def test_visualizers(self): visualizers = self.plugin.visualizers self.assertEqual(visualizers.keys(), {'most_common_viz', 'mapping_viz', 'params_only_viz', 'no_input_viz', 'constrained_input_visualization'}) for viz in visualizers.values(): self.assertIsInstance(viz, qiime2.sdk.Visualizer) def test_pipelines(self): pipelines = self.plugin.pipelines self.assertEqual(pipelines.keys(), {'parameter_only_pipeline', 'typical_pipeline', 'optional_artifact_pipeline', 'pointless_pipeline', 'visualizer_only_pipeline', 'pipelines_in_pipeline', 'failing_pipeline'}) for pipeline in pipelines.values(): self.assertIsInstance(pipeline, qiime2.sdk.Pipeline) # TODO test registration of directory formats. def test_type_fragments(self): types = self.plugin.type_fragments.keys() self.assertEqual( set(types), set(['IntSequence1', 'IntSequence2', 'IntSequence3', 'Mapping', 'FourInts', 'Kennel', 'Dog', 'Cat', 'SingleInt', 'C1', 'C2', 'C3', 'Foo', 'Bar', 'Baz', 'AscIntSequence', 'Squid', 'Octopus', 'Cuttlefish'])) def test_types(self): types = self.plugin.types # Get just the SemanticTypeRecords out of the types dictionary, then # get just the types out of the SemanticTypeRecord namedtuples types = {type_.semantic_type for type_ in types.values()} exp = {IntSequence1, IntSequence2, FourInts, Mapping, Kennel[Dog], Kennel[Cat], SingleInt} self.assertLessEqual(exp, types) self.assertNotIn(Cat, types) self.assertNotIn(Dog, types) self.assertNotIn(Kennel, types) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/plugin/tests/test_tests.py000066400000000000000000000020671412141660100214520ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import unittest import tempfile from qiime2.core.testing.util import get_dummy_plugin from qiime2.plugin.testing import TestPluginBase class TestTesting(TestPluginBase): def setUp(self): self.plugin = get_dummy_plugin() # TODO standardize temporary directories created by QIIME 2 # create a temporary data_dir for sample Visualizations self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-') self.data_dir = os.path.join(self.test_dir.name, 'viz-output') os.mkdir(self.data_dir) def tearDown(self): self.test_dir.cleanup() def test_examples(self): self.execute_examples() if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/plugin/util.py000066400000000000000000000013261412141660100170610ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from qiime2.core.transform import ModelType def transform(data, *, from_type=None, to_type): from_type = type(data) if from_type is None else from_type from_model_type = ModelType.from_view_type(from_type) to_model_type = ModelType.from_view_type(to_type) transformation = from_model_type.make_transformation(to_model_type) return transformation(data) qiime-2021.8.0/qiime2/plugins.py000066400000000000000000000147211412141660100162720ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import sys import importlib.machinery from qiime2.sdk import usage __all__ = ['available_plugins', 'ArtifactAPIUsage'] __path__ = [] def available_plugins(): import qiime2.sdk pm = qiime2.sdk.PluginManager() return set('qiime2.plugins.' + s.replace('-', '_') for s in pm.plugins) class ArtifactAPIUsage(usage.Usage): def __init__(self): super().__init__() self._imports = set() self._recorder = [] self._init_data_refs = dict() def _init_data_(self, ref, factory): self._init_data_refs[ref] = factory # Don't need to compute anything, so just pass along the ref return ref def _init_metadata_(self, ref, factory): self._init_data_refs[ref] = factory return ref def _init_data_collection_(self, ref, collection_type, records): t = ', '.join(sorted([r.ref for r in records])) t = '[%s]' % t if collection_type is list else '{%s}' % t return t def _merge_metadata_(self, ref, records): first_md = records[0].ref remaining_records = ', '.join([r.ref for r in records[1:]]) t = '%s = %s.merge(%s)\n' % (ref, first_md, remaining_records) self._recorder.append(t) return ref def _get_metadata_column_(self, column_name, record): t = '%s = %s.get_column(%r)\n' % (column_name, record.ref, column_name) self._recorder.append(t) return column_name def _comment_(self, text: str): self._recorder.append('# %s' % (text, )) def _action_(self, action: usage.UsageAction, input_opts: dict, output_opts: dict): action_f, action_sig = action.get_action() self._update_imports(action_f) t = self._template_action(action_f, input_opts, output_opts) self._recorder.append(t) return output_opts def _assert_has_line_matching_(self, ref, label, path, expression): pass def render(self): sorted_imps = sorted(self._imports, key=lambda x: x[0]) imps = ['from %s import %s\n' % i for i in sorted_imps] return '\n'.join(imps + self._recorder) def get_example_data(self): return {r: f() for r, f in self._init_data_refs.items()} def _template_action(self, action_f, input_opts, output_opts): output_opts = list(output_opts.values()) if len(output_opts) == 1: output_opts.append('') output_vars = ', '.join(output_opts) t = '%s = %s(\n' % (output_vars.strip(), action_f.id) for k, v in input_opts.items(): t += ' %s=%s,\n' % (k, v) t += ')\n' return t def _update_imports(self, action_f): full_import = action_f.get_import_path() import_path, action_api_name = full_import.rsplit('.', 1) self._imports.add((import_path, action_api_name)) class QIIMEArtifactAPIImporter: def _plugin_lookup(self, plugin_name): import qiime2.sdk pm = qiime2.sdk.PluginManager() lookup = {s.replace('-', '_'): s for s in pm.plugins} if plugin_name not in lookup: return None return pm.plugins[lookup[plugin_name]] def find_spec(self, name, path=None, target=None): # Don't waste time doing anything if it's not a qiime2 plugin if not name.startswith('qiime2.plugins.'): return None if target is not None: # TODO: experiment with this to see if it is possible raise ImportError("Reloading the QIIME 2 Artifact API is not" " currently supported.") # We couldn't care less about path, it is useless to us # (It is the __path__ of the parent module) fqn = name.split('.') plugin_details = fqn[2:] # fqn[len(['qiime2', 'plugins']):] plugin_name = plugin_details[0] plugin = self._plugin_lookup(plugin_name) if plugin is None or len(plugin_details) > 2: return None if len(plugin_details) == 1: return self._make_spec(name, plugin) elif plugin_details[1] == 'visualizers': return self._make_spec(name, plugin, ('visualizers',)) elif plugin_details[1] == 'methods': return self._make_spec(name, plugin, ('methods',)) elif plugin_details[1] == 'pipelines': return self._make_spec(name, plugin, ('pipelines',)) elif plugin_details[1] == 'actions': return self._make_spec(name, plugin, ('methods', 'visualizers', 'pipelines')) return None def _make_spec(self, name, plugin, action_types=None): # See PEP 451 for explanation of what is happening: # https://www.python.org/dev/peps/pep-0451/#modulespec return importlib.machinery.ModuleSpec( name, loader=self, origin='generated QIIME 2 API', loader_state={'plugin': plugin, 'action_types': action_types}, is_package=action_types is None ) def create_module(self, spec): # Required by Python 3.6, we just need the default behavior return None def exec_module(self, module): spec = module.__spec__ plugin = spec.loader_state['plugin'] action_types = spec.loader_state['action_types'] module.__plugin__ = plugin if action_types is None: module.methods = importlib.import_module('.methods', package=spec.name) module.visualizers = importlib.import_module('.visualizers', package=spec.name) module.pipelines = importlib.import_module('.pipelines', package=spec.name) module.actions = importlib.import_module('.actions', package=spec.name) else: for action_type in action_types: actions = getattr(plugin, action_type) for key, value in actions.items(): setattr(module, key, value) sys.meta_path += [QIIMEArtifactAPIImporter()] qiime-2021.8.0/qiime2/sdk/000077500000000000000000000000001412141660100150135ustar00rootroot00000000000000qiime-2021.8.0/qiime2/sdk/__init__.py000066400000000000000000000017371412141660100171340ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from .context import Context from .action import Action, Method, Visualizer, Pipeline from .plugin_manager import PluginManager from .result import Result, Artifact, Visualization from .results import Results from .util import parse_type, parse_format, type_from_ast from ..core.cite import Citations from ..core.exceptions import ValidationError, ImplementationError __all__ = ['Result', 'Results', 'Artifact', 'Visualization', 'Action', 'Method', 'Visualizer', 'Pipeline', 'PluginManager', 'parse_type', 'parse_format', 'type_from_ast', 'Context', 'Citations', 'ValidationError', 'ImplementationError', ] qiime-2021.8.0/qiime2/sdk/action.py000066400000000000000000000541121412141660100166450ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import abc import concurrent.futures import inspect import tempfile import textwrap import itertools import decorator import dill import qiime2.sdk import qiime2.core.type as qtype import qiime2.core.archive as archive from qiime2.core.util import LateBindingAttribute, DropFirstParameter, tuplize def _subprocess_apply(action, args, kwargs): # Preprocess input artifacts as we've got pickled clones which shouldn't # self-destruct. for arg in itertools.chain(args, kwargs.values()): if isinstance(arg, qiime2.sdk.Artifact): # We can't rely on the subprocess preventing atexit hooks as the # destructor is also called when the artifact goes out of scope # (which happens). arg._destructor.detach() results = action(*args, **kwargs) for r in results: # The destructor doesn't keep its detatched state when sent back to the # main process. Something about the context-manager from ctx seems to # cause a GC of the artifacts before the process actually ends, so we # do need to detach these. The specifics are not understood. r._destructor.detach() return results class Action(metaclass=abc.ABCMeta): """QIIME 2 Action""" type = 'action' _ProvCaptureCls = archive.ActionProvenanceCapture __call__ = LateBindingAttribute('_dynamic_call') asynchronous = LateBindingAttribute('_dynamic_async') # Converts a callable's signature into its wrapper's signature (i.e. # converts the "view API" signature into the "artifact API" signature). # Accepts a callable as input and returns a callable as output with # converted signature. @abc.abstractmethod def _callable_sig_converter_(self, callable): raise NotImplementedError # Executes a callable on the provided `view_args`, wrapping and returning # the callable's outputs. In other words, executes the "view API", wrapping # and returning the outputs as the "artifact API". `view_args` is a dict # mapping parameter name to unwrapped value (i.e. view). `view_args` # contains an entry for each parameter accepted by the wrapper. It is the # executor's responsibility to perform any additional transformations on # these parameters, or provide extra parameters, in order to execute the # callable. `output_types` is an OrderedDict mapping output name to QIIME # type (e.g. semantic type). @abc.abstractmethod def _callable_executor_(self, scope, view_args, output_types): raise NotImplementedError # Private constructor @classmethod def _init(cls, callable, signature, plugin_id, name, description, citations, deprecated, examples): """ Parameters ---------- callable : callable signature : qiime2.core.type.Signature plugin_id : str name : str Human-readable name for this action. description : str Human-readable description for this action. """ self = cls.__new__(cls) self.__init(callable, signature, plugin_id, name, description, citations, deprecated, examples) return self # This "extra private" constructor is necessary because `Action` objects # can be initialized from a static (classmethod) context or on an # existing instance (see `_init` and `__setstate__`, respectively). def __init(self, callable, signature, plugin_id, name, description, citations, deprecated, examples): self._callable = callable self.signature = signature self.plugin_id = plugin_id self.name = name self.description = description self.citations = citations self.deprecated = deprecated self.examples = examples self.id = callable.__name__ self._dynamic_call = self._get_callable_wrapper() self._dynamic_async = self._get_async_wrapper() def __init__(self): raise NotImplementedError( "%s constructor is private." % self.__class__.__name__) @property def source(self): """ The source code for the action's callable. Returns ------- str The source code of this action's callable formatted as Markdown text. """ try: source = inspect.getsource(self._callable) except OSError: raise TypeError( "Cannot retrieve source code for callable %r" % self._callable.__name__) return markdown_source_template % {'source': source} def get_import_path(self, include_self=True): path = f'qiime2.plugins.{self.plugin_id}.{self.type}s' if include_self: path += f'.{self.id}' return path def __repr__(self): return "<%s %s>" % (self.type, self.get_import_path()) def __getstate__(self): return dill.dumps({ 'callable': self._callable, 'signature': self.signature, 'plugin_id': self.plugin_id, 'name': self.name, 'description': self.description, 'citations': self.citations, 'deprecated': self.deprecated, 'examples': self.examples, }) def __setstate__(self, state): self.__init(**dill.loads(state)) def _bind(self, context_factory): """Bind an action to a Context factory, returning a decorated function. This is a very primitive API and should be used primarily by the framework and very advanced interfaces which need deep control over the calling semantics of pipelines and garbage collection. The basic idea behind this is outlined as follows: Every action is defined as an *instance* that a plugin constructs. This means that `self` represents the internal details as to what the action is. If you need to associate additional state with the *application* of an action, you cannot mutate `self` without changing all future applications. So there needs to be an additional instance variable that can serve as the state of a given application. We call this a Context object. It is also important that each application of an action has *independent* state, so providing an instance of Context won't work. We need a factory. Parameterizing the context is necessary because it is possible for an action to call other actions. The details need to be coordinated behind the scenes to the user, so we can parameterize the behavior by providing different context factories to `bind` at different points in the "call stack". """ def bound_callable(*args, **kwargs): # This function's signature is rewritten below using # `decorator.decorator`. When the signature is rewritten, # args[0] is the function whose signature was used to rewrite # this function's signature. args = args[1:] ctx = context_factory() # Set up a scope under which we can track destructable references # if something goes wrong, the __exit__ handler of this context # manager will clean up. (It also cleans up when things go right) with ctx as scope: provenance = self._ProvCaptureCls( self.type, self.plugin_id, self.id) scope.add_reference(provenance) # Collate user arguments user_input = {name: value for value, name in zip(args, self.signature.signature_order)} user_input.update(kwargs) # Type management self.signature.check_types(**user_input) output_types = self.signature.solve_output(**user_input) callable_args = {} # Record parameters for name, spec in self.signature.parameters.items(): parameter = callable_args[name] = user_input[name] provenance.add_parameter(name, spec.qiime_type, parameter) # Record and transform inputs for name, spec in self.signature.inputs.items(): artifact = user_input[name] provenance.add_input(name, artifact) if artifact is None: callable_args[name] = None elif spec.has_view_type(): recorder = provenance.transformation_recorder(name) if qtype.is_collection_type(spec.qiime_type): # Always put in a list. Sometimes the view isn't # hashable, which isn't relevant, but would break # a Set[SomeType]. callable_args[name] = [ a._view(spec.view_type, recorder) for a in user_input[name]] else: callable_args[name] = artifact._view( spec.view_type, recorder) else: callable_args[name] = artifact if self.deprecated: with qiime2.core.util.warning() as warn: warn(self._build_deprecation_message(), FutureWarning) # Execute outputs = self._callable_executor_(scope, callable_args, output_types, provenance) if len(outputs) != len(self.signature.outputs): raise ValueError( "Number of callable outputs must match number of " "outputs defined in signature: %d != %d" % (len(outputs), len(self.signature.outputs))) # Wrap in a Results object mapping output name to value so # users have access to outputs by name or position. return qiime2.sdk.Results(self.signature.outputs.keys(), outputs) bound_callable = self._rewrite_wrapper_signature(bound_callable) self._set_wrapper_properties(bound_callable) self._set_wrapper_name(bound_callable, self.id) return bound_callable def _get_callable_wrapper(self): # This is a "root" level invocation (not a nested call within a # pipeline), so no special factory is needed. callable_wrapper = self._bind(qiime2.sdk.Context) self._set_wrapper_name(callable_wrapper, '__call__') return callable_wrapper def _get_async_wrapper(self): def async_wrapper(*args, **kwargs): # TODO handle this better in the future, but stop the massive error # caused by MacOSX asynchronous runs for now. try: import matplotlib as plt if plt.rcParams['backend'].lower() == 'macosx': raise EnvironmentError(backend_error_template % plt.matplotlib_fname()) except ImportError: pass # This function's signature is rewritten below using # `decorator.decorator`. When the signature is rewritten, args[0] # is the function whose signature was used to rewrite this # function's signature. args = args[1:] pool = concurrent.futures.ProcessPoolExecutor(max_workers=1) future = pool.submit(_subprocess_apply, self, args, kwargs) # TODO: pool.shutdown(wait=False) caused the child process to # hang unrecoverably. This seems to be a bug in Python 3.7 # It's probably best to gut concurrent.futures entirely, so we're # ignoring the resource leakage for the moment. return future async_wrapper = self._rewrite_wrapper_signature(async_wrapper) self._set_wrapper_properties(async_wrapper) self._set_wrapper_name(async_wrapper, 'asynchronous') return async_wrapper def _rewrite_wrapper_signature(self, wrapper): # Convert the callable's signature into the wrapper's signature and set # it on the wrapper. return decorator.decorator( wrapper, self._callable_sig_converter_(self._callable)) def _set_wrapper_name(self, wrapper, name): wrapper.__name__ = wrapper.__qualname__ = name def _set_wrapper_properties(self, wrapper): wrapper.__module__ = self.get_import_path(include_self=False) wrapper.__doc__ = self._build_numpydoc() wrapper.__annotations__ = self._build_annotations() # This is necessary so that `inspect` doesn't display the wrapped # function's annotations (the annotations apply to the "view API" and # not the "artifact API"). del wrapper.__wrapped__ def _build_annotations(self): annotations = {} for name, spec in self.signature.signature_order.items(): annotations[name] = spec.qiime_type output = [] for spec in self.signature.outputs.values(): output.append(spec.qiime_type) output = tuple(output) annotations["return"] = output return annotations def _build_numpydoc(self): numpydoc = [] numpydoc.append(textwrap.fill(self.name, width=75)) if self.deprecated: base_msg = textwrap.indent( textwrap.fill(self._build_deprecation_message(), width=72), ' ') numpydoc.append('.. deprecated::\n' + base_msg) numpydoc.append(textwrap.fill(self.description, width=75)) sig = self.signature parameters = self._build_section("Parameters", sig.signature_order) returns = self._build_section("Returns", sig.outputs) # TODO: include Usage-rendered examples here for section in (parameters, returns): if section: numpydoc.append(section) return '\n\n'.join(numpydoc) + '\n' def _build_section(self, header, iterable): section = [] if iterable: section.append(header) section.append('-'*len(header)) for key, value in iterable.items(): variable_line = ( "{item} : {type}".format(item=key, type=value.qiime_type)) if value.has_default(): variable_line += ", optional" section.append(variable_line) if value.has_description(): section.append(textwrap.indent(textwrap.fill( str(value.description), width=71), ' ')) return '\n'.join(section).strip() def _build_deprecation_message(self): return (f'This {self.type.title()} is deprecated and will be removed ' 'in a future version of this plugin.') class Method(Action): """QIIME 2 Method""" type = 'method' # Abstract method implementations: def _callable_sig_converter_(self, callable): # No conversion necessary. return callable def _callable_executor_(self, scope, view_args, output_types, provenance): output_views = self._callable(**view_args) output_views = tuplize(output_views) # TODO this won't work if the user has annotated their "view API" to # return a `typing.Tuple` with some number of components. Python will # return a tuple when there are multiple return values, and this length # check will fail because the tuple as a whole should be matched up to # a single output type instead of its components. This is an edgecase # due to how Python handles multiple returns, and can be worked around # by using something like `typing.List` instead. if len(output_views) != len(output_types): raise TypeError( "Number of output views must match number of output " "semantic types: %d != %d" % (len(output_views), len(output_types))) output_artifacts = [] for output_view, (name, spec) in zip(output_views, output_types.items()): if type(output_view) is not spec.view_type: raise TypeError( "Expected output view type %r, received %r" % (spec.view_type.__name__, type(output_view).__name__)) prov = provenance.fork(name) scope.add_reference(prov) artifact = qiime2.sdk.Artifact._from_view( spec.qiime_type, output_view, spec.view_type, prov) scope.add_parent_reference(artifact) output_artifacts.append(artifact) return tuple(output_artifacts) @classmethod def _init(cls, callable, inputs, parameters, outputs, plugin_id, name, description, input_descriptions, parameter_descriptions, output_descriptions, citations, deprecated, examples): signature = qtype.MethodSignature(callable, inputs, parameters, outputs, input_descriptions, parameter_descriptions, output_descriptions) return super()._init(callable, signature, plugin_id, name, description, citations, deprecated, examples) class Visualizer(Action): """QIIME 2 Visualizer""" type = 'visualizer' # Abstract method implementations: def _callable_sig_converter_(self, callable): return DropFirstParameter.from_function(callable) def _callable_executor_(self, scope, view_args, output_types, provenance): # TODO use qiime2.plugin.OutPath when it exists, and update visualizers # to work with OutPath instead of str. Visualization._from_data_dir # will also need to be updated to support OutPath instead of str. with tempfile.TemporaryDirectory(prefix='qiime2-temp-') as temp_dir: ret_val = self._callable(output_dir=temp_dir, **view_args) if ret_val is not None: raise TypeError( "Visualizer %r should not return anything. " "Received %r as a return value." % (self, ret_val)) provenance.output_name = 'visualization' viz = qiime2.sdk.Visualization._from_data_dir(temp_dir, provenance) scope.add_parent_reference(viz) return (viz,) @classmethod def _init(cls, callable, inputs, parameters, plugin_id, name, description, input_descriptions, parameter_descriptions, citations, deprecated, examples): signature = qtype.VisualizerSignature(callable, inputs, parameters, input_descriptions, parameter_descriptions) return super()._init(callable, signature, plugin_id, name, description, citations, deprecated, examples) class Pipeline(Action): """QIIME 2 Pipeline""" type = 'pipeline' _ProvCaptureCls = archive.PipelineProvenanceCapture def _callable_sig_converter_(self, callable): return DropFirstParameter.from_function(callable) def _callable_executor_(self, scope, view_args, output_types, provenance): outputs = self._callable(scope.ctx, **view_args) outputs = tuplize(outputs) for output in outputs: if not isinstance(output, qiime2.sdk.Result): raise TypeError("Pipelines must return `Result` objects, " "not %s" % (type(output), )) # This condition *is* tested by the caller of _callable_executor_, but # the kinds of errors a plugin developer see will make more sense if # this check happens before the subtype check. Otherwise forgetting an # output would more likely error as a wrong type, which while correct, # isn't root of the problem. if len(outputs) != len(output_types): raise TypeError( "Number of outputs must match number of output " "semantic types: %d != %d" % (len(outputs), len(output_types))) results = [] for output, (name, spec) in zip(outputs, output_types.items()): if not (output.type <= spec.qiime_type): raise TypeError( "Expected output type %r, received %r" % (spec.qiime_type, output.type)) prov = provenance.fork(name, output) scope.add_reference(prov) aliased_result = output._alias(prov) scope.add_parent_reference(aliased_result) results.append(aliased_result) return tuple(results) @classmethod def _init(cls, callable, inputs, parameters, outputs, plugin_id, name, description, input_descriptions, parameter_descriptions, output_descriptions, citations, deprecated, examples): signature = qtype.PipelineSignature(callable, inputs, parameters, outputs, input_descriptions, parameter_descriptions, output_descriptions) return super()._init(callable, signature, plugin_id, name, description, citations, deprecated, examples) markdown_source_template = """ ```python %(source)s ``` """ # TODO add unit test for callables raising this backend_error_template = """ Your current matplotlib backend (MacOSX) does not work with asynchronous calls. A recommended backend is Agg, and can be changed by modifying your matplotlibrc "backend" parameter, which can be found at: \n\n %s """ qiime-2021.8.0/qiime2/sdk/actiongraph.py000066400000000000000000000156451412141660100176770ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from itertools import product, chain import networkx as nx import copy import qiime2 def get_next_arguments(action, type="input"): """ Get a tuple of required/nonrequired inputs or outputs for each method Parameters ---------- action : Qiime2.action type : {"input", "param", "output"} Delineates if getting the action input, param, or output types Returns ------- List of tuples containing name and required semantic types List of tuples containing name and optional semantic types """ req = [] non_req = [] if type == "input": for k, v in action.signature.inputs.items(): if not v.has_default(): req.append([k, v.qiime_type]) else: non_req.append(["."+k, v.qiime_type]) elif type == "param": for k, v in action.signature.parameters.items(): if not v.has_default(): req.append([k, v.qiime_type]) else: non_req.append(["."+k, v.qiime_type]) else: for k, v in action.signature.outputs.items(): if not v.has_default(): req.append([k, v.qiime_type]) else: non_req.append(["."+k, v.qiime_type]) return req, non_req def unravel(list_): """ Unravel Union node to get all permutations of types for each action Parameters ---------- list : list of Qiime2.types Returns ------- list of lists - list of permuations of types for each action """ result = [list_] for i, x in enumerate(list_): if len(list(x[1])) > 1: members = list(x[1]) temp = copy.deepcopy(result) # update result with first element of types in member for each_list in result: each_list[i][1] = members[0] # add in other permutations of types in member for n in range(1, len(members)): copy_result = copy.deepcopy(temp) for each_list in copy_result: each_list[i][1] = members[n] result += copy_result return result def generate_nodes_by_action(action, opt=False): """ Given a method, generates all combinations of inputs and outputs for that particular method and and stores the combinations as dictionaries in a resulting list. Parameters ---------- method : Qiime2.action opt : {True, False} Delineates if optional types should be included Returns ------- list of dictionaries - each dictionary is a combination inputs and outputs for particular node """ input, input_nr = get_next_arguments(action, "input") param, param_nr = get_next_arguments(action, "param") output, output_nr = get_next_arguments(action, "output") input = unravel(input) param = unravel(param) opt_in_list = [] if opt: opt_in_list += input_nr opt_in_list += param_nr opt_in_list = unravel(opt_in_list) ins = [dict(x) for x in [list(chain.from_iterable(i)) for i in list(product(input, param, opt_in_list))]] outs = dict(output + output_nr) results = [{'inputs': i, 'outputs': outs} for i in ins] return results ins = [dict(x) for x in [list(chain.from_iterable(i)) for i in list(product(input, param))]] outs = dict(output) results = [{'inputs': i, 'outputs': outs} for i in ins] return results def build_graph(action_list=[], opt=False): """ Constructs a networkx graph with different semantic types and actions as nodes Parameters ---------- action_list : list of Qiime2.action If list is empty, will pull from all methods in the Qiime2 plugin opt : {True, False} Delineates if optional types should be included in the graph Returns ------- nx.DiGraph - networkx graph connected based on all or specified methods """ G = nx.DiGraph() G.edges(data=True) # get all actions or specifc actions if specified in sigs pm = qiime2.sdk.PluginManager() if not action_list: for _, pg in pm.plugins.items(): action_list += list(pg.actions.values()) for action in action_list: results = generate_nodes_by_action(action, opt) for dict_ in results: for k, v in dict_.items(): if not v: continue # renaming dictionary to remove '.' action_node = {} for x, y in v.items(): if x[0] == '.': action_node[x[1:]] = y else: action_node[x] = y dict_[k] = action_node if not G.has_node(str(dict_)): G.add_node(str(dict_), value=action, node='action') if k == 'inputs': for in_k, in_v in v.items(): if not in_v: continue if in_k[0] == '.': name = "opt_"+str(in_v) G.add_edge(name, str(dict_)) G[name][str(dict_)]['name'] = in_k[1:] G.nodes[name]['type'] = in_v G.nodes[name]['optional'] = True G.nodes[name]['node'] = 'type' else: G.add_edge(in_v, str(dict_)) G[in_v][str(dict_)]['name'] = in_k G.nodes[in_v]['type'] = in_v G.nodes[in_v]['optional'] = False G.nodes[in_v]['node'] = 'type' else: for out_k, out_v in v.items(): if not out_v: continue if out_k[0] == '.': name = "opt_"+str(out_v) G.add_edge("opt_"+str(out_v), str(dict_)) G[str(dict_)][name]['name'] = out_k[1:] G.nodes[name]['type'] = in_v G.nodes[name]['optional'] = True G.nodes[name]['node'] = 'type' else: G.add_edge(str(dict_), out_v) G[str(dict_)][out_v]['name'] = out_k G.nodes[out_v]['type'] = out_v G.nodes[out_v]['optional'] = False G.nodes[out_v]['node'] = 'type' return G qiime-2021.8.0/qiime2/sdk/context.py000066400000000000000000000106251412141660100170550ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import qiime2.sdk class Context: def __init__(self, parent=None): self._parent = parent self._scope = None def get_action(self, plugin: str, action: str): """Return a function matching the callable API of an action. This function is aware of the pipeline context and manages its own cleanup as appropriate. """ pm = qiime2.sdk.PluginManager() plugin = plugin.replace('_', '-') try: plugin_obj = pm.plugins[plugin] except KeyError: raise ValueError("A plugin named %r could not be found." % plugin) try: action_obj = plugin_obj.actions[action] except KeyError: raise ValueError("An action named %r was not found for plugin %r" % (action, plugin)) # This factory will create new Contexts with this context as their # parent. This allows scope cleanup to happen recursively. # A factory is necessary so that independent applications of the # returned callable recieve their own Context objects. return action_obj._bind(lambda: Context(parent=self)) def make_artifact(self, type, view, view_type=None): """Return a new artifact from a given view. This artifact is automatically tracked and cleaned by the pipeline context. """ artifact = qiime2.sdk.Artifact.import_data(type, view, view_type) # self._scope WILL be defined at this point, as pipelines always enter # a scope before deferring to plugin code. (Otherwise cleanup wouldn't # happen) self._scope.add_reference(artifact) return artifact def __enter__(self): """For internal use only. Creates a scope API that can track references that need to be destroyed. """ if self._scope is not None: # Prevent odd things from happening to lifecycle cleanup raise Exception('Cannot enter a context twice.') self._scope = Scope(self) return self._scope def __exit__(self, exc_type, exc_value, exc_tb): if exc_type is not None: # Something went wrong, teardown everything self._scope.destroy() else: # Everything is fine, just cleanup internal references and pass # ownership off to the parent context. parent_refs = self._scope.destroy(local_references_only=True) if self._parent is not None: for ref in parent_refs: self._parent._scope.add_reference(ref) class Scope: def __init__(self, ctx): self.ctx = ctx self._locals = [] self._parent_locals = [] def add_reference(self, ref): """Add a reference to something destructable that is owned by this scope. """ self._locals.append(ref) def add_parent_reference(self, ref): """Add a reference to something destructable that will be owned by the parent scope. The reason it needs to be tracked is so that on failure, a context can still identify what will (no longer) be returned. """ self._parent_locals.append(ref) def destroy(self, local_references_only=False): """Destroy all references and clear state. Parameters ---------- local_references_only : bool Whether to destroy references that will belong to the parent scope. Returns ------- list The list of references that were not destroyed. """ local_refs = self._locals parent_refs = self._parent_locals # Unset instance state, handy to prevent cycles in GC, and also causes # catastrophic failure if some invariant is violated. del self._locals del self._parent_locals del self.ctx for ref in local_refs: ref._destructor() if local_references_only: return parent_refs for ref in parent_refs: ref._destructor() return [] qiime-2021.8.0/qiime2/sdk/plugin_manager.py000066400000000000000000000321351412141660100203610ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import collections import os import pkg_resources import enum import qiime2.core.type from qiime2.core.format import FormatBase from qiime2.plugin.model import SingleFileDirectoryFormatBase from qiime2.core.validate import ValidationObject from qiime2.sdk.util import parse_type from qiime2.core.type import is_semantic_type class GetFormatFilters(enum.Flag): EXPORTABLE = enum.auto() IMPORTABLE = enum.auto() class PluginManager: entry_point_group = 'qiime2.plugins' __instance = None @classmethod def iter_entry_points(cls): """Yield QIIME 2 plugin entry points. If the QIIMETEST environment variable is set, only the framework testing plugin entry point (`dummy-plugin`) will be yielded. Otherwise, all available plugin entry points (excluding `dummy-plugin`) will be yielded. """ for entry_point in pkg_resources.iter_entry_points( group=cls.entry_point_group): if 'QIIMETEST' in os.environ: if entry_point.name == 'dummy-plugin': yield entry_point else: if entry_point.name != 'dummy-plugin': yield entry_point # This class is a singleton as it is slow to create, represents the # state of a qiime2 installation, and is needed *everywhere* def __new__(cls, add_plugins=True): if cls.__instance is None: self = super().__new__(cls) cls.__instance = self try: self._init(add_plugins=add_plugins) except Exception: cls.__instance = None raise else: if add_plugins is False: raise ValueError( 'PluginManager singleton already exists, cannot change ' 'default value for `add_plugins`.') return cls.__instance def forget_singleton(self): """Allows later instatiation of PluginManager to produce new object This is done by clearing class member which saves the instance. This will NOT invalidate or remove the object this method is called on. """ self.__class__.__instance = None def _init(self, add_plugins): self.plugins = {} self.type_fragments = {} self._plugin_by_id = {} self.semantic_types = {} self.transformers = collections.defaultdict(dict) self._reverse_transformers = collections.defaultdict(dict) self.formats = {} self.views = {} self.type_formats = [] self._ff_to_sfdf = {} self.validators = {} if add_plugins: # These are all dependent loops, each requires the loop above it to # be completed. for entry_point in self.iter_entry_points(): project_name = entry_point.dist.project_name package = entry_point.module_name.split('.')[0] plugin = entry_point.load() self.add_plugin(plugin, package, project_name, consistency_check=False) self._consistency_check() def _consistency_check(self): for semantic_type, validator_obj in self.validators.items(): validator_obj.assert_transformation_available( self.get_directory_format(semantic_type)) def add_plugin(self, plugin, package=None, project_name=None, consistency_check=True): self.plugins[plugin.name] = plugin self._plugin_by_id[plugin.id] = plugin if plugin.package is None: plugin.package = package if plugin.project_name is None: plugin.project_name = project_name # validate _after_ applying arguments if plugin.package is None: raise ValueError( 'No value specified for package - must provide a value for ' '`package` or set `plugin.package`.') if plugin.project_name is None: raise ValueError( 'No value specified for project_name - must proved a value ' 'for `project_name` or set `plugin.project_name`.') self._integrate_plugin(plugin) plugin.freeze() if consistency_check is True: return self._consistency_check() def get_plugin(self, *, id=None, name=None): if id is None and name is None: raise ValueError("No plugin requested.") elif id is not None: try: return self._plugin_by_id[id] except KeyError: raise KeyError('No plugin currently registered ' 'with id: "%s".' % (id,)) else: try: return self.plugins[name] except KeyError: raise KeyError('No plugin currently registered ' 'with name: "%s".' % (name,)) def _integrate_plugin(self, plugin): for type_name, type_record in plugin.type_fragments.items(): if type_name in self.type_fragments: conflicting_type_record = \ self.type_fragments[type_name] raise ValueError("Duplicate semantic type (%r) defined in" " plugins: %r and %r" % (type_name, type_record.plugin.name, conflicting_type_record.plugin.name)) self.type_fragments[type_name] = type_record for (input, output), transformer_record in plugin.transformers.items(): if output in self.transformers[input]: raise ValueError("Transformer from %r to %r already exists." % (input, output)) self.transformers[input][output] = transformer_record self._reverse_transformers[output][input] = transformer_record for name, record in plugin.views.items(): if name in self.views: raise NameError( "Duplicate view registration (%r) defined in plugins: %r" " and %r" % (name, record.plugin.name, self.formats[name].plugin.name) ) self.views[name] = record for name, record in plugin.formats.items(): fmt = record.format if issubclass( fmt, qiime2.plugin.model.SingleFileDirectoryFormatBase): if fmt.file.format in self._ff_to_sfdf.keys(): self._ff_to_sfdf[fmt.file.format].add(fmt) else: self._ff_to_sfdf[fmt.file.format] = {fmt} # TODO: remove this when `sniff` is removed if hasattr(fmt, 'sniff') and hasattr(fmt, '_validate_'): raise RuntimeError( 'Format %r registered in plugin %r defines sniff and' '_validate_ methods - only one is permitted.' % (name, record.plugin.name) ) self.formats[name] = record self.type_formats.extend(plugin.type_formats) for semantic_type, validation_object in plugin.validators.items(): if semantic_type not in self.validators: self.validators[semantic_type] = \ ValidationObject(semantic_type) self.validators[semantic_type].add_validation_object( validation_object) def get_semantic_types(self): types = {} for plugin in self.plugins.values(): for type_record in plugin.types.values(): types[str(type_record.semantic_type)] = type_record return types # TODO: Should plugin loading be transactional? i.e. if there's # something wrong, the entire plugin fails to load any piece, like a # databases rollback/commit def get_formats(self, *, filter=None, semantic_type=None): """ get_formats(self, *, filter=None, semantic_type=None) filter : enum filter is an enum integer that will be used to determine user input to output specified formats semantic_type : TypeExpression | String The semantic type is used to filter the formats associated with that specific semantic type This method will filter out the formats using the filter provided by the user and the semantic type. The return is a dictionary of filtered formats keyed on their string names. """ if filter is not None and not isinstance(filter, GetFormatFilters): raise ValueError("The format filter provided: %s is not " "valid.", (filter)) if semantic_type is None: formats = set(f.format for f in self.type_formats) else: formats = set() if isinstance(semantic_type, str): semantic_type = parse_type(semantic_type, "semantic") if is_semantic_type(semantic_type): for type_format in self.type_formats: if semantic_type <= type_format.type_expression: formats.add(type_format.format) break if not formats: raise ValueError("No formats associated with the type " f"{semantic_type}.") else: raise ValueError(f"{semantic_type} is not a valid semantic " "type.") transformable_formats = set(formats) if filter is None or GetFormatFilters.IMPORTABLE in filter: transformable_formats.update( self._get_formats_helper(formats, self._reverse_transformers)) if filter is None or GetFormatFilters.EXPORTABLE in filter: transformable_formats.update( self._get_formats_helper(formats, self.transformers)) result_formats = {} for format_ in transformable_formats: format_ = format_.__name__ result_formats[format_] = self.formats[format_] return result_formats def _get_formats_helper(self, formats, transformer_dict): """ _get_formats_helper(self, formats, transformer_dict) formats : Set[DirectoryFormat] We are finding all formats that are one transformer away from formats in this set tranformer_dict : Dict[ str, Dict[str, TransformerReord]] The dictionary of transformers allows the method to get formats that are transformable from the given format This method creates a set utilizing the transformers dictionary and the formats set to get related formats for a specific format. """ query_set = set(formats) for format_ in formats: if issubclass(format_, SingleFileDirectoryFormatBase): if format_.file.format.__name__ in self.formats: query_set.add(format_.file.format) result_formats = set(query_set) for format_ in query_set: for transformed_format in transformer_dict[format_]: if issubclass(transformed_format, FormatBase): result_formats.add(transformed_format) if issubclass(transformed_format, SingleFileDirectoryFormatBase): result_formats.add(transformed_format.file.format) if transformed_format in self._ff_to_sfdf: result_formats.update( self._ff_to_sfdf[transformed_format]) return result_formats @property def importable_formats(self): """Return formats that are importable. A format is importable in a QIIME 2 deployment if it can be transformed into at least one of the canonical semantic type formats. """ return self.get_formats(filter=GetFormatFilters.IMPORTABLE) @property def importable_types(self): """Return set of concrete semantic types that are importable. A concrete semantic type is importable if it has an associated directory format. """ return self.get_semantic_types() def get_directory_format(self, semantic_type): if not qiime2.core.type.is_semantic_type(semantic_type): raise TypeError( "Must provide a semantic type via `semantic_type`, not %r" % semantic_type) dir_fmt = None for type_format_record in self.type_formats: if semantic_type <= type_format_record.type_expression: dir_fmt = type_format_record.format break if dir_fmt is None: raise TypeError( "Semantic type %r does not have a compatible directory format." % semantic_type) return dir_fmt qiime-2021.8.0/qiime2/sdk/result.py000066400000000000000000000331571412141660100167140ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import shutil import collections import distutils.dir_util import pathlib import qiime2.metadata import qiime2.plugin import qiime2.sdk import qiime2.core.type import qiime2.core.transform as transform import qiime2.core.archive as archive import qiime2.plugin.model as model import qiime2.core.util as util import qiime2.core.exceptions as exceptions # Note: Result, Artifact, and Visualization classes are in this file to avoid # circular dependencies between Result and its subclasses. Result is tightly # coupled to Artifact and Visualization because it is a base class and a # factory, so having the classes in the same file helps make this coupling # explicit. ResultMetadata = collections.namedtuple('ResultMetadata', ['uuid', 'type', 'format']) class Result: """Base class for QIIME 2 result classes (Artifact and Visualization). This class is not intended to be instantiated. Instead, it acts as a public factory and namespace for interacting with Artifacts and Visualizations in a generic way. It also acts as a base class for code reuse and provides an API shared by Artifact and Visualization. """ # Subclasses must override to provide a file extension. extension = None @classmethod def _is_valid_type(cls, type_): """Subclasses should override this method.""" return True @classmethod def peek(cls, filepath): return ResultMetadata(*archive.Archiver.peek(filepath)) @classmethod def extract(cls, filepath, output_dir): """Unzip contents of Artifacts and Visualizations.""" return archive.Archiver.extract(filepath, output_dir) @classmethod def load(cls, filepath): """Factory for loading Artifacts and Visualizations.""" archiver = archive.Archiver.load(filepath) if Artifact._is_valid_type(archiver.type): result = Artifact.__new__(Artifact) elif Visualization._is_valid_type(archiver.type): result = Visualization.__new__(Visualization) else: raise TypeError( "Cannot load filepath %r into an Artifact or Visualization " "because type %r is not supported." % (filepath, archiver.type)) if type(result) is not cls and cls is not Result: raise TypeError( "Attempting to load %s with `%s.load`. Use `%s.load` instead." % (type(result).__name__, cls.__name__, type(result).__name__)) result._archiver = archiver return result @property def type(self): return self._archiver.type @property def uuid(self): return self._archiver.uuid @property def format(self): return self._archiver.format @property def citations(self): return self._archiver.citations def __init__(self): raise NotImplementedError( "%(classname)s constructor is private, use `%(classname)s.load`, " "`%(classname)s.peek`, or `%(classname)s.extract`." % {'classname': self.__class__.__name__}) def __new__(cls): result = object.__new__(cls) result._archiver = None return result def __repr__(self): return ("<%s: %r uuid: %s>" % (self.__class__.__name__.lower(), self.type, self.uuid)) def __hash__(self): return hash(self.uuid) def __eq__(self, other): # Checking the UUID is mostly sufficient but requiring an exact type # match makes it safer in case `other` is a subclass or a completely # different type that happens to have a `.uuid` property. We want to # ensure (as best as we can) that the UUIDs we are comparing are linked # to the same type of QIIME 2 object. return ( type(self) == type(other) and self.uuid == other.uuid ) def __ne__(self, other): return not (self == other) def export_data(self, output_dir): distutils.dir_util.copy_tree( str(self._archiver.data_dir), str(output_dir)) # Return None for now, although future implementations that include # format tranformations may return the invoked transformers return None @property def _destructor(self): return self._archiver._destructor def save(self, filepath, ext=None): """Save to a file. Parameters ---------- filepath : str Path to save file at. extension : str Preferred file extension (.qza, .qzv, .txt, etc). If no preferred extension input is included, Artifact extension will default to .qza and Visualization extension will default to .qzv. Including a period in the extension is optional, and any additional periods delimiting the filepath and the extension will be reduced to a single period. Returns ------- str Filepath and extension (if provided) that the file was saved to. See Also -------- load """ if ext is None: ext = self.extension # This accounts for edge cases in the filename extension # and ensures that there is only a single period in the ext. filepath = filepath.rstrip('.') ext = '.' + ext.lstrip('.') if not filepath.endswith(ext): filepath += ext self._archiver.save(filepath) return filepath def _alias(self, provenance_capture): def clone_original(into): # directory is empty, this function is meant to fix that, so we # can rmdir so that copytree is happy into.rmdir() shutil.copytree(str(self._archiver.data_dir), str(into), copy_function=os.link) # Use hardlinks cls = type(self) alias = cls.__new__(cls) alias._archiver = archive.Archiver.from_data( self.type, self.format, clone_original, provenance_capture) return alias def validate(self, level=NotImplemented): diff = self._archiver.validate_checksums() if diff.changed or diff.added or diff.removed: error = "" if diff.added: error += "Unrecognized files:\n" for key in diff.added: error += " - %r\n" % key if diff.removed: error += "Missing files:\n" for key in diff.removed: error += " - %r\n" % key if diff.changed: error += "Changed files:\n" for (key, (exp, obs)) in diff.changed.items(): error += " - %r: %s -> %s\n" % (key, exp, obs) raise exceptions.ValidationError(error) class Artifact(Result): extension = '.qza' @classmethod def _is_valid_type(cls, type_): if qiime2.core.type.is_semantic_type(type_) and type_.is_concrete(): return True else: return False @classmethod def import_data(cls, type, view, view_type=None): type_, type = type, __builtins__['type'] is_format = False if isinstance(type_, str): type_ = qiime2.sdk.parse_type(type_) if isinstance(view_type, str): view_type = qiime2.sdk.parse_format(view_type) is_format = True if view_type is None: if type(view) is str or isinstance(view, pathlib.PurePath): is_format = True pm = qiime2.sdk.PluginManager() output_dir_fmt = pm.get_directory_format(type_) if pathlib.Path(view).is_file(): if not issubclass(output_dir_fmt, model.SingleFileDirectoryFormatBase): raise qiime2.plugin.ValidationError( "Importing %r requires a directory, not %s" % (output_dir_fmt.__name__, view)) view_type = output_dir_fmt.file.format else: view_type = output_dir_fmt else: view_type = type(view) format_ = None md5sums = None if is_format: path = pathlib.Path(view) if path.is_file(): md5sums = {path.name: util.md5sum(path)} elif path.is_dir(): md5sums = util.md5sum_directory(path) else: raise qiime2.plugin.ValidationError( "Path '%s' does not exist." % path) format_ = view_type provenance_capture = archive.ImportProvenanceCapture(format_, md5sums) return cls._from_view(type_, view, view_type, provenance_capture, validate_level='max') @classmethod def _from_view(cls, type, view, view_type, provenance_capture, validate_level='min'): type_raw = type if isinstance(type, str): type = qiime2.sdk.parse_type(type) if not cls._is_valid_type(type): raise TypeError( "An artifact requires a concrete semantic type, not type %r." % type) pm = qiime2.sdk.PluginManager() output_dir_fmt = pm.get_directory_format(type) if view_type is None: # lookup default format for the type view_type = output_dir_fmt from_type = transform.ModelType.from_view_type(view_type) to_type = transform.ModelType.from_view_type(output_dir_fmt) recorder = provenance_capture.transformation_recorder('return') transformation = from_type.make_transformation(to_type, recorder=recorder) result = transformation(view, validate_level) if type_raw in pm.validators: validation_object = pm.validators[type] validation_object(data=result, level=validate_level) artifact = cls.__new__(cls) artifact._archiver = archive.Archiver.from_data( type, output_dir_fmt, data_initializer=result.path._move_or_copy, provenance_capture=provenance_capture) return artifact def view(self, view_type): return self._view(view_type) def _view(self, view_type, recorder=None): if view_type is qiime2.Metadata and not self.has_metadata(): raise TypeError( "Artifact %r cannot be viewed as QIIME 2 Metadata." % self) from_type = transform.ModelType.from_view_type(self.format) to_type = transform.ModelType.from_view_type(view_type) transformation = from_type.make_transformation(to_type, recorder=recorder) result = transformation(self._archiver.data_dir) if view_type is qiime2.Metadata: result._add_artifacts([self]) to_type.set_user_owned(result, True) return result def has_metadata(self): """ Checks for metadata within an artifact Returns ------- bool True if the artifact has metadata (i.e. can be viewed as ``qiime2.Metadata``), False otherwise. """ from_type = transform.ModelType.from_view_type(self.format) to_type = transform.ModelType.from_view_type(qiime2.Metadata) return from_type.has_transformation(to_type) def validate(self, level='max'): """ Validates the data contents of an artifact Raises ------ ValidationError If the artifact is invalid at the specified level of validation. """ super().validate() self.format.validate(self.view(self.format), level) class Visualization(Result): extension = '.qzv' @classmethod def _is_valid_type(cls, type_): return type_ == qiime2.core.type.Visualization @classmethod def _from_data_dir(cls, data_dir, provenance_capture): # shutil.copytree doesn't allow the destination directory to exist. def data_initializer(destination): return distutils.dir_util.copy_tree( str(data_dir), str(destination)) viz = cls.__new__(cls) viz._archiver = archive.Archiver.from_data( qiime2.core.type.Visualization, None, data_initializer=data_initializer, provenance_capture=provenance_capture) return viz def get_index_paths(self, relative=True): result = {} for abspath in self._archiver.data_dir.iterdir(): data_path = str(abspath.relative_to(self._archiver.data_dir)) if data_path.startswith('index.'): relpath = abspath.relative_to(self._archiver.root_dir) ext = relpath.suffix[1:] if ext in result: raise ValueError( "Multiple index files identified with %s " "extension (%s, %s). This is currently " "unsupported." % (ext, result[ext], relpath)) else: result[ext] = str(relpath) if relative else str(abspath) return result def _repr_html_(self): from qiime2.jupyter import make_html return make_html(str(self._archiver.path)) qiime-2021.8.0/qiime2/sdk/results.py000066400000000000000000000104571412141660100170750ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- # This class provides an interface similar to a `namedtuple` type. We can't use # `namedtuple` directly because each `Action` will return a `Results` object # with `Action`-specific fields (`Action.signature` determines the fields). # Dynamically-defined namedtuple types aren't pickleable, which is necessary # for `asynchronous`. They aren't pickleable because the namedtuple type must # be accessible as a module global, but this global type would be redefined # each time an `Action` is instantiated. class Results(tuple): """Tuple class representing the named results of an ``Action``. Provides an interface similar to a ``namedtuple`` type (e.g. fields are accessible as attributes). Users should not need to instantiate this class directly. """ # Subclassing `tuple` requires `__new__` override. def __new__(cls, fields, values): fields = tuple(fields) values = tuple(values) if len(fields) != len(values): raise ValueError( "`fields` and `values` must have matching length: %d != %d" % (len(fields), len(values))) # Create tuple instance, store fields, and create read-only attributes # for each field name. Fields must be stored for pickling/copying (see # `__getnewargs__`). # # Note: setting field names as attributes allows for tab-completion in # interactive contexts! Using `__getattr__` does not support this. self = super().__new__(cls, values) # Must set attributes this way because `__setattr__` prevents # setting directly (necessary for immutability). object.__setattr__(self, '_fields', fields) # Attach field names as instance attributes. for field, value in zip(fields, values): object.__setattr__(self, field, value) return self def __getnewargs__(self): """Arguments to pass to `__new__`. Used by copy and pickle.""" # `tuple(self)` returns `values`. return self._fields, tuple(self) # `__setattr__` and `__delattr__` must be defined to prevent users from # creating or deleting attributes after this class has been instantiated. # `tuple` and `namedtuple` do not have this problem because they are # immutable (`__slots__ = ()`). We cannot make this class immutable because # we cannot define nonempty `__slots__` when subclassing `tuple`, and we # need the `_fields` attribute. We work around this issue by disallowing # setting and deleting attributes. The error messages here match those # raised by `namedtuple` in Python 3.5.1. def __setattr__(self, name, value): raise AttributeError("can't set attribute") def __delattr__(self, name): raise AttributeError("can't delete attribute") def __eq__(self, other): # Results with different field names should not compare equal, even if # their values are equal. return ( isinstance(other, Results) and self._fields == other._fields and tuple(self) == tuple(other) ) def __ne__(self, other): return not (self == other) def __repr__(self): # It is possible to provide an evalable repr but this type of repr does # not make the field/value pairs apparent. If the constructor accepted # **kwargs, the order of field/value pairs would be lost. lines = [] lines.append('%s (name = value)' % self.__class__.__name__) lines.append('') max_len = -1 for field in self._fields: if len(field) > max_len: max_len = len(field) for field, value in zip(self._fields, self): field_padding = ' ' * (max_len - len(field)) lines.append('%s%s = %r' % (field, field_padding, value)) max_len = -1 for line in lines: if len(line) > max_len: max_len = len(line) lines[1] = '-' * max_len return '\n'.join(lines) qiime-2021.8.0/qiime2/sdk/tests/000077500000000000000000000000001412141660100161555ustar00rootroot00000000000000qiime-2021.8.0/qiime2/sdk/tests/__init__.py000066400000000000000000000005351412141660100202710ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- qiime-2021.8.0/qiime2/sdk/tests/data/000077500000000000000000000000001412141660100170665ustar00rootroot00000000000000qiime-2021.8.0/qiime2/sdk/tests/data/intsequence-fail-max-validation.txt000066400000000000000000000000141412141660100257710ustar00rootroot000000000000001 2 3 4 5 8 qiime-2021.8.0/qiime2/sdk/tests/test_action.py000066400000000000000000000166251412141660100210550ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import collections import tempfile import unittest import warnings import qiime2.core.archive as archive from qiime2.core.testing.util import get_dummy_plugin from qiime2.plugin.testing import TestPluginBase from qiime2.sdk import Artifact, Visualization from qiime2.core.testing.type import IntSequence1, IntSequence2, SingleInt from qiime2.core.testing.visualizer import most_common_viz from qiime2 import Metadata from qiime2.metadata.tests.test_io import get_data_path # NOTE: This test suite exists for tests not easily split into # test_method, test_visualizer, test_pipeline # TestBadInputs tests type mismatches between Action signatures and passed args class TestBadInputs(TestPluginBase): def make_provenance_capture(self): # importing visualizations is not supported, but we do that here to # simplify testing machinery return archive.ImportProvenanceCapture() def setUp(self): self.plugin = get_dummy_plugin() # TODO standardize temporary directories created by QIIME 2 # create a temporary data_dir for sample Visualizations self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-') self.data_dir = os.path.join(self.test_dir.name, 'viz-output') os.mkdir(self.data_dir) most_common_viz(self.data_dir, collections.Counter(range(42))) def tearDown(self): self.test_dir.cleanup() def test_viz_passed_as_input(self): saved_viz = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) method = self.plugin.methods['optional_artifacts_method'] ints1 = Artifact.import_data(IntSequence1, [0, 42, 43]) # tests Viz passed as primitive parameter with self.assertRaisesRegex( TypeError, 'Visualizations may not be used as inputs.'): method(saved_viz, 42) # tests Viz passed as Artifact input with self.assertRaisesRegex( TypeError, 'Visualizations may not be used as inputs.'): method(ints1, 42, optional1=saved_viz) # tests Viz passed as metadata method = self.plugin.methods['identity_with_optional_metadata'] with self.assertRaisesRegex( TypeError, 'Visualizations may not be used as inputs.'): method(ints1, metadata=saved_viz) def test_artifact_passed_incorrectly(self): concatenate_ints = self.plugin.methods['concatenate_ints'] identity_with_metadata = self.plugin.methods['identity_with_metadata'] ints1 = Artifact.import_data(IntSequence1, [0, 42, 43]) ints2 = Artifact.import_data(IntSequence1, [99, -22]) ints3 = Artifact.import_data(IntSequence2, [12, 111]) inappropriate_Artifact = Artifact.import_data(IntSequence1, [-9999999]) int1 = 4 int2 = 5 # tests Artifact passed as integer with self.assertRaisesRegex( TypeError, 'int1.*type Int.*IntSequence1'): concatenate_ints(ints1, ints2, ints3, inappropriate_Artifact, int2) # tests Artifact passed as metadata with self.assertRaisesRegex( TypeError, '\'metadata\'.*type Metadata.*IntSequence1'): identity_with_metadata(ints1, inappropriate_Artifact) # tests wrong type of Artifact passed with self.assertRaisesRegex( TypeError, 'ints3.*IntSequence2.*IntSequence1'): concatenate_ints(ints1, ints2, inappropriate_Artifact, int1, int2) def test_primitive_passed_incorrectly(self): concatenate_ints = self.plugin.methods['concatenate_ints'] identity_with_metadata = self.plugin.methods['identity_with_metadata'] params_only_method = self.plugin.methods['params_only_method'] md_fp = get_data_path('valid/simple.tsv') inappropriate_metadata = Metadata.load(md_fp) ints1 = Artifact.import_data(IntSequence1, [0, 42, 43]) ints3 = Artifact.import_data(IntSequence1, [12, 111]) int1 = 4 int2 = 5 arbitrary_int = 43 # tests primitive int passed as IntSequence artifact with self.assertRaisesRegex(TypeError, 'ints2.*43.*incompatible.*IntSequence1'): concatenate_ints(ints1, arbitrary_int, ints3, int1, int2) # tests primitive passed as metadata with self.assertRaisesRegex(TypeError, 'metadata.*43.*incompatible.*Metadata'): identity_with_metadata(ints1, arbitrary_int) # tests wrong type of primitive passed with self.assertRaisesRegex(TypeError, 'age.*arbitraryString.*incompatible.*Int'): params_only_method('key string', 'arbitraryString') # tests metadata passed as artifact with self.assertRaisesRegex(TypeError, '\'ints2\'.*Metadata.*IntSequence1'): concatenate_ints(ints1, inappropriate_metadata, ints3, int1, int2) def test_primitive_param_out_of_range(self): range_nested_in_list = self.plugin.methods['variadic_input_method'] range_not_nested_in_list = self.plugin.visualizers['params_only_viz'] ints_list = [Artifact.import_data(IntSequence1, [0, 42, 43]), Artifact.import_data(IntSequence2, [4, 5, 6])] int_set = {Artifact.import_data(SingleInt, 7), Artifact.import_data(SingleInt, 8)} nums = {9, 10} bad_range_val = [11, 12, -9999] invalid_age = -99999 # Tests primitives of correct type but outside of Range... # ... in a list with self.assertRaisesRegex( TypeError, 'opt_nums.*-9999.*incompatible.*List'): range_nested_in_list(ints_list, int_set, nums, bad_range_val) # ... not in a list with self.assertRaisesRegex( TypeError, r'\'age\'.*-99999.*incompatible.*Int % Range\(0, None\)'): range_not_nested_in_list('John Doe', invalid_age) def test_primitive_param_not_valid_choice(self): pipeline = self.plugin.pipelines['failing_pipeline'] int_sequence = Artifact.import_data(IntSequence1, [0, 42, 43]) break_from = "invalid choice" # test String not a valid choice with self.assertRaisesRegex( TypeError, 'break_from.*\'invalid choice\''): pipeline(int_sequence, break_from) class TestDeprecation(unittest.TestCase): def setUp(self): self.plugin = get_dummy_plugin() self.method = self.plugin.methods['deprecated_method'] def test_successful_registration(self): self.assertTrue(self.method.deprecated) def test_deprecation_warning(self): with warnings.catch_warnings(record=True) as w: self.method() self.assertEqual(1, len(w)) warning = w[0] self.assertEqual(warning.category, FutureWarning) self.assertTrue('Method is deprecated' in str(warning.message)) def test_docstring(self): self.assertIn('Method is deprecated', self.method.__call__.__doc__) qiime-2021.8.0/qiime2/sdk/tests/test_actiongraph.py000066400000000000000000000123651412141660100220740ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest from qiime2.core.testing.type import (Mapping, IntSequence1, IntSequence2) from qiime2.core.type.primitive import (Int, Str, Metadata) from qiime2.core.type.visualization import (Visualization) from qiime2.core.testing.util import get_dummy_plugin from qiime2.sdk.actiongraph import build_graph class TestActiongraph(unittest.TestCase): def setUp(self): self.plugin = get_dummy_plugin() self.g = None def test_simple_graph(self): methods = [self.plugin.actions['no_input_method']] self.g = build_graph(methods) obs = list(self.g.nodes) exp_node = str({ 'inputs': {}, 'outputs': { 'out': Mapping }, }) type_node = Mapping exp = [type_node, exp_node] for item in obs: assert item in exp assert self.g.has_edge(str(exp_node), type_node) def test_cycle_in_graph_no_params(self): methods = [self.plugin.actions['docstring_order_method']] self.g = build_graph(methods) obs = list(self.g.nodes) exp = [Mapping, Str] exp_node = str({ 'inputs': { 'req_input': Mapping, 'req_param': Str, }, 'outputs': { 'out': Mapping }, }) exp += [exp_node] for item in obs: assert item in exp assert self.g.in_degree(exp_node) == 2 assert self.g.out_degree(exp_node) == 1 def test_cycle_in_graph_with_params(self): methods = [self.plugin.actions['docstring_order_method']] self.g = build_graph(methods, True) obs = list(self.g.nodes) exp = [Mapping, Int, Str, 'opt_Mapping', 'opt_Int'] exp_node = str({ 'inputs': { 'req_input': Mapping, 'req_param': Str, 'opt_input': Mapping, 'opt_param': Int }, 'outputs': { 'out': Mapping }, }) exp += [exp_node] for item in obs: assert item in exp assert self.g.in_degree(exp_node) == 4 assert self.g.out_degree(exp_node) == 1 def test_union(self): vis = [self.plugin.actions['most_common_viz']] self.g = build_graph(vis) obs = list(self.g.nodes) exp = [Visualization, IntSequence1, IntSequence2] exp_node_1 = str({ 'inputs': { 'ints': IntSequence1, }, 'outputs': { 'visualization': Visualization }, }) exp_node_2 = str({ 'inputs': { 'ints': IntSequence2, }, 'outputs': { 'visualization': Visualization }, }) exp += [exp_node_1, exp_node_2] for item in obs: assert item in exp assert self.g.in_degree(exp_node_1) == 1 assert self.g.out_degree(exp_node_1) == 1 assert self.g.in_degree(exp_node_2) == 1 assert self.g.out_degree(exp_node_2) == 1 assert self.g.in_degree(Visualization) == 2 assert self.g.out_degree(Visualization) == 0 def test_multiple_outputs(self): actions = [self.plugin.actions['visualizer_only_pipeline']] self.g = build_graph(actions) obs = list(self.g.nodes) exp = [Visualization, Mapping] exp_node = str({ 'inputs': { 'mapping': Mapping }, 'outputs': { 'viz1': Visualization, 'viz2': Visualization }, }) exp += [exp_node] for item in obs: assert item in exp assert self.g.in_degree(exp_node) == 1 assert self.g.out_degree(exp_node) == 1 def test_metadata(self): actions = [self.plugin.actions['identity_with_metadata']] self.g = build_graph(actions) obs = list(self.g.nodes) exp = [Metadata, IntSequence1, IntSequence2] exp_node_1 = str({ 'inputs': { 'ints': IntSequence1, 'metadata': Metadata }, 'outputs': { 'out': IntSequence1 }, }) exp_node_2 = str({ 'inputs': { 'ints': IntSequence2, 'metadata': Metadata }, 'outputs': { 'out': IntSequence1 }, }) exp += [exp_node_1, exp_node_2] for item in obs: assert item in exp assert self.g.in_degree(exp_node_1) == 2 assert self.g.out_degree(exp_node_1) == 1 assert self.g.in_degree(exp_node_1) == 2 assert self.g.out_degree(exp_node_1) == 1 assert self.g.in_degree(IntSequence1) == 2 assert self.g.out_degree(IntSequence1) == 1 if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/sdk/tests/test_artifact.py000066400000000000000000000533261412141660100213740ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import collections import os import tempfile import unittest import uuid import pathlib import pkg_resources import pandas as pd import qiime2.plugin import qiime2.core.type from qiime2 import Metadata from qiime2.sdk import Artifact from qiime2.sdk.result import ResultMetadata from qiime2.plugin.model import ValidationError import qiime2.core.archive as archive from qiime2.core.testing.format import IntSequenceFormat from qiime2.core.testing.type import IntSequence1, FourInts, Mapping, SingleInt from qiime2.core.testing.util import get_dummy_plugin, ArchiveTestingMixin def get_data_path(filename): return pkg_resources.resource_filename('qiime2.sdk.tests', 'data/%s' % filename) class TestArtifact(unittest.TestCase, ArchiveTestingMixin): def setUp(self): # Ignore the returned dummy plugin object, just run this to verify the # plugin exists as the tests rely on it being loaded. get_dummy_plugin() # TODO standardize temporary directories created by QIIME 2 self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-') self.provenance_capture = archive.ImportProvenanceCapture() def tearDown(self): self.test_dir.cleanup() def test_private_constructor(self): with self.assertRaisesRegex( NotImplementedError, 'Artifact constructor.*private.*Artifact.load'): Artifact() # Note on testing strategy below: many of the tests for `_from_view` and # `load` are similar, with the exception that when `load`ing, the # artifact's UUID is known so more specific assertions can be performed. # While these tests appear somewhat redundant, they are important because # they exercise the same operations on Artifact objects constructed from # different sources, whose codepaths have very different internal behavior. # This internal behavior could be tested explicitly but it is safer to test # the public API behavior (e.g. as a user would interact with the object) # in case the internals change. def test_from_view(self): artifact = Artifact._from_view(FourInts, [-1, 42, 0, 43], list, self.provenance_capture) self.assertEqual(artifact.type, FourInts) # We don't know what the UUID is because it's generated within # Artifact._from_view. self.assertIsInstance(artifact.uuid, uuid.UUID) self.assertEqual(artifact.view(list), [-1, 42, 0, 43]) # Can produce same view if called again. self.assertEqual(artifact.view(list), [-1, 42, 0, 43]) def test_from_view_different_type_with_multiple_view_types(self): artifact = Artifact._from_view(IntSequence1, [42, 42, 43, -999, 42], list, self.provenance_capture) self.assertEqual(artifact.type, IntSequence1) self.assertIsInstance(artifact.uuid, uuid.UUID) self.assertEqual(artifact.view(list), [42, 42, 43, -999, 42]) self.assertEqual(artifact.view(list), [42, 42, 43, -999, 42]) self.assertEqual(artifact.view(collections.Counter), collections.Counter({42: 3, 43: 1, -999: 1})) self.assertEqual(artifact.view(collections.Counter), collections.Counter({42: 3, 43: 1, -999: 1})) def test_from_view_and_save(self): fp = os.path.join(self.test_dir.name, 'artifact.qza') # Using four-ints data layout because it has multiple files, some of # which are in a nested directory. artifact = Artifact._from_view(FourInts, [-1, 42, 0, 43], list, self.provenance_capture) artifact.save(fp) root_dir = str(artifact.uuid) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/file1.txt', 'data/file2.txt', 'data/nested/file3.txt', 'data/nested/file4.txt', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertArchiveMembers(fp, root_dir, expected) def test_load(self): saved_artifact = Artifact.import_data(FourInts, [-1, 42, 0, 43]) fp = os.path.join(self.test_dir.name, 'artifact.qza') saved_artifact.save(fp) artifact = Artifact.load(fp) self.assertEqual(artifact.type, FourInts) self.assertEqual(artifact.uuid, saved_artifact.uuid) self.assertEqual(artifact.view(list), [-1, 42, 0, 43]) self.assertEqual(artifact.view(list), [-1, 42, 0, 43]) def test_load_different_type_with_multiple_view_types(self): saved_artifact = Artifact.import_data(IntSequence1, [42, 42, 43, -999, 42]) fp = os.path.join(self.test_dir.name, 'artifact.qza') saved_artifact.save(fp) artifact = Artifact.load(fp) self.assertEqual(artifact.type, IntSequence1) self.assertEqual(artifact.uuid, saved_artifact.uuid) self.assertEqual(artifact.view(list), [42, 42, 43, -999, 42]) self.assertEqual(artifact.view(list), [42, 42, 43, -999, 42]) self.assertEqual(artifact.view(collections.Counter), collections.Counter({42: 3, 43: 1, -999: 1})) self.assertEqual(artifact.view(collections.Counter), collections.Counter({42: 3, 43: 1, -999: 1})) def test_load_and_save(self): fp1 = os.path.join(self.test_dir.name, 'artifact1.qza') fp2 = os.path.join(self.test_dir.name, 'artifact2.qza') artifact = Artifact.import_data(FourInts, [-1, 42, 0, 43]) artifact.save(fp1) artifact = Artifact.load(fp1) # Overwriting its source file works. artifact.save(fp1) # Saving to a new file works. artifact.save(fp2) root_dir = str(artifact.uuid) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/file1.txt', 'data/file2.txt', 'data/nested/file3.txt', 'data/nested/file4.txt', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertArchiveMembers(fp1, root_dir, expected) root_dir = str(artifact.uuid) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/file1.txt', 'data/file2.txt', 'data/nested/file3.txt', 'data/nested/file4.txt', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertArchiveMembers(fp2, root_dir, expected) def test_roundtrip(self): fp1 = os.path.join(self.test_dir.name, 'artifact1.qza') fp2 = os.path.join(self.test_dir.name, 'artifact2.qza') artifact = Artifact.import_data(FourInts, [-1, 42, 0, 43]) artifact.save(fp1) artifact1 = Artifact.load(fp1) artifact1.save(fp2) artifact2 = Artifact.load(fp2) self.assertEqual(artifact1.type, artifact2.type) self.assertEqual(artifact1.format, artifact2.format) self.assertEqual(artifact1.uuid, artifact2.uuid) self.assertEqual(artifact1.view(list), artifact2.view(list)) # double view to make sure multiple views can be taken self.assertEqual(artifact1.view(list), artifact2.view(list)) def test_load_with_archive_filepath_modified(self): # Save an artifact for use in the following test case. fp = os.path.join(self.test_dir.name, 'artifact.qza') Artifact.import_data(FourInts, [-1, 42, 0, 43]).save(fp) # Load the artifact from a filepath then save a different artifact to # the same filepath. Assert that both artifacts produce the correct # views of their data. # # `load` used to be lazy, only extracting data when it needed to (e.g. # when `save` or `view` was called). This was buggy as the filepath # could have been deleted, or worse, modified to contain a different # .qza file. Thus, the wrong archive could be extracted on demand, or # the archive could be missing altogether. There isn't an easy # cross-platform compatible way to solve this problem, so Artifact.load # is no longer lazy and always extracts its data immediately. The real # motivation for lazy loading was for quick inspection of archives # without extracting/copying data, so that API is now provided through # Artifact.peek. artifact1 = Artifact.load(fp) Artifact.import_data(FourInts, [10, 11, 12, 13]).save(fp) artifact2 = Artifact.load(fp) self.assertEqual(artifact1.view(list), [-1, 42, 0, 43]) self.assertEqual(artifact2.view(list), [10, 11, 12, 13]) def test_extract(self): fp = os.path.join(self.test_dir.name, 'artifact.qza') artifact = Artifact.import_data(FourInts, [-1, 42, 0, 43]) artifact.save(fp) root_dir = str(artifact.uuid) # pathlib normalizes away the `.`, it doesn't matter, but this is the # implementation we're using, so let's test against that assumption. output_dir = pathlib.Path(self.test_dir.name) / 'artifact-extract-test' result_dir = Artifact.extract(fp, output_dir=output_dir) self.assertEqual(result_dir, str(output_dir / root_dir)) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/file1.txt', 'data/file2.txt', 'data/nested/file3.txt', 'data/nested/file4.txt', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertExtractedArchiveMembers(output_dir, root_dir, expected) def test_peek(self): artifact = Artifact.import_data(FourInts, [0, 0, 42, 1000]) fp = os.path.join(self.test_dir.name, 'artifact.qza') artifact.save(fp) metadata = Artifact.peek(fp) self.assertIsInstance(metadata, ResultMetadata) self.assertEqual(metadata.type, 'FourInts') self.assertEqual(metadata.uuid, str(artifact.uuid)) self.assertEqual(metadata.format, 'FourIntsDirectoryFormat') def test_import_data_invalid_type(self): with self.assertRaisesRegex(TypeError, 'concrete semantic type.*Visualization'): Artifact.import_data(qiime2.core.type.Visualization, self.test_dir) with self.assertRaisesRegex(TypeError, 'concrete semantic type.*Visualization'): Artifact.import_data('Visualization', self.test_dir) def test_import_data_with_filepath_multi_file_data_layout(self): fp = os.path.join(self.test_dir.name, 'test.txt') with open(fp, 'w') as fh: fh.write('42\n') with self.assertRaisesRegex(qiime2.plugin.ValidationError, "FourIntsDirectoryFormat.*directory"): Artifact.import_data(FourInts, fp) def test_import_data_with_wrong_number_of_files(self): data_dir = os.path.join(self.test_dir.name, 'test') os.mkdir(data_dir) error_regex = ("Missing.*MappingDirectoryFormat.*mapping.tsv") with self.assertRaisesRegex(ValidationError, error_regex): Artifact.import_data(Mapping, data_dir) def test_import_data_with_unrecognized_files(self): data_dir = os.path.join(self.test_dir.name, 'test') os.mkdir(data_dir) with open(os.path.join(data_dir, 'file1.txt'), 'w') as fh: fh.write('42\n') with open(os.path.join(data_dir, 'file2.txt'), 'w') as fh: fh.write('43\n') nested = os.path.join(data_dir, 'nested') os.mkdir(nested) with open(os.path.join(nested, 'file3.txt'), 'w') as fh: fh.write('44\n') with open(os.path.join(nested, 'foo.txt'), 'w') as fh: fh.write('45\n') error_regex = ("Unrecognized.*foo.txt.*FourIntsDirectoryFormat") with self.assertRaisesRegex(ValidationError, error_regex): Artifact.import_data(FourInts, data_dir) def test_import_data_with_unreachable_path(self): with self.assertRaisesRegex(qiime2.plugin.ValidationError, "does not exist"): Artifact.import_data(IntSequence1, os.path.join(self.test_dir.name, 'foo.txt')) with self.assertRaisesRegex(qiime2.plugin.ValidationError, "does not exist"): Artifact.import_data(FourInts, os.path.join(self.test_dir.name, 'bar', '')) def test_import_data_with_invalid_format_single_file(self): fp = os.path.join(self.test_dir.name, 'foo.txt') with open(fp, 'w') as fh: fh.write('42\n') fh.write('43\n') fh.write('abc\n') fh.write('123\n') error_regex = "foo.txt.*IntSequenceFormat.*\n\n.*Line 3" with self.assertRaisesRegex(ValidationError, error_regex): Artifact.import_data(IntSequence1, fp) def test_import_data_with_invalid_format_multi_file(self): data_dir = os.path.join(self.test_dir.name, 'test') os.mkdir(data_dir) with open(os.path.join(data_dir, 'file1.txt'), 'w') as fh: fh.write('42\n') with open(os.path.join(data_dir, 'file2.txt'), 'w') as fh: fh.write('43\n') nested = os.path.join(data_dir, 'nested') os.mkdir(nested) with open(os.path.join(nested, 'file3.txt'), 'w') as fh: fh.write('44\n') with open(os.path.join(nested, 'file4.txt'), 'w') as fh: fh.write('foo\n') error_regex = "file4.txt.*SingleIntFormat.*\n\n.*integer" with self.assertRaisesRegex(ValidationError, error_regex): Artifact.import_data(FourInts, data_dir) def test_import_data_with_good_validation_multi_files(self): data_dir = os.path.join(self.test_dir.name, 'test') os.mkdir(data_dir) with open(os.path.join(data_dir, 'file1.txt'), 'w') as fh: fh.write('1\n') with open(os.path.join(data_dir, 'file2.txt'), 'w') as fh: fh.write('1\n') a = Artifact.import_data(SingleInt, data_dir) self.assertEqual(1, a.view(int)) def test_import_data_with_bad_validation_multi_files(self): data_dir = os.path.join(self.test_dir.name, 'test') os.mkdir(data_dir) with open(os.path.join(data_dir, 'file1.txt'), 'w') as fh: fh.write('1\n') with open(os.path.join(data_dir, 'file2.txt'), 'w') as fh: fh.write('2\n') error_regex = ("test.*RedundantSingleIntDirectoryFormat.*\n\n" ".*does not match") with self.assertRaisesRegex(ValidationError, error_regex): Artifact.import_data(SingleInt, data_dir) def test_import_data_with_filepath(self): data_dir = os.path.join(self.test_dir.name, 'test') os.mkdir(data_dir) # Filename shouldn't matter for single-file case. fp = os.path.join(data_dir, 'foo.txt') with open(fp, 'w') as fh: fh.write('42\n') fh.write('43\n') fh.write('42\n') fh.write('0\n') artifact = Artifact.import_data(IntSequence1, fp) self.assertEqual(artifact.type, IntSequence1) self.assertIsInstance(artifact.uuid, uuid.UUID) self.assertEqual(artifact.view(list), [42, 43, 42, 0]) def test_import_data_with_directory_single_file(self): data_dir = os.path.join(self.test_dir.name, 'test') os.mkdir(data_dir) fp = os.path.join(data_dir, 'ints.txt') with open(fp, 'w') as fh: fh.write('-1\n') fh.write('-2\n') fh.write('10\n') fh.write('100\n') artifact = Artifact.import_data(IntSequence1, data_dir) self.assertEqual(artifact.type, IntSequence1) self.assertIsInstance(artifact.uuid, uuid.UUID) self.assertEqual(artifact.view(list), [-1, -2, 10, 100]) def test_import_data_with_directory_multi_file(self): data_dir = os.path.join(self.test_dir.name, 'test') os.mkdir(data_dir) with open(os.path.join(data_dir, 'file1.txt'), 'w') as fh: fh.write('42\n') with open(os.path.join(data_dir, 'file2.txt'), 'w') as fh: fh.write('41\n') nested = os.path.join(data_dir, 'nested') os.mkdir(nested) with open(os.path.join(nested, 'file3.txt'), 'w') as fh: fh.write('43\n') with open(os.path.join(nested, 'file4.txt'), 'w') as fh: fh.write('40\n') artifact = Artifact.import_data(FourInts, data_dir) self.assertEqual(artifact.type, FourInts) self.assertIsInstance(artifact.uuid, uuid.UUID) self.assertEqual(artifact.view(list), [42, 41, 43, 40]) def test_eq_identity(self): artifact = Artifact.import_data(FourInts, [-1, 42, 0, 43]) self.assertEqual(artifact, artifact) def test_eq_same_uuid(self): fp = os.path.join(self.test_dir.name, 'artifact.qza') artifact1 = Artifact.import_data(FourInts, [-1, 42, 0, 43]) artifact1.save(fp) artifact2 = Artifact.load(fp) self.assertEqual(artifact1, artifact2) def test_ne_same_data_different_uuid(self): artifact1 = Artifact.import_data(FourInts, [-1, 42, 0, 43]) artifact2 = Artifact.import_data(FourInts, [-1, 42, 0, 43]) self.assertNotEqual(artifact1, artifact2) def test_ne_different_data_different_uuid(self): artifact1 = Artifact.import_data(FourInts, [-1, 42, 0, 43]) artifact2 = Artifact.import_data(FourInts, [1, 2, 3, 4]) self.assertNotEqual(artifact1, artifact2) def test_ne_subclass_same_uuid(self): class ArtifactSubclass(Artifact): pass fp = os.path.join(self.test_dir.name, 'artifact.qza') artifact1 = ArtifactSubclass.import_data(FourInts, [-1, 42, 0, 43]) artifact1.save(fp) artifact2 = Artifact.load(fp) self.assertNotEqual(artifact1, artifact2) self.assertNotEqual(artifact2, artifact1) def test_ne_different_type_same_uuid(self): artifact = Artifact.import_data(FourInts, [-1, 42, 0, 43]) class Faker: @property def uuid(self): return artifact.uuid faker = Faker() self.assertNotEqual(artifact, faker) def test_artifact_validate_max(self): A = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) A.validate() self.assertTrue(True) # Checkpoint assertion A.validate(level='max') self.assertTrue(True) # Checkpoint assertion A = Artifact.import_data('IntSequence1', [1, 2, 3, 4, 5, 6, 7, 10]) with self.assertRaisesRegex(ValidationError, '3 more'): A.validate(level='max') def test_artifact_validate_max_on_import(self): fp = get_data_path('intsequence-fail-max-validation.txt') fmt = IntSequenceFormat(fp, mode='r') fmt.validate(level='min') self.assertTrue(True) # Checkpoint assertion with self.assertRaisesRegex(ValidationError, '3 more'): Artifact.import_data('IntSequence1', fp) def test_artifact_validate_min(self): A = Artifact.import_data('IntSequence1', [1, 2, 3, 4]) A.validate(level='min') self.assertTrue(True) # Checkpoint assertion A = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) A.validate(level='min') self.assertTrue(True) # Checkpoint assertion def test_artifact_validate_invalid_level(self): A = Artifact.import_data('IntSequence1', [1, 2, 3, 4]) with self.assertRaisesRegex(ValueError, 'peanut'): A.validate(level='peanut') def test_view_as_metadata(self): A = Artifact.import_data('Mapping', {'a': '1', 'b': '3'}) obs_md = A.view(Metadata) exp_df = pd.DataFrame({'a': '1', 'b': '3'}, index=pd.Index(['0'], name='id', dtype=object), dtype=object) exp_md = Metadata(exp_df) exp_md._add_artifacts([A]) self.assertEqual(obs_md, exp_md) # This check is redundant because `Metadata.__eq__` being used above # takes source artifacts into account. Doesn't hurt to have an explicit # check though, since this API didn't always track source artifacts # (this check also future-proofs the test in case `Metadata.__eq__` # changes in the future). self.assertEqual(obs_md.artifacts, (A,)) def test_cannot_be_viewed_as_metadata(self): A = Artifact.import_data('IntSequence1', [1, 2, 3, 4]) with self.assertRaisesRegex(TypeError, 'Artifact.*IntSequence1.*cannot be viewed ' 'as QIIME 2 Metadata'): A.view(Metadata) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/sdk/tests/test_method.py000066400000000000000000000571431412141660100210600ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import collections import concurrent.futures import inspect import unittest import uuid import qiime2.plugin from qiime2.core.type import MethodSignature, Int from qiime2.sdk import Artifact, Method, Results from qiime2.core.testing.method import (concatenate_ints, merge_mappings, params_only_method, no_input_method) from qiime2.core.testing.type import ( IntSequence1, IntSequence2, SingleInt, Mapping) from qiime2.core.testing.util import get_dummy_plugin # TODO refactor these tests along with Visualizer tests to remove duplication. class TestMethod(unittest.TestCase): def setUp(self): self.plugin = get_dummy_plugin() def test_private_constructor(self): with self.assertRaisesRegex(NotImplementedError, 'Method constructor.*private'): Method() def test_from_function_with_artifacts_and_parameters(self): concatenate_ints_sig = MethodSignature( concatenate_ints, inputs={ 'ints1': IntSequence1 | IntSequence2, 'ints2': IntSequence1, 'ints3': IntSequence2 }, parameters={ 'int1': qiime2.plugin.Int, 'int2': qiime2.plugin.Int }, outputs=[ ('concatenated_ints', IntSequence1) ] ) method = self.plugin.methods['concatenate_ints'] self.assertEqual(method.id, 'concatenate_ints') self.assertEqual(method.signature, concatenate_ints_sig) self.assertEqual(method.name, 'Concatenate integers') self.assertTrue( method.description.startswith('This method concatenates integers')) self.assertTrue( method.source.startswith('\n```python\ndef concatenate_ints(')) def test_from_function_with_multiple_outputs(self): method = self.plugin.methods['split_ints'] sig_input = method.signature.inputs['ints'].qiime_type self.assertEqual(list(method.signature.inputs.keys()), ['ints']) self.assertLessEqual(IntSequence1, sig_input) self.assertLessEqual(IntSequence2, sig_input) self.assertEqual({}, method.signature.parameters) self.assertEqual(list(method.signature.outputs.keys()), ['left', 'right']) self.assertIs(sig_input, method.signature.outputs['left'].qiime_type) self.assertIs(sig_input, method.signature.outputs['right'].qiime_type) self.assertEqual(method.id, 'split_ints') self.assertEqual(method.name, 'Split sequence of integers in half') self.assertTrue( method.description.startswith('This method splits a sequence')) self.assertTrue( method.source.startswith('\n```python\ndef split_ints(')) def test_from_function_without_parameters(self): method = self.plugin.methods['merge_mappings'] self.assertEqual(method.id, 'merge_mappings') exp_sig = MethodSignature( merge_mappings, inputs={ 'mapping1': Mapping, 'mapping2': Mapping }, input_descriptions={ 'mapping1': 'Mapping object to be merged' }, parameters={}, outputs=[ ('merged_mapping', Mapping) ], output_descriptions={ 'merged_mapping': 'Resulting merged Mapping object' } ) self.assertEqual(method.signature, exp_sig) self.assertEqual(method.name, 'Merge mappings') self.assertTrue( method.description.startswith('This method merges two mappings')) self.assertTrue( method.source.startswith('\n```python\ndef merge_mappings(')) def test_from_function_with_parameters_only(self): method = self.plugin.methods['params_only_method'] self.assertEqual(method.id, 'params_only_method') exp_sig = MethodSignature( params_only_method, inputs={}, parameters={ 'name': qiime2.plugin.Str, 'age': qiime2.plugin.Int }, outputs=[ ('out', Mapping) ] ) self.assertEqual(method.signature, exp_sig) self.assertEqual(method.name, 'Parameters only method') self.assertTrue( method.description.startswith('This method only accepts')) self.assertTrue( method.source.startswith('\n```python\ndef params_only_method(')) def test_from_function_without_inputs_or_parameters(self): method = self.plugin.methods['no_input_method'] self.assertEqual(method.id, 'no_input_method') exp_sig = MethodSignature( no_input_method, inputs={}, parameters={}, outputs=[ ('out', Mapping) ] ) self.assertEqual(method.signature, exp_sig) self.assertEqual(method.name, 'No input method') self.assertTrue( method.description.startswith('This method does not accept any')) self.assertTrue( method.source.startswith('\n```python\ndef no_input_method(')) def test_is_callable(self): self.assertTrue(callable(self.plugin.methods['concatenate_ints'])) def test_callable_properties(self): concatenate_ints = self.plugin.methods['concatenate_ints'] merge_mappings = self.plugin.methods['merge_mappings'] concatenate_exp = { 'int2': Int, 'ints2': IntSequence1, 'return': (IntSequence1,), 'int1': Int, 'ints3': IntSequence2, 'ints1': IntSequence1 | IntSequence2} merge_exp = { 'mapping2': Mapping, 'mapping1': Mapping, 'return': (Mapping,)} mapper = { concatenate_ints: concatenate_exp, merge_mappings: merge_exp} for method, exp in mapper.items(): self.assertEqual(method.__call__.__name__, '__call__') self.assertEqual(method.__call__.__annotations__, exp) self.assertFalse(hasattr(method.__call__, '__wrapped__')) def test_async_properties(self): concatenate_ints = self.plugin.methods['concatenate_ints'] merge_mappings = self.plugin.methods['merge_mappings'] concatenate_exp = { 'int2': Int, 'ints2': IntSequence1, 'return': (IntSequence1,), 'int1': Int, 'ints3': IntSequence2, 'ints1': IntSequence1 | IntSequence2} merge_exp = { 'mapping2': Mapping, 'mapping1': Mapping, 'return': (Mapping,)} mapper = { concatenate_ints: concatenate_exp, merge_mappings: merge_exp} for method, exp in mapper.items(): self.assertEqual(method.asynchronous.__name__, 'asynchronous') self.assertEqual(method.asynchronous.__annotations__, exp) self.assertFalse(hasattr(method.asynchronous, '__wrapped__')) def test_callable_and_async_signature_with_artifacts_and_parameters(self): # Signature with input artifacts and parameters (i.e. primitives). concatenate_ints = self.plugin.methods['concatenate_ints'] for callable_attr in '__call__', 'asynchronous': signature = inspect.Signature.from_callable( getattr(concatenate_ints, callable_attr)) parameters = list(signature.parameters.items()) kind = inspect.Parameter.POSITIONAL_OR_KEYWORD exp_parameters = [ ('ints1', inspect.Parameter( 'ints1', kind, annotation=IntSequence1 | IntSequence2)), ('ints2', inspect.Parameter( 'ints2', kind, annotation=IntSequence1)), ('ints3', inspect.Parameter( 'ints3', kind, annotation=IntSequence2)), ('int1', inspect.Parameter( 'int1', kind, annotation=Int)), ('int2', inspect.Parameter( 'int2', kind, annotation=Int)) ] self.assertEqual(parameters, exp_parameters) def test_callable_and_async_signature_with_no_parameters(self): # Signature without parameters (i.e. primitives), only input artifacts. method = self.plugin.methods['merge_mappings'] for callable_attr in '__call__', 'asynchronous': signature = inspect.Signature.from_callable( getattr(method, callable_attr)) parameters = list(signature.parameters.items()) kind = inspect.Parameter.POSITIONAL_OR_KEYWORD exp_parameters = [ ('mapping1', inspect.Parameter( 'mapping1', kind, annotation=Mapping)), ('mapping2', inspect.Parameter( 'mapping2', kind, annotation=Mapping)) ] self.assertEqual(parameters, exp_parameters) def test_call_with_artifacts_and_parameters(self): concatenate_ints = self.plugin.methods['concatenate_ints'] artifact1 = Artifact.import_data(IntSequence1, [0, 42, 43]) artifact2 = Artifact.import_data(IntSequence2, [99, -22]) result = concatenate_ints(artifact1, artifact1, artifact2, 55, 1) # Test properties of the `Results` object. self.assertIsInstance(result, tuple) self.assertIsInstance(result, Results) self.assertEqual(len(result), 1) self.assertEqual(result.concatenated_ints.view(list), [0, 42, 43, 0, 42, 43, 99, -22, 55, 1]) result = result[0] self.assertIsInstance(result, Artifact) self.assertEqual(result.type, IntSequence1) self.assertIsInstance(result.uuid, uuid.UUID) # Can retrieve multiple views of different type. exp_list_view = [0, 42, 43, 0, 42, 43, 99, -22, 55, 1] self.assertEqual(result.view(list), exp_list_view) self.assertEqual(result.view(list), exp_list_view) exp_counter_view = collections.Counter( {0: 2, 42: 2, 43: 2, 99: 1, -22: 1, 55: 1, 1: 1}) self.assertEqual(result.view(collections.Counter), exp_counter_view) self.assertEqual(result.view(collections.Counter), exp_counter_view) # Accepts IntSequence1 | IntSequence2 artifact3 = Artifact.import_data(IntSequence2, [10, 20]) result, = concatenate_ints(artifact3, artifact1, artifact2, 55, 1) self.assertEqual(result.type, IntSequence1) self.assertEqual(result.view(list), [10, 20, 0, 42, 43, 99, -22, 55, 1]) def test_call_with_multiple_outputs(self): split_ints = self.plugin.methods['split_ints'] artifact = Artifact.import_data(IntSequence1, [0, 42, -2, 43, 6]) result = split_ints(artifact) self.assertIsInstance(result, tuple) self.assertEqual(len(result), 2) for output_artifact in result: self.assertIsInstance(output_artifact, Artifact) self.assertEqual(output_artifact.type, IntSequence1) self.assertIsInstance(output_artifact.uuid, uuid.UUID) # Output artifacts have different UUIDs. self.assertNotEqual(result[0].uuid, result[1].uuid) # Index lookup. self.assertEqual(result[0].view(list), [0, 42]) self.assertEqual(result[1].view(list), [-2, 43, 6]) # Test properties of the `Results` object. self.assertIsInstance(result, Results) self.assertEqual(result.left.view(list), [0, 42]) self.assertEqual(result.right.view(list), [-2, 43, 6]) def test_call_with_multiple_outputs_matched_types(self): split_ints = self.plugin.methods['split_ints'] artifact = Artifact.import_data(IntSequence2, [0, 42, -2, 43, 6]) result = split_ints(artifact) self.assertIsInstance(result, tuple) self.assertEqual(len(result), 2) for output_artifact in result: self.assertIsInstance(output_artifact, Artifact) self.assertEqual(output_artifact.type, IntSequence2) self.assertIsInstance(output_artifact.uuid, uuid.UUID) # Output artifacts have different UUIDs. self.assertNotEqual(result[0].uuid, result[1].uuid) # Index lookup. self.assertEqual(result[0].view(list), [0, 42]) self.assertEqual(result[1].view(list), [-2, 43, 6]) # Test properties of the `Results` object. self.assertIsInstance(result, Results) self.assertEqual(result.left.view(list), [0, 42]) self.assertEqual(result.right.view(list), [-2, 43, 6]) def test_call_with_no_parameters(self): merge_mappings = self.plugin.methods['merge_mappings'] artifact1 = Artifact.import_data(Mapping, {'foo': 'abc', 'bar': 'def'}) artifact2 = Artifact.import_data(Mapping, {'bazz': 'abc'}) result = merge_mappings(artifact1, artifact2) # Test properties of the `Results` object. self.assertIsInstance(result, tuple) self.assertIsInstance(result, Results) self.assertEqual(len(result), 1) self.assertEqual(result.merged_mapping.view(dict), {'foo': 'abc', 'bar': 'def', 'bazz': 'abc'}) result = result[0] self.assertIsInstance(result, Artifact) self.assertEqual(result.type, Mapping) self.assertIsInstance(result.uuid, uuid.UUID) self.assertEqual(result.view(dict), {'foo': 'abc', 'bar': 'def', 'bazz': 'abc'}) def test_call_with_parameters_only(self): params_only_method = self.plugin.methods['params_only_method'] result, = params_only_method("Someone's Name", 999) self.assertIsInstance(result, Artifact) self.assertEqual(result.type, Mapping) self.assertIsInstance(result.uuid, uuid.UUID) self.assertEqual(result.view(dict), {"Someone's Name": '999'}) def test_call_without_inputs_or_parameters(self): no_input_method = self.plugin.methods['no_input_method'] result, = no_input_method() self.assertIsInstance(result, Artifact) self.assertEqual(result.type, Mapping) self.assertIsInstance(result.uuid, uuid.UUID) self.assertEqual(result.view(dict), {'foo': '42'}) def test_call_with_optional_artifacts(self): method = self.plugin.methods['optional_artifacts_method'] ints1 = Artifact.import_data(IntSequence1, [0, 42, 43]) ints2 = Artifact.import_data(IntSequence1, [99, -22]) ints3 = Artifact.import_data(IntSequence2, [43, 43]) # No optional artifacts provided. obs = method(ints1, 42).output self.assertEqual(obs.view(list), [0, 42, 43, 42]) # One optional artifact provided. obs = method(ints1, 42, optional1=ints2).output self.assertEqual(obs.view(list), [0, 42, 43, 42, 99, -22]) # All optional artifacts provided. obs = method( ints1, 42, optional1=ints2, optional2=ints3, num2=111).output self.assertEqual(obs.view(list), [0, 42, 43, 42, 99, -22, 43, 43, 111]) # Invalid type provided as optional artifact. with self.assertRaisesRegex(TypeError, 'type IntSequence1.*type IntSequence2'): method(ints1, 42, optional1=ints3) def test_call_with_variadic_inputs(self): method = self.plugin.methods['variadic_input_method'] ints = [Artifact.import_data(IntSequence1, [1, 2, 3]), Artifact.import_data(IntSequence2, [4, 5, 6])] int_set = {Artifact.import_data(SingleInt, 7), Artifact.import_data(SingleInt, 8)} nums = {9, 10} opt_nums = [11, 12, 13] result, = method(ints, int_set, nums, opt_nums) self.assertEqual(result.view(list), list(range(1, 14))) def test_asynchronous(self): concatenate_ints = self.plugin.methods['concatenate_ints'] artifact1 = Artifact.import_data(IntSequence1, [0, 42, 43]) artifact2 = Artifact.import_data(IntSequence2, [99, -22]) future = concatenate_ints.asynchronous( artifact1, artifact1, artifact2, 55, 1) self.assertIsInstance(future, concurrent.futures.Future) result = future.result() # Test properties of the `Results` object. self.assertIsInstance(result, tuple) self.assertIsInstance(result, Results) self.assertEqual(len(result), 1) self.assertEqual(result.concatenated_ints.view(list), [0, 42, 43, 0, 42, 43, 99, -22, 55, 1]) result = result[0] self.assertIsInstance(result, Artifact) self.assertEqual(result.type, IntSequence1) self.assertIsInstance(result.uuid, uuid.UUID) # Can retrieve multiple views of different type. exp_list_view = [0, 42, 43, 0, 42, 43, 99, -22, 55, 1] self.assertEqual(result.view(list), exp_list_view) self.assertEqual(result.view(list), exp_list_view) exp_counter_view = collections.Counter( {0: 2, 42: 2, 43: 2, 99: 1, -22: 1, 55: 1, 1: 1}) self.assertEqual(result.view(collections.Counter), exp_counter_view) self.assertEqual(result.view(collections.Counter), exp_counter_view) # Accepts IntSequence1 | IntSequence2 artifact3 = Artifact.import_data(IntSequence2, [10, 20]) future = concatenate_ints.asynchronous(artifact3, artifact1, artifact2, 55, 1) result, = future.result() self.assertEqual(result.type, IntSequence1) self.assertEqual(result.view(list), [10, 20, 0, 42, 43, 99, -22, 55, 1]) def test_async_with_multiple_outputs(self): split_ints = self.plugin.methods['split_ints'] artifact = Artifact.import_data(IntSequence1, [0, 42, -2, 43, 6]) future = split_ints.asynchronous(artifact) self.assertIsInstance(future, concurrent.futures.Future) result = future.result() self.assertIsInstance(result, tuple) self.assertEqual(len(result), 2) for output_artifact in result: self.assertIsInstance(output_artifact, Artifact) self.assertEqual(output_artifact.type, IntSequence1) self.assertIsInstance(output_artifact.uuid, uuid.UUID) # Output artifacts have different UUIDs. self.assertNotEqual(result[0].uuid, result[1].uuid) # Index lookup. self.assertEqual(result[0].view(list), [0, 42]) self.assertEqual(result[1].view(list), [-2, 43, 6]) # Test properties of the `Results` object. self.assertIsInstance(result, Results) self.assertEqual(result.left.view(list), [0, 42]) self.assertEqual(result.right.view(list), [-2, 43, 6]) def test_async_with_multiple_outputs_matched_types(self): split_ints = self.plugin.methods['split_ints'] artifact = Artifact.import_data(IntSequence2, [0, 42, -2, 43, 6]) future = split_ints.asynchronous(artifact) self.assertIsInstance(future, concurrent.futures.Future) result = future.result() self.assertIsInstance(result, tuple) self.assertEqual(len(result), 2) for output_artifact in result: self.assertIsInstance(output_artifact, Artifact) self.assertEqual(output_artifact.type, IntSequence2) self.assertIsInstance(output_artifact.uuid, uuid.UUID) # Output artifacts have different UUIDs. self.assertNotEqual(result[0].uuid, result[1].uuid) # Index lookup. self.assertEqual(result[0].view(list), [0, 42]) self.assertEqual(result[1].view(list), [-2, 43, 6]) # Test properties of the `Results` object. self.assertIsInstance(result, Results) self.assertEqual(result.left.view(list), [0, 42]) self.assertEqual(result.right.view(list), [-2, 43, 6]) def test_docstring(self): merge_mappings = self.plugin.methods['merge_mappings'] split_ints = self.plugin.methods['split_ints'] identity_with_optional_metadata = ( self.plugin.methods['identity_with_optional_metadata']) no_input_method = self.plugin.methods['no_input_method'] params_only_method = self.plugin.methods['params_only_method'] long_description_method = self.plugin.methods[ 'long_description_method'] docstring_order_method = self.plugin.methods['docstring_order_method'] self.assertEqual(merge_mappings.__doc__, 'QIIME 2 Method') merge_calldoc = merge_mappings.__call__.__doc__ self.assertEqual(exp_merge_calldoc, merge_calldoc) split_ints_return = split_ints.__call__.__doc__.split('\n\n')[3] self.assertEqual(exp_split_ints_return, split_ints_return) optional_params = ( identity_with_optional_metadata.__call__.__doc__.split('\n\n')[2]) self.assertEqual(exp_optional_params, optional_params) no_input_method = no_input_method.__call__.__doc__ self.assertEqual(exp_no_input_method, no_input_method) params_only = params_only_method.__call__.__doc__ self.assertEqual(exp_params_only, params_only) long_desc = long_description_method.__call__.__doc__ self.assertEqual(exp_long_description, long_desc) docstring_order = docstring_order_method.__call__.__doc__ self.assertEqual(exp_docstring_order, docstring_order) exp_merge_calldoc = """\ Merge mappings This method merges two mappings into a single new mapping. If a key is shared between mappings and the values differ, an error will be raised. Parameters ---------- mapping1 : Mapping Mapping object to be merged mapping2 : Mapping Returns ------- merged_mapping : Mapping Resulting merged Mapping object """ exp_split_ints_return = """\ Returns ------- left : IntSequence1\xb9 | IntSequence2\xb2 right : IntSequence1\xb9 | IntSequence2\xb2 """ exp_optional_params = """\ Parameters ---------- ints : IntSequence1 | IntSequence2 metadata : Metadata, optional\ """ exp_no_input_method = """\ No input method This method does not accept any type of input. Returns ------- out : Mapping """ exp_params_only = """\ Parameters only method This method only accepts parameters. Parameters ---------- name : Str age : Int Returns ------- out : Mapping """ exp_long_description = """\ Long Description This is a very long description. If asked about its length, I would have to say it is greater than 79 characters. Parameters ---------- mapping1 : Mapping This is a very long description. If asked about its length, I would have to say it is greater than 79 characters. name : Str This is a very long description. If asked about its length, I would have to say it is greater than 79 characters. age : Int Returns ------- out : Mapping This is a very long description. If asked about its length, I would have to say it is greater than 79 characters. """ exp_docstring_order = """\ Docstring Order Tests whether inputs and parameters are rendered in signature order Parameters ---------- req_input : Mapping This should show up first. req_param : Str This should show up second. opt_input : Mapping, optional This should show up third. opt_param : Int, optional This should show up fourth. Returns ------- out : Mapping This should show up last, in it's own section. """ if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/sdk/tests/test_pipeline.py000066400000000000000000000207231412141660100213770ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest import inspect import pandas as pd import qiime2 import qiime2.sdk from qiime2.core.testing.util import get_dummy_plugin from qiime2.core.testing.type import IntSequence1, SingleInt, Mapping from qiime2.plugin import Visualization, Int, Bool class TestPipeline(unittest.TestCase): def setUp(self): self.plugin = get_dummy_plugin() self.single_int = qiime2.Artifact.import_data(SingleInt, -1) self.int_sequence = qiime2.Artifact.import_data(IntSequence1, [1, 2, 3]) self.mapping = qiime2.Artifact.import_data(Mapping, {'foo': '42'}) def test_private_constructor(self): with self.assertRaisesRegex(NotImplementedError, 'Pipeline constructor.*private'): qiime2.sdk.Pipeline() def test_from_function_spot_check(self): typical_pipeline = self.plugin.pipelines['typical_pipeline'] self.assertEqual(typical_pipeline.id, 'typical_pipeline') assert typical_pipeline.signature.inputs for spec in typical_pipeline.signature.inputs.values(): assert spec.has_description() assert spec.has_qiime_type() assert not spec.has_view_type() assert not spec.has_default() spec = typical_pipeline.signature.parameters['add'] assert spec.has_default() def test_from_function_optional(self): optional_artifact_pipeline = self.plugin.pipelines[ 'optional_artifact_pipeline'] spec = optional_artifact_pipeline.signature.inputs['single_int'] assert spec.has_default() def test_is_callable(self): assert callable(self.plugin.pipelines['typical_pipeline']) def test_callable_and_async_signature(self): # Shouldn't include `ctx` typical_pipeline = self.plugin.pipelines['typical_pipeline'] kind = inspect.Parameter.POSITIONAL_OR_KEYWORD exp_parameters = [ ('int_sequence', inspect.Parameter( 'int_sequence', kind, annotation=IntSequence1)), ('mapping', inspect.Parameter( 'mapping', kind, annotation=Mapping)), ('do_extra_thing', inspect.Parameter( 'do_extra_thing', kind, annotation=Bool)), ('add', inspect.Parameter( 'add', kind, default=1, annotation=Int)) ] for callable_attr in '__call__', 'asynchronous': signature = inspect.Signature.from_callable( getattr(typical_pipeline, callable_attr)) parameters = list(signature.parameters.items()) self.assertEqual(parameters, exp_parameters) def test_signatures_independent(self): typical_pipeline = self.plugin.pipelines['typical_pipeline'] parameter_only_pipeline = self.plugin.pipelines[ 'parameter_only_pipeline'] for callable_attr in '__call__', 'asynchronous': signature_a = inspect.Signature.from_callable( getattr(typical_pipeline, callable_attr)) signature_b = inspect.Signature.from_callable( getattr(parameter_only_pipeline, callable_attr)) self.assertNotEqual(signature_a, signature_b) def iter_callables(self, name): pipeline = self.plugin.pipelines[name] yield pipeline yield lambda *args, **kwargs: pipeline.asynchronous( *args, **kwargs).result() def test_parameter_only_pipeline(self): index = pd.Index(['a', 'b', 'c'], name='id', dtype=object) df = pd.DataFrame({'col1': ['2', '1', '3']}, index=index, dtype=object) metadata = qiime2.Metadata(df) for call in self.iter_callables('parameter_only_pipeline'): results = call(100) self.assertEqual(results.foo.view(list), [100, 2, 3]) self.assertEqual(results.bar.view(list), [100, 2, 3, 100, 2, 3, 100, 2, 3, 100, 2]) results = call(3, int2=4, metadata=metadata) self.assertEqual(results.foo.view(list), [3, 4, 3]) self.assertEqual(results.bar.view(list), [3, 4, 3, 3, 4, 3, 3, 4, 3, 3, 4]) def test_typical_pipeline(self): for call in self.iter_callables('typical_pipeline'): results = call(self.int_sequence, self.mapping, False) self.assertEqual(results.left_viz.type, Visualization) self.assertEqual(results.left.view(list), [1]) self.assertEqual(results.right.view(list), [2, 3]) self.assertNotEqual(results.out_map.uuid, self.mapping.uuid) self.assertEqual(results.out_map.view(dict), self.mapping.view(dict)) results = call(self.int_sequence, self.mapping, True, add=5) self.assertEqual(results.left.view(list), [6]) self.assertEqual(results.right.view(list), [2, 3]) with self.assertRaisesRegex(ValueError, 'Bad mapping'): m = qiime2.Artifact.import_data(Mapping, {'a': 1}) call(self.int_sequence, m, False) def test_optional_artifact_pipeline(self): for call in self.iter_callables('optional_artifact_pipeline'): ints, = call(self.int_sequence) self.assertEqual(ints.view(list), [1, 2, 3, 4]) ints, = call(self.int_sequence, single_int=self.single_int) self.assertEqual(ints.view(list), [1, 2, 3, -1]) def test_visualizer_only_pipeline(self): for call in self.iter_callables('visualizer_only_pipeline'): viz1, viz2 = call(self.mapping) self.assertEqual(viz1.type, Visualization) self.assertEqual(viz2.type, Visualization) def test_pipeline_in_pipeline(self): for call in self.iter_callables('pipelines_in_pipeline'): results = call(self.int_sequence, self.mapping) self.assertEqual(results.int1.view(int), 4) self.assertEqual(results.right_viz.type, Visualization) self.assertEqual(len(results), 8) with self.assertRaisesRegex(ValueError, 'Bad mapping'): m = qiime2.Artifact.import_data(Mapping, {1: 1}) call(self.int_sequence, m) def test_pointless_pipeline(self): for call in self.iter_callables('pointless_pipeline'): single_int, = call() self.assertEqual(single_int.type, SingleInt) self.assertEqual(single_int.view(int), 4) def test_failing_from_arity(self): for call in self.iter_callables('failing_pipeline'): with self.assertRaisesRegex(TypeError, 'match number.*3.*1'): call(self.int_sequence, break_from='arity') def test_failing_from_return_view(self): for call in self.iter_callables('failing_pipeline'): with self.assertRaisesRegex(TypeError, 'Result.*objects.*None'): call(self.int_sequence, break_from='return-view') def test_failing_from_method(self): for call in self.iter_callables('failing_pipeline'): with self.assertRaisesRegex(ValueError, "Key 'foo' exists"): call(self.int_sequence, break_from='method') def test_failing_from_type(self): for call in self.iter_callables('failing_pipeline'): with self.assertRaisesRegex(TypeError, 'Mapping.*SingleInt'): call(self.int_sequence, break_from='type') def test_failing_from_internal(self): for call in self.iter_callables('failing_pipeline'): with self.assertRaisesRegex(ValueError, 'this never works'): call(self.int_sequence, break_from='internal') def test_failing_from_missing_plugin(self): for call in self.iter_callables('failing_pipeline'): with self.assertRaisesRegex(ValueError, r'plugin.*not\%a\$plugin'): call(self.int_sequence, break_from='no-plugin') def test_failing_from_missing_action(self): for call in self.iter_callables('failing_pipeline'): with self.assertRaisesRegex(ValueError, r'action.*not\%a\$method'): call(self.int_sequence, break_from='no-action') if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/sdk/tests/test_plugin_manager.py000066400000000000000000000336771412141660100225760ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest import qiime2.plugin import qiime2.sdk from qiime2.plugin.plugin import SemanticTypeRecord, FormatRecord from qiime2.sdk.plugin_manager import GetFormatFilters from qiime2.core.testing.type import (IntSequence1, IntSequence2, IntSequence3, Mapping, FourInts, Kennel, Dog, Cat, SingleInt, C1, C2, C3, Foo, Bar, Baz, AscIntSequence, Squid, Octopus, Cuttlefish) from qiime2.core.testing.format import (Cephalapod, IntSequenceDirectoryFormat, MappingDirectoryFormat, IntSequenceV2DirectoryFormat, IntSequenceFormatV2, IntSequenceMultiFileDirectoryFormat, FourIntsDirectoryFormat, IntSequenceFormat, RedundantSingleIntDirectoryFormat, EchoFormat, EchoDirectoryFormat, CephalapodDirectoryFormat) from qiime2.core.testing.validator import (validator_example_null1, validate_ascending_seq, validator_example_null2) from qiime2.core.testing.util import get_dummy_plugin class TestPluginManager(unittest.TestCase): def setUp(self): self.plugin = get_dummy_plugin() # PluginManager is a singleton so there's no issue creating it again. self.pm = qiime2.sdk.PluginManager() def test_plugins(self): plugins = self.pm.plugins exp = {'dummy-plugin': self.plugin} self.assertEqual(plugins, exp) def test_validators(self): self.assertEqual({Kennel[Dog], Kennel[Cat], AscIntSequence, Squid, Octopus, Cuttlefish}, set(self.pm.validators)) self.assertEqual( set([r.validator for r in self.pm.validators[Kennel[Dog]]._validators]), {validator_example_null1, validator_example_null2}) self.assertEqual( [r.validator for r in self.pm.validators[Kennel[Cat]]._validators], [validator_example_null1]) self.assertEqual( [r.validator for r in self.pm.validators[AscIntSequence]._validators], [validate_ascending_seq]) def test_type_fragments(self): types = self.pm.type_fragments exp = { 'IntSequence1': SemanticTypeRecord(semantic_type=IntSequence1, plugin=self.plugin), 'IntSequence2': SemanticTypeRecord(semantic_type=IntSequence2, plugin=self.plugin), 'IntSequence3': SemanticTypeRecord(semantic_type=IntSequence3, plugin=self.plugin), 'Mapping': SemanticTypeRecord(semantic_type=Mapping, plugin=self.plugin), 'FourInts': SemanticTypeRecord(semantic_type=FourInts, plugin=self.plugin), 'Kennel': SemanticTypeRecord(semantic_type=Kennel, plugin=self.plugin), 'Dog': SemanticTypeRecord(semantic_type=Dog, plugin=self.plugin), 'Cat': SemanticTypeRecord(semantic_type=Cat, plugin=self.plugin), 'SingleInt': SemanticTypeRecord(semantic_type=SingleInt, plugin=self.plugin), 'C1': SemanticTypeRecord(semantic_type=C1, plugin=self.plugin), 'C2': SemanticTypeRecord(semantic_type=C2, plugin=self.plugin), 'C3': SemanticTypeRecord(semantic_type=C3, plugin=self.plugin), 'Foo': SemanticTypeRecord(semantic_type=Foo, plugin=self.plugin), 'Bar': SemanticTypeRecord(semantic_type=Bar, plugin=self.plugin), 'Baz': SemanticTypeRecord(semantic_type=Baz, plugin=self.plugin), 'AscIntSequence': SemanticTypeRecord(semantic_type=AscIntSequence, plugin=self.plugin), 'Squid': SemanticTypeRecord(semantic_type=Squid, plugin=self.plugin), 'Octopus': SemanticTypeRecord(semantic_type=Octopus, plugin=self.plugin), 'Cuttlefish': SemanticTypeRecord(semantic_type=Cuttlefish, plugin=self.plugin), } self.assertEqual(types, exp) def test_get_semantic_types(self): types = self.pm.get_semantic_types() exp = { 'IntSequence1': SemanticTypeRecord(semantic_type=IntSequence1, plugin=self.plugin), 'IntSequence2': SemanticTypeRecord(semantic_type=IntSequence2, plugin=self.plugin), 'Mapping': SemanticTypeRecord(semantic_type=Mapping, plugin=self.plugin), 'FourInts': SemanticTypeRecord(semantic_type=FourInts, plugin=self.plugin), 'Kennel[Dog]': SemanticTypeRecord(semantic_type=Kennel[Dog], plugin=self.plugin), 'Kennel[Cat]': SemanticTypeRecord(semantic_type=Kennel[Cat], plugin=self.plugin), 'SingleInt': SemanticTypeRecord(semantic_type=SingleInt, plugin=self.plugin), } self.assertLessEqual(exp.keys(), types.keys()) self.assertNotIn(Cat, types) self.assertNotIn(Dog, types) self.assertNotIn(Kennel, types) # TODO: add tests for type/directory/transformer registrations def test_get_formats_no_type_or_filter(self): exp = { 'IntSequenceFormat': FormatRecord(format=IntSequenceFormat, plugin=self.plugin), 'IntSequenceDirectoryFormat': FormatRecord(format=IntSequenceDirectoryFormat, plugin=self.plugin), 'IntSequenceFormatV2': FormatRecord(format=IntSequenceFormatV2, plugin=self.plugin), 'IntSequenceV2DirectoryFormat': FormatRecord(format=IntSequenceV2DirectoryFormat, plugin=self.plugin), 'IntSequenceMultiFileDirectoryFormat': FormatRecord(format=IntSequenceMultiFileDirectoryFormat, plugin=self.plugin), 'RedundantSingleIntDirectoryFormat': FormatRecord(format=RedundantSingleIntDirectoryFormat, plugin=self.plugin), 'FourIntsDirectoryFormat': FormatRecord(format=FourIntsDirectoryFormat, plugin=self.plugin), 'EchoFormat': FormatRecord(format=EchoFormat, plugin=self.plugin), 'EchoDirectoryFormat': FormatRecord(format=EchoDirectoryFormat, plugin=self.plugin), 'MappingDirectoryFormat': FormatRecord(format=MappingDirectoryFormat, plugin=self.plugin), 'Cephalapod': FormatRecord(format=Cephalapod, plugin=self.plugin), 'CephalapodDirectoryFormat': FormatRecord(format=CephalapodDirectoryFormat, plugin=self.plugin), } obs = self.pm.get_formats() self.assertEqual(obs, exp) def test_get_formats_SFDF(self): exp = { 'IntSequenceFormat': FormatRecord(format=IntSequenceFormat, plugin=self.plugin), 'IntSequenceFormatV2': FormatRecord(format=IntSequenceFormatV2, plugin=self.plugin), 'IntSequenceDirectoryFormat': FormatRecord(format=IntSequenceDirectoryFormat, plugin=self.plugin), 'IntSequenceV2DirectoryFormat': FormatRecord(format=IntSequenceV2DirectoryFormat, plugin=self.plugin), 'IntSequenceMultiFileDirectoryFormat': FormatRecord(format=IntSequenceMultiFileDirectoryFormat, plugin=self.plugin) } obs = self.pm.get_formats(semantic_type='IntSequence1') self.assertEqual(exp, obs) def test_get_formats_SFDF_EXPORTABLE(self): exp = { 'IntSequenceFormat': FormatRecord(format=IntSequenceFormat, plugin=self.plugin), 'IntSequenceFormatV2': FormatRecord(format=IntSequenceFormatV2, plugin=self.plugin), 'IntSequenceDirectoryFormat': FormatRecord(format=IntSequenceDirectoryFormat, plugin=self.plugin), 'IntSequenceV2DirectoryFormat': FormatRecord(format=IntSequenceV2DirectoryFormat, plugin=self.plugin) } obs = self.pm.get_formats(filter=GetFormatFilters.EXPORTABLE, semantic_type=IntSequence1) self.assertEqual(exp, obs) def test_get_formats_SFDF_IMPORTABLE(self): exp = { 'IntSequenceFormat': FormatRecord(format=IntSequenceFormat, plugin=self.plugin), 'IntSequenceDirectoryFormat': FormatRecord(format=IntSequenceDirectoryFormat, plugin=self.plugin), 'IntSequenceMultiFileDirectoryFormat': FormatRecord(format=IntSequenceMultiFileDirectoryFormat, plugin=self.plugin) } obs = self.pm.get_formats(filter=GetFormatFilters.IMPORTABLE, semantic_type=IntSequence1) self.assertEqual(exp, obs) def test_get_formats_DF(self): exp = { 'IntSequenceFormat': FormatRecord(format=IntSequenceFormat, plugin=self.plugin), 'IntSequenceFormatV2': FormatRecord(format=IntSequenceFormatV2, plugin=self.plugin), 'IntSequenceDirectoryFormat': FormatRecord(format=IntSequenceDirectoryFormat, plugin=self.plugin), 'IntSequenceV2DirectoryFormat': FormatRecord(format=IntSequenceV2DirectoryFormat, plugin=self.plugin), 'IntSequenceMultiFileDirectoryFormat': FormatRecord(format=IntSequenceMultiFileDirectoryFormat, plugin=self.plugin) } obs = self.pm.get_formats(semantic_type='IntSequence3') self.assertEqual(exp, obs) def test_get_formats_DF_EXPORTABLE(self): exp = { 'IntSequenceFormat': FormatRecord(format=IntSequenceFormat, plugin=self.plugin), 'IntSequenceDirectoryFormat': FormatRecord(format=IntSequenceDirectoryFormat, plugin=self.plugin), 'IntSequenceMultiFileDirectoryFormat': FormatRecord(format=IntSequenceMultiFileDirectoryFormat, plugin=self.plugin) } obs = self.pm.get_formats(filter=GetFormatFilters.EXPORTABLE, semantic_type=IntSequence3) self.assertEqual(exp, obs) def test_get_formats_DF_IMPORTABLE(self): exp = { 'IntSequenceFormatV2': FormatRecord(format=IntSequenceFormatV2, plugin=self.plugin), 'IntSequenceV2DirectoryFormat': FormatRecord(format=IntSequenceV2DirectoryFormat, plugin=self.plugin), 'IntSequenceMultiFileDirectoryFormat': FormatRecord(format=IntSequenceMultiFileDirectoryFormat, plugin=self.plugin) } obs = self.pm.get_formats(filter=GetFormatFilters.IMPORTABLE, semantic_type=IntSequence3) self.assertEqual(exp, obs) def test_get_formats_invalid_type(self): with self.assertRaisesRegex(ValueError, "No formats associated"): self.pm.get_formats(semantic_type='Random[Frequency]') def test_get_formats_invalid_filter(self): with self.assertRaisesRegex(ValueError, "filter.*is not valid"): self.pm.get_formats(filter="EXPORTABLE") if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/sdk/tests/test_result.py000066400000000000000000000447151412141660100211170ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import tempfile import unittest import pathlib import qiime2.core.type from qiime2.sdk import Result, Artifact, Visualization from qiime2.sdk.result import ResultMetadata import qiime2.core.archive as archive import qiime2.core.exceptions as exceptions from qiime2.core.testing.type import FourInts from qiime2.core.testing.util import get_dummy_plugin, ArchiveTestingMixin from qiime2.core.testing.visualizer import mapping_viz class TestResult(unittest.TestCase, ArchiveTestingMixin): def make_provenance_capture(self): # You can't actually import a visualization, but I won't tell # visualization if you don't... return archive.ImportProvenanceCapture() def setUp(self): # Ignore the returned dummy plugin object, just run this to verify the # plugin exists as the tests rely on it being loaded. get_dummy_plugin() # TODO standardize temporary directories created by QIIME 2 self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-') self.data_dir = os.path.join(self.test_dir.name, 'viz-output') os.mkdir(self.data_dir) mapping_viz(self.data_dir, {'abc': 'foo', 'def': 'bar'}, {'ghi': 'baz', 'jkl': 'bazz'}, key_label='Key', value_label='Value') def tearDown(self): self.test_dir.cleanup() def test_private_constructor(self): with self.assertRaisesRegex( NotImplementedError, 'Result constructor.*private.*Result.load'): Result() def test_load_artifact(self): saved_artifact = Artifact.import_data(FourInts, [-1, 42, 0, 43]) fp = os.path.join(self.test_dir.name, 'artifact.qza') saved_artifact.save(fp) artifact = Result.load(fp) self.assertIsInstance(artifact, Artifact) self.assertEqual(artifact.type, FourInts) self.assertEqual(artifact.uuid, saved_artifact.uuid) self.assertEqual(artifact.view(list), [-1, 42, 0, 43]) def test_load_visualization(self): saved_visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) fp = os.path.join(self.test_dir.name, 'visualization.qzv') saved_visualization.save(fp) visualization = Result.load(fp) self.assertIsInstance(visualization, Visualization) self.assertEqual(visualization.type, qiime2.core.type.Visualization) self.assertEqual(visualization.uuid, saved_visualization.uuid) def test_extract_artifact(self): fp = os.path.join(self.test_dir.name, 'artifact.qza') artifact = Artifact.import_data(FourInts, [-1, 42, 0, 43]) artifact.save(fp) root_dir = str(artifact.uuid) # pathlib normalizes away the `.`, it doesn't matter, but this is the # implementation we're using, so let's test against that assumption. output_dir = pathlib.Path(self.test_dir.name) / 'artifact-extract-test' result_dir = Result.extract(fp, output_dir=output_dir) self.assertEqual(result_dir, str(output_dir / root_dir)) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/file1.txt', 'data/file2.txt', 'data/nested/file3.txt', 'data/nested/file4.txt', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertExtractedArchiveMembers(output_dir, root_dir, expected) def test_extract_visualization(self): fp = os.path.join(self.test_dir.name, 'visualization.qzv') visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) visualization.save(fp) root_dir = str(visualization.uuid) output_dir = pathlib.Path(self.test_dir.name) / 'viz-extract-test' result_dir = Result.extract(fp, output_dir=output_dir) self.assertEqual(result_dir, str(output_dir / root_dir)) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/index.html', 'data/css/style.css', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertExtractedArchiveMembers(output_dir, root_dir, expected) def test_peek_artifact(self): artifact = Artifact.import_data(FourInts, [0, 0, 42, 1000]) fp = os.path.join(self.test_dir.name, 'artifact.qza') artifact.save(fp) metadata = Result.peek(fp) self.assertIsInstance(metadata, ResultMetadata) self.assertEqual(metadata.type, 'FourInts') self.assertEqual(metadata.uuid, str(artifact.uuid)) self.assertEqual(metadata.format, 'FourIntsDirectoryFormat') def test_peek_visualization(self): visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) fp = os.path.join(self.test_dir.name, 'visualization.qzv') visualization.save(fp) metadata = Result.peek(fp) self.assertIsInstance(metadata, ResultMetadata) self.assertEqual(metadata.type, 'Visualization') self.assertEqual(metadata.uuid, str(visualization.uuid)) self.assertIsNone(metadata.format) def test_save_artifact_auto_extension(self): artifact = Artifact.import_data(FourInts, [0, 0, 42, 1000]) # Filename & extension endswith is matching (default). fp = os.path.join(self.test_dir.name, 'artifactqza') obs_fp = artifact.save(fp) obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifactqza.qza') # Filename & extension endswith is matching (non-default). fp = os.path.join(self.test_dir.name, 'artifacttxt') obs_fp = artifact.save(fp, 'txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifacttxt.txt') # No period in filename; no period in extension. fp = os.path.join(self.test_dir.name, 'artifact') obs_fp = artifact.save(fp, 'txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.txt') # No period in filename; multiple periods in extension. fp = os.path.join(self.test_dir.name, 'artifact') obs_fp = artifact.save(fp, '..txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.txt') # Single period in filename; no period in extension. fp = os.path.join(self.test_dir.name, 'artifact.') obs_fp = artifact.save(fp, 'txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.txt') # Single period in filename; single period in extension. fp = os.path.join(self.test_dir.name, 'artifact.') obs_fp = artifact.save(fp, '.txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.txt') # Single period in filename; multiple periods in extension. fp = os.path.join(self.test_dir.name, 'artifact.') obs_fp = artifact.save(fp, '..txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.txt') # Multiple periods in filename; single period in extension. fp = os.path.join(self.test_dir.name, 'artifact..') obs_fp = artifact.save(fp, '.txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.txt') # Multiple periods in filename; multiple periods in extension. fp = os.path.join(self.test_dir.name, 'artifact..') obs_fp = artifact.save(fp, '..txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.txt') # No extension in filename; no extension input. fp = os.path.join(self.test_dir.name, 'artifact') obs_fp = artifact.save(fp) obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.qza') # No extension in filename; different extension input. fp = os.path.join(self.test_dir.name, 'artifact') obs_fp = artifact.save(fp, '.txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.txt') # No extension in filename; default extension input. fp = os.path.join(self.test_dir.name, 'artifact') obs_fp = artifact.save(fp, '.qza') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.qza') # Different extension in filename; no extension input. fp = os.path.join(self.test_dir.name, 'artifact.zip') obs_fp = artifact.save(fp) obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.zip.qza') # Different extension in filename; # Different extension input (non-matching). fp = os.path.join(self.test_dir.name, 'artifact.zip') obs_fp = artifact.save(fp, '.txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.zip.txt') # Different extension in filename; # Different extension input (matching). fp = os.path.join(self.test_dir.name, 'artifact.zip') obs_fp = artifact.save(fp, '.zip') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.zip') # Different extension in filename; default extension input. fp = os.path.join(self.test_dir.name, 'artifact.zip') obs_fp = artifact.save(fp, '.qza') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.zip.qza') # Default extension in filename; no extension input. fp = os.path.join(self.test_dir.name, 'artifact.qza') obs_fp = artifact.save(fp) obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.qza') # Default extension in filename; different extension input. fp = os.path.join(self.test_dir.name, 'artifact.qza') obs_fp = artifact.save(fp, '.txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.qza.txt') # Default extension in filename; default extension input. fp = os.path.join(self.test_dir.name, 'artifact.qza') obs_fp = artifact.save(fp, '.qza') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'artifact.qza') def test_save_visualization_auto_extension(self): visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) # Filename & extension endswith is matching (default). fp = os.path.join(self.test_dir.name, 'visualizationqzv') obs_fp = visualization.save(fp) obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualizationqzv.qzv') # Filename & extension endswith is matching (non-default). fp = os.path.join(self.test_dir.name, 'visualizationtxt') obs_fp = visualization.save(fp, 'txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualizationtxt.txt') # No period in filename; no period in extension. fp = os.path.join(self.test_dir.name, 'visualization') obs_fp = visualization.save(fp, 'txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.txt') # No period in filename; multiple periods in extension. fp = os.path.join(self.test_dir.name, 'visualization') obs_fp = visualization.save(fp, '..txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.txt') # Single period in filename; no period in extension. fp = os.path.join(self.test_dir.name, 'visualization.') obs_fp = visualization.save(fp, 'txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.txt') # Single period in filename; single period in extension. fp = os.path.join(self.test_dir.name, 'visualization.') obs_fp = visualization.save(fp, '.txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.txt') # Single period in filename; multiple periods in extension. fp = os.path.join(self.test_dir.name, 'visualization.') obs_fp = visualization.save(fp, '..txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.txt') # Multiple periods in filename; single period in extension. fp = os.path.join(self.test_dir.name, 'visualization..') obs_fp = visualization.save(fp, '.txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.txt') # Multiple periods in filename; multiple periods in extension. fp = os.path.join(self.test_dir.name, 'visualization..') obs_fp = visualization.save(fp, '..txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.txt') # No extension in filename; no extension input. fp = os.path.join(self.test_dir.name, 'visualization') obs_fp = visualization.save(fp) obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.qzv') # No extension in filename; different extension input. fp = os.path.join(self.test_dir.name, 'visualization') obs_fp = visualization.save(fp, '.txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.txt') # No extension in filename; default extension input. fp = os.path.join(self.test_dir.name, 'visualization') obs_fp = visualization.save(fp, '.qzv') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.qzv') # Different extension in filename; no extension input. fp = os.path.join(self.test_dir.name, 'visualization.zip') obs_fp = visualization.save(fp) obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.zip.qzv') # Different extension in filename; # Different extension input (non-matching). fp = os.path.join(self.test_dir.name, 'visualization.zip') obs_fp = visualization.save(fp, '.txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.zip.txt') # Different extension in filename; # Different extension input (matching). fp = os.path.join(self.test_dir.name, 'visualization.zip') obs_fp = visualization.save(fp, '.zip') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.zip') # Different extension in filename; default extension input. fp = os.path.join(self.test_dir.name, 'visualization.zip') obs_fp = visualization.save(fp, '.qzv') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.zip.qzv') # Default extension in filename; no extension input. fp = os.path.join(self.test_dir.name, 'visualization.qzv') obs_fp = visualization.save(fp) obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.qzv') # Default extension in filename; different extension input. fp = os.path.join(self.test_dir.name, 'visualization.qzv') obs_fp = visualization.save(fp, '.txt') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.qzv.txt') # Default extension in filename; default extension input. fp = os.path.join(self.test_dir.name, 'visualization.qzv') obs_fp = visualization.save(fp, '.qzv') obs_filename = os.path.basename(obs_fp) self.assertEqual(obs_filename, 'visualization.qzv') def test_import_data_single_dirfmt_to_single_dirfmt(self): temp_data_dir = os.path.join(self.test_dir.name, 'import') os.mkdir(temp_data_dir) with open(os.path.join(temp_data_dir, 'ints.txt'), 'w') as fh: fh.write("1\n2\n3\n") qiime2.Artifact.import_data('IntSequence2', temp_data_dir, view_type="IntSequenceDirectoryFormat") def test_artifact_has_metadata_true(self): A = Artifact.import_data('Mapping', {'a': '1', 'b': '2'}) self.assertTrue(A.has_metadata()) def test_artifact_has_metadata_false(self): A = Artifact.import_data('IntSequence1', [1, 2, 3, 4]) self.assertFalse(A.has_metadata()) def test_validate_artifact_good(self): artifact = Artifact.import_data('IntSequence1', [1, 2, 3, 4]) artifact.validate() self.assertTrue(True) # Checkpoint def test_validate_artifact_bad(self): artifact = Artifact.import_data('IntSequence1', [1, 2, 3, 4]) with (artifact._archiver.root_dir / 'extra.file').open('w') as fh: fh.write('uh oh') with self.assertRaisesRegex(exceptions.ValidationError, r'extra\.file'): artifact.validate() def test_validate_vizualization_good(self): visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) visualization.validate() self.assertTrue(True) # Checkpoint def test_validate_vizualization_bad(self): visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) with (visualization._archiver.root_dir / 'extra.file').open('w') as fh: fh.write('uh oh') with self.assertRaisesRegex(exceptions.ValidationError, r'extra\.file'): visualization.validate() if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/sdk/tests/test_results.py000066400000000000000000000116331412141660100212730ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import pickle import unittest from qiime2.sdk import Results class TestResults(unittest.TestCase): def test_tuple_subclass(self): self.assertTrue(issubclass(Results, tuple)) self.assertIsInstance(Results(['a', 'b'], [42, 43]), tuple) def test_tuple_cast(self): r = Results(['a', 'b'], [42, 43]) t = tuple(r) self.assertIs(type(t), tuple) self.assertEqual(t, (42, 43)) def test_callable_return_and_unpacking(self): def f(): return Results(['a', 'b'], [42, 43]) a, b = f() self.assertEqual(a, 42) self.assertEqual(b, 43) def test_constructor_iterable(self): r = Results(iter(['a', 'b']), iter([42, 43])) self.assertEqual(tuple(r), (42, 43)) self.assertEqual(r._fields, ('a', 'b')) def test_constructor_len_mismatch(self): with self.assertRaises(ValueError): Results(['a', 'b'], [42]) def test_pickle(self): r = Results(['a', 'b'], [42, 'abc']) pickled = pickle.dumps(r) unpickled = pickle.loads(pickled) self.assertEqual(unpickled, r) def test_field_attributes(self): r = Results(['foo', 'bar'], [42, 'abc']) self.assertEqual(r.foo, 42) self.assertEqual(r.bar, 'abc') with self.assertRaises(AttributeError): r.baz def test_per_instance_field_attributes(self): # Field attributes are added to a `Results` instance, not the type. r1 = Results(['foo', 'bar'], [42, 'abc']) r2 = Results(['x'], [42.0]) for attr in 'foo', 'bar', 'x': self.assertFalse(hasattr(Results, attr)) self.assertTrue(hasattr(r1, 'foo')) self.assertTrue(hasattr(r1, 'bar')) self.assertFalse(hasattr(r1, 'x')) self.assertFalse(hasattr(r2, 'foo')) self.assertFalse(hasattr(r2, 'bar')) self.assertTrue(hasattr(r2, 'x')) def test_index_access(self): r = Results(['foo', 'bar'], [42, 'abc']) self.assertEqual(r[0], 42) self.assertEqual(r[1], 'abc') with self.assertRaises(IndexError): r[2] def test_immutability(self): r = Results(['foo', 'bar'], [42, 'abc']) # Setter for existing attribute. with self.assertRaises(AttributeError): r.bar = 999 # Setter for new attribute. with self.assertRaises(AttributeError): r.baz = 999 # Deleter for existing attribute. with self.assertRaises(AttributeError): del r.bar # Deleter for new attribute. with self.assertRaises(AttributeError): del r.baz with self.assertRaises(TypeError): r[0] = 999 def test_eq_same_obj(self): r = Results(['a', 'b'], [1, 2]) self.assertEqual(r, r) def test_eq_subclass(self): class ResultsSubclass(Results): pass r1 = Results(['foo'], ['abc']) r2 = ResultsSubclass(['foo'], ['abc']) self.assertEqual(r1, r2) def test_eq_different_source_types(self): r1 = Results(iter(['a', 'b']), iter([42, 43])) r2 = Results(['a', 'b'], [42, 43]) self.assertEqual(r1, r2) def test_eq_empty(self): r1 = Results([], []) r2 = Results([], []) self.assertEqual(r1, r2) def test_eq_nonempty(self): r1 = Results(['foo', 'bar'], ['abc', 'def']) r2 = Results(['foo', 'bar'], ['abc', 'def']) self.assertEqual(r1, r2) def test_ne_type(self): r1 = Results(['foo', 'bar'], ['abc', 'def']) r2 = ('abc', 'def') self.assertNotEqual(r1, r2) def test_ne_fields(self): r1 = Results(['foo', 'bar'], ['abc', 'def']) r2 = Results(['foo', 'baz'], ['abc', 'def']) self.assertNotEqual(r1, r2) def test_ne_values(self): r1 = Results(['foo', 'bar'], ['abc', 'def']) r2 = Results(['foo', 'bar'], ['abc', 'xyz']) self.assertNotEqual(r1, r2) def test_repr_empty(self): r = Results([], []) self.assertTrue(repr(r).startswith('Results')) self.assertTrue(repr(r).endswith('---')) def test_repr_single(self): r = Results(['a'], [42]) self.assertTrue(repr(r).startswith('Results')) self.assertTrue(repr(r).endswith('a = 42')) def test_repr_multiple(self): r = Results(['a', 'foo'], [42, 'abc']) self.assertTrue(repr(r).startswith('Results')) self.assertTrue('a = 42' in repr(r)) self.assertTrue(repr(r).endswith("foo = 'abc'")) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/sdk/tests/test_usage.py000066400000000000000000000435221412141660100207000ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest import tempfile from qiime2.core.type import signature from qiime2.core.testing.util import get_dummy_plugin from qiime2.core.testing.type import Mapping import qiime2.core.testing.examples as examples from qiime2.sdk import usage, action from qiime2 import plugin, Metadata, Artifact class TestCaseUsage(unittest.TestCase): def setUp(self): self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-') self.plugin = get_dummy_plugin() def tearDown(self): self.test_dir.cleanup() class TestUsage(TestCaseUsage): def test_basic(self): action = self.plugin.actions['concatenate_ints'] use = usage.DiagnosticUsage() action.examples['concatenate_ints_simple'](use) records = use._get_records() self.assertEqual(5, len(records)) obs1, obs2, obs3, obs4, obs5 = records.values() self.assertEqual('init_data', obs1.source) self.assertEqual('init_data', obs2.source) self.assertEqual('init_data', obs3.source) self.assertEqual('comment', obs4.source) self.assertEqual('action', obs5.source) self.assertTrue('basic usage' in obs4.result['text']) self.assertEqual('dummy_plugin', obs5.result['plugin_id']) self.assertEqual('concatenate_ints', obs5.result['action_id']) self.assertEqual({'int1': 4, 'int2': 2, 'ints1': {'ref': 'ints_a', 'source': 'init_data'}, 'ints2': {'ref': 'ints_b', 'source': 'init_data'}, 'ints3': {'ref': 'ints_c', 'source': 'init_data'}}, obs5.result['input_opts']) self.assertEqual({'concatenated_ints': 'ints_d'}, obs5.result['output_opts']) def test_chained(self): action = self.plugin.actions['concatenate_ints'] use = usage.DiagnosticUsage() action.examples['concatenate_ints_complex'](use) records = use._get_records() self.assertEqual(7, len(records)) obs1, obs2, obs3, obs4, obs5, obs6, obs7 = records.values() self.assertEqual('init_data', obs1.source) self.assertEqual('init_data', obs2.source) self.assertEqual('init_data', obs3.source) self.assertEqual('comment', obs4.source) self.assertEqual('action', obs5.source) self.assertEqual('comment', obs6.source) self.assertEqual('action', obs7.source) self.assertTrue('chained usage (pt 1)' in obs4.result['text']) self.assertEqual('dummy_plugin', obs5.result['plugin_id']) self.assertEqual('concatenate_ints', obs5.result['action_id']) self.assertEqual({'int1': 4, 'int2': 2, 'ints1': {'ref': 'ints_a', 'source': 'init_data'}, 'ints2': {'ref': 'ints_b', 'source': 'init_data'}, 'ints3': {'ref': 'ints_c', 'source': 'init_data'}}, obs5.result['input_opts']) self.assertEqual({'concatenated_ints': 'ints_d'}, obs5.result['output_opts']) self.assertTrue('chained usage (pt 2)' in obs6.result['text']) self.assertEqual('dummy_plugin', obs7.result['plugin_id']) self.assertEqual('concatenate_ints', obs7.result['action_id']) exp7 = {'int1': 41, 'int2': 0, 'ints1': {'action_id': 'concatenate_ints', 'input_opts': {'int1': 4, 'int2': 2, 'ints1': {'ref': 'ints_a', 'source': 'init_data'}, 'ints2': {'ref': 'ints_b', 'source': 'init_data'}, 'ints3': {'ref': 'ints_c', 'source': 'init_data'}}, 'output_opt': 'concatenated_ints', 'output_opts': {'concatenated_ints': 'ints_d'}, 'plugin_id': 'dummy_plugin', 'source': 'action'}, 'ints2': {'ref': 'ints_b', 'source': 'init_data'}, 'ints3': {'ref': 'ints_c', 'source': 'init_data'}} self.assertEqual(exp7, obs7.result['input_opts']) self.assertEqual({'concatenated_ints': 'concatenated_ints'}, obs7.result['output_opts']) def test_comments_only(self): action = self.plugin.actions['concatenate_ints'] use = usage.DiagnosticUsage() action.examples['comments_only'](use) records = use._get_records() self.assertEqual(2, len(records)) obs1, obs2 = records.values() self.assertEqual('comment', obs1.source) self.assertEqual('comment', obs2.source) self.assertEqual('comment 1', obs1.result['text']) self.assertEqual('comment 2', obs2.result['text']) def test_metadata_merging(self): action = self.plugin.actions['identity_with_metadata'] use = usage.DiagnosticUsage() action.examples['identity_with_metadata_merging'](use) records = use._get_records() self.assertEqual(5, len(records)) obs1, obs2, obs3, obs4, obs5 = records.values() self.assertEqual('init_data', obs1.source) self.assertEqual('init_metadata', obs2.source) self.assertEqual('init_metadata', obs3.source) self.assertEqual('merge_metadata', obs4.source) self.assertEqual('action', obs5.source) def test_get_metadata_column(self): action = self.plugin.actions['identity_with_metadata_column'] use = usage.DiagnosticUsage() action.examples['identity_with_metadata_column_get_mdc'](use) records = use._get_records() self.assertEqual(4, len(records)) obs1, obs2, obs3, obs4 = records.values() self.assertEqual('init_data', obs1.source) self.assertEqual('init_metadata', obs2.source) self.assertEqual('get_metadata_column', obs3.source) self.assertEqual('action', obs4.source) def test_use_init_collection_data(self): action = self.plugin.actions['variadic_input_method'] use = usage.DiagnosticUsage() action.examples['variadic_input_simple'](use) records = use._get_records() self.assertEqual(7, len(records)) obs1, obs2, obs3, obs4, obs5, obs6, obs7 = records.values() self.assertEqual('init_data', obs1.source) self.assertEqual('init_data', obs2.source) self.assertEqual('init_data_collection', obs3.source) self.assertEqual('init_data', obs4.source) self.assertEqual('init_data', obs5.source) self.assertEqual('init_data_collection', obs6.source) self.assertEqual('action', obs7.source) self.assertEqual(set, type(obs7.result['input_opts']['nums'])) self.assertEqual('ints', obs7.result['input_opts']['ints'][0]['ref']) self.assertEqual('int_set', obs7.result['input_opts']['int_set'][0]['ref']) def test_optional_inputs(self): action = self.plugin.actions['optional_artifacts_method'] use = usage.DiagnosticUsage() records = use._get_records() action.examples['optional_inputs'](use) self.assertEqual(3, len(records)) obs1, obs2, obs3 = records.values() self.assertEqual('init_data', obs1.source) self.assertEqual('action', obs2.source) self.assertEqual('action', obs3.source) class TestUsageAction(TestCaseUsage): def test_successful_init(self): obs = usage.UsageAction(plugin_id='foo', action_id='bar') self.assertEqual('foo', obs.plugin_id) self.assertEqual('bar', obs.action_id) def test_invalid_plugin_id(self): with self.assertRaisesRegex(ValueError, 'specify a value for plugin_id'): usage.UsageAction(plugin_id='', action_id='bar') def test_invalid_action_id(self): with self.assertRaisesRegex(ValueError, 'specify a value for action_id'): usage.UsageAction(plugin_id='foo', action_id='') def test_successful_get_action(self): ua = usage.UsageAction( plugin_id='dummy_plugin', action_id='concatenate_ints') obs_action_f, obs_sig = ua.get_action() self.assertTrue(isinstance(obs_action_f, action.Method)) self.assertTrue(isinstance(obs_sig, signature.MethodSignature)) def test_unknown_action_get_action(self): ua = usage.UsageAction( plugin_id='dummy_plugin', action_id='concatenate_spleens') with self.assertRaisesRegex(KeyError, 'No action.*concatenate_spleens'): ua.get_action() def test_validate_invalid_inputs(self): ua = usage.UsageAction( plugin_id='dummy_plugin', action_id='concatenate_ints') with self.assertRaisesRegex(TypeError, 'instance of UsageInputs'): ua.validate({}, usage.UsageOutputNames()) def test_validate_invalid_outputs(self): ua = usage.UsageAction( plugin_id='dummy_plugin', action_id='concatenate_ints') with self.assertRaisesRegex(TypeError, 'instance of UsageOutputNames'): ua.validate(usage.UsageInputs(), {}) class TestUsageInputs(TestCaseUsage): def setUp(self): super().setUp() def foo(x: dict, z: str, optional_input: dict = None, optional_param: str = None) -> dict: return x self.signature = signature.MethodSignature( foo, inputs={'x': Mapping, 'optional_input': Mapping}, parameters={'z': plugin.Str, 'optional_param': plugin.Str}, outputs=[('y', Mapping)], ) def test_successful_init(self): obs = usage.UsageInputs(foo='bar') self.assertEqual(['foo'], list(obs.values.keys())) self.assertEqual(['bar'], list(obs.values.values())) def test_validate_missing_required_input(self): ui = usage.UsageInputs(y='hello', optional_input='a', optional_param='b') with self.assertRaisesRegex(ValueError, 'Missing input.*x'): ui.validate(self.signature) def test_validate_missing_required_parameter(self): ui = usage.UsageInputs(x='hello', optional_input='a', optional_param='b') with self.assertRaisesRegex(ValueError, 'Missing parameter.*z'): ui.validate(self.signature) def test_validate_extra_values(self): ui = usage.UsageInputs(x='hello', z='goodbye', foo=True, optional_input='a', optional_param='b') with self.assertRaisesRegex(ValueError, 'Extra input.*parameter.*foo'): ui.validate(self.signature) def test_validate_missing_optional_input(self): ui = usage.UsageInputs(x='hello', z='goodbye', optional_param='a') ui.validate(self.signature) self.assertTrue(True) def test_validate_missing_optional_parameter(self): ui = usage.UsageInputs(x='hello', z='goodbye', optional_input='a') ui.validate(self.signature) self.assertTrue(True) def test_type_of_input(self): test_inputs = usage.UsageInputs(x=[1, 2, 3], z={7, 8, 9}) test_inputs.validate(self.signature) self.assertIsInstance(test_inputs.values['x'], list) self.assertIsInstance(test_inputs.values['z'], set) class TestUsageOutputNames(TestCaseUsage): def setUp(self): super().setUp() def foo(x: dict, z: str, optional: str = None) -> (dict, dict): return x self.signature = signature.MethodSignature( foo, inputs={'x': Mapping}, parameters={'z': plugin.Str, 'optional': plugin.Str}, outputs=[('y', Mapping), ('a', Mapping)], ) def test_successful_init(self): obs = usage.UsageOutputNames(foo='bar') self.assertEqual(['foo'], list(obs.values.keys())) self.assertEqual(['bar'], list(obs.values.values())) def test_invalid_init(self): with self.assertRaisesRegex(TypeError, 'key.*foo.*string, not.*bool'): usage.UsageOutputNames(foo=True) def test_validate_missing_output(self): uo = usage.UsageOutputNames(y='hello') with self.assertRaisesRegex(ValueError, 'Missing output.*a'): uo.validate(self.signature) def test_validate_extra_output(self): uo = usage.UsageOutputNames(y='goodbye', a='hello', peanut='noeyes') with self.assertRaisesRegex(ValueError, 'Extra output.*peanut'): uo.validate(self.signature) def test_validate_derived_missing_output(self): uo = usage.UsageOutputNames(x='goodbye', y='hello') with self.assertRaisesRegex(ValueError, 'SDK.*missing output.*y'): uo.validate_computed({'x': 'val'}) def test_validate_derived_extra_output(self): uo = usage.UsageOutputNames(x='goodbye', y='hello') with self.assertRaisesRegex(ValueError, 'SDK.*extra output.*peanut'): uo.validate_computed({'x': 'val', 'y': 'val', 'peanut': 'val'}) class TestUsageBaseClass(TestCaseUsage): def setUp(self): super().setUp() class Usage(usage.Usage): pass self.Usage = Usage def test_get_result_invalid(self): use = self.Usage() with self.assertRaisesRegex(KeyError, 'No record with ref id: "peanut"'): use.get_result('peanut') def test_action_invalid_action_provided(self): use = self.Usage() with self.assertRaisesRegex(TypeError, 'provide.*UsageAction'): use.action({}, {}, {}) def test_merge_metadata_one_input(self): use = self.Usage() with self.assertRaisesRegex(ValueError, 'two or more'): use.merge_metadata('foo') class TestScopeRecord(TestCaseUsage): def test_invalid_assert_has_line_matching(self): with self.assertRaisesRegex(TypeError, 'should be a `callable`'): usage.ScopeRecord('foo', 'value', 'source', assert_has_line_matching='spleen') class TestExecutionUsage(TestCaseUsage): def test_init_data(self): use = usage.ExecutionUsage() with self.assertRaisesRegex(ValueError, 'expected an Artifact'): use.init_data('name', lambda: object) with self.assertRaisesRegex(ValueError, 'not all .* Artifacts'): use.init_data('name', lambda: [object]) with self.assertRaisesRegex(TypeError, 'expected Metadata'): use.init_metadata('name', lambda: object) with self.assertRaisesRegex(ValueError, 'expected a ScopeRecord.'): use.init_data_collection('', list, object) with self.assertRaisesRegex(ValueError, 'expected a ScopeRecord.'): use.init_data_collection('', list, usage.ScopeRecord('', object, ''), object) def test_merge_metadata(self): use = usage.ExecutionUsage() md1 = use.init_metadata('md1', examples.md1_factory) md2 = use.init_metadata('md2', examples.md2_factory) merged = use.merge_metadata('md3', md1, md2) self.assertIsInstance(merged.result, Metadata) def test_variadic_input_simple(self): use = usage.ExecutionUsage() action = self.plugin.actions['variadic_input_method'] action.examples['variadic_input_simple'](use) ints_a = use._get_record('ints_a') ints_b = use._get_record('ints_b') ints = use._get_record('ints') single_int1 = use._get_record('single_int1') single_int2 = use._get_record('single_int2') int_set = use._get_record('int_set') out = use._get_record('out') self.assertIsInstance(ints_a.result, Artifact) self.assertIsInstance(ints_b.result, Artifact) self.assertIsInstance(ints.result, list) self.assertEqual(ints.result[0], ints_a.result) self.assertEqual(ints.result[1], ints_b.result) self.assertIsInstance(single_int1.result, Artifact) self.assertIsInstance(single_int2.result, Artifact) self.assertIsInstance(int_set.result, set) self.assertIn(single_int1.result, int_set.result) self.assertIn(single_int2.result, int_set.result) self.assertIsInstance(out.result, Artifact) def test_variadic_input_simple_async(self): use = usage.ExecutionUsage(asynchronous=True) action = self.plugin.actions['variadic_input_method'] action.examples['variadic_input_simple'](use) ints_a = use._get_record('ints_a') ints_b = use._get_record('ints_b') ints = use._get_record('ints') single_int1 = use._get_record('single_int1') single_int2 = use._get_record('single_int2') int_set = use._get_record('int_set') out = use._get_record('out') self.assertIsInstance(ints_a.result, Artifact) self.assertIsInstance(ints_b.result, Artifact) self.assertIsInstance(ints.result, list) self.assertEqual(ints.result[0], ints_a.result) self.assertEqual(ints.result[1], ints_b.result) self.assertIsInstance(single_int1.result, Artifact) self.assertIsInstance(single_int2.result, Artifact) self.assertIsInstance(int_set.result, set) self.assertIn(single_int1.result, int_set.result) self.assertIn(single_int2.result, int_set.result) self.assertIsInstance(out.result, Artifact) qiime-2021.8.0/qiime2/sdk/tests/test_util.py000066400000000000000000000037001412141660100205430ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import unittest import qiime2 import qiime2.sdk class TestUtil(unittest.TestCase): def test_artifact_actions(self): obs = qiime2.sdk.util.actions_by_input_type(None) self.assertEqual(obs, []) # For simplicity, we are gonna test the names of the plugin and # the actions obs = [(x.name, [yy.name for yy in y]) for x, y in qiime2.sdk.util.actions_by_input_type('SingleInt')] exp = [('dummy-plugin', [ 'Do stuff normally, but override this one step sometimes'])] self.assertEqual(obs, exp) obs = [(x.name, [yy.name for yy in y]) for x, y in qiime2.sdk.util.actions_by_input_type( 'Kennel[Cat]')] self.assertEqual(obs, []) obs = [(x.name, [yy.name for yy in y]) for x, y in qiime2.sdk.util.actions_by_input_type( 'IntSequence1')] exp = [('dummy-plugin', [ 'A typical pipeline with the potential to raise an error', 'Concatenate integers', 'Identity', 'Identity', 'Identity', 'Do a great many things', 'Identity', 'Identity', 'Identity', 'Visualize most common integers', 'Split sequence of integers in half', 'Test different ways of failing', 'Optional artifacts method', 'Do stuff normally, but override this one step sometimes', 'TypeMatch with list and set params'])] self.assertEqual(len(obs), 1) self.assertEqual(obs[0][0], exp[0][0]) self.assertCountEqual(obs[0][1], exp[0][1]) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/sdk/tests/test_visualization.py000066400000000000000000000356211412141660100224760ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import tempfile import unittest import uuid import collections import pathlib import qiime2.core.type from qiime2.sdk import Visualization from qiime2.sdk.result import ResultMetadata import qiime2.core.archive as archive from qiime2.core.testing.visualizer import ( mapping_viz, most_common_viz, multi_html_viz) from qiime2.core.testing.util import ArchiveTestingMixin class TestVisualization(unittest.TestCase, ArchiveTestingMixin): def make_provenance_capture(self): # You can't actually import a visualization, but I won't tell # visualization if you don't... return archive.ImportProvenanceCapture() def setUp(self): # TODO standardize temporary directories created by QIIME 2 self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-') # Using `mapping_viz` because it produces multiple files, including a # nested directory. self.data_dir = os.path.join(self.test_dir.name, 'viz-output') os.mkdir(self.data_dir) mapping_viz(self.data_dir, {'abc': 'foo', 'def': 'bar'}, {'ghi': 'baz', 'jkl': 'bazz'}, key_label='Key', value_label='Value') def tearDown(self): self.test_dir.cleanup() def test_private_constructor(self): with self.assertRaisesRegex( NotImplementedError, 'Visualization constructor.*private.*Visualization.load'): Visualization() # Note on testing strategy below: many of the tests for `_from_data_dir` # and `load` are similar, with the exception that when `load`ing, the # visualization's UUID is known so more specific assertions can be # performed. While these tests appear somewhat redundant, they are # important because they exercise the same operations on Visualization # objects constructed from different sources, whose codepaths have very # different internal behavior. This internal behavior could be tested # explicitly but it is safer to test the public API behavior (e.g. as a # user would interact with the object) in case the internals change. def test_from_data_dir(self): visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) self.assertEqual(visualization.type, qiime2.core.type.Visualization) self.assertIsInstance(visualization.uuid, uuid.UUID) def test_from_data_dir_and_save(self): fp = os.path.join(self.test_dir.name, 'visualization.qzv') visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) visualization.save(fp) root_dir = str(visualization.uuid) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/index.html', 'data/css/style.css', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertArchiveMembers(fp, root_dir, expected) def test_load(self): saved_visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) fp = os.path.join(self.test_dir.name, 'visualization.qzv') saved_visualization.save(fp) visualization = Visualization.load(fp) self.assertEqual(visualization.type, qiime2.core.type.Visualization) self.assertEqual(visualization.uuid, saved_visualization.uuid) def test_load_and_save(self): fp1 = os.path.join(self.test_dir.name, 'visualization1.qzv') fp2 = os.path.join(self.test_dir.name, 'visualization2.qzv') visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) visualization.save(fp1) visualization = Visualization.load(fp1) # Overwriting its source file works. visualization.save(fp1) # Saving to a new file works. visualization.save(fp2) root_dir = str(visualization.uuid) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/index.html', 'data/css/style.css', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertArchiveMembers(fp1, root_dir, expected) root_dir = str(visualization.uuid) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/index.html', 'data/css/style.css', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertArchiveMembers(fp2, root_dir, expected) def test_roundtrip(self): fp1 = os.path.join(self.test_dir.name, 'visualization1.qzv') fp2 = os.path.join(self.test_dir.name, 'visualization2.qzv') visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) visualization.save(fp1) visualization1 = Visualization.load(fp1) visualization1.save(fp2) visualization2 = Visualization.load(fp2) self.assertEqual(visualization1.type, visualization2.type) self.assertEqual(visualization1.uuid, visualization2.uuid) def test_load_with_archive_filepath_modified(self): # Save a visualization for use in the following test case. fp = os.path.join(self.test_dir.name, 'visualization.qzv') Visualization._from_data_dir(self.data_dir, self.make_provenance_capture()).save(fp) # Load the visualization from a filepath then save a different # visualization to the same filepath. Assert that both visualizations # access the correct data. # # `load` used to be lazy, only extracting data when it needed to (e.g. # when `save` or `get_index_paths` was called). This was buggy as the # filepath could have been deleted, or worse, modified to contain a # different .qzv file. Thus, the wrong archive could be extracted on # demand, or the archive could be missing altogether. There isn't an # easy cross-platform compatible way to solve this problem, so # Visualization.load is no longer lazy and always extracts its data # immediately. The real motivation for lazy loading was for quick # inspection of archives without extracting/copying data, so that API # is now provided through Visualization.peek. visualization1 = Visualization.load(fp) new_data_dir = os.path.join(self.test_dir.name, 'viz-output2') os.mkdir(new_data_dir) most_common_viz(new_data_dir, collections.Counter(range(42))) Visualization._from_data_dir(new_data_dir, self.make_provenance_capture()).save(fp) visualization2 = Visualization.load(fp) self.assertEqual(visualization1.get_index_paths(), {'html': 'data/index.html'}) self.assertEqual(visualization2.get_index_paths(), {'html': 'data/index.html', 'tsv': 'data/index.tsv'}) def test_extract(self): fp = os.path.join(self.test_dir.name, 'visualization.qzv') visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) visualization.save(fp) root_dir = str(visualization.uuid) # pathlib normalizes away the `.`, it doesn't matter, but this is the # implementation we're using, so let's test against that assumption. output_dir = pathlib.Path(self.test_dir.name) / 'viz-extract-test' result_dir = Visualization.extract(fp, output_dir=output_dir) self.assertEqual(result_dir, str(output_dir / root_dir)) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/index.html', 'data/css/style.css', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertExtractedArchiveMembers(output_dir, root_dir, expected) def test_get_index_paths_single_load(self): fp = os.path.join(self.test_dir.name, 'visualization.qzv') visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) visualization.save(fp) visualization = Visualization.load(fp) actual = visualization.get_index_paths() expected = {'html': 'data/index.html'} self.assertEqual(actual, expected) def test_get_index_paths_single_from_data_dir(self): visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) actual = visualization.get_index_paths() expected = {'html': 'data/index.html'} self.assertEqual(actual, expected) def test_get_index_paths_multiple_load(self): data_dir = os.path.join(self.test_dir.name, 'mc-viz-output1') os.mkdir(data_dir) most_common_viz(data_dir, collections.Counter(range(42))) fp = os.path.join(self.test_dir.name, 'visualization.qzv') visualization = Visualization._from_data_dir( data_dir, self.make_provenance_capture()) visualization.save(fp) visualization = Visualization.load(fp) actual = visualization.get_index_paths() expected = {'html': 'data/index.html', 'tsv': 'data/index.tsv'} self.assertEqual(actual, expected) def test_get_index_paths_multiple_from_data_dir(self): data_dir = os.path.join(self.test_dir.name, 'mc-viz-output2') os.mkdir(data_dir) most_common_viz(data_dir, collections.Counter(range(42))) visualization = Visualization._from_data_dir( data_dir, self.make_provenance_capture()) actual = visualization.get_index_paths() expected = {'html': 'data/index.html', 'tsv': 'data/index.tsv'} self.assertEqual(actual, expected) def test_get_index_paths_multiple_html_load(self): data_dir = os.path.join(self.test_dir.name, 'multi-html-viz1') os.mkdir(data_dir) multi_html_viz(data_dir, [1, 42]) fp = os.path.join(self.test_dir.name, 'visualization.qzv') visualization = Visualization._from_data_dir( data_dir, self.make_provenance_capture()) visualization.save(fp) visualization = Visualization.load(fp) with self.assertRaises(ValueError): visualization.get_index_paths() def test_get_index_paths_multiple_html_from_data_dir(self): data_dir = os.path.join(self.test_dir.name, 'multi-html-viz2') os.mkdir(data_dir) multi_html_viz(data_dir, [1, 42]) visualization = Visualization._from_data_dir( data_dir, self.make_provenance_capture()) with self.assertRaises(ValueError): visualization.get_index_paths() def test_get_index_paths_relative_false(self): data_dir = os.path.join(self.test_dir.name, 'mc-viz-output2') os.mkdir(data_dir) most_common_viz(data_dir, collections.Counter(range(42))) visualization = Visualization._from_data_dir( data_dir, self.make_provenance_capture()) def get_abs_path(rel): return str(visualization._archiver.root_dir / rel) actual = visualization.get_index_paths(relative=False) expected = {'html': get_abs_path('data/index.html'), 'tsv': get_abs_path('data/index.tsv')} self.assertEqual(actual, expected) def test_peek(self): visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) fp = os.path.join(self.test_dir.name, 'visualization.qzv') visualization.save(fp) metadata = Visualization.peek(fp) self.assertIsInstance(metadata, ResultMetadata) self.assertEqual(metadata.type, 'Visualization') self.assertEqual(metadata.uuid, str(visualization.uuid)) self.assertIsNone(metadata.format) def test_eq_identity(self): visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) self.assertEqual(visualization, visualization) def test_eq_same_uuid(self): fp = os.path.join(self.test_dir.name, 'visualization.qzv') visualization1 = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) visualization1.save(fp) visualization2 = Visualization.load(fp) self.assertEqual(visualization1, visualization2) def test_ne_same_data_different_uuid(self): visualization1 = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) visualization2 = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) self.assertNotEqual(visualization1, visualization2) def test_ne_different_data_different_uuid(self): visualization1 = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) data_dir = os.path.join(self.test_dir.name, 'mc-viz-output1') os.mkdir(data_dir) most_common_viz(data_dir, collections.Counter(range(42))) visualization2 = Visualization._from_data_dir( data_dir, self.make_provenance_capture()) self.assertNotEqual(visualization1, visualization2) def test_ne_subclass_same_uuid(self): class VisualizationSubclass(Visualization): pass fp = os.path.join(self.test_dir.name, 'visualization.qzv') visualization1 = VisualizationSubclass._from_data_dir( self.data_dir, self.make_provenance_capture()) visualization1.save(fp) visualization2 = Visualization.load(fp) self.assertNotEqual(visualization1, visualization2) self.assertNotEqual(visualization2, visualization1) def test_ne_different_type_same_uuid(self): visualization = Visualization._from_data_dir( self.data_dir, self.make_provenance_capture()) class Faker: @property def uuid(self): return visualization.uuid faker = Faker() self.assertNotEqual(visualization, faker) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/sdk/tests/test_visualizer.py000066400000000000000000000444021412141660100217670ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import concurrent.futures import inspect import os.path import tempfile import unittest import uuid import qiime2.plugin import qiime2.core.type from qiime2.core.type import VisualizerSignature, Str, Range from qiime2.core.type.visualization import Visualization as VisualizationType from qiime2.sdk import Artifact, Visualization, Visualizer, Results from qiime2.core.testing.visualizer import (most_common_viz, mapping_viz, params_only_viz, no_input_viz) from qiime2.core.testing.type import IntSequence1, IntSequence2, Mapping from qiime2.core.testing.util import get_dummy_plugin, ArchiveTestingMixin class TestVisualizer(unittest.TestCase, ArchiveTestingMixin): def setUp(self): # TODO standardize temporary directories created by QIIME 2 self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-') self.plugin = get_dummy_plugin() def tearDown(self): self.test_dir.cleanup() def test_private_constructor(self): with self.assertRaisesRegex(NotImplementedError, 'Visualizer constructor.*private'): Visualizer() def test_from_function_with_artifacts_and_parameters(self): visualizer = self.plugin.visualizers['mapping_viz'] self.assertEqual(visualizer.id, 'mapping_viz') exp_sig = VisualizerSignature( mapping_viz, inputs={ 'mapping1': Mapping, 'mapping2': Mapping }, parameters={ 'key_label': qiime2.plugin.Str, 'value_label': qiime2.plugin.Str }, ) self.assertEqual(visualizer.signature, exp_sig) self.assertEqual(visualizer.name, 'Visualize two mappings') self.assertTrue( visualizer.description.startswith('This visualizer produces an ' 'HTML visualization')) self.assertTrue( visualizer.source.startswith('\n```python\ndef mapping_viz(')) def test_from_function_without_parameters(self): visualizer = self.plugin.visualizers['most_common_viz'] self.assertEqual(visualizer.id, 'most_common_viz') exp_sig = VisualizerSignature( most_common_viz, inputs={ 'ints': IntSequence1 | IntSequence2 }, parameters={} ) self.assertEqual(visualizer.signature, exp_sig) self.assertEqual(visualizer.name, 'Visualize most common integers') self.assertTrue( visualizer.description.startswith('This visualizer produces HTML ' 'and TSV')) self.assertTrue( visualizer.source.startswith('\n```python\ndef most_common_viz(')) def test_from_function_with_parameters_only(self): visualizer = self.plugin.visualizers['params_only_viz'] self.assertEqual(visualizer.id, 'params_only_viz') exp_sig = VisualizerSignature( params_only_viz, inputs={}, parameters={ 'name': qiime2.plugin.Str, 'age': qiime2.plugin.Int % Range(0, None) } ) self.assertEqual(visualizer.signature, exp_sig) self.assertEqual(visualizer.name, 'Parameters only viz') self.assertTrue( visualizer.description.startswith('This visualizer only accepts ' 'parameters.')) self.assertTrue( visualizer.source.startswith('\n```python\ndef params_only_viz(')) def test_from_function_without_inputs_or_parameters(self): visualizer = self.plugin.visualizers['no_input_viz'] self.assertEqual(visualizer.id, 'no_input_viz') exp_sig = VisualizerSignature( no_input_viz, inputs={}, parameters={} ) self.assertEqual(visualizer.signature, exp_sig) self.assertEqual(visualizer.name, 'No input viz') self.assertTrue( visualizer.description.startswith('This visualizer does not ' 'accept any')) self.assertTrue( visualizer.source.startswith('\n```python\ndef no_input_viz(')) def test_is_callable(self): self.assertTrue(callable(self.plugin.visualizers['mapping_viz'])) self.assertTrue(callable(self.plugin.visualizers['most_common_viz'])) def test_callable_properties(self): mapping_viz = self.plugin.visualizers['mapping_viz'] most_common_viz = self.plugin.visualizers['most_common_viz'] mapping_exp = { 'mapping1': Mapping, 'return': (VisualizationType,), 'key_label': Str, 'mapping2': Mapping, 'value_label': Str} most_common_exp = { 'ints': IntSequence1 | IntSequence2, 'return': (VisualizationType,)} mapper = { mapping_viz: mapping_exp, most_common_viz: most_common_exp} for visualizer, exp in mapper.items(): self.assertEqual(visualizer.__call__.__name__, '__call__') self.assertEqual(visualizer.__call__.__annotations__, exp) self.assertFalse(hasattr(visualizer.__call__, '__wrapped__')) def test_async_properties(self): mapping_viz = self.plugin.visualizers['mapping_viz'] most_common_viz = self.plugin.visualizers['most_common_viz'] mapping_exp = { 'mapping1': Mapping, 'return': (VisualizationType,), 'key_label': Str, 'mapping2': Mapping, 'value_label': Str} most_common_exp = { 'ints': IntSequence1 | IntSequence2, 'return': (VisualizationType,)} mapper = { mapping_viz: mapping_exp, most_common_viz: most_common_exp} for visualizer, exp in mapper.items(): self.assertEqual(visualizer.asynchronous.__name__, 'asynchronous') self.assertEqual(visualizer.asynchronous.__annotations__, exp) self.assertFalse(hasattr(visualizer.asynchronous, '__wrapped__')) def test_callable_and_async_signature(self): mapping_viz = self.plugin.visualizers['mapping_viz'] for callable_attr in '__call__', 'asynchronous': signature = inspect.Signature.from_callable( getattr(mapping_viz, callable_attr)) parameters = list(signature.parameters.items()) kind = inspect.Parameter.POSITIONAL_OR_KEYWORD exp_parameters = [ ('mapping1', inspect.Parameter( 'mapping1', kind, annotation=Mapping)), ('mapping2', inspect.Parameter( 'mapping2', kind, annotation=Mapping)), ('key_label', inspect.Parameter( 'key_label', kind, annotation=Str)), ('value_label', inspect.Parameter( 'value_label', kind, annotation=Str)) ] self.assertEqual(parameters, exp_parameters) def test_callable_and_async_different_signature(self): # Test that a different Visualizer object has a different dynamic # signature. most_common_viz = self.plugin.visualizers['most_common_viz'] for callable_attr in '__call__', 'asynchronous': signature = inspect.Signature.from_callable( getattr(most_common_viz, callable_attr)) parameters = list(signature.parameters.items()) kind = inspect.Parameter.POSITIONAL_OR_KEYWORD exp_parameters = [ ('ints', inspect.Parameter( 'ints', kind, annotation=IntSequence1 | IntSequence2)) ] self.assertEqual(parameters, exp_parameters) def test_call_with_artifacts_and_parameters(self): mapping_viz = self.plugin.visualizers['mapping_viz'] artifact1 = Artifact.import_data(Mapping, {'foo': 'abc', 'bar': 'def'}) artifact2 = Artifact.import_data( Mapping, {'baz': 'abc', 'bazz': 'ghi'}) result = mapping_viz(artifact1, artifact2, 'Key', 'Value') # Test properties of the `Results` object. self.assertIsInstance(result, tuple) self.assertIsInstance(result, Results) self.assertEqual(len(result), 1) self.assertEqual(result.visualization, result[0]) result = result[0] self.assertIsInstance(result, Visualization) self.assertEqual(result.type, qiime2.core.type.Visualization) self.assertIsInstance(result.uuid, uuid.UUID) # TODO qiime2.sdk.Visualization doesn't have an API to access its # contents yet. For now, save and assert the correct files are present. filepath = os.path.join(self.test_dir.name, 'visualization.qzv') result.save(filepath) root_dir = str(result.uuid) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/index.html', 'data/css/style.css', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml', 'provenance/artifacts/%s/metadata.yaml' % artifact1.uuid, 'provenance/artifacts/%s/VERSION' % artifact1.uuid, 'provenance/artifacts/%s/citations.bib' % artifact1.uuid, 'provenance/artifacts/%s/action/action.yaml' % artifact1.uuid, 'provenance/artifacts/%s/metadata.yaml' % artifact2.uuid, 'provenance/artifacts/%s/VERSION' % artifact2.uuid, 'provenance/artifacts/%s/citations.bib' % artifact2.uuid, 'provenance/artifacts/%s/action/action.yaml' % artifact2.uuid } self.assertArchiveMembers(filepath, root_dir, expected) def test_call_with_no_parameters(self): most_common_viz = self.plugin.visualizers['most_common_viz'] artifact = Artifact.import_data( IntSequence1, [42, 42, 10, 0, 42, 5, 0]) result = most_common_viz(artifact) # Test properties of the `Results` object. self.assertIsInstance(result, tuple) self.assertIsInstance(result, Results) self.assertEqual(len(result), 1) self.assertEqual(result.visualization, result[0]) result = result[0] self.assertIsInstance(result, Visualization) self.assertEqual(result.type, qiime2.core.type.Visualization) self.assertIsInstance(result.uuid, uuid.UUID) # TODO qiime2.sdk.Visualization doesn't have an API to access its # contents yet. For now, save and assert the correct files are present. filepath = os.path.join(self.test_dir.name, 'visualization.qzv') result.save(filepath) root_dir = str(result.uuid) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/index.html', 'data/index.tsv', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml', 'provenance/artifacts/%s/metadata.yaml' % artifact.uuid, 'provenance/artifacts/%s/VERSION' % artifact.uuid, 'provenance/artifacts/%s/citations.bib' % artifact.uuid, 'provenance/artifacts/%s/action/action.yaml' % artifact.uuid } self.assertArchiveMembers(filepath, root_dir, expected) def test_call_with_parameters_only(self): params_only_viz = self.plugin.visualizers['params_only_viz'] # Parameters all have default values. result, = params_only_viz() self.assertIsInstance(result, Visualization) self.assertEqual(result.type, qiime2.core.type.Visualization) self.assertIsInstance(result.uuid, uuid.UUID) filepath = os.path.join(self.test_dir.name, 'visualization.qzv') result.save(filepath) root_dir = str(result.uuid) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/index.html', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertArchiveMembers(filepath, root_dir, expected) def test_call_without_inputs_or_parameters(self): no_input_viz = self.plugin.visualizers['no_input_viz'] result, = no_input_viz() self.assertIsInstance(result, Visualization) self.assertEqual(result.type, qiime2.core.type.Visualization) self.assertIsInstance(result.uuid, uuid.UUID) filepath = os.path.join(self.test_dir.name, 'visualization.qzv') result.save(filepath) root_dir = str(result.uuid) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/index.html', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml' } self.assertArchiveMembers(filepath, root_dir, expected) def test_asynchronous(self): mapping_viz = self.plugin.visualizers['mapping_viz'] artifact1 = Artifact.import_data(Mapping, {'foo': 'abc', 'bar': 'def'}) artifact2 = Artifact.import_data( Mapping, {'baz': 'abc', 'bazz': 'ghi'}) future = mapping_viz.asynchronous(artifact1, artifact2, 'Key', 'Value') self.assertIsInstance(future, concurrent.futures.Future) result = future.result() # Test properties of the `Results` object. self.assertIsInstance(result, tuple) self.assertIsInstance(result, Results) self.assertEqual(len(result), 1) self.assertEqual(result.visualization, result[0]) result = result[0] self.assertIsInstance(result, Visualization) self.assertEqual(result.type, qiime2.core.type.Visualization) self.assertIsInstance(result.uuid, uuid.UUID) # TODO qiime2.sdk.Visualization doesn't have an API to access its # contents yet. For now, save and assert the correct files are present. filepath = os.path.join(self.test_dir.name, 'visualization.qzv') result.save(filepath) root_dir = str(result.uuid) expected = { 'VERSION', 'checksums.md5', 'metadata.yaml', 'data/index.html', 'data/css/style.css', 'provenance/metadata.yaml', 'provenance/VERSION', 'provenance/citations.bib', 'provenance/action/action.yaml', 'provenance/artifacts/%s/metadata.yaml' % artifact1.uuid, 'provenance/artifacts/%s/VERSION' % artifact1.uuid, 'provenance/artifacts/%s/citations.bib' % artifact1.uuid, 'provenance/artifacts/%s/action/action.yaml' % artifact1.uuid, 'provenance/artifacts/%s/metadata.yaml' % artifact2.uuid, 'provenance/artifacts/%s/VERSION' % artifact2.uuid, 'provenance/artifacts/%s/citations.bib' % artifact2.uuid, 'provenance/artifacts/%s/action/action.yaml' % artifact2.uuid } self.assertArchiveMembers(filepath, root_dir, expected) def test_visualizer_callable_output(self): artifact = Artifact.import_data(Mapping, {'foo': 'abc', 'bar': 'def'}) # Callable returns a value from `return_vals` return_vals = (True, False, [], {}, '', 0, 0.0) for return_val in return_vals: def func(output_dir: str, foo: dict) -> None: return return_val self.plugin.visualizers.register_function( func, {'foo': Mapping}, {}, '', '' ) visualizer = self.plugin.visualizers['func'] with self.assertRaisesRegex(TypeError, "should not return"): visualizer(foo=artifact) # Callable returns None (default function return) def func(output_dir: str, foo: dict) -> None: return None self.plugin.visualizers.register_function( func, {'foo': Mapping}, {}, '', '' ) visualizer = self.plugin.visualizers['func'] # Should not raise an exception output = visualizer(foo=artifact) self.assertIsInstance(output, Results) self.assertIsInstance(output.visualization, Visualization) def test_docstring(self): mapping_viz = self.plugin.visualizers['mapping_viz'] common_viz = self.plugin.visualizers['most_common_viz'] params_only_viz = self.plugin.visualizers['params_only_viz'] no_input_viz = self.plugin.visualizers['no_input_viz'] obs = mapping_viz.__call__.__doc__ self.assertEqual(obs, exp_mapping_viz) obs = common_viz.__call__.__doc__ self.assertEqual(obs, exp_common_viz) obs = params_only_viz.__call__.__doc__ self.assertEqual(obs, exp_params_only_viz) obs = no_input_viz.__call__.__doc__ self.assertEqual(obs, exp_no_input_viz) exp_mapping_viz = """\ Visualize two mappings This visualizer produces an HTML visualization of two key-value mappings, each sorted in alphabetical order by key. Parameters ---------- mapping1 : Mapping mapping2 : Mapping key_label : Str value_label : Str Returns ------- visualization : Visualization """ exp_common_viz = """\ Visualize most common integers This visualizer produces HTML and TSV outputs containing the input sequence of integers ordered from most- to least-frequently occurring, along with their respective frequencies. Parameters ---------- ints : IntSequence1 | IntSequence2 Returns ------- visualization : Visualization """ exp_params_only_viz = """\ Parameters only viz This visualizer only accepts parameters. Parameters ---------- name : Str, optional age : Int % Range(0, None), optional Returns ------- visualization : Visualization """ exp_no_input_viz = """\ No input viz This visualizer does not accept any type of input. Returns ------- visualization : Visualization """ if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/sdk/usage.py000066400000000000000000000675561412141660100165140ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- """ The Usage module enables generating examples for multiple QIIME 2 interfaces. """ import abc import re import types import typing from qiime2 import core, sdk, metadata class UsageAction: """ Parameters ---------- plugin_id : str Plugin ID. action_id : str Action ID. """ def __init__(self, *, plugin_id: str, action_id: str): if plugin_id == '': raise ValueError('Must specify a value for plugin_id.') if action_id == '': raise ValueError('Must specify a value for action_id.') self.plugin_id = plugin_id self.action_id = action_id self._plugin_manager = sdk.PluginManager() def __repr__(self): return 'UsageAction(plugin_id=%r, action_id=%r)' %\ (self.plugin_id, self.action_id) def get_action(self) -> typing.Tuple[ typing.Union['sdk.Method', 'sdk.Pipeline'], typing.Type['core.PipelineSignature']]: """ Get this example's action and signature. Returns ------- action_f : QIIME 2 Method, Visualizer, or Pipeline The plugin action. action_f.signature : QIIME 2 Method, Visualizer, or Pipeline Signature The method signature for the plugin action. """ plugin = self._plugin_manager.get_plugin(id=self.plugin_id) # TODO: should this validation be pushed up into # plugin.py or action.py? try: action_f = plugin.actions[self.action_id] except KeyError: raise KeyError('No action currently registered with ' 'id: "%s".' % (self.action_id,)) return (action_f, action_f.signature) def validate(self, inputs: 'UsageInputs', outputs: 'UsageOutputNames') -> None: """ Ensure that all required inputs and outputs are declared and reference appropriate scope records, if necessary. Parameters ---------- inputs : UsageInputs outputs : UsageOutputNames """ if not isinstance(inputs, UsageInputs): raise TypeError('Must provide an instance of UsageInputs.') if not isinstance(outputs, UsageOutputNames): raise TypeError('Must provide an instance of UsageOutputNames.') _, sig = self.get_action() inputs.validate(sig) outputs.validate(sig) class UsageInputs: """ A convenience object for assisting with construction of input options and performing signature validation. Parameters ---------- kwargs : Example inputs Inputs to be passed in to ``Usage.action``. """ def __init__(self, **kwargs: typing.Any): self.values = kwargs def __repr__(self): return 'UsageInputs(**%r)' % (self.values,) def validate(self, signature: typing.Type['core.PipelineSignature']) -> None: """ Ensure that all required inputs are accounted for and that there are no unnecessary or extra parameters provided. Parameters ---------- signature : QIIME 2 Method, Visualizer, or Pipeline Signature The plugin action's signature. Raises ------ ValueError If there are missing or extra inputs or parameters. """ provided = set(self.values.keys()) inputs, params = signature.inputs, signature.parameters exp_inputs, optional_inputs = set(), set() for name, sig in inputs.items(): if sig.has_default(): optional_inputs.add(name) else: exp_inputs.add(name) exp_params, optional_params = set(), set() for name, sig in params.items(): if sig.has_default(): optional_params.add(name) else: exp_params.add(name) missing = exp_inputs - provided if len(missing) > 0: raise ValueError('Missing input(s): %r' % (missing, )) missing = exp_params - provided if len(missing) > 0: raise ValueError('Missing parameter(s): %r' % (missing, )) all_vals = exp_inputs | optional_inputs | exp_params | optional_params extra = provided - all_vals if len(extra) > 0: raise ValueError('Extra input(s) or parameter(s): %r' % (extra, )) def build_opts(self, signature: typing.Type['core.PipelineSignature'], scope: 'Scope') -> dict: """ Build a dictionary mapping example ``ScopeRecord``s and primitives to action inputs and parameters, respectively. Values are derived from either an input's ``ScopeRecord`` (``ScopeRecord.value``), or the value a keyword argument passed into the ``UsageInputs`` constructor. Parameters ---------- signature : QIIME 2 Method, Visualizer, or Pipeline Signature The plugin action's signature. scope : Scope A Usage example's current scope. Returns ------- dict Input identifiers and their example values. """ opts = {} for name, signature in signature.signature_order.items(): if name in self.values: v = self.values[name] if isinstance(v, ScopeRecord) and v.ref in scope.records: value = self.values[name].result else: value = v opts[name] = value return opts class UsageOutputNames: """ Parameters ---------- kwargs : str A mapping between the Action's output name and the desired scope record identifier in which to "store" the results. """ def __init__(self, **kwargs: str): for key, val in kwargs.items(): if not isinstance(val, str): raise TypeError( 'Name provided for key %r must be a string, not a %r.' % (key, type(val))) self.values = kwargs def __repr__(self): return 'UsageOutputNames(**%r)' % (self.values, ) def get(self, key) -> str: """ Get an example output's identifier. Returns ------- str The identifier for an example output. """ return self.values[key] def validate(self, signature: typing.Type['core.PipelineSignature']) -> None: """ Ensure that all required outputs are accounted for and that there are no unnecessary or extra outputs provided. Parameters ---------- signature Action signature. Raises ------ ValueError If the example has missing or extra outputs as per the action signature. """ provided = set(self.values.keys()) exp_outputs = set(signature.outputs) missing = exp_outputs - provided if len(missing) > 0: raise ValueError('Missing output(s): %r' % (missing, )) extra = provided - exp_outputs if len(extra) > 0: raise ValueError('Extra output(s): %r' % (extra, )) def validate_computed(self, computed_outputs: typing.Dict[ str, typing.Union['sdk.Artifact', 'sdk.Visualization', 'metadata.Metadata']] ) -> None: """ Check that outputs are still valid after being processed by a Usage driver's ``_action_`` method. Parameters ---------- computed_outputs : dict of outputs Outputs returned by the Usage driver's ``_action_`` method. Raises ------ ValueError If there are missing or extra outputs as per the action signature. """ provided = set(computed_outputs.keys()) exp_outputs = set(self.values.keys()) missing = exp_outputs - provided if len(missing) > 0: raise ValueError('SDK implementation is missing output(s): %r' % (missing, )) extra = provided - exp_outputs if len(extra) > 0: raise ValueError('SDK implementation has specified extra ' 'output(s): %r' % (extra, )) def build_opts(self, action_signature: typing.Type['core.PipelineSignature'], scope: 'Scope') -> dict: """ Build a dictionary mapping action output identifiers to example output value. Parameters ---------- action_signature : QIIME 2 Method, Visualizer, or Pipeline Signature The plugin action's signature. scope : Scope A Usage example's current scope. Returns ------- dict Output identifiers and their example values. """ opts = {} for output in action_signature.outputs.keys(): opts[output] = self.get(output) return opts class ScopeRecord: """ Represents a discrete step in a Usage example. Provides information needed by Usage drivers to render Usage examples. Parameters ---------- ref : str A unique identifier for referring to the record. value : Artifact, Visualization, or Metadata The value referred to by ``ref``. source : str The Usage method called to initialize example data. assert_has_line_matching : callable A function for asserting something about rendered example data. Notes ----- ``ScopeRecord`` is an internal implementation and need not be instantiated manually. """ def __init__(self, ref: str, value: typing.Union['sdk.Artifact', 'sdk.Visualization', 'metadata.Metadata'], source: str, assert_has_line_matching: typing.Optional[ typing.Callable] = None): if assert_has_line_matching is not None and \ not callable(assert_has_line_matching): raise TypeError('Value for `assert_has_line_matching` should be a ' '`callable`.') self.ref = ref self._result = value self._source = source self._assert_has_line_matching_ = assert_has_line_matching def __repr__(self): return 'ScopeRecord' % (self.ref, self.result, self.source) @property def result(self) -> typing.Union[ 'sdk.Artifact', 'sdk.Visualization', 'metadata.Metadata']: """ Artifact, Visualization, or Metadata value referred to by ``self.ref``. """ return self._result @property def source(self) -> str: """ The origin of a ``ScopeRecord``. Possible values for this property are ``init_data``, ``init_metadata``, ``init_data_collection``, ``merge_metadata``, ``get_metadata_column``, or ``action``. This information is required by some Usage drivers in order to render examples properly. See Also -------- Scope.push_record q2cli.q2cli.core.usage.CLIRenderer """ return self._source def assert_has_line_matching(self, label: str, path: str, expression: str) -> None: """ A proxy for a Usage driver's reference implementation for verifying that the file at ``path`` contains a line matching ``expression``. Parameters ---------- label : str A label for describing this assertion. Interface drivers may choose to omit this information in the rendered output. path : str Path to example data file. expression : str A regex pattern to be passed as the first argument to ``re.search``. Raises ______ AssertionError If ``expression`` is not found in ``path``. See Also -------- See ``ExecutionUsage`` for an example implementation. """ return self._assert_has_line_matching_(self.ref, label, path, expression) class Scope: """ Sequentially track all ``ScopeRecord``s for a Usage example. The Scope provides a detailed, start-to-finish accounting of all of the steps necessary for translating the Usage example into an interface-specific representation. Notes ----- ``Scope`` is an internal implementation and need not be instantiated manually. """ def __init__(self): self._records: typing.Dict[str, 'ScopeRecord'] = dict() def __repr__(self): return '%r' % (self._records, ) @property def records(self) -> types.MappingProxyType: """ A read-only view of ``ScopeRecords`` in the current scope. """ return types.MappingProxyType(self._records) def push_record(self, ref: str, value: typing.Union['sdk.Artifact', 'sdk.Visualization', 'metadata.Metadata'], source: str, assert_has_line_matching: typing.Callable = None, ) -> 'ScopeRecord': """ Appends a new ``ScopeRecord`` to the Usage Example's scope. Parameters ---------- ref : str value : Artifact, Visualization, or Metadata Data from a Usage data initialization method. source : str The origin of Usage example data. assert_has_line_matching : callable Verify that the file at ``path`` contains a line matching ``expression`` within an Artifact. See ``ScopeRecord.assert_has_line_matching``. This is a proxy for a Usage driver's reference implementation. Returns ------- record : ScopeRecord """ record = ScopeRecord(ref=ref, value=value, source=source, assert_has_line_matching=assert_has_line_matching) self._records[ref] = record return record def get_record(self, ref: str) -> ScopeRecord: """ Look up a ``ScopeRecord`` from the current scope by identifier. Parameters ---------- ref : str The identifier for a ``ScopeRecord``. Raises ------ KeyError If the record identifier isn't in the scope. Returns ------- record : ScopeRecord """ try: return self.records[ref] except KeyError: raise KeyError('No record with ref id: "%s" in scope.' % (ref, )) class Usage(metaclass=abc.ABCMeta): """ ``Usage`` is the base class for Usage driver implementations and should be customized for the interface or implementation. """ UsageAction = UsageAction UsageInputs = UsageInputs UsageOutputNames = UsageOutputNames def __init__(self): self._scope = Scope() def init_data(self, ref: str, factory: typing.Callable[[], 'sdk.Artifact']) \ -> 'ScopeRecord': """ Initialize example Artifact data from a factory. Whether or not the example data is actually created is dependent on the driver executing the example. Parameters ---------- ref : str Unique identifier for example data. factory : callable A factory that returns an example Artifact. Returns ------- record : ScopeRecord A record with information about example data. """ value = self._init_data_(ref, factory) return self._push_record(ref, value, 'init_data') def _init_data_(self, ref, factory): raise NotImplementedError def init_metadata(self, ref: str, factory: typing.Callable[[], 'metadata.Metadata']) \ -> 'ScopeRecord': """ Initialize Metadata for a Usage example. Whether or not the example metadata is actually created is dependent on the driver executing the example. Parameters ---------- ref : str Unique identifier for example metadata. factory : callable A factory that returns example Metadata. Returns ------- record : ScopeRecord A record with information about example Metadata. """ value = self._init_metadata_(ref, factory) return self._push_record(ref, value, 'init_metadata') def _init_metadata_(self, ref, factory): raise NotImplementedError def init_data_collection(self, ref: str, collection_type: typing.Union[list, set], *records: ScopeRecord) -> 'ScopeRecord': """ Initialize a collection of data for a Usage example. Parameters ---------- ref : str Unique identifier for example data collection. collection_type : list or set The type of collection required by an action. records : ScopeRecords belonging to the collection The records associated with data to be initialized in the collection. Returns ------- record : ScopeRecord A record with information about an example of collection data. """ if len(records) < 1: raise ValueError('Must provide at least one ScopeRecord input.') for record in records: if not isinstance(record, ScopeRecord): raise ValueError('Record (%r) returned a %s, expected a ' 'ScopeRecord.' % (record, type(record))) value = self._init_data_collection_(ref, collection_type, records) return self._push_record(ref, value, 'init_data_collection') def _init_data_collection_(self, ref, collection_type, records): raise NotImplementedError def merge_metadata(self, ref: str, *records: typing.List['ScopeRecord']) -> 'ScopeRecord': """ Merge previously initialized example Metadata. Parameters ---------- ref : str Unique identifier for merged Metadata. records : ScopeRecords Records for the example Metadata to be merged. Returns ------- record : ScopeRecord A new record with information about the merged example Metadata. """ if len(records) < 2: raise ValueError('Must provide two or more Metadata inputs.') value = self._merge_metadata_(ref, records) return self._push_record(ref, value, 'merge_metadata') def _merge_metadata_(self, ref, records): raise NotImplementedError def get_metadata_column(self, column_name: str, record: ScopeRecord) -> 'ScopeRecord': """ Get a Metadata column from previously initialized example Metadata. Parameters ---------- column_name : str The name of a column in example Metadata. record : ScopeRecord The record associated with example Metadata. Returns ------- record : ScopeRecord A new scope record for example Metadata column ``column_name``. """ value = self._get_metadata_column_(column_name, record) return self._push_record(column_name, value, 'get_metadata_column') def _get_metadata_column_(self, column_name, record): raise NotImplementedError def comment(self, text: str): comment = self._comment_(text) return self._push_record(str(comment), comment, 'comment') def _comment_(self, text: str): raise NotImplementedError def action(self, action: UsageAction, inputs: UsageInputs, outputs: UsageOutputNames) -> None: """ This method is a proxy for invoking a QIIME 2 Action. Whether or not the Action is actually invoked is dependent on the driver executing the example. Parameters ---------- action : UsageAction Specifies the Plugin and Action for a Usage example. inputs : UsageInputs Specifies the inputs to an Action for a Usage example. outputs : UsageOutputNames Species the outputs of an Action for a Usage example. Examples -------- qiime2.core.testing.examples : Usage examples """ if not isinstance(action, UsageAction): raise TypeError('Must provide an instance of UsageAction.') action.validate(inputs, outputs) _, action_signature = action.get_action() input_opts = inputs.build_opts(action_signature, self._scope) output_opts = outputs.build_opts(action_signature, self._scope) computed_outputs = self._action_(action, input_opts, output_opts) self._add_outputs_to_scope(outputs, computed_outputs) def _action_(self, action: UsageAction, input_opts: dict, output_opts: dict) -> dict: raise NotImplementedError def _assert_has_line_matching_(self, ref, label, path, expression): raise NotImplementedError def get_result(self, ref: str) -> 'ScopeRecord': """ Get the record for a Usage example output. This is a convenience method used to access records generated after running ``Usage.action``. Parameters ---------- ref : str Output identifier. Raises ------ KeyError If ``ref`` is not associated with a record generated by ``Usage.action``. TypeError If the source type is not an Action. """ record = self._get_record(ref) source = record.source if source != 'action': raise TypeError('source == %s but must be "action"' % source) return record def _add_outputs_to_scope(self, outputs: UsageOutputNames, computed_outputs): outputs.validate_computed(computed_outputs) for output, result in computed_outputs.items(): ref = outputs.get(output) self._push_record(ref, result, 'action') def _push_record(self, ref, value, source): return self._scope.push_record( ref=ref, value=value, source=source, assert_has_line_matching=self._assert_has_line_matching_) def _get_record(self, ref): return self._scope.get_record(ref) def _get_records(self): return self._scope.records class DiagnosticUsage(Usage): """ Reference implementation of a Usage driver. Used to generate information for testing the Usage API. See Also -------- qiime2.sdk.tests.test_usage.TestUsage : Unit tests using this driver. """ def __init__(self): super().__init__() def _init_data_(self, ref, factory): return { 'source': 'init_data', 'ref': ref, } def _init_metadata_(self, ref, factory): return { 'source': 'init_metadata', 'ref': ref, } def _init_data_collection_(self, ref, collection_type, records): return { 'source': 'init_data_collection', 'ref': ref, }, collection_type([i.ref for i in records]) def _merge_metadata_(self, ref, records): return { 'source': 'merge_metadata', 'ref': ref, 'records_refs': [r.ref for r in records], } def _get_metadata_column_(self, column_name, record): return { 'source': 'get_metadata_column', 'ref': column_name, 'record_ref': record.ref, 'column_name': column_name, } def _comment_(self, text): return { 'source': 'comment', 'text': text, } def _action_(self, action, input_opts, output_opts): results = dict() for output_opt in output_opts.keys(): results[output_opt] = { 'source': 'action', 'plugin_id': action.plugin_id, 'action_id': action.action_id, 'input_opts': input_opts, 'output_opts': output_opts, 'output_opt': output_opt, } return results def _assert_has_line_matching_(self, ref, label, path, expression): return { 'source': 'assert_has_line_matching', 'ref': ref, 'label': label, 'path': path, 'expression': expression, } class ExecutionUsage(Usage): """ Execute and test rendered examples. See Also -------- qiime2.sdk.tests.test_usage.TestExecutionUsage : Tests using this driver. qiime2.plugin.testing.TestPluginBase.execute_examples : Executes examples. """ def __init__(self, asynchronous=False): super().__init__() self.asynchronous = asynchronous def _init_data_(self, ref, factory): result = factory() result_type = type(result) if result_type not in (list, set, sdk.Artifact): raise ValueError('Factory (%r) returned a %s, expected an ' 'Artifact.' % (factory, result_type)) if result_type in (list, set): if not all([isinstance(i, sdk.Artifact) for i in result]): raise ValueError('Factory (%r) returned a %s where not all ' 'elements were Artifacts.' % (factory, result_type)) return result def _init_metadata_(self, ref, factory): result = factory() result_type = type(result) if not isinstance(result, metadata.Metadata): raise TypeError('Factory (%r) returned a %s, but expected ' 'Metadata.' % (factory, result_type)) return result def _init_data_collection_(self, ref, collection_type, records): collection = [] for record in records: collection.append(record.result) return collection_type(collection) def _merge_metadata_(self, ref, records): mds = [r.result for r in records] return mds[0].merge(*mds[1:]) def _get_metadata_column_(self, column_name, record): return record.result.get_column(column_name) def _comment_(self, text): pass def _action_(self, action: UsageAction, input_opts: dict, output_opts: dict): action_f, _ = action.get_action() if self.asynchronous: results = action_f.asynchronous(**input_opts).result() else: results = action_f(**input_opts) return {k: getattr(results, k) for k in output_opts.keys()} def _assert_has_line_matching_(self, ref, label, path, expression): data = self._get_record(ref).result hits = sorted(data._archiver.data_dir.glob(path)) if len(hits) != 1: raise ValueError('Value provided for path (%s) did not produce ' 'exactly one hit: %s' % (path, hits)) target = hits[0].read_text() match = re.search(expression, target, flags=re.MULTILINE) if match is None: raise AssertionError('Expression %r not found in %s.' % (expression, path)) qiime-2021.8.0/qiime2/sdk/util.py000066400000000000000000000073761412141660100163570ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import qiime2.sdk import qiime2.core.type as qtype import qiime2.core.type.parse as _parse from qiime2.core.type import ( is_semantic_type, is_primitive_type, is_collection_type, is_metadata_type, is_visualization_type, interrogate_collection_type, parse_primitive, is_union, is_metadata_column_type) __all__ = [ 'is_semantic_type', 'is_primitive_type', 'is_collection_type', 'is_metadata_type', 'is_visualization_type', 'interrogate_collection_type', 'type_from_ast', 'parse_primitive', 'parse_type', 'parse_format', 'actions_by_input_type', 'is_union', 'is_metadata_column_type', ] def type_from_ast(ast, scope=None): """Convert a type ast (from `.to_ast()`) to a type expression. Parameters ---------- ast : json compatible object The abstract syntax tree produced by `to_ast` on a type. scope : dict A dictionary to use between multiple calls to share scope between different types. This allows type variables from the same type map to be constructed from an equivalent type map. Scope should be shared within a given call signature, but not between call signatures. Returns ------- type expression """ return _parse.ast_to_type(ast, scope=scope) def parse_type(string, expect=None): """Convert a string into a type expression Parameters ---------- string : str The string type expression to convert into a TypeExpression expect : {'semantic', 'primitive', 'visualization'}, optional Will raise a TypeError if the resulting TypeExpression is not a member of `expect`. Returns ------- type expression """ if expect is not None and expect not in {'semantic', 'primitive', 'visualization'}: raise ValueError("`expect` got %r, must be 'semantic', 'primitive'," " 'visualization', or None." % (expect,)) type_expr = _parse.ast_to_type(_parse.string_to_ast(string)) if expect is None: pass elif expect == 'semantic' and qtype.is_semantic_type(type_expr): pass elif expect == 'primitive' and qtype.is_primitive_type(type_expr): pass elif expect == 'visualization' and type_expr == qtype.Visualization: pass else: raise TypeError("Type expression %r is not a %s type." % (type_expr, expect)) return type_expr def parse_format(format_str): if format_str is None: return None pm = qiime2.sdk.PluginManager() try: format_record = pm.formats[format_str] except KeyError: raise TypeError("No format: %s" % format_str) return format_record.format def actions_by_input_type(string): """Plugins and actions that have as input the artifact type (string) Parameters ---------- string : str QIIME2 artifact type Returns ------- list of tuples: [(q2.plugin, [q2.actions, ...]), ...] """ commands = [] if string is not None: query_type = qiime2.sdk.util.parse_type(string) pm = qiime2.sdk.PluginManager() for pgn, pg in pm.plugins.items(): actions = list({a for an, a in pg.actions.items() for iname, i in a.signature.inputs.items() if i.qiime_type >= query_type}) if actions: commands.append((pg, actions)) return commands qiime-2021.8.0/qiime2/tests/000077500000000000000000000000001412141660100153745ustar00rootroot00000000000000qiime-2021.8.0/qiime2/tests/__init__.py000066400000000000000000000005351412141660100175100ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- qiime-2021.8.0/qiime2/tests/test_artifact_api.py000066400000000000000000000217431412141660100214420ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import importlib import sys import tempfile import types import unittest import qiime2.sdk from qiime2.core.testing.util import get_dummy_plugin from qiime2.plugins import ArtifactAPIUsage class TestImports(unittest.TestCase): def setUp(self): self._sys_modules = sys.modules.copy() # Ignore the returned dummy plugin object, just run this to verify the # plugin exists as the tests rely on it being loaded. get_dummy_plugin() def tearDown(self): # This allows us to reset the state of our imports to test different # expressions to make sure they work independently. remove = [] for key in sys.modules: if key not in self._sys_modules: remove.append(key) # can't delete in place while we iterate for key in remove: del sys.modules[key] def _check_spec(self, module, name, is_package): self.assertIsInstance(module, types.ModuleType) self.assertEqual(module.__name__, name) self.assertEqual(module.__spec__.name, name) self.assertEqual(module.__spec__.submodule_search_locations, [] if is_package else None) self.assertFalse(module.__spec__.has_location) self.assertIn('generated QIIME 2 API', repr(module)) def _check_plugin(self, module): self._check_spec(module, 'qiime2.plugins.dummy_plugin', True) self._check_methods(module.methods) self._check_visualizers(module.visualizers) self.assertEqual(set(x for x in dir(module) if not x.startswith('_')), {'visualizers', 'methods', 'actions', 'pipelines'}) def _check_methods(self, module): self._check_spec(module, 'qiime2.plugins.dummy_plugin.methods', False) self.assertTrue(hasattr(module, 'concatenate_ints')) self.assertFalse(hasattr(module, 'most_common_viz')) self.assertIsInstance(module.concatenate_ints, qiime2.sdk.Action) def _check_visualizers(self, module): self._check_spec( module, 'qiime2.plugins.dummy_plugin.visualizers', False) self.assertTrue(hasattr(module, 'most_common_viz')) self.assertFalse(hasattr(module, 'concatenate_ints')) self.assertIsInstance(module.most_common_viz, qiime2.sdk.Action) def test_import_root(self): import qiime2.plugins.dummy_plugin self._check_plugin(qiime2.plugins.dummy_plugin) def test_import_root_from(self): from qiime2.plugins import dummy_plugin self._check_plugin(dummy_plugin) def test_import_methods(self): import qiime2.plugins.dummy_plugin.methods self._check_methods(qiime2.plugins.dummy_plugin.methods) def test_import_visualizers(self): import qiime2.plugins.dummy_plugin.visualizers self._check_visualizers(qiime2.plugins.dummy_plugin.visualizers) def test_import_methods_from(self): from qiime2.plugins.dummy_plugin import methods self._check_methods(methods) def test_import_visualizers_from(self): from qiime2.plugins.dummy_plugin import visualizers self._check_visualizers(visualizers) def test_import_non_plugin(self): with self.assertRaises(ImportError): import qiime2.plugins.dummy_not_plugin # noqa def test_import_non_action(self): with self.assertRaises(ImportError): import qiime2.plugins.dummy_plugin.non_action # noqa def test_import_side_module(self): # Certain implementations of __PATH__ can cause a module to load # siblings (__PATH__ = ['.'] for example) import qiime2.metadata self.assertIsInstance(qiime2.metadata, types.ModuleType) with self.assertRaises(ImportError): import qiime2.plugins.metadata # noqa def test_import_too_deep(self): with self.assertRaises(ImportError): import qiime2.plugins.dummy_plugin.methods.too_deep # noqa def test_import_non_module(self): with self.assertRaises(ImportError): import qiime2.plugins.dummy_plugin.methods.concatenate_ints # noqa def test_reload_fails(self): import qiime2.plugins.dummy_plugin with self.assertRaises(ImportError): importlib.reload(qiime2.plugins.dummy_plugin) class TestArtifactAPIUsage(unittest.TestCase): def setUp(self): # TODO standardize temporary directories created by QIIME 2 self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-') self.plugin = get_dummy_plugin() def tearDown(self): self.test_dir.cleanup() def test_basic(self): action = self.plugin.actions['concatenate_ints'] use = ArtifactAPIUsage() action.examples['concatenate_ints_simple'](use) exp = """\ from qiime2.plugins.dummy_plugin.methods import concatenate_ints # This example demonstrates basic usage. ints_d, = concatenate_ints( ints1=ints_a, ints2=ints_b, ints3=ints_c, int1=4, int2=2, ) """ self.assertEqual(exp, use.render()) def test_chained(self): action = self.plugin.actions['concatenate_ints'] use = ArtifactAPIUsage() action.examples['concatenate_ints_complex'](use) exp = """\ from qiime2.plugins.dummy_plugin.methods import concatenate_ints # This example demonstrates chained usage (pt 1). ints_d, = concatenate_ints( ints1=ints_a, ints2=ints_b, ints3=ints_c, int1=4, int2=2, ) # This example demonstrates chained usage (pt 2). concatenated_ints, = concatenate_ints( ints1=ints_d, ints2=ints_b, ints3=ints_c, int1=41, int2=0, ) """ self.assertEqual(exp, use.render()) def test_dereferencing(self): action = self.plugin.actions['typical_pipeline'] use = ArtifactAPIUsage() action.examples['typical_pipeline_simple'](use) exp = """\ from qiime2.plugins.dummy_plugin.pipelines import typical_pipeline out_map, left, right, left_viz, right_viz = typical_pipeline( int_sequence=ints, mapping=mapper, do_extra_thing=True, ) """ self.assertEqual(exp, use.render()) def test_chained_dereferencing(self): action = self.plugin.actions['typical_pipeline'] use = ArtifactAPIUsage() action.examples['typical_pipeline_complex'](use) exp = """\ from qiime2.plugins.dummy_plugin.pipelines import typical_pipeline out_map1, left1, right1, left_viz1, right_viz1 = typical_pipeline( int_sequence=ints1, mapping=mapper1, do_extra_thing=True, ) out_map2, left2, right2, left_viz2, right_viz2 = typical_pipeline( int_sequence=left1, mapping=out_map1, do_extra_thing=False, ) """ self.assertEqual(exp, use.render()) def test_metadata_merging(self): action = self.plugin.actions['identity_with_metadata'] use = ArtifactAPIUsage() action.examples['identity_with_metadata_merging'](use) exp = """\ from qiime2.plugins.dummy_plugin.methods import identity_with_metadata md3 = md1.merge(md2) out, = identity_with_metadata( ints=ints, metadata=md3, ) """ self.assertEqual(exp, use.render()) def test_metadata_column_from_helper(self): action = self.plugin.actions['identity_with_metadata_column'] use = ArtifactAPIUsage() action.examples['identity_with_metadata_column_get_mdc'](use) exp = """\ from qiime2.plugins.dummy_plugin.methods import identity_with_metadata_column a = md.get_column('a') out, = identity_with_metadata_column( ints=ints, metadata=a, ) """ self.assertEqual(exp, use.render()) def test_use_init_collection_data(self): action = self.plugin.actions['variadic_input_method'] use = ArtifactAPIUsage() action.examples['variadic_input_simple'](use) exp = """\ from qiime2.plugins.dummy_plugin.methods import variadic_input_method out, = variadic_input_method( ints=[ints_a, ints_b], int_set={single_int1, single_int2}, nums={8, 9, 7}, ) """ self.assertEqual(exp, use.render()) def test_optional_inputs(self): action = self.plugin.actions['optional_artifacts_method'] use = ArtifactAPIUsage() action.examples['optional_inputs'](use) exp = """\ from qiime2.plugins.dummy_plugin.methods import optional_artifacts_method output, = optional_artifacts_method( ints=ints, num1=1, ) output, = optional_artifacts_method( ints=ints, num1=1, num2=2, ) ints_b, = optional_artifacts_method( ints=ints, num1=1, num2=None, ) output, = optional_artifacts_method( ints=ints, num1=3, optional1=ints_b, num2=4, ) """ self.assertEqual(exp, use.render()) if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/tests/test_util.py000066400000000000000000000107041412141660100177640ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import errno import tempfile import unittest import unittest.mock as mock import qiime2.util as util EXDEV = OSError(errno.EXDEV, "Invalid cross-device link") EPERM = PermissionError(errno.EPERM, "unsafe operation e.g. using link wrong") EACCES = PermissionError(errno.EACCES, "insufficient r/w permissions") SECRET = "this is a secret for testing, don't tell anyone!" class TestDuplicate(unittest.TestCase): def setUp(self): self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-') self.dst1 = os.path.join(self.test_dir.name, 'dst1') self.dst2 = os.path.join(self.test_dir.name, 'dst2') self.dir = os.path.join(self.test_dir.name, 'dir') with open(self.dst2, 'w') as fh: fh.write("This is not the secret") os.mkdir(self.dir) self.src = os.path.join(self.test_dir.name, 'src') self.missing = os.path.join(self.test_dir.name, 'missing') with open(self.src, 'w') as fh: fh.write(SECRET) def tearDown(self): self.test_dir.cleanup() def test_src_not_exists(self): with self.assertRaisesRegex(FileNotFoundError, self.missing): util.duplicate(self.missing, self.dst1) def test_src_dir(self): with self.assertRaisesRegex(IsADirectoryError, self.dir): util.duplicate(self.dir, self.dst1) def test_dst_not_exists(self): util.duplicate(self.src, self.dst1) assert os.path.exists(self.dst1) with open(self.dst1) as fh: self.assertEqual(fh.read(), SECRET) def test_dst_exists(self): with self.assertRaisesRegex(FileExistsError, self.dst2): util.duplicate(self.src, self.dst2) def test_dst_dir(self): with self.assertRaisesRegex(IsADirectoryError, self.dir): util.duplicate(self.src, self.dir) @mock.patch('qiime2.util.os.link', side_effect=EACCES) def test_perm_error_EACCES(self, mocked_link): with self.assertRaisesRegex( PermissionError, "insufficient r/w permissions"): util.duplicate(self.src, self.dst1) assert mocked_link.called @mock.patch('qiime2.util.os.link', side_effect=EPERM) def test_perm_error_EPERM(self, mocked_link): util.duplicate(self.src, self.dst1) assert mocked_link.called assert os.path.exists(self.dst1) with open(self.dst1) as fh: self.assertEqual(fh.read(), SECRET) @mock.patch('qiime2.util.os.link', side_effect=EXDEV) def test_cross_device_src_not_exists(self, mocked_link): with self.assertRaisesRegex(FileNotFoundError, self.missing): util.duplicate(self.missing, self.dst1) @mock.patch('qiime2.util.os.link', side_effect=EXDEV) def test_cross_device_src_dir(self, mocked_link): with self.assertRaisesRegex(IsADirectoryError, self.dir): util.duplicate(self.dir, self.dst1) @mock.patch('qiime2.util.os.link', side_effect=EXDEV) def test_cross_device_dst_not_exists(self, mocked_link): util.duplicate(self.src, self.dst1) assert mocked_link.called assert os.path.exists(self.dst1) with open(self.dst1) as fh: self.assertEqual(fh.read(), SECRET) @mock.patch('qiime2.util.os.link', side_effect=EXDEV) def test_cross_device_dst_exists(self, mocked_link): with self.assertRaisesRegex(FileExistsError, self.dst2): util.duplicate(self.src, self.dst2) @mock.patch('qiime2.util.os.link', side_effect=EXDEV) def test_cross_device_dst_dir(self, mocked_link): with self.assertRaisesRegex(IsADirectoryError, self.dir): util.duplicate(self.src, self.dir) @mock.patch('qiime2.util.os.link', side_effect=EXDEV) @mock.patch('qiime2.util.shutil.copyfile', side_effect=EACCES) def test_cross_device_perm_error(self, mocked_link, mocked_copyfile): with self.assertRaisesRegex( PermissionError, "insufficient r/w permissions"): util.duplicate(self.src, self.dst1) assert mocked_link.called assert mocked_copyfile.called if __name__ == '__main__': unittest.main() qiime-2021.8.0/qiime2/util.py000066400000000000000000000062741412141660100155720ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- import os import sys import errno import shutil import threading import contextlib _REDIRECTED_STDIO_LOCK = threading.Lock() @contextlib.contextmanager def redirected_stdio(stdout=None, stderr=None): with _REDIRECTED_STDIO_LOCK: if stdout is not None: with _redirected_fd(to=stdout, stdio=sys.stdout): if stderr is not None: with _redirected_fd(to=stderr, stdio=sys.stderr): yield else: yield elif stderr is not None: with _redirected_fd(to=stderr, stdio=sys.stderr): yield else: yield # Taken whole-sale from: http://stackoverflow.com/a/22434262/579416 @contextlib.contextmanager def _redirected_fd(to=os.devnull, stdio=None): if stdio is None: stdio = sys.stdout stdio_fd = _get_fileno(stdio) # copy stdio_fd before it is overwritten # NOTE: `copied` is inheritable on Windows when duplicating a standard # stream with os.fdopen(os.dup(stdio_fd), 'wb') as copied: stdio.flush() # flush library buffers that dup2 knows nothing about try: os.dup2(_get_fileno(to), stdio_fd) # $ exec >&to except ValueError: # filename with open(to, 'wb') as to_file: os.dup2(to_file.fileno(), stdio_fd) # $ exec > to try: yield stdio # allow code to be run with the redirected stdio finally: # restore stdio to its previous value # NOTE: dup2 makes stdio_fd inheritable unconditionally stdio.flush() os.dup2(copied.fileno(), stdio_fd) # $ exec >&copied def _get_fileno(file_or_fd): fd = getattr(file_or_fd, 'fileno', lambda: file_or_fd)() if not isinstance(fd, int): raise ValueError("Expected a file (`.fileno()`) or a file descriptor") return fd def duplicate(src, dst): """Alternative to shutil.copyfile, this will use os.link when possible. See shutil.copyfile for documention. Only `src` and `dst` are supported. Unlike copyfile, this will not overwrite the destination if it exists. """ if os.path.isdir(src): # os.link will give a permission error raise OSError(errno.EISDIR, "Is a directory", src) if os.path.isdir(dst): # os.link will give a FileExists error raise OSError(errno.EISDIR, "Is a directory", dst) if os.path.exists(dst): # shutil.copyfile will overwrite the existing file raise OSError(errno.EEXIST, "File exists", src, "File exists", dst) try: os.link(src, dst) except OSError as e: if e.errno == errno.EXDEV: # Invalid cross-device link shutil.copyfile(src, dst) elif e.errno == errno.EPERM: # Permissions/ownership error shutil.copyfile(src, dst) else: raise qiime-2021.8.0/setup.cfg000066400000000000000000000002311412141660100146610ustar00rootroot00000000000000[versioneer] VCS=git style=pep440 versionfile_source = qiime2/_version.py versionfile_build = qiime2/_version.py tag_prefix = parentdir_prefix = qiime2- qiime-2021.8.0/setup.py000066400000000000000000000016761412141660100145700ustar00rootroot00000000000000# ---------------------------------------------------------------------------- # Copyright (c) 2016-2021, QIIME 2 development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE, distributed with this software. # ---------------------------------------------------------------------------- from setuptools import find_packages, setup import versioneer setup( name='qiime2', version=versioneer.get_version(), cmdclass=versioneer.get_cmdclass(), license='BSD-3-Clause', url='https://qiime2.org', packages=find_packages(), entry_points={ 'qiime2.plugins': [ 'dummy-plugin=qiime2.core.testing.plugin:dummy_plugin' ] }, package_data={ 'qiime2.metadata.tests': ['data/*/*'], 'qiime2.core.testing': ['citations.bib'], 'qiime2.sdk.tests': ['data/*'], 'qiime2': ['citations.bib'] }, zip_safe=False, ) qiime-2021.8.0/versioneer.py000066400000000000000000002060221412141660100156010ustar00rootroot00000000000000 # Version: 0.18 # flake8: noqa """The Versioneer - like a rocketeer, but for versions. The Versioneer ============== * like a rocketeer, but for versions! * https://github.com/warner/python-versioneer * Brian Warner * License: Public Domain * Compatible With: python2.6, 2.7, 3.2, 3.3, 3.4, 3.5, 3.6, and pypy * [![Latest Version] (https://pypip.in/version/versioneer/badge.svg?style=flat) ](https://pypi.python.org/pypi/versioneer/) * [![Build Status] (https://travis-ci.org/warner/python-versioneer.png?branch=master) ](https://travis-ci.org/warner/python-versioneer) This is a tool for managing a recorded version number in distutils-based python projects. The goal is to remove the tedious and error-prone "update the embedded version string" step from your release process. Making a new release should be as easy as recording a new tag in your version-control system, and maybe making new tarballs. ## Quick Install * `pip install versioneer` to somewhere to your $PATH * add a `[versioneer]` section to your setup.cfg (see below) * run `versioneer install` in your source tree, commit the results ## Version Identifiers Source trees come from a variety of places: * a version-control system checkout (mostly used by developers) * a nightly tarball, produced by build automation * a snapshot tarball, produced by a web-based VCS browser, like github's "tarball from tag" feature * a release tarball, produced by "setup.py sdist", distributed through PyPI Within each source tree, the version identifier (either a string or a number, this tool is format-agnostic) can come from a variety of places: * ask the VCS tool itself, e.g. "git describe" (for checkouts), which knows about recent "tags" and an absolute revision-id * the name of the directory into which the tarball was unpacked * an expanded VCS keyword ($Id$, etc) * a `_version.py` created by some earlier build step For released software, the version identifier is closely related to a VCS tag. Some projects use tag names that include more than just the version string (e.g. "myproject-1.2" instead of just "1.2"), in which case the tool needs to strip the tag prefix to extract the version identifier. For unreleased software (between tags), the version identifier should provide enough information to help developers recreate the same tree, while also giving them an idea of roughly how old the tree is (after version 1.2, before version 1.3). Many VCS systems can report a description that captures this, for example `git describe --tags --dirty --always` reports things like "0.7-1-g574ab98-dirty" to indicate that the checkout is one revision past the 0.7 tag, has a unique revision id of "574ab98", and is "dirty" (it has uncommitted changes. The version identifier is used for multiple purposes: * to allow the module to self-identify its version: `myproject.__version__` * to choose a name and prefix for a 'setup.py sdist' tarball ## Theory of Operation Versioneer works by adding a special `_version.py` file into your source tree, where your `__init__.py` can import it. This `_version.py` knows how to dynamically ask the VCS tool for version information at import time. `_version.py` also contains `$Revision$` markers, and the installation process marks `_version.py` to have this marker rewritten with a tag name during the `git archive` command. As a result, generated tarballs will contain enough information to get the proper version. To allow `setup.py` to compute a version too, a `versioneer.py` is added to the top level of your source tree, next to `setup.py` and the `setup.cfg` that configures it. This overrides several distutils/setuptools commands to compute the version when invoked, and changes `setup.py build` and `setup.py sdist` to replace `_version.py` with a small static file that contains just the generated version data. ## Installation See [INSTALL.md](./INSTALL.md) for detailed installation instructions. ## Version-String Flavors Code which uses Versioneer can learn about its version string at runtime by importing `_version` from your main `__init__.py` file and running the `get_versions()` function. From the "outside" (e.g. in `setup.py`), you can import the top-level `versioneer.py` and run `get_versions()`. Both functions return a dictionary with different flavors of version information: * `['version']`: A condensed version string, rendered using the selected style. This is the most commonly used value for the project's version string. The default "pep440" style yields strings like `0.11`, `0.11+2.g1076c97`, or `0.11+2.g1076c97.dirty`. See the "Styles" section below for alternative styles. * `['full-revisionid']`: detailed revision identifier. For Git, this is the full SHA1 commit id, e.g. "1076c978a8d3cfc70f408fe5974aa6c092c949ac". * `['date']`: Date and time of the latest `HEAD` commit. For Git, it is the commit date in ISO 8601 format. This will be None if the date is not available. * `['dirty']`: a boolean, True if the tree has uncommitted changes. Note that this is only accurate if run in a VCS checkout, otherwise it is likely to be False or None * `['error']`: if the version string could not be computed, this will be set to a string describing the problem, otherwise it will be None. It may be useful to throw an exception in setup.py if this is set, to avoid e.g. creating tarballs with a version string of "unknown". Some variants are more useful than others. Including `full-revisionid` in a bug report should allow developers to reconstruct the exact code being tested (or indicate the presence of local changes that should be shared with the developers). `version` is suitable for display in an "about" box or a CLI `--version` output: it can be easily compared against release notes and lists of bugs fixed in various releases. The installer adds the following text to your `__init__.py` to place a basic version in `YOURPROJECT.__version__`: from ._version import get_versions __version__ = get_versions()['version'] del get_versions ## Styles The setup.cfg `style=` configuration controls how the VCS information is rendered into a version string. The default style, "pep440", produces a PEP440-compliant string, equal to the un-prefixed tag name for actual releases, and containing an additional "local version" section with more detail for in-between builds. For Git, this is TAG[+DISTANCE.gHEX[.dirty]] , using information from `git describe --tags --dirty --always`. For example "0.11+2.g1076c97.dirty" indicates that the tree is like the "1076c97" commit but has uncommitted changes (".dirty"), and that this commit is two revisions ("+2") beyond the "0.11" tag. For released software (exactly equal to a known tag), the identifier will only contain the stripped tag, e.g. "0.11". Other styles are available. See [details.md](details.md) in the Versioneer source tree for descriptions. ## Debugging Versioneer tries to avoid fatal errors: if something goes wrong, it will tend to return a version of "0+unknown". To investigate the problem, run `setup.py version`, which will run the version-lookup code in a verbose mode, and will display the full contents of `get_versions()` (including the `error` string, which may help identify what went wrong). ## Known Limitations Some situations are known to cause problems for Versioneer. This details the most significant ones. More can be found on Github [issues page](https://github.com/warner/python-versioneer/issues). ### Subprojects Versioneer has limited support for source trees in which `setup.py` is not in the root directory (e.g. `setup.py` and `.git/` are *not* siblings). The are two common reasons why `setup.py` might not be in the root: * Source trees which contain multiple subprojects, such as [Buildbot](https://github.com/buildbot/buildbot), which contains both "master" and "slave" subprojects, each with their own `setup.py`, `setup.cfg`, and `tox.ini`. Projects like these produce multiple PyPI distributions (and upload multiple independently-installable tarballs). * Source trees whose main purpose is to contain a C library, but which also provide bindings to Python (and perhaps other langauges) in subdirectories. Versioneer will look for `.git` in parent directories, and most operations should get the right version string. However `pip` and `setuptools` have bugs and implementation details which frequently cause `pip install .` from a subproject directory to fail to find a correct version string (so it usually defaults to `0+unknown`). `pip install --editable .` should work correctly. `setup.py install` might work too. Pip-8.1.1 is known to have this problem, but hopefully it will get fixed in some later version. [Bug #38](https://github.com/warner/python-versioneer/issues/38) is tracking this issue. The discussion in [PR #61](https://github.com/warner/python-versioneer/pull/61) describes the issue from the Versioneer side in more detail. [pip PR#3176](https://github.com/pypa/pip/pull/3176) and [pip PR#3615](https://github.com/pypa/pip/pull/3615) contain work to improve pip to let Versioneer work correctly. Versioneer-0.16 and earlier only looked for a `.git` directory next to the `setup.cfg`, so subprojects were completely unsupported with those releases. ### Editable installs with setuptools <= 18.5 `setup.py develop` and `pip install --editable .` allow you to install a project into a virtualenv once, then continue editing the source code (and test) without re-installing after every change. "Entry-point scripts" (`setup(entry_points={"console_scripts": ..})`) are a convenient way to specify executable scripts that should be installed along with the python package. These both work as expected when using modern setuptools. When using setuptools-18.5 or earlier, however, certain operations will cause `pkg_resources.DistributionNotFound` errors when running the entrypoint script, which must be resolved by re-installing the package. This happens when the install happens with one version, then the egg_info data is regenerated while a different version is checked out. Many setup.py commands cause egg_info to be rebuilt (including `sdist`, `wheel`, and installing into a different virtualenv), so this can be surprising. [Bug #83](https://github.com/warner/python-versioneer/issues/83) describes this one, but upgrading to a newer version of setuptools should probably resolve it. ### Unicode version strings While Versioneer works (and is continually tested) with both Python 2 and Python 3, it is not entirely consistent with bytes-vs-unicode distinctions. Newer releases probably generate unicode version strings on py2. It's not clear that this is wrong, but it may be surprising for applications when then write these strings to a network connection or include them in bytes-oriented APIs like cryptographic checksums. [Bug #71](https://github.com/warner/python-versioneer/issues/71) investigates this question. ## Updating Versioneer To upgrade your project to a new release of Versioneer, do the following: * install the new Versioneer (`pip install -U versioneer` or equivalent) * edit `setup.cfg`, if necessary, to include any new configuration settings indicated by the release notes. See [UPGRADING](./UPGRADING.md) for details. * re-run `versioneer install` in your source tree, to replace `SRC/_version.py` * commit any changed files ## Future Directions This tool is designed to make it easily extended to other version-control systems: all VCS-specific components are in separate directories like src/git/ . The top-level `versioneer.py` script is assembled from these components by running make-versioneer.py . In the future, make-versioneer.py will take a VCS name as an argument, and will construct a version of `versioneer.py` that is specific to the given VCS. It might also take the configuration arguments that are currently provided manually during installation by editing setup.py . Alternatively, it might go the other direction and include code from all supported VCS systems, reducing the number of intermediate scripts. ## License To make Versioneer easier to embed, all its code is dedicated to the public domain. The `_version.py` that it creates is also in the public domain. Specifically, both are released under the Creative Commons "Public Domain Dedication" license (CC0-1.0), as described in https://creativecommons.org/publicdomain/zero/1.0/ . """ from __future__ import print_function try: import configparser except ImportError: import ConfigParser as configparser import errno import json import os import re import subprocess import sys class VersioneerConfig: """Container for Versioneer configuration parameters.""" def get_root(): """Get the project root directory. We require that all commands are run from the project root, i.e. the directory that contains setup.py, setup.cfg, and versioneer.py . """ root = os.path.realpath(os.path.abspath(os.getcwd())) setup_py = os.path.join(root, "setup.py") versioneer_py = os.path.join(root, "versioneer.py") if not (os.path.exists(setup_py) or os.path.exists(versioneer_py)): # allow 'python path/to/setup.py COMMAND' root = os.path.dirname(os.path.realpath(os.path.abspath(sys.argv[0]))) setup_py = os.path.join(root, "setup.py") versioneer_py = os.path.join(root, "versioneer.py") if not (os.path.exists(setup_py) or os.path.exists(versioneer_py)): err = ("Versioneer was unable to run the project root directory. " "Versioneer requires setup.py to be executed from " "its immediate directory (like 'python setup.py COMMAND'), " "or in a way that lets it use sys.argv[0] to find the root " "(like 'python path/to/setup.py COMMAND').") raise VersioneerBadRootError(err) try: # Certain runtime workflows (setup.py install/develop in a setuptools # tree) execute all dependencies in a single python process, so # "versioneer" may be imported multiple times, and python's shared # module-import table will cache the first one. So we can't use # os.path.dirname(__file__), as that will find whichever # versioneer.py was first imported, even in later projects. me = os.path.realpath(os.path.abspath(__file__)) me_dir = os.path.normcase(os.path.splitext(me)[0]) vsr_dir = os.path.normcase(os.path.splitext(versioneer_py)[0]) if me_dir != vsr_dir: print("Warning: build in %s is using versioneer.py from %s" % (os.path.dirname(me), versioneer_py)) except NameError: pass return root def get_config_from_root(root): """Read the project setup.cfg file to determine Versioneer config.""" # This might raise EnvironmentError (if setup.cfg is missing), or # configparser.NoSectionError (if it lacks a [versioneer] section), or # configparser.NoOptionError (if it lacks "VCS="). See the docstring at # the top of versioneer.py for instructions on writing your setup.cfg . setup_cfg = os.path.join(root, "setup.cfg") parser = configparser.SafeConfigParser() with open(setup_cfg, "r") as f: parser.readfp(f) VCS = parser.get("versioneer", "VCS") # mandatory def get(parser, name): if parser.has_option("versioneer", name): return parser.get("versioneer", name) return None cfg = VersioneerConfig() cfg.VCS = VCS cfg.style = get(parser, "style") or "" cfg.versionfile_source = get(parser, "versionfile_source") cfg.versionfile_build = get(parser, "versionfile_build") cfg.tag_prefix = get(parser, "tag_prefix") if cfg.tag_prefix in ("''", '""'): cfg.tag_prefix = "" cfg.parentdir_prefix = get(parser, "parentdir_prefix") cfg.verbose = get(parser, "verbose") return cfg class NotThisMethod(Exception): """Exception raised if a method is not valid for the current scenario.""" # these dictionaries contain VCS-specific tools LONG_VERSION_PY = {} HANDLERS = {} def register_vcs_handler(vcs, method): # decorator """Decorator to mark a method as the handler for a particular VCS.""" def decorate(f): """Store f in HANDLERS[vcs][method].""" if vcs not in HANDLERS: HANDLERS[vcs] = {} HANDLERS[vcs][method] = f return f return decorate def run_command(commands, args, cwd=None, verbose=False, hide_stderr=False, env=None): """Call the given command(s).""" assert isinstance(commands, list) p = None for c in commands: try: dispcmd = str([c] + args) # remember shell=False, so use git.cmd on windows, not just git p = subprocess.Popen([c] + args, cwd=cwd, env=env, stdout=subprocess.PIPE, stderr=(subprocess.PIPE if hide_stderr else None)) break except EnvironmentError: e = sys.exc_info()[1] if e.errno == errno.ENOENT: continue if verbose: print("unable to run %s" % dispcmd) print(e) return None, None else: if verbose: print("unable to find command, tried %s" % (commands,)) return None, None stdout = p.communicate()[0].strip() if sys.version_info[0] >= 3: stdout = stdout.decode() if p.returncode != 0: if verbose: print("unable to run %s (error)" % dispcmd) print("stdout was %s" % stdout) return None, p.returncode return stdout, p.returncode LONG_VERSION_PY['git'] = ''' # This file helps to compute a version number in source trees obtained from # git-archive tarball (such as those provided by githubs download-from-tag # feature). Distribution tarballs (built by setup.py sdist) and build # directories (produced by setup.py build) will contain a much shorter file # that just contains the computed version number. # This file is released into the public domain. Generated by # versioneer-0.18 (https://github.com/warner/python-versioneer) """Git implementation of _version.py.""" import errno import os import re import subprocess import sys def get_keywords(): """Get the keywords needed to look up the version information.""" # these strings will be replaced by git during git-archive. # setup.py/versioneer.py will grep for the variable names, so they must # each be defined on a line of their own. _version.py will just call # get_keywords(). git_refnames = "%(DOLLAR)sFormat:%%d%(DOLLAR)s" git_full = "%(DOLLAR)sFormat:%%H%(DOLLAR)s" git_date = "%(DOLLAR)sFormat:%%ci%(DOLLAR)s" keywords = {"refnames": git_refnames, "full": git_full, "date": git_date} return keywords class VersioneerConfig: """Container for Versioneer configuration parameters.""" def get_config(): """Create, populate and return the VersioneerConfig() object.""" # these strings are filled in when 'setup.py versioneer' creates # _version.py cfg = VersioneerConfig() cfg.VCS = "git" cfg.style = "%(STYLE)s" cfg.tag_prefix = "%(TAG_PREFIX)s" cfg.parentdir_prefix = "%(PARENTDIR_PREFIX)s" cfg.versionfile_source = "%(VERSIONFILE_SOURCE)s" cfg.verbose = False return cfg class NotThisMethod(Exception): """Exception raised if a method is not valid for the current scenario.""" LONG_VERSION_PY = {} HANDLERS = {} def register_vcs_handler(vcs, method): # decorator """Decorator to mark a method as the handler for a particular VCS.""" def decorate(f): """Store f in HANDLERS[vcs][method].""" if vcs not in HANDLERS: HANDLERS[vcs] = {} HANDLERS[vcs][method] = f return f return decorate def run_command(commands, args, cwd=None, verbose=False, hide_stderr=False, env=None): """Call the given command(s).""" assert isinstance(commands, list) p = None for c in commands: try: dispcmd = str([c] + args) # remember shell=False, so use git.cmd on windows, not just git p = subprocess.Popen([c] + args, cwd=cwd, env=env, stdout=subprocess.PIPE, stderr=(subprocess.PIPE if hide_stderr else None)) break except EnvironmentError: e = sys.exc_info()[1] if e.errno == errno.ENOENT: continue if verbose: print("unable to run %%s" %% dispcmd) print(e) return None, None else: if verbose: print("unable to find command, tried %%s" %% (commands,)) return None, None stdout = p.communicate()[0].strip() if sys.version_info[0] >= 3: stdout = stdout.decode() if p.returncode != 0: if verbose: print("unable to run %%s (error)" %% dispcmd) print("stdout was %%s" %% stdout) return None, p.returncode return stdout, p.returncode def versions_from_parentdir(parentdir_prefix, root, verbose): """Try to determine the version from the parent directory name. Source tarballs conventionally unpack into a directory that includes both the project name and a version string. We will also support searching up two directory levels for an appropriately named parent directory """ rootdirs = [] for i in range(3): dirname = os.path.basename(root) if dirname.startswith(parentdir_prefix): return {"version": dirname[len(parentdir_prefix):], "full-revisionid": None, "dirty": False, "error": None, "date": None} else: rootdirs.append(root) root = os.path.dirname(root) # up a level if verbose: print("Tried directories %%s but none started with prefix %%s" %% (str(rootdirs), parentdir_prefix)) raise NotThisMethod("rootdir doesn't start with parentdir_prefix") @register_vcs_handler("git", "get_keywords") def git_get_keywords(versionfile_abs): """Extract version information from the given file.""" # the code embedded in _version.py can just fetch the value of these # keywords. When used from setup.py, we don't want to import _version.py, # so we do it with a regexp instead. This function is not used from # _version.py. keywords = {} try: f = open(versionfile_abs, "r") for line in f.readlines(): if line.strip().startswith("git_refnames ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["refnames"] = mo.group(1) if line.strip().startswith("git_full ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["full"] = mo.group(1) if line.strip().startswith("git_date ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["date"] = mo.group(1) f.close() except EnvironmentError: pass return keywords @register_vcs_handler("git", "keywords") def git_versions_from_keywords(keywords, tag_prefix, verbose): """Get version information from git keywords.""" if not keywords: raise NotThisMethod("no keywords at all, weird") date = keywords.get("date") if date is not None: # git-2.2.0 added "%%cI", which expands to an ISO-8601 -compliant # datestamp. However we prefer "%%ci" (which expands to an "ISO-8601 # -like" string, which we must then edit to make compliant), because # it's been around since git-1.5.3, and it's too difficult to # discover which version we're using, or to work around using an # older one. date = date.strip().replace(" ", "T", 1).replace(" ", "", 1) refnames = keywords["refnames"].strip() if refnames.startswith("$Format"): if verbose: print("keywords are unexpanded, not using") raise NotThisMethod("unexpanded keywords, not a git-archive tarball") refs = set([r.strip() for r in refnames.strip("()").split(",")]) # starting in git-1.8.3, tags are listed as "tag: foo-1.0" instead of # just "foo-1.0". If we see a "tag: " prefix, prefer those. TAG = "tag: " tags = set([r[len(TAG):] for r in refs if r.startswith(TAG)]) if not tags: # Either we're using git < 1.8.3, or there really are no tags. We use # a heuristic: assume all version tags have a digit. The old git %%d # expansion behaves like git log --decorate=short and strips out the # refs/heads/ and refs/tags/ prefixes that would let us distinguish # between branches and tags. By ignoring refnames without digits, we # filter out many common branch names like "release" and # "stabilization", as well as "HEAD" and "master". tags = set([r for r in refs if re.search(r'\d', r)]) if verbose: print("discarding '%%s', no digits" %% ",".join(refs - tags)) if verbose: print("likely tags: %%s" %% ",".join(sorted(tags))) for ref in sorted(tags): # sorting will prefer e.g. "2.0" over "2.0rc1" if ref.startswith(tag_prefix): r = ref[len(tag_prefix):] if verbose: print("picking %%s" %% r) return {"version": r, "full-revisionid": keywords["full"].strip(), "dirty": False, "error": None, "date": date} # no suitable tags, so version is "0+unknown", but full hex is still there if verbose: print("no suitable tags, using unknown + full revision id") return {"version": "0+unknown", "full-revisionid": keywords["full"].strip(), "dirty": False, "error": "no suitable tags", "date": None} @register_vcs_handler("git", "pieces_from_vcs") def git_pieces_from_vcs(tag_prefix, root, verbose, run_command=run_command): """Get version from 'git describe' in the root of the source tree. This only gets called if the git-archive 'subst' keywords were *not* expanded, and _version.py hasn't already been rewritten with a short version string, meaning we're inside a checked out source tree. """ GITS = ["git"] if sys.platform == "win32": GITS = ["git.cmd", "git.exe"] out, rc = run_command(GITS, ["rev-parse", "--git-dir"], cwd=root, hide_stderr=True) if rc != 0: if verbose: print("Directory %%s not under git control" %% root) raise NotThisMethod("'git rev-parse --git-dir' returned error") # if there is a tag matching tag_prefix, this yields TAG-NUM-gHEX[-dirty] # if there isn't one, this yields HEX[-dirty] (no NUM) describe_out, rc = run_command(GITS, ["describe", "--tags", "--dirty", "--always", "--long", "--match", "%%s*" %% tag_prefix], cwd=root) # --long was added in git-1.5.5 if describe_out is None: raise NotThisMethod("'git describe' failed") describe_out = describe_out.strip() full_out, rc = run_command(GITS, ["rev-parse", "HEAD"], cwd=root) if full_out is None: raise NotThisMethod("'git rev-parse' failed") full_out = full_out.strip() pieces = {} pieces["long"] = full_out pieces["short"] = full_out[:7] # maybe improved later pieces["error"] = None # parse describe_out. It will be like TAG-NUM-gHEX[-dirty] or HEX[-dirty] # TAG might have hyphens. git_describe = describe_out # look for -dirty suffix dirty = git_describe.endswith("-dirty") pieces["dirty"] = dirty if dirty: git_describe = git_describe[:git_describe.rindex("-dirty")] # now we have TAG-NUM-gHEX or HEX if "-" in git_describe: # TAG-NUM-gHEX mo = re.search(r'^(.+)-(\d+)-g([0-9a-f]+)$', git_describe) if not mo: # unparseable. Maybe git-describe is misbehaving? pieces["error"] = ("unable to parse git-describe output: '%%s'" %% describe_out) return pieces # tag full_tag = mo.group(1) if not full_tag.startswith(tag_prefix): if verbose: fmt = "tag '%%s' doesn't start with prefix '%%s'" print(fmt %% (full_tag, tag_prefix)) pieces["error"] = ("tag '%%s' doesn't start with prefix '%%s'" %% (full_tag, tag_prefix)) return pieces pieces["closest-tag"] = full_tag[len(tag_prefix):] # distance: number of commits since tag pieces["distance"] = int(mo.group(2)) # commit: short hex revision ID pieces["short"] = mo.group(3) else: # HEX: no tags pieces["closest-tag"] = None count_out, rc = run_command(GITS, ["rev-list", "HEAD", "--count"], cwd=root) pieces["distance"] = int(count_out) # total number of commits # commit date: see ISO-8601 comment in git_versions_from_keywords() date = run_command(GITS, ["show", "-s", "--format=%%ci", "HEAD"], cwd=root)[0].strip() pieces["date"] = date.strip().replace(" ", "T", 1).replace(" ", "", 1) return pieces def plus_or_dot(pieces): """Return a + if we don't already have one, else return a .""" if "+" in pieces.get("closest-tag", ""): return "." return "+" def render_pep440(pieces): """Build up version string, with post-release "local version identifier". Our goal: TAG[+DISTANCE.gHEX[.dirty]] . Note that if you get a tagged build and then dirty it, you'll get TAG+0.gHEX.dirty Exceptions: 1: no tags. git_describe was just HEX. 0+untagged.DISTANCE.gHEX[.dirty] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += plus_or_dot(pieces) rendered += "%%d.g%%s" %% (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" else: # exception #1 rendered = "0+untagged.%%d.g%%s" %% (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" return rendered def render_pep440_pre(pieces): """TAG[.post.devDISTANCE] -- No -dirty. Exceptions: 1: no tags. 0.post.devDISTANCE """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += ".post.dev%%d" %% pieces["distance"] else: # exception #1 rendered = "0.post.dev%%d" %% pieces["distance"] return rendered def render_pep440_post(pieces): """TAG[.postDISTANCE[.dev0]+gHEX] . The ".dev0" means dirty. Note that .dev0 sorts backwards (a dirty tree will appear "older" than the corresponding clean one), but you shouldn't be releasing software with -dirty anyways. Exceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += plus_or_dot(pieces) rendered += "g%%s" %% pieces["short"] else: # exception #1 rendered = "0.post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += "+g%%s" %% pieces["short"] return rendered def render_pep440_old(pieces): """TAG[.postDISTANCE[.dev0]] . The ".dev0" means dirty. Eexceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" else: # exception #1 rendered = "0.post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" return rendered def render_git_describe(pieces): """TAG[-DISTANCE-gHEX][-dirty]. Like 'git describe --tags --dirty --always'. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += "-%%d-g%%s" %% (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render_git_describe_long(pieces): """TAG-DISTANCE-gHEX[-dirty]. Like 'git describe --tags --dirty --always -long'. The distance/hash is unconditional. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] rendered += "-%%d-g%%s" %% (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render(pieces, style): """Render the given version pieces into the requested style.""" if pieces["error"]: return {"version": "unknown", "full-revisionid": pieces.get("long"), "dirty": None, "error": pieces["error"], "date": None} if not style or style == "default": style = "pep440" # the default if style == "pep440": rendered = render_pep440(pieces) elif style == "pep440-pre": rendered = render_pep440_pre(pieces) elif style == "pep440-post": rendered = render_pep440_post(pieces) elif style == "pep440-old": rendered = render_pep440_old(pieces) elif style == "git-describe": rendered = render_git_describe(pieces) elif style == "git-describe-long": rendered = render_git_describe_long(pieces) else: raise ValueError("unknown style '%%s'" %% style) return {"version": rendered, "full-revisionid": pieces["long"], "dirty": pieces["dirty"], "error": None, "date": pieces.get("date")} def get_versions(): """Get version information or return default if unable to do so.""" # I am in _version.py, which lives at ROOT/VERSIONFILE_SOURCE. If we have # __file__, we can work backwards from there to the root. Some # py2exe/bbfreeze/non-CPython implementations don't do __file__, in which # case we can only use expanded keywords. cfg = get_config() verbose = cfg.verbose try: return git_versions_from_keywords(get_keywords(), cfg.tag_prefix, verbose) except NotThisMethod: pass try: root = os.path.realpath(__file__) # versionfile_source is the relative path from the top of the source # tree (where the .git directory might live) to this file. Invert # this to find the root from __file__. for i in cfg.versionfile_source.split('/'): root = os.path.dirname(root) except NameError: return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to find root of source tree", "date": None} try: pieces = git_pieces_from_vcs(cfg.tag_prefix, root, verbose) return render(pieces, cfg.style) except NotThisMethod: pass try: if cfg.parentdir_prefix: return versions_from_parentdir(cfg.parentdir_prefix, root, verbose) except NotThisMethod: pass return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to compute version", "date": None} ''' @register_vcs_handler("git", "get_keywords") def git_get_keywords(versionfile_abs): """Extract version information from the given file.""" # the code embedded in _version.py can just fetch the value of these # keywords. When used from setup.py, we don't want to import _version.py, # so we do it with a regexp instead. This function is not used from # _version.py. keywords = {} try: f = open(versionfile_abs, "r") for line in f.readlines(): if line.strip().startswith("git_refnames ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["refnames"] = mo.group(1) if line.strip().startswith("git_full ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["full"] = mo.group(1) if line.strip().startswith("git_date ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["date"] = mo.group(1) f.close() except EnvironmentError: pass return keywords @register_vcs_handler("git", "keywords") def git_versions_from_keywords(keywords, tag_prefix, verbose): """Get version information from git keywords.""" if not keywords: raise NotThisMethod("no keywords at all, weird") date = keywords.get("date") if date is not None: # git-2.2.0 added "%cI", which expands to an ISO-8601 -compliant # datestamp. However we prefer "%ci" (which expands to an "ISO-8601 # -like" string, which we must then edit to make compliant), because # it's been around since git-1.5.3, and it's too difficult to # discover which version we're using, or to work around using an # older one. date = date.strip().replace(" ", "T", 1).replace(" ", "", 1) refnames = keywords["refnames"].strip() if refnames.startswith("$Format"): if verbose: print("keywords are unexpanded, not using") raise NotThisMethod("unexpanded keywords, not a git-archive tarball") refs = set([r.strip() for r in refnames.strip("()").split(",")]) # starting in git-1.8.3, tags are listed as "tag: foo-1.0" instead of # just "foo-1.0". If we see a "tag: " prefix, prefer those. TAG = "tag: " tags = set([r[len(TAG):] for r in refs if r.startswith(TAG)]) if not tags: # Either we're using git < 1.8.3, or there really are no tags. We use # a heuristic: assume all version tags have a digit. The old git %d # expansion behaves like git log --decorate=short and strips out the # refs/heads/ and refs/tags/ prefixes that would let us distinguish # between branches and tags. By ignoring refnames without digits, we # filter out many common branch names like "release" and # "stabilization", as well as "HEAD" and "master". tags = set([r for r in refs if re.search(r'\d', r)]) if verbose: print("discarding '%s', no digits" % ",".join(refs - tags)) if verbose: print("likely tags: %s" % ",".join(sorted(tags))) for ref in sorted(tags): # sorting will prefer e.g. "2.0" over "2.0rc1" if ref.startswith(tag_prefix): r = ref[len(tag_prefix):] if verbose: print("picking %s" % r) return {"version": r, "full-revisionid": keywords["full"].strip(), "dirty": False, "error": None, "date": date} # no suitable tags, so version is "0+unknown", but full hex is still there if verbose: print("no suitable tags, using unknown + full revision id") return {"version": "0+unknown", "full-revisionid": keywords["full"].strip(), "dirty": False, "error": "no suitable tags", "date": None} @register_vcs_handler("git", "pieces_from_vcs") def git_pieces_from_vcs(tag_prefix, root, verbose, run_command=run_command): """Get version from 'git describe' in the root of the source tree. This only gets called if the git-archive 'subst' keywords were *not* expanded, and _version.py hasn't already been rewritten with a short version string, meaning we're inside a checked out source tree. """ GITS = ["git"] if sys.platform == "win32": GITS = ["git.cmd", "git.exe"] out, rc = run_command(GITS, ["rev-parse", "--git-dir"], cwd=root, hide_stderr=True) if rc != 0: if verbose: print("Directory %s not under git control" % root) raise NotThisMethod("'git rev-parse --git-dir' returned error") # if there is a tag matching tag_prefix, this yields TAG-NUM-gHEX[-dirty] # if there isn't one, this yields HEX[-dirty] (no NUM) describe_out, rc = run_command(GITS, ["describe", "--tags", "--dirty", "--always", "--long", "--match", "%s*" % tag_prefix], cwd=root) # --long was added in git-1.5.5 if describe_out is None: raise NotThisMethod("'git describe' failed") describe_out = describe_out.strip() full_out, rc = run_command(GITS, ["rev-parse", "HEAD"], cwd=root) if full_out is None: raise NotThisMethod("'git rev-parse' failed") full_out = full_out.strip() pieces = {} pieces["long"] = full_out pieces["short"] = full_out[:7] # maybe improved later pieces["error"] = None # parse describe_out. It will be like TAG-NUM-gHEX[-dirty] or HEX[-dirty] # TAG might have hyphens. git_describe = describe_out # look for -dirty suffix dirty = git_describe.endswith("-dirty") pieces["dirty"] = dirty if dirty: git_describe = git_describe[:git_describe.rindex("-dirty")] # now we have TAG-NUM-gHEX or HEX if "-" in git_describe: # TAG-NUM-gHEX mo = re.search(r'^(.+)-(\d+)-g([0-9a-f]+)$', git_describe) if not mo: # unparseable. Maybe git-describe is misbehaving? pieces["error"] = ("unable to parse git-describe output: '%s'" % describe_out) return pieces # tag full_tag = mo.group(1) if not full_tag.startswith(tag_prefix): if verbose: fmt = "tag '%s' doesn't start with prefix '%s'" print(fmt % (full_tag, tag_prefix)) pieces["error"] = ("tag '%s' doesn't start with prefix '%s'" % (full_tag, tag_prefix)) return pieces pieces["closest-tag"] = full_tag[len(tag_prefix):] # distance: number of commits since tag pieces["distance"] = int(mo.group(2)) # commit: short hex revision ID pieces["short"] = mo.group(3) else: # HEX: no tags pieces["closest-tag"] = None count_out, rc = run_command(GITS, ["rev-list", "HEAD", "--count"], cwd=root) pieces["distance"] = int(count_out) # total number of commits # commit date: see ISO-8601 comment in git_versions_from_keywords() date = run_command(GITS, ["show", "-s", "--format=%ci", "HEAD"], cwd=root)[0].strip() pieces["date"] = date.strip().replace(" ", "T", 1).replace(" ", "", 1) return pieces def do_vcs_install(manifest_in, versionfile_source, ipy): """Git-specific installation logic for Versioneer. For Git, this means creating/changing .gitattributes to mark _version.py for export-subst keyword substitution. """ GITS = ["git"] if sys.platform == "win32": GITS = ["git.cmd", "git.exe"] files = [manifest_in, versionfile_source] if ipy: files.append(ipy) try: me = __file__ if me.endswith(".pyc") or me.endswith(".pyo"): me = os.path.splitext(me)[0] + ".py" versioneer_file = os.path.relpath(me) except NameError: versioneer_file = "versioneer.py" files.append(versioneer_file) present = False try: f = open(".gitattributes", "r") for line in f.readlines(): if line.strip().startswith(versionfile_source): if "export-subst" in line.strip().split()[1:]: present = True f.close() except EnvironmentError: pass if not present: f = open(".gitattributes", "a+") f.write("%s export-subst\n" % versionfile_source) f.close() files.append(".gitattributes") run_command(GITS, ["add", "--"] + files) def versions_from_parentdir(parentdir_prefix, root, verbose): """Try to determine the version from the parent directory name. Source tarballs conventionally unpack into a directory that includes both the project name and a version string. We will also support searching up two directory levels for an appropriately named parent directory """ rootdirs = [] for i in range(3): dirname = os.path.basename(root) if dirname.startswith(parentdir_prefix): return {"version": dirname[len(parentdir_prefix):], "full-revisionid": None, "dirty": False, "error": None, "date": None} else: rootdirs.append(root) root = os.path.dirname(root) # up a level if verbose: print("Tried directories %s but none started with prefix %s" % (str(rootdirs), parentdir_prefix)) raise NotThisMethod("rootdir doesn't start with parentdir_prefix") SHORT_VERSION_PY = """ # This file was generated by 'versioneer.py' (0.18) from # revision-control system data, or from the parent directory name of an # unpacked source archive. Distribution tarballs contain a pre-generated copy # of this file. import json version_json = ''' %s ''' # END VERSION_JSON def get_versions(): return json.loads(version_json) """ def versions_from_file(filename): """Try to determine the version from _version.py if present.""" try: with open(filename) as f: contents = f.read() except EnvironmentError: raise NotThisMethod("unable to read _version.py") mo = re.search(r"version_json = '''\n(.*)''' # END VERSION_JSON", contents, re.M | re.S) if not mo: mo = re.search(r"version_json = '''\r\n(.*)''' # END VERSION_JSON", contents, re.M | re.S) if not mo: raise NotThisMethod("no version_json in _version.py") return json.loads(mo.group(1)) def write_to_version_file(filename, versions): """Write the given version number to the given _version.py file.""" os.unlink(filename) contents = json.dumps(versions, sort_keys=True, indent=1, separators=(",", ": ")) with open(filename, "w") as f: f.write(SHORT_VERSION_PY % contents) print("set %s to '%s'" % (filename, versions["version"])) def plus_or_dot(pieces): """Return a + if we don't already have one, else return a .""" if "+" in pieces.get("closest-tag", ""): return "." return "+" def render_pep440(pieces): """Build up version string, with post-release "local version identifier". Our goal: TAG[+DISTANCE.gHEX[.dirty]] . Note that if you get a tagged build and then dirty it, you'll get TAG+0.gHEX.dirty Exceptions: 1: no tags. git_describe was just HEX. 0+untagged.DISTANCE.gHEX[.dirty] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += plus_or_dot(pieces) rendered += "%d.g%s" % (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" else: # exception #1 rendered = "0+untagged.%d.g%s" % (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" return rendered def render_pep440_pre(pieces): """TAG[.post.devDISTANCE] -- No -dirty. Exceptions: 1: no tags. 0.post.devDISTANCE """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += ".post.dev%d" % pieces["distance"] else: # exception #1 rendered = "0.post.dev%d" % pieces["distance"] return rendered def render_pep440_post(pieces): """TAG[.postDISTANCE[.dev0]+gHEX] . The ".dev0" means dirty. Note that .dev0 sorts backwards (a dirty tree will appear "older" than the corresponding clean one), but you shouldn't be releasing software with -dirty anyways. Exceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += plus_or_dot(pieces) rendered += "g%s" % pieces["short"] else: # exception #1 rendered = "0.post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += "+g%s" % pieces["short"] return rendered def render_pep440_old(pieces): """TAG[.postDISTANCE[.dev0]] . The ".dev0" means dirty. Eexceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" else: # exception #1 rendered = "0.post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" return rendered def render_git_describe(pieces): """TAG[-DISTANCE-gHEX][-dirty]. Like 'git describe --tags --dirty --always'. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += "-%d-g%s" % (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render_git_describe_long(pieces): """TAG-DISTANCE-gHEX[-dirty]. Like 'git describe --tags --dirty --always -long'. The distance/hash is unconditional. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] rendered += "-%d-g%s" % (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render(pieces, style): """Render the given version pieces into the requested style.""" if pieces["error"]: return {"version": "unknown", "full-revisionid": pieces.get("long"), "dirty": None, "error": pieces["error"], "date": None} if not style or style == "default": style = "pep440" # the default if style == "pep440": rendered = render_pep440(pieces) elif style == "pep440-pre": rendered = render_pep440_pre(pieces) elif style == "pep440-post": rendered = render_pep440_post(pieces) elif style == "pep440-old": rendered = render_pep440_old(pieces) elif style == "git-describe": rendered = render_git_describe(pieces) elif style == "git-describe-long": rendered = render_git_describe_long(pieces) else: raise ValueError("unknown style '%s'" % style) return {"version": rendered, "full-revisionid": pieces["long"], "dirty": pieces["dirty"], "error": None, "date": pieces.get("date")} class VersioneerBadRootError(Exception): """The project root directory is unknown or missing key files.""" def get_versions(verbose=False): """Get the project version from whatever source is available. Returns dict with two keys: 'version' and 'full'. """ if "versioneer" in sys.modules: # see the discussion in cmdclass.py:get_cmdclass() del sys.modules["versioneer"] root = get_root() cfg = get_config_from_root(root) assert cfg.VCS is not None, "please set [versioneer]VCS= in setup.cfg" handlers = HANDLERS.get(cfg.VCS) assert handlers, "unrecognized VCS '%s'" % cfg.VCS verbose = verbose or cfg.verbose assert cfg.versionfile_source is not None, \ "please set versioneer.versionfile_source" assert cfg.tag_prefix is not None, "please set versioneer.tag_prefix" versionfile_abs = os.path.join(root, cfg.versionfile_source) # extract version from first of: _version.py, VCS command (e.g. 'git # describe'), parentdir. This is meant to work for developers using a # source checkout, for users of a tarball created by 'setup.py sdist', # and for users of a tarball/zipball created by 'git archive' or github's # download-from-tag feature or the equivalent in other VCSes. get_keywords_f = handlers.get("get_keywords") from_keywords_f = handlers.get("keywords") if get_keywords_f and from_keywords_f: try: keywords = get_keywords_f(versionfile_abs) ver = from_keywords_f(keywords, cfg.tag_prefix, verbose) if verbose: print("got version from expanded keyword %s" % ver) return ver except NotThisMethod: pass try: ver = versions_from_file(versionfile_abs) if verbose: print("got version from file %s %s" % (versionfile_abs, ver)) return ver except NotThisMethod: pass from_vcs_f = handlers.get("pieces_from_vcs") if from_vcs_f: try: pieces = from_vcs_f(cfg.tag_prefix, root, verbose) ver = render(pieces, cfg.style) if verbose: print("got version from VCS %s" % ver) return ver except NotThisMethod: pass try: if cfg.parentdir_prefix: ver = versions_from_parentdir(cfg.parentdir_prefix, root, verbose) if verbose: print("got version from parentdir %s" % ver) return ver except NotThisMethod: pass if verbose: print("unable to compute version") return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to compute version", "date": None} def get_version(): """Get the short version string for this project.""" return get_versions()["version"] def get_cmdclass(): """Get the custom setuptools/distutils subclasses used by Versioneer.""" if "versioneer" in sys.modules: del sys.modules["versioneer"] # this fixes the "python setup.py develop" case (also 'install' and # 'easy_install .'), in which subdependencies of the main project are # built (using setup.py bdist_egg) in the same python process. Assume # a main project A and a dependency B, which use different versions # of Versioneer. A's setup.py imports A's Versioneer, leaving it in # sys.modules by the time B's setup.py is executed, causing B to run # with the wrong versioneer. Setuptools wraps the sub-dep builds in a # sandbox that restores sys.modules to it's pre-build state, so the # parent is protected against the child's "import versioneer". By # removing ourselves from sys.modules here, before the child build # happens, we protect the child from the parent's versioneer too. # Also see https://github.com/warner/python-versioneer/issues/52 cmds = {} # we add "version" to both distutils and setuptools from distutils.core import Command class cmd_version(Command): description = "report generated version string" user_options = [] boolean_options = [] def initialize_options(self): pass def finalize_options(self): pass def run(self): vers = get_versions(verbose=True) print("Version: %s" % vers["version"]) print(" full-revisionid: %s" % vers.get("full-revisionid")) print(" dirty: %s" % vers.get("dirty")) print(" date: %s" % vers.get("date")) if vers["error"]: print(" error: %s" % vers["error"]) cmds["version"] = cmd_version # we override "build_py" in both distutils and setuptools # # most invocation pathways end up running build_py: # distutils/build -> build_py # distutils/install -> distutils/build ->.. # setuptools/bdist_wheel -> distutils/install ->.. # setuptools/bdist_egg -> distutils/install_lib -> build_py # setuptools/install -> bdist_egg ->.. # setuptools/develop -> ? # pip install: # copies source tree to a tempdir before running egg_info/etc # if .git isn't copied too, 'git describe' will fail # then does setup.py bdist_wheel, or sometimes setup.py install # setup.py egg_info -> ? # we override different "build_py" commands for both environments if "setuptools" in sys.modules: from setuptools.command.build_py import build_py as _build_py else: from distutils.command.build_py import build_py as _build_py class cmd_build_py(_build_py): def run(self): root = get_root() cfg = get_config_from_root(root) versions = get_versions() _build_py.run(self) # now locate _version.py in the new build/ directory and replace # it with an updated value if cfg.versionfile_build: target_versionfile = os.path.join(self.build_lib, cfg.versionfile_build) print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, versions) cmds["build_py"] = cmd_build_py if "cx_Freeze" in sys.modules: # cx_freeze enabled? from cx_Freeze.dist import build_exe as _build_exe # nczeczulin reports that py2exe won't like the pep440-style string # as FILEVERSION, but it can be used for PRODUCTVERSION, e.g. # setup(console=[{ # "version": versioneer.get_version().split("+", 1)[0], # FILEVERSION # "product_version": versioneer.get_version(), # ... class cmd_build_exe(_build_exe): def run(self): root = get_root() cfg = get_config_from_root(root) versions = get_versions() target_versionfile = cfg.versionfile_source print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, versions) _build_exe.run(self) os.unlink(target_versionfile) with open(cfg.versionfile_source, "w") as f: LONG = LONG_VERSION_PY[cfg.VCS] f.write(LONG % {"DOLLAR": "$", "STYLE": cfg.style, "TAG_PREFIX": cfg.tag_prefix, "PARENTDIR_PREFIX": cfg.parentdir_prefix, "VERSIONFILE_SOURCE": cfg.versionfile_source, }) cmds["build_exe"] = cmd_build_exe del cmds["build_py"] if 'py2exe' in sys.modules: # py2exe enabled? try: from py2exe.distutils_buildexe import py2exe as _py2exe # py3 except ImportError: from py2exe.build_exe import py2exe as _py2exe # py2 class cmd_py2exe(_py2exe): def run(self): root = get_root() cfg = get_config_from_root(root) versions = get_versions() target_versionfile = cfg.versionfile_source print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, versions) _py2exe.run(self) os.unlink(target_versionfile) with open(cfg.versionfile_source, "w") as f: LONG = LONG_VERSION_PY[cfg.VCS] f.write(LONG % {"DOLLAR": "$", "STYLE": cfg.style, "TAG_PREFIX": cfg.tag_prefix, "PARENTDIR_PREFIX": cfg.parentdir_prefix, "VERSIONFILE_SOURCE": cfg.versionfile_source, }) cmds["py2exe"] = cmd_py2exe # we override different "sdist" commands for both environments if "setuptools" in sys.modules: from setuptools.command.sdist import sdist as _sdist else: from distutils.command.sdist import sdist as _sdist class cmd_sdist(_sdist): def run(self): versions = get_versions() self._versioneer_generated_versions = versions # unless we update this, the command will keep using the old # version self.distribution.metadata.version = versions["version"] return _sdist.run(self) def make_release_tree(self, base_dir, files): root = get_root() cfg = get_config_from_root(root) _sdist.make_release_tree(self, base_dir, files) # now locate _version.py in the new base_dir directory # (remembering that it may be a hardlink) and replace it with an # updated value target_versionfile = os.path.join(base_dir, cfg.versionfile_source) print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, self._versioneer_generated_versions) cmds["sdist"] = cmd_sdist return cmds CONFIG_ERROR = """ setup.cfg is missing the necessary Versioneer configuration. You need a section like: [versioneer] VCS = git style = pep440 versionfile_source = src/myproject/_version.py versionfile_build = myproject/_version.py tag_prefix = parentdir_prefix = myproject- You will also need to edit your setup.py to use the results: import versioneer setup(version=versioneer.get_version(), cmdclass=versioneer.get_cmdclass(), ...) Please read the docstring in ./versioneer.py for configuration instructions, edit setup.cfg, and re-run the installer or 'python versioneer.py setup'. """ SAMPLE_CONFIG = """ # See the docstring in versioneer.py for instructions. Note that you must # re-run 'versioneer.py setup' after changing this section, and commit the # resulting files. [versioneer] #VCS = git #style = pep440 #versionfile_source = #versionfile_build = #tag_prefix = #parentdir_prefix = """ INIT_PY_SNIPPET = """ from ._version import get_versions __version__ = get_versions()['version'] del get_versions """ def do_setup(): """Main VCS-independent setup function for installing Versioneer.""" root = get_root() try: cfg = get_config_from_root(root) except (EnvironmentError, configparser.NoSectionError, configparser.NoOptionError) as e: if isinstance(e, (EnvironmentError, configparser.NoSectionError)): print("Adding sample versioneer config to setup.cfg", file=sys.stderr) with open(os.path.join(root, "setup.cfg"), "a") as f: f.write(SAMPLE_CONFIG) print(CONFIG_ERROR, file=sys.stderr) return 1 print(" creating %s" % cfg.versionfile_source) with open(cfg.versionfile_source, "w") as f: LONG = LONG_VERSION_PY[cfg.VCS] f.write(LONG % {"DOLLAR": "$", "STYLE": cfg.style, "TAG_PREFIX": cfg.tag_prefix, "PARENTDIR_PREFIX": cfg.parentdir_prefix, "VERSIONFILE_SOURCE": cfg.versionfile_source, }) ipy = os.path.join(os.path.dirname(cfg.versionfile_source), "__init__.py") if os.path.exists(ipy): try: with open(ipy, "r") as f: old = f.read() except EnvironmentError: old = "" if INIT_PY_SNIPPET not in old: print(" appending to %s" % ipy) with open(ipy, "a") as f: f.write(INIT_PY_SNIPPET) else: print(" %s unmodified" % ipy) else: print(" %s doesn't exist, ok" % ipy) ipy = None # Make sure both the top-level "versioneer.py" and versionfile_source # (PKG/_version.py, used by runtime code) are in MANIFEST.in, so # they'll be copied into source distributions. Pip won't be able to # install the package without this. manifest_in = os.path.join(root, "MANIFEST.in") simple_includes = set() try: with open(manifest_in, "r") as f: for line in f: if line.startswith("include "): for include in line.split()[1:]: simple_includes.add(include) except EnvironmentError: pass # That doesn't cover everything MANIFEST.in can do # (http://docs.python.org/2/distutils/sourcedist.html#commands), so # it might give some false negatives. Appending redundant 'include' # lines is safe, though. if "versioneer.py" not in simple_includes: print(" appending 'versioneer.py' to MANIFEST.in") with open(manifest_in, "a") as f: f.write("include versioneer.py\n") else: print(" 'versioneer.py' already in MANIFEST.in") if cfg.versionfile_source not in simple_includes: print(" appending versionfile_source ('%s') to MANIFEST.in" % cfg.versionfile_source) with open(manifest_in, "a") as f: f.write("include %s\n" % cfg.versionfile_source) else: print(" versionfile_source already in MANIFEST.in") # Make VCS-specific changes. For git, this means creating/changing # .gitattributes to mark _version.py for export-subst keyword # substitution. do_vcs_install(manifest_in, cfg.versionfile_source, ipy) return 0 def scan_setup_py(): """Validate the contents of setup.py against Versioneer's expectations.""" found = set() setters = False errors = 0 with open("setup.py", "r") as f: for line in f.readlines(): if "import versioneer" in line: found.add("import") if "versioneer.get_cmdclass()" in line: found.add("cmdclass") if "versioneer.get_version()" in line: found.add("get_version") if "versioneer.VCS" in line: setters = True if "versioneer.versionfile_source" in line: setters = True if len(found) != 3: print("") print("Your setup.py appears to be missing some important items") print("(but I might be wrong). Please make sure it has something") print("roughly like the following:") print("") print(" import versioneer") print(" setup( version=versioneer.get_version(),") print(" cmdclass=versioneer.get_cmdclass(), ...)") print("") errors += 1 if setters: print("You should remove lines like 'versioneer.VCS = ' and") print("'versioneer.versionfile_source = ' . This configuration") print("now lives in setup.cfg, and should be removed from setup.py") print("") errors += 1 return errors if __name__ == "__main__": cmd = sys.argv[1] if cmd == "setup": errors = do_setup() errors += scan_setup_py() if errors: sys.exit(1)