zmat-0.9.9/0000755000175200007730000000000014515254731012737 5ustar rlaboissrlaboisszmat-0.9.9/AUTHORS.txt0000644000175200007730000000160214515254731014624 0ustar rlaboissrlaboissZMAT is written by Qianqian Fang. Qianqian is currently an Associate Professor in the Department of Bioengineering at Northeastern University. Address: Dept. of Bioengineering Northeastern University 360 Huntington Ave, ISEC 206, Boston, MA 02115, USA URL: http://fanglab.org Email: ZMAT Website : https://github.com/NeuroJSON/zmat The base64_{encode,decode} functions in the zmat.cpp functions were copied from http://web.mit.edu/freebsd/head/contrib/wpa/src/utils/base64.c http://web.mit.edu/freebsd/head/contrib/wpa/src/utils/base64.h written by Jouni Malinen licensed under the BSD license The optional LZMA/LZIP compression support is provided by the EazyLZMA library written by Lloyd Hilaiel. EazyLZMA was built on Igor Pavlov's LZMA library (both previous works were released under the public domain) https://github.com/lloyd/easylzma zmat-0.9.9/images/0000755000175200007730000000000014515254731014204 5ustar rlaboissrlaboisszmat-0.9.9/images/zmat_media.png0000755000175200007730000034762214515254731017045 0ustar rlaboissrlaboissPNG  IHDRx@/VgsBIT|d pHYs IDATxm$yܪ] .Ii%B ױ- DKBOz㓜>t#$ɇ}b?I?9$&w  Mp?YJ\vzEOOuݧnWﮔy~u}{ 7W ҂SBsAk#\Tȼqef<8lU8 ]]_;l)b]E 7Džҍ ȭ XÇRgqC pcS`6Az.⸍ȫMß?<yqGe]X5J* .?ղGh{y7[pDq]X]BFQEx& [%E*켯wl 6~ݦRqnm|ǹpV8~ '8V`'zj)g7Nt==Dͽ wd_?Mn[%H%C8"Vہp58qaE(Sg疤NEQ:?X V^ ׿zyݏ}:?u2z_ kS!UWM"}d_\YI{Jk@5"{u߸!o BhA^xF;j"T @)줷 Ehױ5 T$F:S؜?.t|E+9npSQd.FTur. /I@iw.-,5%EFACI2R;({`sm8$eKsP+w"h l"֍+VS5Jh7|5f‹s&q^\m;(>\Jn.$KD=mTE9_m0Md//=6ҞI"&hp6/ex)N8.<W?ngms⮯a x%u"jO$E*֛FN3Kp[(otϻr>/r8/$~vk3KRV!jbɫX*/KȇPB`gmX wF%M{b:A262ոb;B.:όN\*UOb-7 N+G2oXzZΨgp)Cee]MDɼyx.ʄ*l{)} ldkxV̓*0@<fSO({&(Q~Ӷ7kjZg=$對 a‰ oMa%o,Hu 6@8Md3 R'G}qC\mL]8)dU}+owcUMpg}Sĺ-e" zY}PD=C(ePC9Ij{`M.F'qyq7&PuJ"ֵK8J˾}d4΋aݗ ^ m{ur QlIřu55.K룒( ڜ[;k8{+Xc+;m>"/H1r\9,Z^-ZU_bT][#a"vQ:-nO)AmULkluF%wPkD}ULAqQ{SGpQܛA@E*I"jkXG^t@ 4Yi{96_0:|0hsߖZ$7wU?719*MwF' .vSq%L0 3{E\h. [@uvIRݏ3 &.w^$lSĚv1W`pw♴hWu]I'P9}^oQ }?V('fS,7g{6 ubs 0D1lXMNGm }ܟ~I*Dz0bXb4' ~c3]B8jFf*Q.f&Ɉ3n!51|lIX*ds)Mv>^~k(^,,v6W^=]6G,gލo7/e?|eIuM9<wK}Aԉ ^^H*ۇ8?#+N\ cAdq^xϩ B*Іt\fٗ NBlNaOy!w7E;>eDlmTXy"Ѥrfr=0 ]kU#nY{?7>΋[Ⱥu46YㆸlhFVN]<<6lr?ʾϫ;;])Ӹgԝ?n5JXSXSőYo^o!8aX w.=/R}ʁ6ঝ/p {Z~ ܑ^|R?,/+%`J-Io=aKa[? ,-B~/bzo1J'}a#Qۅ]˟P36 mOY4 =xdV(!i[dh> a8 ]ĕ#eq'ͩ/U'/]/2_MF{8̉"VqܮԠNScr=uǐ{ 1=\zNm_K^<2E,8?nr;l5lHpe@{ȯNsfkM?5fW_#\Şɗ֟TSְXLCpw#xi&n&{HSr$ T?5ғ;T\ğ|_<U:4CtFGI 0@ Uޓcܛpi MxJm^٪Zޙ  tޫ,c9򈍧a_NqVS*,ֱ\ƍᛗ wHߤ(c=x#gՖ Z>n6܊mkoso BV͊ʑxo`DԶNN _;s]zNRsN xoE? iX y4`cϭXd1EWY@wT3~۳p ~{1t2 s:8/=VRʓS*K%}'yxѵ& gT%5,7P?l,wP?^)ձst])J3>u^_$d_VRڋS*(CLT*~!kؔ)4 NyC[&w ͐tA;ig/Ua"k6zll%%W*j,I714ɲWJ.\kkH-"_w!n=86Oܫ)bڎFCGJv7΋ͤgت 0.׻fǸ8_>o?,'4'gq[/?6hm7E,hՏg~򏆷F1#{Q[Z%0:^a8t+Lhi\3ԋ(2e85|-N`3)SS6 `iP-[FH&*bby> #.˸d*ORW~=K~S >viK襕`>4t Xrj3;7{L>A0dJ;K.N#d$!XKWL9sP=\,]G=\VǾ6_wġRl8h]3^Y9{`-k:4yL/piPRĔ 9 =}n5}lْhŽ-bGQZ3K:;om{^ט\>d`r#%:cj#T!*{fY`lYY6E,&Ɔ"ĺۄakH${3Im\~׹ Qsao|l՘ryDܓS+3GjIk[#0j-CFtYb3 g\oscqN  aadKI0I6,vFTɘ7Z'dOxW#xOrVx~M{%=hxryf<tێ糩gn&a(&&gRބ͓[-SwM>J%}#?؋U T|C3 ,gdL)O2=e1V#O$bj4?˚lj0YI=#{_^վ"!JeùeApHϤ) .K$aa9C I22i#O(zvLlYQagf#xL*zJM%fLgM6YoXG{bޛ&ǓSz'Gus_~=3D[=eoYlh#opU#C2q_osX,S=,aa9cʐ{6=M,<Sj:2eW:<1|T39y`/'IiAEϐf6c{#Ɛ[PI`Qe.$mrc?@d:`=3(rOz4 2s~DٴU77>}g3 s$LoP#,g g.8qk?$s^GF>s>cc [7(N"L2C2eOqL8G]~5\dƎ0 3Ug^ Y" 8<)#n$,12~Ԇ}A˜D!,L=K²H{:0h b7sݩo<ؠ2 O Ky {c?J6S|jfj=F-xǕl6WrϬ{CG!A>G8FM53Yg)sdN]q8S?M䞊a#3ؤ@ʘpdLI@2 &18='q" g1Kyt\g=Ƚ G:+kN/ q{ȊL'k7ךf3QǽXӘ՞0uc"査Y7I=Eb rĬX#\gv(#OH3X&91COg!SWI'$qbйyff'f=^j^g҄601Lt C Y\1rT0uw"CoC:<$"qzdY-8Dhl<24zL|rX0!{#KCb\}oK;%Ds&Hz}/Sb6m5q{.zb!3U)bgg*gFߺ˚}%n`*ܛes!d ;ƪ{615vp<"?*I #$LSVdZb#3Noߙ282GE^_unU2Qg={a 161/'F IDAT=BAuWJ#oE1!AXkjfN7n7yD ޠaUܳa^I/{1?Oro(z̃k:֮|w30?D*bvXkn'sOAo0^ro/J}mNY<c;ecE202'-Eh%CE;dͦ)3)e65cDuJT;7E,ĎgcgƟ*?h2ɐK/9ۦsoLa9Эs>XCb'` 0uL¤oX챆-n_Gځ~^BOA}AN:fg}dKM#*!rr ~L4ַ]w&$B=;̬LazdƲ1=1dža\X9^Ǽ_֏IEJ%@h\ɀHdXvJ^SĊ4=7K/uHI-(%6az5"|L-2Iae*]_d8+ A=bGF7e$&vtSԶ s#„ܗ:orXKH3jb s2_!7n0||ۑ~yzFv\fncipyfȲ0!Ʋ.CeM my"b%uG4" ^uNfgX ;P)gC#؏#xMIj- ^!0UL-XF}g9A:('(3;3 qa$qRWP>̟HnqY%x7uE$K?"\,Ϛld$L߬ ֓JҜ)(%f?<$;e)H'׎<Ɣl6EhC5$C-B%bݼ"-bcYe__G*z٫Br17\2+c?2qlMtj(:ߋ3'2y;Rb*c.<ɽ7S-W׍Eg)fU<ϟTk{8Uvρ굠v'`;L܉mX3_xE3okTw:#/6KA%_8kg0 w~_Rw 闤Ж|vN(*<$.ZELƅ3oboV8Geߊ`{m~zW08/H/0@v?_;\w<^}ٟYr_tL=D%)Xao`}ρbH `ٓ^zBi,HuȺ%%/=&)b]_!7pcǀ9nێ`ڮp"Ȱ LlO*~i,1x=կ^ʁOQu}ljNO*Ί^w!:<~UL3X wǭLS]KRi]HA^x9$՛p'a%ͽvL _:P m>Gz@X\j3#w贁S*q]B`v{u~ǃecC0,mWKAC˾ R8g0#NyiԘ]{N~[/\ GDCP! ß_)$`uD ؞6v|X6`s~dϾ( I?Bn @VO!Hocّ8_rQ3K2"/$ɞ%J^xJ Ԟe>Q 6IP%dڗK ━߇BBQ.\FWyz_}F{)ַ4C=07h\=} }[3T7#@xzoؿ7.Ki$9Pv.sk>q?;0ܿVgPYrRb2Jz`^Hr,XKWB/ NNBQJ"Awm "$r׸!$w@rWaGEG,ml!ѻFaKl N@I=^7$yq% c} 8]#eϒ;Kr$WT쁾\7? l> \r )>G$i6U-^fBm]K)?""LvMa˸D C2P9"11Ra'/ȑe?BˊVJ=5)Ci0erGB|ɾ aX9wJ鰁pXH](%[YRB{0-. /F{\׆3IsOwx6*½ ASœ*C+Sh t23`Hb:rW85fu,GV_yLh Hvawd )W4XwH돩,(.sY^~Vm10 ?:$(:5vػD]T(KncAO=3܎n}/Q !:vp>WӉ VۣTa7JYaON6:7 !ݑ}^^L'.c̓43]yw {b=T]z<]ưz,w(H#pgU/@:~Խ[.=VaٴpVfZ#}߻"z%|5E;KU~~l,$W騨C+XˋDc}p}!ʧz$OJh'$ׇͨ/S*Q|"~k +Y]Tnob%kjKOo C:. opRؘʾ Ւv^tń )b-.<K5Dˆ7D}A=W!? leI:wQ-Nj/KR'm̲ou (UQu/3Q@* 0@[Wr$˾)VSp8u_BH>uSbhބS,9Pa SfH%xͦX7>zsf@M/A'kqB)[TU"1$ xNZN4'j.mh\RG%-'.qN֜[CU5/|Q\f{z_}F )NG۶8߲?KQUT .'{?B/-IwP ԰qduJ(Tz iy0 J ]i GkXoºkX_D1Kn tAօ30xML2}b#u#o/sBxV /KNyn6n\^A.GGd rE"{ƺGCDU@NOD= &\%о8Z$!b)TSh12=dr"MMu?լGKR8ct v0ܭV*Y {?*X5<ߑISb E~3=ƃmP™\NX9 a-nBlwPkㆸ*y|c_ŔX^uP/HB$7ُB5 J.6KRz,pꁛX?fะ>NQDKO7%rP3g.H9 3]Kf57E.F!Rc9mNZ pXg#{Srlǽ.*7qYJy˻b"ՋT3"1FIutk9}}Ay]lR=IJ3š(䞵MYQ8.{`ۡQ#Pg,Qw엥0#JUBq>m~:JiFAMK,PqzZGb '72zQ_gl˾؛* rjrhm7.eX\9 ͧaNlG*}B\m,y)IĶ>Nr%S q޿tJ]c>@*X P(|/a#8ʁ_YJx 15#[%QaTѡȐ['5lޭᎳ-f/ dA YOeSf>7%ehʾl2DDHQXԳ~M{ )oX,p W]xӎO˾ lR}64 j."AV*u,K~3*]_tiALSEnޱKjSroG<#O?gVJBD\n!:^lhoJ/mc=?=DTUuVVlD~+}7"sU&HѸ!mKHEB?ú+$Y}[=JUCU;]BYk /oLĸ.?-@qYqB؉)DTUB_Js@^"݋=b#Dr qw q Wjl*tw.iz#uؔWQpb@% geN}A䢊(OqX `nE6 mt0@j_F!*j"*y"%- TeǚSƶ\ A)!(@=!>рG}ACҩ=T#@/C5QJKuFE6rqs0zxeJg047c,n{bMP_#dE}$VF_úv"ěg^D)wA(y .ee$?3 F ٶy|[C̤)<. Kw":6^_L'x4D\5*XaPJZ VFml\uK~vLqAń+'ÛhvK`hXOJXԷgӋl驪OݘGo^R._+Dexabk~E@E} n!xA Ϫ^n<$xI7PغVt7SV0Nc&Ƃ-Mɽ'+y[\ *鶵.ZBSe t{dܒY8i8/n87=1rI%Qiq(qo69?Sh]YIzogf瞑ڂsCd t{< 9T&sKR5&rSHݡfeop!+/wJ̟UdKiDaO$_߄uKmaB?ܡxXNRq];v쮾O)]C3ϪTo2zj%.vu: l{t14P\'wͦC dKoC'{&xÀ75}s|"QP"s[/{'\ܐ*C3vd?$o_Џ9z/B'^ccB4Σ _BtcTohr׳L|M BqcJSD?) IDATJ7. %. fE_ŧ{F^4YNl;_u?_w7-{)%32!Xީ lA0r?{"Cވ5 45G~`E8N؇1x'chZk\䞹p~\ۛ} .W6 1KX#=F<.er Gk&~<&_k1Ys+l~cF~x&g=9\cELK z QuϐxX1y$u)sŢnBr0:tI85Kѯ[5r;UZ v$ֽw\"zU}Tw:~3 z8~L)|J;͕+C"xNnϼjwTEi?G,xDm=)ERc {+RaSOIğZ&nyL<P\9'g`LyeR~bƃD!dF`7 sGx~իBD:cDEO~]ەcثW~}W)=Fe#0edʻOfdBFĿ#"_._zu|VavEƝ 7~ͻk:Ty%Tz &Șe̤4bRaIG86wQpV3Oy&N4c8v(AaL߀9#QȽY >+rd d5د\g=իWɶx*o2]_$p"rv.O;^<|Lȥ˗/qo'Q>}V/[&m'OVz_IwnA#r=86իϟ3 vڵïED$A)٫W>%nw_+9ւMByfcU% h} }}{‹$u" ` DV=- ̂)7E_)}8~ 7'dNq» ͦ `MPO/:?+>wg |uc8kpwr'Oeӛ?מRV@m/!(#DY"2'"s;4m3=P{Һ{͈tkg" {+(k]SÃa۞oѯ|xn7%7eTdϱM ?F6P/5R&m;gsLN;{Inqy/Yd=gD8I?)]e X7mz`'_} >CJJ/]&2{ys.b*[&=`XkFdڹ8""G Cي~ oV 1LoGk 'n^-rp?Ñ_iᑿ1?心o:%v4FK6߃KM9Θ`LMg[,"E\Ɓ(EY64L!e&~ O@p־\b8zpYPqQp{j*S1AnM=9&b&Ǚ(%'9qQ<$!p >sW +_}/3ĞfP>|Bo,ovVo6Svɠ+wvn;C 9h;WF\Yn LBnK3=dG 8^mWpX"dx|] VsSmra3*,Ye<^,x)_0:_p72^ aj7jDܣ8/y>rNTq3sk[VP11}dsMDNL"p CM Ɨ u'Lw{))W*ey<{]}`X"~1ͽjZy;0y^w~d?#/ֻMr^-&8Wp}d7cWޗf8 V&dhY+SVhaDUw] n'I )EQ y^Q'N.U(x+wm,FKC`Q w*. mjI6+* f:HۓCS. dhC8癘 `Y]lK+Ҡ>rPNw Z~l;=Ө}dP]J:fATio; ҽ`w̽ X':;Uݳk\X,uB%sܝ?M0s2֘Ӈ1xr(@8t.?$sDC$Veּop,3gqf @)7wܹ80T*>C6 Ɉ2 /i?Z!:; foҋ}H胥z6MygY ${^ؾ+gV>`5 `;w6f,J{m16}j;>|ݤt\ޓp-Z<^%4aa&D$RtEn}63(ho۶0`SyuQ=R/~cYFo3) mNZL4Mɏ$],(f%݂1B*` vq W>@ 2۶7B$ H: 3 kw@lbAf@;]4|-3ȫ/ZJO njM Ht߹l:1im,̔q7gm^JB53= ǝs,yz0h+O,bX^@|m^["Fek='I˿^GCTvg!O#H.<4"frLZ"i ,Xhrʍ`<"My:`_3'YJUfZ5g\ ;;ۧySۖc^A@.i Di,̵sh/CGu(:ػu(JG\m"#2qv;Ldg,k7' !hNIsSx81xLC[΂#Y~{CHI f_3., }":bEE6h I g_ez@8۩)F|~#hH8k~Mv Vq0M՗d@8C`ggG8Lp2O` K.h7W} IDATmb S:zEoY3a2xDPC7wC3=J] @܂ucN/W@݇A! MW>Z$T 25g}d1_k0J VtSlxȂXn칙we$f  PܝJMqu>ywRyIg!t7⍩6耛~OЎ- ^/Kz<03u޼y.y[c"u5Hp,x}~Ӯ7Ÿ́;SH D &udήt5لju9mq= 3uH^4:n@KŴgO6|mw>_GRogoN&wDH)Y Bg͟?Jw̡OA}D rS3&(v-sHC^_3w05V-A%p$aa8NvEQ>Z>.\i`kdv`s4`p!3H_Ś[^`%'-4 (֟3{vd "" ƅTS.e_$͔?[M@b:$x5ǬH?W|kԫ*B $41;/N[B)ڼSOX̅&v ysи#-dZ2 r#fSӬ"DXnW) N5a`UY)SJ=0 s^)N`549bM9#7QPxnRO$g<+Z4;lQ9UǒkB{i-aݸ^$#U((&E9# vm2N ƙ?ۘ=n4ŷ쥨c9TH,AV +Ș&Rv.O۶mL$,\Z5' ?-@[72 '` b`Dz6x -ՄԱ_r& {q7EF lMrO8ۓ{')*(6 n94mflf˗Gt8pVpecmsY} BۃWMc&]x.7\#,l&*n:$f0&2mP" 1S Yg\yO KDĂNzZ 7nHF >uPfxyN>_BgаoX+@h^-'< ڡvs~,/a/ )jq 3>rPHC8  ,}iȏk3ylЙ][URQ:<6;2ۮ5+`=] .n$kg Q4}V{v>=pM8pJE X׵ݴ|<KJίqZ ?=Mы?ȇz Z!c_D`?51s>5aֈyvK_ԙ+·HʳVW0DY+ ޝ׬~ԳKQTWZ2x{,5dT0 Iaa'aF.Vm~7Y/؈w-#}sq=vO,_ /7~rZݽ( 5kA"-:RR~z9H% Cס\FUy9fÃO8I#x?m*B0g4 |dk \w:?xk5®%A$y>IoţW'|:e6.v4[قﰍk"w76k/?~r-. ʥ+>ק"O&@sD"|a* ;6`<_y؍Woqö=E"7|T Zߍ@dh mks_Jg3Tت[t^$O~'Ͷ[r%/:cxUDZ1DbtuKh2Xt0-`P&BS_EsD8ly3 á(0=f($7@dH`>]y] ,__ڽB x<̉hE7R %XsEȃ!Ĕ H [> VaH9&ŴW! ( Eoᤱ+Ɛ1>nqkö y#$z|[u}… OɧJ(d b $HKqZ䳊{$l-nSЇ0 K,/ wy1fA2pY}m-hda09:S݈\;LMd[Wa/ya#5l}ӴC=BwX4BoI Hd$1$# C~Yn+-8닺0A  4 p0e[)ꮢsWY6@D Zs=TO  & Af`Jޙqhlv2s  8i(Nx/rZ^?w=|pZ) WYw I#lc6[ضgf[2OR\ XwR-vWcVExqӔA'e˖<[޼y; t͓Zv^oڟ lٲ3~\uDӰ}Ι3+ۉX У(n^=<_dWE^~t$WGO2mz <tKIV`䊝OSw3n70*ЖJWsbԓ Iw3y?tm v+)զX񾌇?`$?HXrHFíON}}7^oC ^b\߂@Nyx5`yT>QhY֗F>3=vQN"ΓҦ2X52iKК'Ӯn0=8)i*܀@1Wh^M}`s*r:3pСe 4F TI"xr-irU?6 |U,2j  y&Z' ]@NڃTN` }?t\5޳};"j>8th 6>>P`J'΀l,6k'@V\tbmȫ-0lioHj-ƵӰ&|e'K veĉ儧_f$?i T]>2~FvMg1O;5s9-ƜYP#)躪FB=\ق24໻zۊSpP1[Uj}g<ݾdU.Mc?|2>7Y>({O&U Up"\% m܌IG{#1ؠɬ FBS^gYhL`>†K˦͋ѾvQ$I( ªnpyZ:L{-y,QkbFEVbyZUU0 @9m+3-]%UU0L%+KT޼Bhkă gL}~=Gm+YX<^Èz~^l!klݐzC?J0:S=n\@ YU]KNh6FG"^ĉ:CdNSpj}%)UBȉĥ-ۦl3]*UQTߘ'8;H6p.A(k^TG!1CH]'#=3@sp èA$ةnIXҮ8=08k&u8XwCUEKT,ȀJUNL fTBַ T]B2TZ`i}8<冄y3eMv]7ÿqr}%maNu*0))t IDAT7hRFaJ=/ z ҮUK_GkhKdȂFyTT_5dƃ^EB])bXbp\ܒ 'oTj=K'/Zy :COi:nQV @!UaĖQ(fI7GodeiwA,m1bW^bA: Xr爱u^V/Z缳&6'mg>gOX}GrB^]Aм q2Ruleo{4j,90&wKPi2T)UTy1>'xLFVj{;D[ G{Hb ?u V.ȢXbV9OMm-XU}lҮ˚nu&}>$STT=>f^ǁ+ ˖0RzSn҈0 ]:)|qk7{T]fMjǪ,;k ]ioqtf4H4IpS[wfXHc05t3N;W6t}nbvG3dPd ģ/[0JM~yB?K[sfm;/T~ZYز1䤭u7 7vwY {ȯ-ݚQU a[V}zn1/dq,L7%ofe#m_X?zS=-BnbqbMi)s n>r%H#GG:```dmI6E>K$;o-f4HHƨ#-w.*F8`ZpiNbdxj?ٛ˜_DIѡ}h`8s8S&h,sopzM]Wh \0@MLX ӝ4Oߗv}/WSqF5,l9ԘF4wr+`7Ͽ F2aљ V5"s+7$D4PLyl!VSn 8W| ˫[5> 8bsa4> M f307eSξfVww6 ogdj z<_EOtW̱! \TlDUȿ4QE1T>5UnS>Eel1Cw7=O W[*.E$lt-_˾Zaڍgw`HU#_zRJ!8A.ptƇrbcʹdNCޠ?͍ !*,8H$Zтw>H8̂O3Е r\}i'KQq2 Aھ)>/ k$,}'exЁo/ZomĂ[yCcG9+Xař_oβ)9'4,e'udD|RGbOP縹5M/{km-`~*iz&|+/u,ǜ9)R_TI;FQ?`1mTE7 `+S%@Feݩ-,g9a m|ƃ??<wh;r۫o 44xBy5x>-̘ѵ/\=v<؆ s K6Nn9ToZxmA+jSa2Ӈ $,}{w>BdJ(KKs=󪾄G-QT`bM\HEեwRh6r6icV2%U_T bmZDKKT5Jf*iڛ.h/$Kg!n WIь5=tMpT߭Ny/G+jpdϯ Ͽ$-fK5]Gru[e|5upCc?)ȇ~Ǻ"J&Ϲ' ^Ud"<=Wo*ljhT7eB*U@o\oyd677L$,W~ Zo ^k;-F[)QI'U[N!x* 쏢o6l[ Q^5b[Wx)/N%H/(fD=<^B>KVPH9u%VμynLWĪ`-38CQk1UŔjJ4nM1b>''oQe8O59v5o2m,F|{V v̡.69zbfbPe'X57s]GgFN@f*h.*uW0G~3P!ΕH#3OK֩7zC5qiOYtǓebiVd b?OAc̴ok}dt/AY_U|^Y i~?-7K dvH<`<ޘM'c *`upzYC)P/ [& 8 za5^?3m;/lΜ09vlWRFvag GNsFkVB 7Uyy{- tzvc.˥ VR!Z1-fSb[56 ҲWC|QėST-qk2@\rI-s#e3["Qp&%#Oxdq4kYI[DWQ~ t׿Ჴ-ͳ&)*>Ff~d|KN Be[׈--C $ hӤ^ζiV>%g90cihS okGnR2pB;oT/nT&%mk"g{ub%څ]![* n)*m,v KfǝyINj2Dv[FBZSP*W`<' $TW?h,FXXyc>{txo%e6\6<*Ot3"K'jړWo2PyUu틯_vF={;9,*r! %I쪥k[[4HeH~_Q6%{P::Woਡ}&0i?<Mnso;oz5K@X?{ëyp g`!J~d\}5)ټ"`@TBzhL_S>dJp-R{oBoҐ.y4ljз_^]r{hًSy mRJfaشmA~V(ѴvwK[((1OXO+%Z c=,` h*=#x_10oh=<±G z]Sr#(^ܰw4Z`F0Zh>Ͻ%?͋Beb+)!T< QWoSe1FWDBMlAZYvM`FnwFr~lHy0N+XZ<%b0$=Ӯ0 q|t 0tَ S|= )U**UA0x)Xd[5 Y"KR%@zcߵWUC]D/JKc+X5:7󠺵KG*:`wӅ:S&;{;q< KX3W%Pܰs"V*9ZIoPn ;ov~aB3jޗP'8a[Il\B̄ 9-\:O0bWi*>]@"'(RFR\I%46 AI8#'6ߺelu?;nTH]h2~Z `BQ1N;GTw>JiWMǻ1Fnftg:P(;=愮jMM) ZCΑRYYe 1"#[Gl TjN QC8~X d g6lX`jD]ANn9X48NxZ2SIW13IiTqP 3eW@Z1׾J/zzK3ƾ9Q|5C-H%7|"Bͅ?SwjEONy!F\MX9[›HK풁qX ]>P*!\:4D `'߲WU#'{)@ӬHo h7@5_= D Ekj -ՈAĂ {PF ᯦;.`f2Z.lQTǖ8#1)U JhZYџ֓Qo;cLkl02]LBe]CSe xILHFY3Q(3Xs/<32 1V|RQdv D; %i}6afR͐C|b1 o>)zsmT{ڭG\]'?GQ2FSC/T/⼜gvv el^cHU<4vţ/~m+}x5(d|]#H>c"*[xe p[:lo(jUhAn2Po;fWT0mv>v́D;$ij-љPBS+wŇMG./;[`-@ߑM<+V"0_]Tަ$ʒ \md!;Veo&朓(E e0mnZ㝝+VXr}kCs"SHlA*P00ip=&ډ_EhN@/ oe-4/0K< 3[Bf}Ľ'Z٠G9`YKg5mY^hZ-N׏&rdUOMNBi&H:q v\;vTw߶[}RG5'Ɣ_Y";rz_>z1 ٙ٘WX+E C3\2sf"Fp(n&9b|}pSfrd#Xy%b1)Au, m+m,F"(޽s|֒ _9DpY6,*M޷_"ہb0KyR#}4QTb={`6T| ĴcO4Sٔ w\aR7' /Jv|Ժإ< T,Ek0@NɊSTjX#xlQ IY0;+i9i-QRTgϭަ{zf܈Q*0*qʀDMb>h$/gch7F`$0 3=Ww}V]u!9T{k{)MKj pzbV]qk<vFprX72qŷ>Hɽ?Isf/*о؎}YW>[xxsʜ"$~*qƶKCrB^۲ܫwo~Tyȶ;d1 1-'1Rp5ʔ]d JyRn!F7C~5l0J#1`9HtPq/r ߊͮaPt{Y"V+04Zא:/+`8t&9>923] 1a!ud݅A | \=?F&ܗ+W B)6OsonpS<߶Pvl;7*vH/xRN}W]1۱׶<{+T"s)Z"?:p8b4r]ĥ#f;%n?y*-ZH΄PV.(oos>o(0|R֙sLEDlGM2xµATˇ͌PAB/w\{?F߃Y$RI*+62#[~`ikx*^׬+˵wm̸imEc3q8p%ӁTR*˛LH[лuoW^Vm|/{}oy76uoC{WRڻ, /"YB3mj?ʏvtfEb!2.ۤ6P[NJ^myۧy-Dw; -0/_͚;v\Y4nFN"mt {Xh VXzWZTKH5+E<.yER#ƅ7n8¥6|lΞD„ɠy IDATŅ[w.=7-|#zݜ] TtٵW)8T8^uVꏑӱ{fб:{O։DWv:o CJG+78wVc#^^aȽ&;R^uHldc;I1Xڌ9n$}/ZY@Nzfa'6+]ɵZ?RO=9)cДk߾`SOȋ|g6TM8*>l'o`xH V$_cX潸̲lҁ yAudͳ:WcBXf׳7]%}) {ȦᒹyvH ܰGlyc9)%;c',s%dYI# ݵ/lӉ-bϜ\ WPMZyߧ_wUfdˌNǸ+/ӷfϹs# ^ 8_,8>Lq׽ly=KdodDe3d?ow~,M9Fۃ./U!q1{O#/boi# x Qf=H5þ }퉬)P;H8 << `cE}߹/3gXK?|K9,!KTo(wl!ϑ ;:a{o-!w]"uJp,δ:WwhցdYڥAփ'OD{Dщo?_l،iSQ* @R3MFA)=KlR>JpkkFvo\qL4:@8%Fq =prC@@C >jwaH=D.W%cI ;!)$H*{MnHj@UYbCêJ$?Mϋ^Cc:E*(=d cL[4amme/tDL~׾-v] YH{UOK~Q=˹]HvtÎ{oVH%l12»$ӌx`C / :qNCP 0\͝޿Dx(O,dm3A>Jdǧn͙y=G[>K]?BKＤмT(ܓ!٘LN}/!oA(Vy4d;%iAMyog=%Q@p[@/?'=!Gy{agJm=;vLcX)A<'lYgO5&xoīj0@LypdZ%[1 F"ۉdDRt1F =P4Pr[D$͓%Pee .h|E{.TJnro s+f=.e=>S꺃r7 vpqp2񱥮P"OG}GLC1[$`3= H5YҾwُMNjsJ0l'Ƃz*{M{PbO+FvWc6dK25śbDk}We?_R' y3ќ!89.&s)tO2h|mw8'+)q˚6].n4p$h5A?hc䳗vs)a[b<8FLxF0rSՈo(!B@ :#׋fkP)V= +ND\¼an T s((hSf;D:tY 7hc]+W+*e得G+Z_ .?)z׽q%%Y!X5m@ Q6Ȗ&Fp[ m%o86̳mZ>wq'~ICNϣ{߭n^vR"3y_gU^xΩkɑyh,ˊBq`4abAߡj1O!۲KX?yJKݣ2,TÒ=7ش"ϖuwiXm؇`()̛?Tc%< JBXg]9>wŮvyn9 ΡG7#g>uoql8lժ CYPIaلI"S:-6rw\oM4Z5d9 %4m1ޤO /+xڕy‚.g xyU+; oj\6zs 7dﾾ9m;6vMG;ڳT#&|i뚵mo3ݪvi}]/+ oE,uR6q#4x́uDdhFmNVO(k6oa&ja]|/gJ`q)dɰH˥O$=snTЇTBr$am+A#+70IաXϾ@ !o[@*r(I"DS*L`נw6'{*c%RN:ybZ2 M͜4lByBh[>QQF2|'w)-]J'A[v"4~ҮvU 춭Wd!v~j"rI&|$v-I5ά\W/$`%>"QP5J9I½JS*Hj>R*BߵK,QDDA&27"-kon!BFz꩏ ˫b$N5jB/PGmu%Тb|"gkyG%"{ 1jȶR-oLK V+i) 6g߶ +m49ј{׏Z[yPZRڅyJZz@{ eN+/Lh/{Q ~T]ʵl/ܻ[,DmwٗI:쓇G= Ih'@^KcTdK|SN-[ܹ1[{_Dd Xz'oAK9\\#GuꩧnPb}Z]]8s: 94x'zE^m\첻 b~]g;0ŕ):jao:{Bw~[ԑɻ>/t]uscE}=.F7r)Fǀ?w@4"sɪL^4e\MAMH^ vHɋ,$bNД!6ӦM ~&RDc$=82aWRc;]lҁs:t]v `&ߜҳ/2gn? v ś ߁9:~;m%b?EudD>oiE~͏ulOyR^-cc]]?pa_QA%=oWd2pP2 P0I ưa܋nbo׿p_:9)Bm7_̠R_WD'D޷KAv՘72QjtvigW^P@bO{c}H:vd>.hP.`n%)y㙄K%`ȰngM\+-@xbM^jչ`l%ͼ!&Oyt4ئ8A.?K"oN }[hDk 㳟_zp{;OǧGY:I8"d ')컟D=XyBbK8ݤkEjI.hXXϛ:ԿuA׎ES0Dt՞jkcEd(h /JS懐q'"87U4 M/{N80ͼp}P%r @DǚRcTED\@0w6`uLEz)ɓ'/rцY#q֝ehqͻBf*n9V(Wx\TYe:PVS׼{~(SO5XU  XO L~0XWQd0R>qm g? E9Qތ!zgCe7 %4}@.9_ڱ2yRuoDGB߾E^$LTs_>/- ѶHa~Co!b߳N:eaGCU(sl\{vJ)um?WR7ajL EX{-YO>ҥK#Fn`iz@מt}꾟:q]W2fzJ^"h]GG$KR} Yf>+%qfgֳW&m A.TJ^u 0Hj* ژ D|l+'L&3ksv8@+#~dv#1\Līc0ÿ:x ysEŭ4I7U Q$ sw+?{1$)xհ$ع.kJ.j(KD( 4AKRW[}9H/XѳfvfJ4έ23SBL/SfdgZ񩖸$۹<]y`76$]qWbp;_eԫzOΈYj+ `Æ `0,ymU~)q 17^gDSSF~#6E5k2eq' T@W/0P/cWsqWW4)\ ~r;>k;SC؊,Icuk_}$%F;8CJ2_@4=s'` {oƷ>'#l: "jYz͓'OKҲ!F َ{;4(Nah#} &Pfֱ |ɨZ0@ڿ ;1TwyX9aTP) IDATj)_>j@ }zR˴ĉVdڌ#7.b\/r7f)ϖwI={Z| ˺~xҗ徽m =kaxB]Gq7÷6 C0U$qXK݋@1Iwdži l:%Fvt{/J975-,c>dOnV!g b[EvRf)GLڔ:Ѻy.looQG'pw<σ ^* Uzv=߳KuE}qbj]9jsƐcL宪WQ;{j>ս!(Os%Fd@y{ 2¾q?a-9Dܶۻ+a }vF6(aΪl|œ@kT֖ea۾WNi;IEKJDQgk*X]7]N:G.%/&M$Evez5 !dg[IVFAlR*c r&M"Ƙ ~뭷^yp=A-jM:* +ZhPtK2NJO|LqhvÄp9Xߞ3,˻WhĔh x̗+F>AM?=9~x3^kRiˁ.TkssE D3iCC.}mwYgsT&wLc}p0YhE@ 7:H;®iKTI 5nh3u8'51U|u٨q8J6^*P7%So?vezՎ80`9T\XN\+ m=T$%5RtVy-=%|)%'KgiYJaJwL+~XrX[e ``E4D,[z6Ѫ $D">xaXho.rġksG>;4x Mҁ\#oO.zVq8eJrwLcw(1 V=xmY}1w Ã8J xꩶdSjې0c91hbܵ,[yHg Qn4W2ύ,o֨莖IrRfyQ6-KIP6.e?s]bICvsf;1 χR >Vv09?CBfO:V@:G4YM{6GZVQ}2`fO~0\#Hhe:XkB6PcYg+I=P"$Ng#,*jil 2ߒO0 'EMC8 Q}t|]|2̍+g,vyv9imm$ O"pղ,Lvˌ[R2MWpCVIXϯ K.ΤZ\Z63-U˒AINep3zӂ]<zj$Q3Zx!X3kiٺaq1޲AjJON+,~D+vy2pL Gˌ}/SpԤղHoDH ;ėr~ƣ%dqۡ_"8S*pzEdq`ې&+B1x |Hl,%9"X%#n4%G/7\kNm:쑘n|UHe\kbA ٱ=Pnů4JLL┱w|w'2ɲM{Mjt"X]jnܣ*p Kw~z8վ u5I՜z0*eOoi:` TjYJΰ-zY^DQI&qO,ew'Ԅx#5["hc|Zr<=ѽ7_sجuv2ɲ]-9) @Cwgo 4q*^: $e uк, "6^Y ĸp[H4x9Y\g}ad}ϟPe[wh"Ϙ% 59@q1)N_:sex(MP q-9Rnr.bZ-1 kL4֥FUWOSnCv0ʙ}KwfW1foL|QSv Hמ )C)j2E.ӣh$&|0K:8wy;j;$Q.-DE1v'{31)ᨩ>):dР?:H*jYoZ隒2_=~ dkE KmUxϑ !_5̇'>:rn g>1+pΌˤ@ hdk0h%58ιߒ G0%nOorƋ.{#f܄uP=L_8%դ^Y \ K xS76Ҹ/MFBlhʅĹljCjG޲6Uz NTȄ T4TL_& X&ʄJ|, W)a?"y!2"*w[Ӑ;tKDHdlWiLvv݀{[pvu. PJeLܳBcۺ"Uf*f tIHKfUY' ;Z} jY"qL`U4T g+ĸ1Bx$a%7vV.B\!FY{4hcFzKU75aycGɎu)b=ih,.l;A@^wIK|6skG x-oy;%C:̞sG[J''xb%y7|w S#Y  CϪW:ά}J))zLΎكK xMLFIlA"K/g_z!Uk~ao߬s'>7vW۟~`*WED4FTQ4Rl*ċh̘Df^ɂZo]I=b$寨/h MftS/8#+ 2e@N1iļ``HŖϴ׾ɓż3RhLm*7 iڋoU[0:"͔3X vx<վϭ}jJmRʇ.kk@8 JK%ŽiI7/eaj$Co Q2\>3^gV[*ǾvT@vlOVDzTNEEsW8טnݯ2?J`w׬6c@zIɄsfzk9\)8=d#ו%JɳL7LK2%C_N[={֚ltR3xoCՆ;jG5Dr_%1f.D']q̛lߚ(=f)dD~w1]PM=qپzG!߽a {P]OE߇oRj]Zqq"$p`PO@.`{k񋿵VA'yJY, 4pj,tAmٯ6QEWp1Ft𯯇ī"/]`ޛdjR:.j,'kC/TXxZRX{e/Emw 1az!(ש‡z]I/kFYЇO̒=Z$ ew Z!sע?b`ɑ.Jˆ)ELn9ٯB+ /5\Lk mhqֿ*=kTC+ڏHks6_v*NssmV!7krS<- '5,2g P:y#/\p+k#\/4'(0i܏ gHgy}Yݚ?~k4#\2JVxy]VfߞlNL͉fǯU #i%$v 9SロK󁺚?sFful޳7}U\wxpvC ׎5Xe7َ+ҩqfY!t ȉy?^?W?!Դ~f-dIq ӥPG?D֧6c=y#gPX^L]n`ӫݺZND%oM).r tPzҌ\S7es oRl0 5 -DB2p&&W a闔 ׽SGL Bd.FbһUQvGQo=y{s7<)382WuU*ZXHD);yS%g(x6\ J|i -ˈ_ͣ{%Gh xi )i Zc],$ ߢs_]fPBgK|޷7_2~FZ^R$d0)^鮪b`Mߐfok3>p=+ hR4WdKpRI:UŸsW'#[YiIB9:)# Quf7?%'py{OB;HuWj\Iq)I{6ROF-I#-$sגcnr:I[f,;h xLT'g"T˂3^3uSw#"yzgOf EӘ&^= -5o2¤)G,+,q``7D/w "VB蛫΋!f~wt;+3UdH; _=Y/qY8_Y?p4ƏƸf\<q2N@ٞ<ue-[3B՗.-˯E(tx͘8nkWq|'!@qsn./6hLJ췿H@ L0BeJ b&0\>G2,bTX_xwH5Pl5o, V>b&HŁ|a8g fo_楋)'.o9GngԺk9.g w>>03|됒pc+;}C1B܍FZ4 )ThLpn2aI.q࿲A$Jt-t9DJʁvwHJj@ 'Q*s=1,h gV~Hi@FqhQ.FOچ>́7p0 >7?= 7d'΅3?Q;mNH:A@6 1dUNbh(.m2t!g%X3j Iyn.ZFc4/,ڂNV;B?' IDAT2gfcw3~5YG6a%o/2'$KЕ$:4:rJ1UQ C]fa{!'ac$6C[Qr `Ve%**˷YU [k3bjjV\k"e6l[ }Y :Ex\@,܊Pg {8 u~Q8k m?mfmR`g-rſ?+>C@z("Td&T'% S`mm%QcȆ?$۾X CW?h4+Ϥ{( X9cC.+H8 9uK(｜V3Zqߜf<:+I60\.a[rېI܃}nG*TF} `'s7V4 p?D,of:> xp0( Xpj,,k~>`be/ߏXUm_R:0πH::m,sU6q2&"aA*`)j[7λ/qci$[IЉf5Y9? n wohn8$5 <󐉀 àw`1c_<-܊X#6_m(s*/Ge^^e8lpl5bFphQd8nкh~do(d$]0ʲn.e=#vHKS߈$;Ɉrdۊ&RNTdNa} r_<SCEmg%&.vf1W-Ȍ]e]+bvΰqZIzfs m/p1S4 ͑2n2#ʩ?|M) 'R[N([S86K$A҇ς]e 7#C x 7!b7GE.jm_] krpXNYgSk|2 !^\$H&1')ft2sW_#3&% *%߽V`v3M` ie<DYLR9Up(ru,aYј@pMDirQzYR"X!|d_ʀ6fq\KRI'4p lh~;(M-K78ǹ"BNȒre'Bf¶]߽=,9 fuK㬨QA F0h 1ˤ5 =c2I&6d3I&qwdȢ(mD@јcl ܽnuνݝ|ý]NU:}yy%}c\C,76V0V$il*kYNе |ird` Ava KK@f[rҞoo$:ni1DmE{ Ä 1;!42vܧ|TMOlQ(@Gup*cS%w6tѸ=Z*cĕ`,C9V6Hם(ڟ dk G.dNTh9W@̴kܧ|"+LMzk=6%cW\m2"?b)KtEE= K.~SR ʯ~ʩtXf1 V>yܣW"U| X!nĺ^8(xyvmM@S(#mE|]ڛANOaR kN?yMNڤx00#UMEjHҫ:V%6Aݯvh٭)-Vcp3U&'pjөȊ2JE,U't1Bs&7ɮDF+%zbTiBD)<!m}fTm1+ODn<޵5;p8j)`Dj:zc`ueT;Wjk6RegZEﯼ8~ Wn]=q*«v4m\(Vxm`P-|M"C.;>]8M\fS}k$-& $|lā)0DLB%T$>Iw40q~nA5gَARXa5;(|͝;feءR"7WBbf|@5ݷgr}6@˜b?<(F bb]+z"v6Y3% K7?Huf*p_]- ;ɿjSU%6ţC.NTb"vqVy6\ςxPk9f2F.kc'-xə1?ҷ}{ yX^\vf*j&RFIyw:@3]ۜG Żv ŻQ*j)E,u]O5EYOoZ}f<ĻӰa7\rFv(ǞU;Ա!`^L0aT~ieͿtl:ޱJf٣`gM`Jϡ_i%S^ag4UrUQ:|D{FP"F`cΉz69UΤH}˓qx`=hIo*CHDb"]}q6hˋ=Fe7w=(dz,8 7d[qD0NEYjA^@7p.,0lk#m'0Su_Bqۚ1_8KuxƆԋFxr/vٲ{,;\O0fs>2Kд硈?RF3W9Wmr\^^H;hjUr -XY獀LN7P'y2?HR~5}_ӿx|ûk3pazp뾺I/Şuسϭbx]/?|K~{đ/s3YSu-sX}i4X0)CƔs[y^O~8 9 588DGZ}_,c1zOH"؊ b~ȄUC!xd&SٜNI_(6pT+2Nso;ݾeP!űWe\܊ RSHx³l{60ܠ:5d_v9kp9[ĴWj^W^nZYnbqd.¾ 1UlWnK8 F@d"ʖ2رD PuiôW&C!c!TXx ƆLGP:Y )ݜqIBkM #C-\I9>ȸvN PҥWamVmj|X?(E~:q3_ysz]NR&&RWFIBQO2s'.~th= $f*q[?jm¾WGƂ&m*T bWm<RJ1w30w-.J/+zR9[1Sf,7X$8etڒI0Ḥ'tN j"x)z9^ ݰ }"4C-;7iL%"x_2p\& R:˦B!wٸ[mjfJ rC (:B42<[1SdK+Y;N`lpbzNFĆkZW3V2q|eTi5'ô"HCc{_CZ.٪C^PiP;?r0:]f߶ow쳗'|ׇ{^|o?l9I8 oݣHV.s[ʾ20i ~ǥ{KM+"OѶ7vN t:0[1Sq,e 3s92z'#Nsa\ν+~RfcDna@|\5 ޭ9{ggXx܀@RaFF>Y3% 랓754ܒ[IY_9TwZ26%-9ﰱNAhzW@2gTTq7wWadz":E^;e|V7ۇ?c?PZDܬ_X@y`8p]U$`ix+YDl׃ /:GMJ|ʘf</ rQvt=={{X^ywlL-dGkC5ƜU( ¦Gs igMrm^ PH)\:߇Ȋ86^~`c,PH"Fx\DweɠE)Uwk*x„z6px(NUŒK\zvw\_|;E+_0l&R\ٮd bxz6{ kB$kN7^!#,uVOY>1Ǿiuٗt,9]7c/[eŜ{g 4JAQ_8s3 i_~"b٭D ݘGa 2+6k02Uh!e]@7{a$-g2C !' IDATq>.>2F*yeEq0]Gc3[ϻggVn,&Р8+pC{o?@eL<;^u{?n}uϧǂ;,x?>+mH!;1Œb5 fIk0s " ?OFK/#c =;xk _Ah1UooG×+@ N)dJ\؆|H}5a08Um?ۚxBθme|VG͗Y"޸++yָnazd֛sK%}iJ+νn .jge"+ }:V_R,dWe^-xU מR7;ØHPs?&4N9"^{gfH%(OaȵU 9VRccqa汜LUk }:<׳ָ9=xF@~8qtj$`Uu9pzw iS0.7{״E Ի}L9XH.Ѩj&\ym(b$=n捗x.LrjnG.֊: ^j1޷T6qY7z)=;qUE QrCyh/Ė~CTH郙܌!r- {MШH[V:6<y2J,2Lq)_9j 2&igwέHQ_޻DP.pqtV3gx9\)͹U ǁb{WzD0Z ŇhsVnmt  Jdv{8L-h!m#- $&Soe3^ٞ:c} 6-Ez?|aI16}{|bڍ }0ݱxR3Z81:6\띅t{^gڮ@W𐔿gGfKp˳?ˮb `"Gc(_49T8&N12&@"4^Ȱĩ2(p \پV8ZWtzm :Xi!TFaؾTeR˛gR;^oNp `˷Cɀsx-4s&-B4ArCV\ƸwC8,u=6x9Uq.l?Dk1u ]ν̞=7h2zTbt`0c$+1ƞ "3ts\Y Ǹ0i5 Fn^Xg0-]k?"@Cq^S8PElŮҠ?˓|=fzgŋ}-ÈVЀs]D'y!"nh@֌ H;K0J-K{2 lUJVbϒg>TU)ߡΏN0NӴ_~yv*)@deJ~?ɽ͟KgVyjg DMy1Pe]ٹ % ,G*Gmwyi[=YN4JwăȺ@ZLʗ,Yr1v Drc_^oe')Xupyқ+GOiOeS>:>+ؚ8DjU br&wSjŃ-d{a]ve+t&c^dɿUA4yOBmPmiB Bm?% WMP^?ŗs9?+Q6UҰvM;*7$Bkv!qhKI#`P˔y3O\a)D1Vwoy烃j(rkƛ>T9z(7lm7e㖪5uFYԫǗ5t>ĀF c˲nZtA ?pDXAwI=*ϫՎ;c*@<Wpg>vv"g%ƙJJ_?_9bwHHdgꅑ?y}O\RAc?m >7ƳE/soL6;oP=( 3~"&;W$Gt}|掃_z`1W8-}Mձ#7cػ_Yc,Vmp kc4Ɔ=;'I9^|a6h̸~׎zWoRo9bQ'ܙ{^sg w'i7b+q\Aa ˗H*[s'8{(u ׺D"oh(ϗ (Hd!Oׂw'"me'83:QRWW(W&8㌧vڵ~ɒ%5 J0XLJ)Jf4'ZУ,\2DL10Ql+a2^{d8y2s]вNdޯ+D>q++,ގ `×*b[+ s5_oxw.؃J, o992ijZvƹu# <5춵5,7Pv܅q|dWҋ<3I53&XY,\\Sү˜+@ސg(nhzjȉl3sW1kEfpYb2Pb/_W >c\&KPm-eN>44 VxX[޽{=\a6o reCRzTg Psg?uOd܉q!w?]"LϬI :)h\^7$u]<"l-y+_Zw]:7BѢvWǷ޽{={NPMV_Ms4tK!}5, G}lZ&r 3KXS:( &8Zk1*1T?~K<ρ׽4:XE# ͆gki9N,{#/&020Cs>hc\We刉UbDv,_˓X6FFˏ*Ev6ܛvn&ޢ$O}ǏQckfǏ"7ovvP߱#:G9 Bm.}PBM Q.2vPxy_K=FR؊:O'G%{aqvCua%%4a`;w|KYSNixY'e}dH+K4n½K>52d{~9l?=}l5915Wk\.ݻCɲZ-\XZ`T)Rl_zǝL3P??J~yrI%2atf620}`. <#=?=Q|1,sTN'oP8(r{ >[n<08#Tkc(]3ayckG~oCwLxgɃ_}ٞ$Mߋ&wq0|p>.G}YWq$,MKƣ_G+3^^p^fe[s~<䰐w6oKfͼؗ7в.Y6ZQrOB_HMrP;C;}c×F\w)OVlGuxܾ'cs#܌޳(`v.@bz0fu;7lQgKH qK awCLxQl{^Fb!JpN0GxvӨxŹTr& =-&ÛJνL#Q9Jz"mQVx ,/W|s3d dKA@Z& VľHu8xӆvVU~jθ+۰se_~/dž <$HctM;l%cL4FRB&ߑsF=i q.2eKc#Vu@V/4}8%$ӨC+梕=#QF`9^+wֈBFv==\a9wTNe(7#}t0O1Ša;cW{UDӜ&{HkuJYz5d¨Ր4ʠ^Ղ}ыT\.>R ݳgOAźS*A`B͇1Iz;g EQEDL4>؏7m$MCj1=zTDDS+b؋:w< T8`/yH/ى(vz r!$|\ʔ UzZd#F2[JU62PCA >ji")m1qOiedÔMx$3G4nI?NҰζDQNgutt%lY`j@쾭\Ju\kl ->w,s)# ْN\ ځmH%L2qɢBRBWnA^ke5>-UeHLbPZŢ"$$% `R'.Zdƒg%Zj 0.k]z&ޜcTw$jޤuB*$N^Dv=&Vl4]iVBH$,B$P[DfBָǍ/2t2^Viبަ'h# (OF[뵺ŧ4[qfO6pyW;yϱ,pdB8GT}-Z-ԑ M?&)^Z +^H=HZdyna'ÉME^2ߟ.u-#r5h'<Ÿkٸzƽ4XZ%u7w 9iJS⠤.1C]b=*d͟O]_ sL6Skeg\fC.$ kGզ4eZQ?[xO*^ dV2VBk '[fe ܘʝ|8qT8V֔]edu渰}_i5i92M!7u|48WXUqDa2u+J=6i0!mh1 %_04se3QʈaCoHWGN81i^sʪ'ɕ/v kew'[')tܴa- GȐXӣG%ngd<Ԛ'fe deyS)a{!U@nJ,f5oh9SJY_.5<6u|Fb'F;Mr, m%Rθo4ꔾK ]zlFLƮ9YZLJ{.M'M8s5|ɖ*kR&e}RjpeҗqK,5nsTIj0ρL";:NDHᢑNZF ,I'-HV9\# >m bD}dzðHI!4,56 G*WbeNr&9ν~hi) -R}Y#2&+82Ex\$+}|"D$w&T'6@dFF\eYr=dc뼛Zlw}cSQlvH}}pdJ x#V!m˶h IDATr У+aƟfui;YF .t+S$ aF1GF3lQJER'3 b/@%n? gs) ?xucV^a;[҃bmJ GrJ/8u+?) .n~t\eQ)@Kx.a?_Y-O#^Ǵ (༭--2zTdIQGk閠X? " !TiNKpŸo-;:[<>#ܐHa5R~cS3ؗc-b>TR@d-rfϞXLWBt"0J@?#?)T_ ~5jlhτͮǓc@+=gPrXoVq6x8 ⡭tpxJ.َ6dhA`PD6㭅d$.\)3Lo j+2i툼 E2pʫ5lzp|õx &ZF=a$͏VѼWL'f3R;~e?[xgąFFޠlmo.YضP[E 3zFL]zed ǴHMٛъ^6j$S'QRQZFY^&j݆fsPc'ʕp4CC)5I\RlEFkkN,/Iˑ-+cG9'ڨIr&~ݶ≎cO:A[\PqϵؾY 2g%5 P>΀7~T{0iLr:RE8?xhr{(mYnl{?HRӴ_k0i+>{]tӞ )1C̖߫ved"߸Ǐn]/.D>##Y69nξ@$mI|hyzU;FT7)vFx9ƾx:uB~9`c?~! ja+d u E)D„+w4{ďn8f<|1O>5luY_9i9a+yU j\ޏ_j]q0:~D*Z(ve0欖ӪgP*Sr֜[l~WU¸2jM'w.OSq7&6J=׼sLXMo=T'DԥۻV^be4Iur ܱ̇,ɥTր"ٶ=}&F+gxu_aT7#`!fg`yHz = kT}cy8Aި[)xe]m}Y:7(/U}Ռa^|a|dJyٓ2l8'kjcSt#˚%< ,r)`b<0bP,L%Xjjs3<I }R3xY!b٭a^0qǝQkbFtjB(Q۾8xyԎr'uQrO>jjfz{(s|o5 ]ꫧ^AӟpOgi} d{-ىfDӄLJHxj鯿4&W UkLce2鷵mA2e}R:wdigv+u0 L4x& Z>w Ea-/.62|sgs T90mHfWg_s̟wھ'_3ryCR >*|Uo1Q',"V}ڋfe*3`O-'N*ӷw} ]HI,aLL a=T!l hl<<myV-r)3G⡙5gĕjhD3|yk<򃒴i$k|y4_,*THLIrZ.5cVWFd|hý{븗 :t &' ӏ;KiDfkNk}ic=_VG>1~)GDf~J嬤3 E=LrT9_o~Oru #ռm(AB##=N զk@q[Tp}ccjZȬ?t..:s.h9r2޷6PLX?"VL9?;DYYν7 zT;p l<JE\GݗuD1D"(nع 2$w* R*)@m $Ö:zT}8]=Ps3]_=.+~f(gYJ"X %oef>fλt=yum<k=[C~ U WV'׳  ?}Lqߛ#}?'{dym򡪝KPW|Պߌ|^GbC]ޘNZj9{Eɵԍ&(@;Ylgf,:FFv9l#uBu|5hP0lm=-9esmū#J2,_Xw^!n.j[u UHJ1Ss~NB/ڼOWMgu0'8X*c؈I'B+t<K*31l34Ն:cJ0VRigNy75C!'"vbmZ]Xd Vӌ[:RV]vuW=0qlC~=)aN:n`P"nkY@(D坦rBrP_pi0T#?9mTRЦ*Gyk/2X* e0 #0^Әsg7,JAR Pg5ľr6]z0Su;bћ ms~4+>ad 3Ǹ3WVa(1L@j  ];m'U%ۊAvm] 3;7kރ֭dxզͫjz0sU=gaʈ E6P,);XjW|J0{լ"RE2Y͛ P<@|3ޮZq{+AVZUjmrVzVN&anYNƋ@2j6f/ҝMu=3ަy7G\*=`E+P]Oհs[VA teb3-W.oGb[ rx=l(R.25f$|[w}I6a$«o{HӇ\1R;X<_N' ZjRk[*b'b\]Hޔ,-\w(U<u5``JD|m@H (ޔ{&yϏDO%QPϦ i͜Q'LCSrn[2qB~N*Й-NHa'LXs ;VnݩZDae,XezÍ!t=[h]s`'SJ~A m?`G[SkwEE9j޺/ʭ8R4c-eUQ.j8vqng?(fwXԪM"q@+ǽ*_PԀyd#W[,lKBF$^5{SRIݖK!"Z0  <*%R}sg񘼇qV皍\#~*j0u}v6%ce1P1'S؆JGgGEe*_yȕ'ĸ'nè\i!0L 5i;;m nM&qrTT囏Rw.k}M:-Ϥ'wRN/!C:狁ʼq ɒLzpDR+0`_,h\b\}ƻ$Mپ_0E=X1#y) c` 0qL?%nZ^[m~vF<ZuZqh_O]cwOoX,dije`(&&pGUVOwp8brBLfG.)$sR|,=G#Z @eF= R YFMSj|U'x{.pܾ}W?GY 17!^h|hILDsVfE uG [>R>"`,&>_w+jaiS6Zb׸*)rfDI `DyiIR㾫Q>MmQ'yLyAz 0< `MQlc]v5Oa/(nݨwª/c6 ˚giMǝwsf :pP=)>)RE 8A?&b(Z)2Z*=k gY X(mCq'ӒV$d$[%{@骂䄱KdcC 4j'h/01F njx~CkI4ǭIG }]ϡ.dMˎx&!n藚@ea xW ! jidF|U=a7ւYh$+]*"r ORO0+^0 G7eܴν=?q{ x6'nJcDһq2IY<9;#= g 'a=&}Je'ƕCV-{M߯WJ t^'R7 Xœ?.1E$=FcFFu%Пg7/699%ԭ)fdy$(R,RcHjm!~UF6 9LQ$@%ٝ2O{33eb/@Spo8*8܊~:'G \5t=!>{ȦzM_A@ppd & ¢B?b T#,ژQZ5 o`At~K%C5FykEHYu?ToM_ř{ڮ/M%gʒ>!~R3ZH?ׄ8?b (}Wopn"Zhρ NB^'"VMx&^a /՝MgPsU""]-c 8xBj4J}r4ϖ1"hjaQ73=yT89ffs3<޵#>q_?p6ƌN/ܟ2B/~bz9 Ü{c߿c %?Z})Au:I~܉[qli:Vlt}q3юR>j7wQop׉Gw3ؽ;$ЬCWu4S1S&jHw$˳r)d!)VDl:bޙ!^/N`M𫊄R\4b: oT2Ltխ3J\w}jq| u,x.fqvYhx+O_(M]|ψO="@^`AAҭls^N2sn!m\ua7NΛ?82 xeKvjI)rOX€GY=tn_cߜy=i4TIWF' t7a.8%u#1vyR㾣?ćJ$ݛ Z[ʪVIU$BpP8"S]/9:%z9簿?umJh{+yZO3b.)d0 Qݏs- 9 'Kƌ/i2'N hwe\$F0W}5xᰏWyo ^,%{|"ܴc{yR^Z,xQ@铑X=>:XDĝvf$ZU]#34XjkBlٕ! Y:=~PׯSJ!56aEFC(\mKUh0 ;d8-)dFRIf~/K3Czk;y#9V0rX2sF"]\ Z$t6׳FdH6oMelas7_|x;oF2 [WpЀ3ß/tbx@WYbQ'2;i2{.9֊0-qJ2` c̅ elX0 WDxA.a鼬Nԕ;W~姼WM}59 ȎKg9O2{jx{zyQP> O6ٕw^cGHeX>^yzD7`1iʼge6 IDATm%k=::j[ƙ /uajܷSTO^ C瓊/^/|vDexG)?DAk[c #x˗f̘1ӹ Di7sq_#A(hz+bV.hm'G}`f?Xs'x_|352lۆSX\vXfͱD~$[iˑ4`ٔ^w/7f} 'O^]>cϨ}ӟJ_G '-݂J|qil K2mmĭ?B4VS'zwޝ>G.e߀K<6J/$9AKۅy4oܑ`I7 FG05 .يvO o+vaX0 p[YmI%[qݩSqPdk!(1f0V@I_%kܿ}V,v=n|/)88duUSq `N}p+^O7wuc=29^Oŵ7Cc^c %ܯ9:Na1w~%eVK#h1;bQW 4)gȡ 5"0Z v%es!'N6N$$C е_=,O`[aŒ-.}nɨl 8+rUJF0 Y.#1= $ ~!n+u'@pG8]8.ed66< 0yܔ|`h>sew`@Pc8 f7J aztq2uzp`^ш 7Qt Ɩ\KX؀p#_7GOGf8 N+<g[COTh+^Ѐ> gސC9sdVpz >>R7##u_a#Иhd)hhάDZ/Dq xMe_L{ǁ41F8m(VOp 1k+h"f2#y0Y:3xp i.?o2za^p--lvDf/݊dMjXt{VcT x_~sN'4mE{~~v0ড%!%8>6Lz׶X܍\"~9‰S)c 3bh1jxҖ9Kp*2 7=mX k4`1 tWqƒtQ9` Z/ҭR0DO^|9 xń;3twZFRx0%[Wb9=퐑 e. # O!A0eVxv_ȡCF."H$ .Z'pE@Zr %PKxDžLt0) dug>d#_4Nu y§i3ڈHcF6M,&߼LcPi5C&s@kʴ?%[q֗0ݦ]jexgl0/!~,{1kii`R-D:`s3{2kjP5 M֒2~n '9,2f4`D8HofB"P I"１CGT(}3eЛ^Dr)7A|`0ϰ"VDھޓh8VC+\UUZ?eݐZ zТKXxT 6a%R<z!" /.B*]R6(h# "zG%!)0I}$5 H:%qouvܾxG>(&h:P5j[z9N@͝RHҥP)rω~Cvv礙KXyYww%nH)aj7~ %"bG4*T ~~pPlY i--㾌'`tגm~Rc<AsTxgcH80-Ak;YX ]OiYTrz\Qz6}A82Q$;=Rqg@~>(TI"Zuh+gCq6"nn-Ҽ'ܩE`z8p|=,h Xs1k?;u:?˜bX5q f(޺N/hQk&Tnl8}[3^z+ð1M۠BQǽ/n@r%5e?iQPUH8nڈ{@ >Xbze$qE (R;" z0|.#AI1]kY+Pzy1浴WY=ubRd><H$C&^]b~ 'C r^ "x]yr2j{s/BlDh[ڈh\ZAa `lI#gȨI!1OfE&KS(JmR ÀZ䀤x(2bF.Ƞ5%N{3P4e^ߏ '1_x$jIic^V5Scxћ IPs@k;ADHKu Jx_:wٞ1zuẺI=C4n/KJWdjJ{f^v%""id 5v☙*ژM ñP##nm'~'vz~1? a^f\x(.s٧@[9Ms' =oX +DžP )%Fvb7D IyQty:)d mhdG<$唥M ѣpJjM'^|>?L'IDY5hm'/u4ub1}H@x,0Rq@ ھAOQ+}pm5 Kf̘!q랅רD+bR8Ę4`Tܢ-B#24wPH&ͫ[b9 ]Ns 'w*ľ+gQ6~ĂΘλ|%I+,h [ZZ^{m!c/{Bt91ĻEjqTò #&Ab~IXfǚk, %ui\4 KLwGPdϹjM};uM{KbRg,eb(s~#&:};;D¢,Bq~x8: 0kysԆ}Gx GaNB@HeYJ?N.xvZRdۏ#yO+dc)]O<Ӌ.V גNZ^F}WU$jęhu#itxPe LkjKȋ!15w0Esrll8,W3e_+HpH[nߋ>F9tPjR3_th''>seh˩|Vd Ft8c Pt–$҃m9藝V=7w,*7wRفI16L{PHRo^A"%!HtMk;isf̘qիQ =fo5<᾵-M1BvbQ K^!3._sFtyYJ++jt]Q\+ yi< Qf!Te<ڣ&RnM$H:إBAsW"2g![.SaxbV|`3|zD[ۉ Fy|W0VSHXi(\V|!=J_;Ix}EVYSQ>|GuGC8U MaEqOy%2'O>&Bpc9{eV9PߏLz0,Mf$-t&%iAgs!^~Y-]nqU7w';|"W{ dB'RwױvcXH~:X6uI^^qښn%?7Q-c##go*K4l-ZE4R`lm'WXհ>XUQ+HJ}#Ab\B1c5kּ5d*yЉuǎ.HF;\"$6Jq8*Q98{USh)M.<ȼ4"l#RK{Ӡ'9TlG@cAX,9 tRP<)}JCپH`༁+H)>KZuɯZg]9 :?vSe6wRPP䢛'46rJl` }@-ϣYśл`O #+Dx^.(]^y4p$}p[r?kk;yhj<,)eǏy+|72KEc\BZ3u$AVrrTlj`d?yqD#}?I%b!7Ũ||Lҹ3Jh5N5k"Yt/1[b\,i87j^` ooo'O|a-Ct]km(Og\B'G&}5J}f F+t9!Nh,F>ktʙDN=TS 1V:8^Y9:g4d Y,C8^фF O׼z\j8Y}@a,G"e'㜾4ƽ phRD4҅A# %e,x5rJzlB(J2Ti)k*l*ybC_d|3Ц6XlS;f2"cٽvuW }~Fw7r3j٧Zɔ~9z$>)w)?_L MKD;@^qt,bM-Z'o28NNV^=b'ZN5c aܹz7cvi/]J\ԷF1=90ܴapB.z5E9{P<(Yx8as'\I;I%e[.Ǥ! esgg*&DnK3{sāw!:3.md\~9 Jᤂw@s R`eٕ3v%Bo\`cpӢn|0{u]yGE dGC$)]mi{wA^/,Z-{?^\ŕK\7^ܿ~`ϕГG >{~a ::p` yt޺w S.Td"02/i% x͝UYdIr!tT>$!9`Dr^fĘ+סW@ SM؁w$0/͜u0 }@p3Gܹq'}Ӿ) -e-jYvd2dBpF#QKk}dzb$'W^֭[ `a1g՟uZb7*kKcAV^?i_{خ^Q@a{LOͨ3 66'zdV n7[hrq>1Yݷ_ͱo>e\8y{ 7ZmD{wd[:;.-T+rǧjg{= V4wnJaSYN 'J h} ҡhuiiژ,' *6ڈ1^T>~B&`Ԇ,!;54.-1('{W)oƛ ¼Ӡ^|-3L N$g">|G$x,}&&;h8޲]ѸK;=S?4Ab\jx;rּ4VΆ:gj0KAn9E盺p!b~FW<]--Zt>waUO;baE48wU&(zpyG1YCDUU_wT0G7VIy2wŊ}ؾ4|޷;P?(Lf)GOQTy.آu'3ERQPYl,dn'x=|!#0[4SgycSjafh`/LgԴbQuY9G(]--]/USjc%3B 1CeIGyߕˍpU#) AXU%a K18NrI5>Jd31EG8#ad.9Q`zkyf TdM!CtUqHĔp,}y69b/o{܉r`@6Dw2/q,)rX$Kɝ;3t150GL`F4i;~ˎx[E57W~וj2O[UVVAR)3ynqaΓԓ/ͬ[9)]--37UT -H2c2(1#sLn9"}*xșzPGIg@my[O4#yRM`gn+._^{BYNm@?Xvֱ]ޕNq|>,KSŰؾ02]pOې:cӛ;)jO#NbHh }NgԑNdS0ia"c @rKT233X$~5+y;caS[R0wLo=b0;uLсU:QA3g"|_0σspr`?s'ƅm@y_kj;N X5K~̡ͩyrwI\Oӕ{QnTư--^;6˃O3ռj4`Lv¡5  :isf;|P<+es|>%y2q: W)"2!PoW G'QMbOƸbX$YYVj1Vtws3äawc F-}@$_Ra@FZa0لV&,?|cҭ< Tr  t[3K1`SBYp@7.~de6^ܚ[8zw#DS=ē-)5f?Jtַա^Y笍1wf +-H vE\\pHu XԢ⼛dΟЉտwh@}GWn(tw-0F97W3 G` NjQes2 f"XaKyC'3' b+6$/:(r$_>U#svx2fmwpiC!C֮DY4ZX,<|zԉI=مkIt)!W ݰuhbYq3 N0ŧ}[9q |lGt ]#|阶m=yL%g[}3GN+;tezǷ9o5'G3H1SzdoqG,+i+ܧ!8UG/(:CII)N,fJx"ZsV̌j$epp gv[ Adt,ia~NdR 8ۻwG=eMz.bć$&$P?ԀSktp]Y0 d%bҊ(0+^}=`=/D{gx,`,ܷv@ 'ݻQ\!ы]ŃY\xB6q"VJ /9-M 놂+͔t|>0ɤ#X6pKՔÍ M=*'Dn@sIrH ø7i:C?W`imWx)'oZTnm`<#4qp"dQ$[XxEb Ĝ{Zu{mdsHx*:h>8 nn#tF7t.9z`D$F2" p~Tf9d=$֚~Ł,crv68oZ !9D,(WU+8oB/Fn(r%HCprѺf<*]f-N½<x`viBz*48?#Tٔ}~/-JiK5?F&D!,`y~ykU9dUuukwFt?w2>x4/Oȸ<]f8'ad<-&.#:Y1qϞ Dv3.{=}њt $sYްS(?FI,&naSfw?͝͹~>4U8ݹ1|vR yA4nBhk#΍ң9oW:t{ueU ~WCuy[3,m0 EqOq<:˱aktN,*,ܷ<<"&ϷTW(1W\'NŚS IU˾3z6 WaZ݂xP\Nh1e3[wM{w*1oʕ& WQO7ؕ'a1 tofd=?7cv%^|?rCrM09yvӢn|pP*Rs17$QFPzNg"Y VM;ׯ>`0li">J@N%KGʎ_nc5i$}6qKZhݾ^r5 < QBEӳF[,Hyn1.Y,)Ecx6:9͸0t Ο>[r_^S{>[o^x>ڄw/~lWu%s?+}w58 :Yz6#Ǹ;91<Ī̜xt+ 'KVCxq{^& 'h'`%{8PK&%b=դjАl̻dT7Z;uINcג!{A`"O,h(έ%iěX0Ư-5 7I+xF] x,Q&XDp k_BQJ%G:_K.68!I*NKrN5 ]шX&f:c%34jr~%2ZŸ< =} ? 1ܗ\T"+B©z8U7B}.;MP+T#)Γm1YW=U7T79λ"} O!/~lw EfgfN8v/ܥSLU>֗ǤP*٬[=&]>Jݣ|{^%4*UNbX؀^"!f* ]#%o>Ng7i,aǧ4]Aܽ7N>Dy>KXN K@L<hV o}.,]\[[pyT; \Cnczm0:Ktbd^vAl`W"h X#T4#bxz۶ @k$U0;u+on8'Yƿۼ43=87W}\ Q nM!$Ov1{]ͺ'p؏ox~VI#%9<=N஍Ѡp?dԌ!7XY7\\J2bPq7L>f1*uoS 5;o[!Ln hYbҺ_gaT834gniѱ۠11f#Iy1'N%09b* ?Ȧ #[0Ycr^(V-π~ a)8kl+~a>?/dJsܾ{9(fED>ַ|APdYW fF >3VoKͮJ97tcՃDof#5DD?D9Hi5+`N C3KG:q]xsΥG<}CHM,p-Ak3X<|) ʟ2XR9ěqk}B@f#%myYwޒ;1,.>p91X>sBu#$ٍZ0]Q sIIM_$EU[ܤ]lrHD#ˢ p= ǂoۻLe4(ċ#JLn&d65_zUk[<6/FHV;yǖ Xq{sW/YNY qIh+]97X؈i \>&og~ Mz_܍_aÚd] P‡Lt.1nP9 yMke=c)|oL_I9ːH 6e3Wn9n1=Q7sP|;n@/f̘lM qWu:ah|}a5lLMj:Q)MȺI 6 0K7c'1| DB/~}g9*L&WG^ZM-o] OD'8 E*qA$d7cQH^rGLcpKx+~C ߁M@J8t 2$ 3od[mžɌ~k5ZHɌ J/ӎ)gF-q6WAOqRA(f!8 M k91*'(|=/>U憎,"Va! C[Ǝ(-pOK*7 EϺ؉\W8XE8&+'b\bw5`69Cu&<0w("6g^5ߒ53Rr\ípby]s6C;LS#j vy?M&ܦi^,ʼn ޵ܼC#G$ZdҴf'_>^2A, Yrx 8vCлΥ G(I_0(o2pNCXv݁^x 64۲3^߶+&. #XEު끚c_ʳ.}JzHpO)CNXwh:M =֟mw𢡊XdП `,ԔQאE-CYX= $~:ؾV=vCM-m7jΰVw'h $·9Oʔ^=ye޼yڵk/I&R!|QȌҽ~}7T+O\j2f/E7":j:ئho㷱7+@Tk `kqkYn8$"cL+mnBҺ'-Ln1ĉ0k=zMiSz|:>η-OtV>AR:p(άHHQ\NxL}6=YVP͙ن<z_̲t X_D@ z=Ы3ǀh ڄs'Ͻ}/P_xS"#]-9t#B/wc@< cĩ3 F1}@h+FN?v04q5" ,Yi#]7χ uo=ćCX8Xx."*9eϼfAo\kn,C*m#v߱93M&0&ǭ/0d7 `ab27sI'Dt5k~f9A!_F+lNc^}ՙ= s'J<1Ϋ42guV+rw9Љ}%1vjuO_Ū?AV@'Wu :ձ *+䞂pdx yi^DAdHwnBRnGx~$P$Z"]p?$8٦oJ7Ӗ{ [&4~1FDFMQx,#k+SDu-Uiܤ6"㞈({40#f{Ͻl?vo}/~ӟc#v8>sŏF|?z_&+rǃ.?6 Rt/71/8<LJ$ybףj@dq 1]C r @mOr|ϚgU!oO/?}}ǔo%~pY4$?_N5qr~wO[[[W>9;0hr9rö@TIUFaǯ۔\hR;>ui;><" eϠ!f qNķ Klb|ͥs83ӸObsP;{|{s_gL~w!1g ͈NsN9 aȘ0i#&_, wW !n➵(xxv, P?jrnpBuƫ۰л]/ƗɛG}oUmWT |q<_{{NY*n=*L_@IN[8GmY<IJoz2>&T)TPg||w>Φ6鸑^3m%nhY_<͊?'/uLv| Cd8^A0 8 DŽ? Kth0'L&oh8Q~n:.$QNȨM,jT| @yAX k\#X84o{ r8ZtP2 0DSuLi506#ycKj$챉@qs<$H2!/[,}d%Ůdy2>&9-1pW=WHub-y冘t: I!n'pB85>=&B8PLnqy+?mCs8ZRK8/ݧvS#_V"nS92L6fʺ~&ӂ3Ñl)V25dq\j p1N2ÅD%DCͿ` riqȻsA n?{_䡱+}8<޼}3^2/&m$)|DYϟĭE_{(+ ?撞SOd&cK܏Iܽ#:bS8 G6qV5dВ}N+.xAAuŐ2Ps=%6e]l~zVFo1 +>gBU^*`n,Yٳ$!eSQq2rK}h^_'WKtƐ.?J;v+E )Ws?:"z3QM\$F>E|<4 LbaX'ILB}sWe'` %Dl])_ c a.bXQo^GpSxi 3!sA] ) &^SŤKQ/[K$8ٞFY8UC!vBN)#AY̫ ^KB{ $1d%DRrsp(dl.g"Mr/Qۀ5[2IEKٞp|M)QI$S֧`%Ov>/\=y;sjja<*DKg Ƿh֐Ǒ_$>+rਁ7JgUrµIぱ:I`K8l_mMls8 XLظ,!\|r+X2wkôql8=u8G!GUG?fQ UQ ,S;ͦpG֖:ޟ\N%_Yz}Y:}~pY-"(#x#unNLsp A0z_ǝ3c }?u|/w}5Ȓ+nA8ŕ#dFe nA.M Fuq;IQE}*YYR%RBD9@52l^, 闷/<̃]4ɽsY3D,jGrJQu/&*argCǯq.VahqY- o򎕛v)*BEpz&t%zcX)_ ?Ȯ)aZcj\ҕP%0Iٴ"$y%q' p``raJ%}Y|,ŲX,tCBfƇpV> 5t2mT3P?Z-=eԵ$ӡDYl"_y[P$ 4_TQ5'8? B>`pyug~+t?Rw('SeH't}@YtL4$؃z5Υm'wHL[ hh+DQK_{ybW [E$Mzfek?9yΜk;߈':-7*+t, V)l^%B#."lsِ4,1î_r|tO־n>oM([οMd;MU0/evФe ?T@ÉxK\ˑ1v>_!z_j>t:@TTQ֘Uyձkz.(]\ {lMkώk/dzoV"U ў*60dM U<`o^o;wӁ8Pi+{1]^E^uNb,c]=;e.]S3P l1Ϭٛc0n`'}|w2}1!lPKHd*W5|]c7ťkܠ__޸OȈ RrɄ+[\6z {fu; O\W*Su(*! I਺Ry=j+tJ`1w57Y(s_`TcfŸlmF۵??tƕv:p AW,K׾4(3܏#$U&=+Tǚǒ;߉Jկ_TB(SbqT,Z%p;ܬlp]uQ mD#U<^(y1 H=kd Od$wܨXo1Ou[bq!veのX1 1do d_4P+8~﯑8Tǚǒ;_f4ry8x[z; 00rW G)&ۂ'sy_i Y O wa_,䨖:ּÑ;~yHy( ه*{[3-/w~5;|rO;YEbLeh~<03 ծؾ'BLiq38jsQ^^&̃4sSW9UA}}il8= MloB;Ms?Ӗ{iqGoVHGZv$}|k-l7S+zNa>D7REkvD^zp)r7V%~8h8W㜲/[xDe i8>`]h_!_P*tV~Ngs'=TIY%=AKo]<\6^h+[,2q4ioւ*2aakGS6^ԓqYJr͊>}q#O{!3Su. 0P6( 5eBJ/3+?wZPlmN+ [dllr.>ewn+r:!FG ]T ^Ҷ,"<}C}9\ZHyVT8zf\_[WoCurt{ !˾L}3}^tKISȵ(;u[y|^Ց$G rp$c/lLjsM}if=81qc81qc81qc81qc81qc81qcH35{qzH'wAfj&p6ʺTn ֫eYSfߡv{P6|[%;Cٳ~u5.U_k5 }<l/L5N= ,hnkY"0l~jT@ {Rp*,`mOޑl:jξusL?c3 yٷ s^J[s{n> ڂ:x\ e_li=8=mUHOO\V!wp@,dYԁ:֍,҉Uнܳ}6VN2(|q.ss4I_.60Q=/kXs4{qo:BM)tqÐd U1nq-4U2ڝ6oY=W8kv>XNa 9yA2U:֍?Bk˝!(ܗ$yoMMf3貂2ob:oN!nͤpHVbYnO|?:mƕٝ36a6=oeD M|Ϳ=3ޱ]R{Tǻi=K8?|@@{Jԅv֍ dysHW]RQk\yY[P0 {eT)X&kV!yNZ^F Hs/?3%M.+7nIVqõMV:VaM㜬|P0uL%1w3yB\%7 o? XW*ݹЌC2;0c^}V[%_~eZ7W4-?g`;vo80tC\95nng){WHK'ۂ9۱cSƹ1oֿI~veŒ; ٧A/] 0{8A~(OmcNƹ],NÃA,oɚ]]%6W׸b-y6 RWH#xOeVlg0Oomk?Y嫏?N:n&%w܎'?l-,ۋYzRNAe._}R-I7773ͿƹԚ래E.Q[ãJ,a8~OeW;@;ga&r(!Bc}OK[@ɞsGyձʔzR?8_e[Q4 ȾݶJ5{j\ d˾ J(S7d|oIdh;7?t{br D;>G|'%yH\]=+v:˸vH2 *4+psָHy ^v IDAT:'|iCRBcβɟ1e_ҙ?P,W{iAٷ7ov1ڬm0^a&E[/[3aV`](&{H;ʬ){k\I2U_xVڗq#u8RgAʾ4KxY 8 ؟%Y Y/3HB[sőrY͡uUSRY[5Jٽ(tfIe|YaF77'V e{V6QI:V4(Lիk,^zt}GO˃1 '2DҍˉR%Э)2t-.yF_Q*k/IF Ls9 ٧U.4ޭnQn#HG_xh5 $LB-zِ[=cP*Oָ͂6d%7"וR,a9쉦QWR:b?.7KXN׬Y{ױr՟< x|]O%;܈?U 4?M@y8 {w|\Q^yc>嗬l*>UkN+%out֛Cٙw ڗ|^ݶfq!J깯q 8Ǽk?/C|\E^uNG,A~cQME\խkoc^חxœeEk6V{:;o΂QQOwZ4Fl U9] |[,x= ڐB1JO^ twn^<]W)'$ Q>dxv&g}WQ5[Q ܚI=!|t`yP.Ho^zdN`a@In{6g*Iθ_$n%zOU jb[o-{\߹ҨޜdHxxs~bWN1N5%A /p.kŰ#>ȹg'y$wM@=xPhܩ_FQGS}d?iI>|t0W)׸}?B MDV4=zܛDzXܓ)I(r{l`J]82Zmձ}]ֿpH2ƨ& .}g!Ŧ>kN-/ 'eoL1sRzE}:n}H4,Q]x}ߚ/t~Fɽ_qVKMqa-yQM|eGxg$a9˛wdBsO \%4w![6wܾX ZC&E%$S(;_-]}ۋ$P}71 o[9^+9F$ `)E~|S,}0y9~kTTGJ#IQa ˾!?'C 0/_td!Kۓ5Ei2"HD$bk u>ة$,t:ߵ.<8-0KvzF q$a99<\G9TW <;9|ݱgI#3e:OIIZ|^x |c=kl]<̘27OFʍEDn#ҏw)y;ͦ`ns 3 /eR =X2{9MC''}d@BXG8B#ʉX9(s(!Q%3"/.ݗy q*VfbMX'異;ǔa!Dϊc1!~XNqsbSuDbgP6/T>R &oEG(z9fM܃wlG5sIb IQ1'KH&fGZBzst/'e⹗*?b >t&niFc V8:Ny[0~䯮y'"R1X.ϝbAܰV%{:'cH]J{<6IQCl(J|I(ez`<2Ǟœۈj5.an蝖xnDt$:8#x|˾%#xYy%CL;Ϋ|,6=iƒiMX>D"‹=,p|N ]Vٝ$( ק (,Klp 3gE8a^ηz9dF=CGc4땊!^B b_c;i@ܰ}GsZ7l=Y` '\{̹WTura!Ƚ(s,*EȬ-v OKs3}r_b0!XYb3O_Z$,hY{?cEXN{u1:EA8!^O<>:{gSA} CíQ*Sqr ’{ Ìlb,"|*b^qԣJPAhѓg 푮C?? .M7:cu>w`^mLĢT* ?Y-G^1ߪO,">Y,5rϨٵm-Eְqb2!-PdG"n=*l\f`' x\V =7A+0}7bv>Þ ӕqq.#%rե /a ]XKK &aIQ^Klt"KcCS;ymR2AV fܘϨM |KI>S~ȪM8_!dM|^8 =h; K1ˍʓJc%$%_lKl|cVrtH`HW?6q('++.kࡔүVVK403*N)e&L֘1^?A@L$d G?f!ܧ l|l"0(mac$ }Vfq@% &X D夛}"LC0ex~Z| d0jR)15ƛҧUzjY4 sV$;0U0yl1P#B7".džR`jwr0q*"q:ʐ{42 Vs Hh,L M7o}VvI&N͉bw™Qbȣ.vDς.Y".? E# #r*1ǡ ˉ Sb\ d(c&oTEIc+?=QckHOpcwt}G9nIo:gqgp-L))%o0^jj\$~ȋ=Iq=^l)}@ #k=Ɏ"_х5HsN0FS8ȧO+?UyRﴓ I ;x'r_nʐ{,a914 Ju:B1&oibriWX*FR>Ih7Ok5-b+y;Iu~Fa96cP]&h/2gIcJQC ]-}YlF3!&cEIc)+zqK+tĊ2aL91; r&?M:zT~rWHk\rct~ye|` k|O"b~PyhEq^ygd|.-@#xXB=,=Λbab/JQ*%YQ2&%b] Sqa/:-;؃]V .m9IY3!(6$h.|mȢOsUɞdzG?{oPLCu=z|=+_ _i(d;^ASˋ#Tpk_XTep7qWȹ18_$0|UCxzxÄ,qx@d_vw!{% ȎcUXrk_](p{LW.&8Kxe&.oFISr69V$CUǟslQFꅹhEC]y0Ass^K} qHoRhW2}W]!ƃ1OG焲g(QnT"w-U8>_=rZbkʪXX;Y;bz g3$QX3 :~b!+sɊjJD $y[Y8_$^76qU|m 3&luclm\j㰵{ԣpTC\)<=4K8񽜘\Sg蹧I1rߗsֽ p|+{f5U_.bsG|r&P58cu/sd|s^Ww0uِ{ 0 tۑ{dJ|<\V˩g,YZ@Qf0Ų)Fܣܣba9 ^tP }2)dՂG=_4ބ]qTx1x^6q)tdIDATf"hTev}MT|sk%㻮.aFU:a8G].*[Y-NLɾ|roLbi@Ư hTP{I_$7(ljuP*r/A8NYA+۴kD6gX\A%\՟pm5N{'PN#X؎9n-.?qWx^u,q,~}f5 B-Ճk_A A\u蘧]AM&T;8v-ȶ"EW6+ύ=(;Iu^TxEswom~Jإk&"I.=>yqAB.(&d:EU_Cн?onGW}"lp}c~:g}$trPy6;jWHu 騈K(rPaV;Rϟ]u~jH/6q.Oz(4s@n">518 RcQ}X\ U9:H ՟7|*po^v !MK).αyN־ЌyCR)hN!e\0[7CZOݵR<%$E9NTS>G(;{;j\z qJo^!(յO,PwH0pIₓ]\v:)[8GЇ nn s[CV,.UH;:?8lZd9:nX2h%:{] GfjYCvyﮯV][E_{յCD8n^zq7ok1exzk d#:~SR%_rJ/3I}FU^ $KHR}(_&pN*:0]LvV{u1:S{e(.E?~t5.]]7o@s/v0[Y[/G^Jt39ɘ)/, >pKA'Rs_'ong͖,q)S?_GɕzHQssϽ8"np앏jvt2Yi:ՓOҍkk 92Tg7^`q}Bů~P*u_\ (`yʵ51"-0X70 xK׻eSCZh7WD)u"1m 9K\71I. AUαȗyT/Oz'(8oaT[,4K׸17p3,eOW;djwCB\z_N z~Leqڗdhtt\ݟV𼧅0r_Xj :z<ǯtJ{[$</0F{C°q, ӰXl<=݃?F@^7PAi",z}2e\ +LrF,<^E lFϞx!F4xFj`ԣ$w͛`cg(r ^^+[ NϗY&q^D\t”K9DWz'IudbJ`+99Mq[>St 1~$;Gl)WŹAj0j]jVO{6 {/{jhy~w61\Gh-65ZkoW\Z|inc9+5P#yJ? E{ yD;-e{]HF~]_%lq~P<5y8xS(|`}uu꘧3:d(k?nmW [pғ='z{\ьwsNZV Na"3 yz !fjwڽN"),Q2L=n*,N Q3rţ:Sǫ<:a (aF,i(]Vzass/M s.<E/z׺#G< 8jt 5zrbQofN p2)ޚpSFdb@vUЧA!l>Փڎ55s2=sx4 ͞F GwJRe澺l>[V#?3֌dIEzk8c1I ΀ݓ:~q91oe>4{{܈K=˿g*tQ E '>¹T"tAۅYJ.Ζ]22.O&l3WtrEckNxz[DH]%Mu}͛轸DK{J`B,K7d_Qs)4'`pO%pMjY|IT'Յe]QBqF^Qy@ߏ{p?yV#'HYc?*9wT'H헉wِ~|c.uGj$WNAbW{bGwPlwPM_6(*iiy 1__}<2V*P#KA;~P0dPΑd8YWGkvU4.߿s_q"yJ28g0T8w$p'>jcd*[DT)`sZ(8qښ|k)DB'ku|旽~XO5.-rH2?y>='WAy ГdU~}9ၯ:˯/ƥj~ʕw2b6_8_nwTe's| IO@J'7]Ek_~%@K;}-gP_u3ɾg?Ǡƕd9Y޼#?;ZP~.G(!z< U<8t5)IENDB`zmat-0.9.9/images/zmat_logo.png0000755000175200007730000001637714515254731016726 0ustar rlaboissrlaboissPNG  IHDRiCCPICC profile(}=H@_S"vqP,t*Bi+`r4iHR\ׂUg]\AIEJ_Rhq?{ܽff$jN\~U "?CHb,f9]gys (>xEAN,u|%^xw_woir% PLTEW$Y&Y-Z']%^-`/a0d5f0g8h@l?pBpItHvK{R}\V\[a]fbjfjlsioqvooޟp~zwx{y{骕ſtյԶѼ½翵ﾩì¨ô°ĪļƯƶŰIJȾɲ˹Ⱥȶ˵η;Ͼҿەx\bKGDH pHYs.#.#x?vtIME:tEXtCommentCreated with GIMPWIDATx͝{PGzqʥ9>?8J\βuE*1%?Hlq689fHe*|,wBb%$Z%ictDBYؠ!xm#eEYec=ɧP힞|ϯ:'-J*IY"lfN zF03-_&OJO Y({b)A!5'3ctTnn$ş!>sDؙEY%_Cʪ jyT6 dm*] ͥB {յuE+lv(Xɔ8KiY&Wd |.68Uaca?똭:]nEK㻽rv*\UdmbnURZN!3l9d!PJ"{d9|ДEZRR?#f(2YyOsx>hCv-=Vm"*iWib;k ;q2gH-]2R*lW԰@Aj}ՍӶj"Ϟ*M(_۞>AA]=`Si }KI:նg!U'a׃i2M'T,‹`MMI'M"4`L)YeN}87{D_=Xum S܋UX$1\cB] YDE>'^g^+W.򿓊(t™ζC6\;y-'M" 379s{vs l`̸s˷Mr(;Ơ2|xsz_=}fkCm ڻ`s]ַ?SQ^^]6UXa7Xka/`k~\*G-v^tAl+ ۳:PK^ 6'3DEf]`O rL_^-o-'E7ta?$h.J3p^U@J|{&`OQ®(NL|;{yz%Է70M|󚃠&V?A-JV/XVu#[jNݠ7WwyS[6lڴl&/mҟlV ŻnxFv~YRG7ua JJ:iXmZ>HM27fgg[R|![l$~yXdX3 ug ё&j2 J?í> 5;a=;֥֛WVX1tgGP_o]v8 {$֘]#lgdcݣ䭰ڈ".gۨ4;N Xiw)Ygt._>Ҩe#R?J8j{`37i:XZ> Vm]/ V,x8Kw?!:X:1a`5[ʰ e*&i]S6yٰi)s\nٺ!n ]+g]Y_ JVVV/XUcQuCF]H|TÞi:XQ&<8Oc>B &7 ;-L¦dۚMp:L.`tlqa @g\ moo=$:(;`8{ ǦfXaU3}\XNgp9i_sj3trH- IwX ,&L5h?{Z@a[p+d`k7k6(8?u?Rc +99'AFa +g&)/q'ʘ@DD)GvqA݁*4#Vn'C5\%veξHU͘ys{IYB0'?K[ o=wS=Yv VdvK)8]Ec+'ۂgn/"-j̜ėE-̸4EoY sVV?]6yRK] ~s=]\D&c@ZnAqE&`eP,COE>U% "d"jk72 _8캅.qۻE]$+%蝄s$%ӣA NMrp@=hVK1Ts`NQL{J9"XwzB%P`YGqIE]"D?3,Sٮ%3ѹMW)hm^h/8|˚#qddKKC\@,J%b=t$vW'YID6W)#~&sQf -f=kwoʴg*p(m=eRrOHV.` &΋Z=f`K.^6W*6Af=J$]n`Y>crOyqE}ǎ[4з5yryXpW#U2أ_~}w}jz_?k q< *;*Lzˢ{W;þu<9g> hqG"wb> ,q! ,X+Wr0YݢٻwwXMz<3xϺB{[RzgHz\a%wXE˄}zd;gKJ/g0X9a`V?;ZNŐ)S40|<]U5CXK뺽@pOhV'Nٮ%+k,K‡}4LpZgoV ba6kadN,I7{y|j{Kt; iuPai}CwyJ)=r?qQgm96X.w!J? afl+fAW)`L.u)"Vr}a'>;DW]] SO?=*k<~_<_r߾zk]zh͚Ͼȃ>|Oˁ? ~j)uq&k,z?tjuZ+_c=60Fx_~w×E/`?$0K5WrM`%"0:+l渇fgLF'VԚEX͚9}FkV<`2l/5WNF׻lX-s0qB:GE_XIEfvX|qyY9+ĘkVI[`WC?ӗg_&qCx?=lfzNJ 4Cvͭ%8, w]֒3b]X `EVޥΊ]kzNZ߁򧟚_Ϯ18%}v@_4ϞF{)*˓?{8cɯt'X<(R_M}_xd٫PDʘFn\#:ȰݒDTx&A 'T;t[C9:QCDA6Q`в7\c&Y3Mbf Mr Ƙ$>RDE箇Ni p=GZa;ŊrspW(; jExz3} [Yx90U@MAewӤ,x4n#Gpa*#Lp$W(͹3#jxh,  5Kj"RIQtFѾ. XoDGUq ^vxM lA.ƣx52ꘞh?ҫdjZchѩyPV"0w1.%ۃuh)VnbZЪev< R[WlO,m+ HNUӗN#h{d]RW !i.׃@;"QPuXL ** \copyright Qianqian Fang, 2019-2023 ** ** ZMat provides an easy-to-use interface for stream compression and decompression. ** ** It can be compiled as a MATLAB/Octave mex function (zipmat.mex/zmat.m) and compresses ** arrays and strings in MATLAB/Octave. It can also be compiled as a lightweight ** C-library (libzmat.a/libzmat.so) that can be called in C/C++/FORTRAN etc to ** provide stream-level compression and decompression. ** ** Currently, zmat/libzmat supports a list of different compression algorthms, including ** - zlib and gzip : the most widely used algorithm algorithms for .zip and .gz files ** - lzma and lzip : high compression ratio LZMA based algorithms for .lzma and .lzip files ** - lz4 and lz4hc : real-time compression based on LZ4 and LZ4HC algorithms ** - zstd : ZStandard compression algorithm ** - blosc2{blosclz,lz4,lz4hc,zlib,zstd}: blosc2 multi-threading meta-compressor/decompressors ** - base64 : base64 encoding and decoding ** ** ZMat is part of the NeuroJSON project (https://neurojson.org) ** More information can be found at https://github.com/NeuroJSON/zmat ** ** Depencency: ZLib library: https://www.zlib.net/ ** \copyright (c) 1995-2017 Jean-loup Gailly and Mark Adler ** ** Depencency: LZ4 library: https://lz4.github.io/lz4/ ** \copyright (c) 2011-2019, Yann Collet, ** ** Depencency: Original LZMA library ** \copyright Igor Pavlov ** ** Depencency: Eazylzma: https://github.com/lloyd/easylzma ** \copyright Lloyd Hilaiel (lloyd) ** ** Depencency: base64_encode()/base64_decode() ** \copyright (c) 2005-2011, Jouni Malinen ** ** Depencency: C-blosc2 ** \copyright (c) 2019-present The Blosc Development Team ** \copyright (c) 2009-2018 Francesc Alted ** ** Depencency: ZStandard ** \copyright (c) Meta Platforms, Inc. and affiliates. ** ** Depencency: miniz ** \copyright (c) 2013-2014 RAD Game Tools and Valve Software ** \copyright (c) 2010-2014 Rich Geldreich and Tenacious Software LLC ** ** \section slicense License ** GPL v3, see LICENSE.txt for details *******************************************************************************/ /***************************************************************************//** \file zmatlib.h @brief zmat library header file *******************************************************************************/ #ifndef ZMAT_LIB_H #define ZMAT_LIB_H #ifdef __cplusplus extern "C" { #endif /** * @brief Compression/encoding methods * * 0: zlib * 1: gzip * 2: base64 * 3: lzip * 4: lzma * 5: lz4 * 6: lz4hc * 7: zstd * 8: blosc2blosclz * 9: blosc2lz4 * 10: blosc2lz4hc * 11: blosc2zlib * 12: blosc2zstd * -1: unknown */ enum TZipMethod {zmZlib, zmGzip, zmBase64, zmLzip, zmLzma, zmLz4, zmLz4hc, zmZstd, zmBlosc2Blosclz, zmBlosc2Lz4, zmBlosc2Lz4hc, zmBlosc2Zlib, zmBlosc2Zstd, zmUnknown = -1}; /** * @brief advanced ZMat parameters needed for blosc2 metacompressor */ union TZMatFlags { int iscompress; /**< combined flag used to pass on to zmat_run */ struct settings { /**< unpacked flags */ char clevel; /**< compression level, 0: decompression, 1: use default level; negative: set compression level (-1 to -19) */ char nthread; /**< number of compression/decompression threads */ char shuffle; /**< byte shuffle length */ char typesize; /**< for ND-array, the byte-size for each array element */ } param; }; /** * @brief Main interface to perform compression/decompression * * @param[in] inputsize: input stream buffer length * @param[in] inputstr: input stream buffer pointer * @param[in] outputsize: output stream buffer length * @param[in] outputbuf: output stream buffer pointer * @param[in] ret: encoder/decoder specific detailed error code (if error occurs) * @param[in] iscompress: 0: decompression, 1: use default compression level; * negative interger: set compression level (-1, less, to -9, more compression). * @return return the coarse grained zmat error code; detailed error code is in ret. */ int zmat_run(const size_t inputsize, unsigned char* inputstr, size_t* outputsize, unsigned char** outputbuf, const int zipid, int* ret, const int iscompress); /** * @brief Simplified interface to perform compression (use default compression level) * * @param[in] inputsize: input stream buffer length * @param[in] inputstr: input stream buffer pointer * @param[in] outputsize: output stream buffer length * @param[in] outputbuf: output stream buffer pointer * @param[in] ret: encoder/decoder specific detailed error code (if error occurs) * @return return the coarse grained zmat error code; detailed error code is in ret. */ int zmat_encode(const size_t inputsize, unsigned char* inputstr, size_t* outputsize, unsigned char** outputbuf, const int zipid, int* ret); /** * @brief Simplified interface to perform decompression * * @param[in] inputsize: input stream buffer length * @param[in] inputstr: input stream buffer pointer * @param[in] outputsize: output stream buffer length * @param[in] outputbuf: output stream buffer pointer * @param[in] ret: encoder/decoder specific detailed error code (if error occurs) * @return return the coarse grained zmat error code; detailed error code is in ret. */ int zmat_decode(const size_t inputsize, unsigned char* inputstr, size_t* outputsize, unsigned char** outputbuf, const int zipid, int* ret); /** * @brief Free the output buffer to facilitate use in fortran * * @param[in,out] outputbuf: the outputbuf buffer's initial address to be freed */ void zmat_free(unsigned char** outputbuf); /** * @brief Look up a string in a string list and return the index * * @param[in] origkey: string to be looked up * @param[out] table: the dictionary where the string is searched * @return if found, return the index of the string in the dictionary, otherwise -1. */ int zmat_keylookup(char* origkey, const char* table[]); /** * @brief Convert error code to a string error message * * @param[in] id: zmat error code */ char* zmat_error(int id); /** * @brief base64_encode - Base64 encode * @src: Data to be encoded * @len: Length of the data to be encoded * @out_len: Pointer to output length variable, or %NULL if not used * @mode: 0 or 1, newline every 72 char and at end; 2: no new line at end, 3: no newline * Returns: Allocated buffer of out_len bytes of encoded data, * or %NULL on failure * * Caller is responsible for freeing the returned buffer. Returned buffer is * nul terminated to make it easier to use as a C string. The nul terminator is * not included in out_len. */ unsigned char* base64_encode(const unsigned char* src, size_t len, size_t* out_len, int mode); /** * base64_decode - Base64 decode * @src: Data to be decoded * @len: Length of the data to be decoded * @out_len: Pointer to output length variable * Returns: Allocated buffer of out_len bytes of decoded data, * or %NULL on failure * * Caller is responsible for freeing the returned buffer. */ unsigned char* base64_decode(const unsigned char* src, size_t len, size_t* out_len); #ifdef __cplusplus } #endif #endif zmat-0.9.9/Makefile0000644000175200007730000000131414515254731014376 0ustar rlaboissrlaboiss############################################################ # ZMat: A portable C-library and MATLAB/Octave toolbox for inline data compression # # Author: Qianqian Fang ############################################################ PKGNAME=zmat LIBNAME=lib$(PKGNAME) MEXNAME=zipmat VERSION=0.9.9 SOURCE=src EXAMPLE=example/c all: mex oct lib dll example lib: -$(MAKE) -C $(SOURCE) lib dll: -$(MAKE) -C $(SOURCE) dll mex: -$(MAKE) -C $(SOURCE) mex oct: -$(MAKE) -C $(SOURCE) oct example: lib -$(MAKE) -C $(EXAMPLE) all clean: -rm -rf $(LIBNAME).* $(MEXNAME).mex* -$(MAKE) -C $(SOURCE) clean -$(MAKE) -C $(EXAMPLE) clean .DEFAULT_GOAL := mex .PHONY: all lib dll mex oct example clean zmat-0.9.9/.github/0000755000175200007730000000000014515254731014277 5ustar rlaboissrlaboisszmat-0.9.9/.github/workflows/0000755000175200007730000000000014515254731016334 5ustar rlaboissrlaboisszmat-0.9.9/.github/workflows/run_test.yml0000644000175200007730000002456014515254731020731 0ustar rlaboissrlaboissname: ZMat CI on: [push, pull_request] jobs: octave_test: name: Octave tests strategy: # tested octave versions: ubuntu-20.04 = 5.2, ubuntu-22.04 = 6.4, macos-11 = 8.1, windows-2019 = 8.3 matrix: os: [ubuntu-20.04, ubuntu-22.04, macos-11, windows-2019] runs-on: ${{ matrix.os }} defaults: run: shell: bash steps: - name: Checkout repo uses: actions/checkout@v3 with: submodules: 'recursive' - name: Install dependencies run: | if [[ "$RUNNER_OS" == "Linux" ]]; then sudo apt-get update && sudo apt-get install -y liboctave-dev valgrind curl --retry 3 -kL https://github.com/upx/upx/releases/download/v4.1.0/upx-4.1.0-amd64_linux.tar.xz --output upx.tar.xz tar -xvf upx.tar.xz sudo mv upx-4.1.0-amd64_linux/upx /usr/bin rm -rf upx-4.1.0-amd64_linux upx.tar.xz fi [[ "$RUNNER_OS" == "macOS" ]] && brew install octave upx if [[ "$RUNNER_OS" == "Windows" ]]; then choco install upx which upx curl --retry 3 -kL http://cdimage.debian.org/mirror/gnu.org/gnu/octave/windows/octave-8.3.0-w64-64.7z --output octave_8.3.0.7z 7z x octave_8.3.0.7z -ooctave -y echo 'C:\msys64\mingw64\bin' >> $GITHUB_PATH echo "$PWD/octave/octave-8.3.0-w64-64/mingw64/bin" >> $GITHUB_PATH fi - name: Install msys2 libraries (Windows only) if: ${{ runner.os == 'Windows' }} uses: msys2/setup-msys2@v2 with: update: true install: >- mingw-w64-x86_64-gcc mingw-w64-x86_64-zlib mingw-w64-x86_64-winpthreads-git base-devel - name: Build static library run: | which gcc which g++ make -C src lib CC=gcc CXX=g++ make -C example/c all CC=gcc CXX=g++ LIBTYPE= if [ "$RUNNER_OS" == "macOS" ]; then DYLD_LIBRARY_PATH=lib example/c/testzmat; else LD_LIBRARY_PATH=lib example/c/testzmat; fi octave-cli --eval "fprintf(1,['OCTAVE_ARCH=' regexprep(computer('arch'), 'darwin[0-9.]+-', 'darwin-')])" octave-cli --eval "fprintf(1,['OCTAVE_ARCH=' regexprep(computer('arch'), 'darwin[0-9.]+-', 'darwin-')])" >> $GITHUB_ENV - name: Build dynamic library run: | make -C src dll CC=gcc CXX=g++ make -C example/c all CC=gcc CXX=g++ LIBTYPE= if [ "$RUNNER_OS" == "macOS" ]; then DYLD_LIBRARY_PATH=lib example/c/testzmat; else LD_LIBRARY_PATH=lib example/c/testzmat; fi - name: Build octave mex (Windows only) if: ${{ runner.os == 'Windows' }} shell: msys2 {0} run: | export PATH="/mingw64/bin":"$PWD/octave/octave-8.3.0-w64-64/mingw64/bin":$PATH which gcc gcc -v make -C src oct CC=gcc CXX=g++ USERLINKOPT="-static-libgcc -static-libstdc++ -fstack-protector -L/mingw64/lib -Wl,-Bstatic -lz -lwinpthread -Wl,-Bdynamic -v -s '$PWD/octave/octave-8.3.0-w64-64/mingw64/lib/octave/8.3.0/liboctinterp.dll.a'" objdump -p zipmat.mex | grep "DLL Name:" strip zipmat.mex echo "$PWD/octave/octave-8.3.0-w64-64/mingw64/bin" >> $GITHUB_PATH - name: Compress octave mex (Windows only) if: ${{ runner.os == 'Windows' }} run: upx -9 zipmat.mex || true - name: Build octave mex (Linux and Mac) if: ${{ runner.os != 'Windows' }} run: | gcc -v if [ "$RUNNER_OS" == "macOS" ]; then make -C src oct CC=gcc CXX=g++ MEXLINKOPT="-static" USERLINKOPT="-v -s" otool -L zipmat.mex upx -9 zipmat.mex || true else make -C src oct CC=gcc CXX=g++ MEXLINKOPT="-static-libgcc -static-libstdc++ -Wl,-Bstatic -lm -lpthread -Wl,-Bdynamic" USERLINKOPT="-v -s" ldd zipmat.mex fi strip -S zipmat.mex upx -9 zipmat.mex || true - name: Run octave test if: ${{ runner.os != 'Linux' }} run: | octave-cli --version if [ "$RUNNER_OS" == "Windows" ]; then export PATH="$PWD/octave/octave-8.3.0-w64-64/mingw64/bin":$PATH fi octave-cli --eval "addpath(pwd);cd test;run_zmat_test;cd ../example;demo_zmat_basic;zmat_speedbench" - name: Run octave test if: ${{ runner.os == 'Linux' }} run: | octave-cli --version valgrind octave-cli --eval "addpath(pwd);cd test;run_zmat_test;cd ../example;demo_zmat_basic" octave-cli --eval "addpath(pwd);cd example;zmat_speedbench" - name: Create package folder run: | mkdir packages mkdir zmat mkdir -p zmat/octave/${{ env.OCTAVE_ARCH }} cp *.{m,txt,rst} zmat/ cp PKG_ADD* zmat/ cp INDEX zmat/ cp DESCRIPTION zmat/ cp COPYING zmat/ cp -a example zmat/ cp -a test zmat/ - name: Create release tag run: perl -e "print 'RELEASE_TAG='. lc('${{ runner.os }}-${{ runner.arch }}-github-latest')" >> $GITHUB_ENV - name: Zip zmat run: | mv zipmat.mex zmat/octave/${{ env.OCTAVE_ARCH }} if [[ "$RUNNER_OS" == "Windows" ]]; then 7z a -tzip packages/zmat-octave-${{ env.RELEASE_TAG }}.zip zmat else zip -FSr --symlink packages/zmat-octave-${{ env.RELEASE_TAG }}.zip zmat fi - name: Upload zmat package if: ${{ matrix.os == 'ubuntu-20.04' || matrix.os == 'macos-11' || matrix.os == 'windows-2019' }} uses: actions/upload-artifact@v3 with: name: all-zmat-packages path: packages/zmat-octave-${{ env.RELEASE_TAG }}.zip matlab_test: name: MATLAB test strategy: matrix: os: [ubuntu-20.04, macos-11, windows-2019] defaults: run: shell: bash runs-on: ${{ matrix.os }} steps: - name: Checkout repo uses: actions/checkout@v3 with: submodules: 'recursive' - name: Install dependencies run: | if [[ "$RUNNER_OS" == "Linux" ]]; then curl --retry 3 -kL https://github.com/upx/upx/releases/download/v4.1.0/upx-4.1.0-amd64_linux.tar.xz --output upx.tar.xz tar -xvf upx.tar.xz sudo mv upx-4.1.0-amd64_linux/upx /usr/bin || true rm -rf upx-4.1.0-amd64_linux upx.tar.xz || true fi [[ "$RUNNER_OS" == "macOS" ]] && brew install upx [[ "$RUNNER_OS" == "Windows" ]] && choco install upx echo "installed upx" - name: Set up MATLAB uses: matlab-actions/setup-matlab@v1 - name: Install msys2 libraries (Windows only) if: ${{ runner.os == 'Windows' }} uses: msys2/setup-msys2@v2 with: update: true install: >- mingw-w64-x86_64-gcc mingw-w64-x86_64-zlib mingw-w64-x86_64-winpthreads-git base-devel - name: Set up mex gcc (Windows only) if: ${{ runner.os == 'Windows' }} run: | echo 'MW_MINGW64_LOC=/c/msys64/mingw64' >> $GITHUB_ENV echo 'COMPFLAGS=""' >> $GITHUB_ENV - name: Build MATLAB mex run: | rm -rf private if [[ "$RUNNER_OS" == "Windows" ]]; then make -C src mex CC=gcc CXX=g++ MEX="cmd //s //c mex -f mexopts_msys2_gcc.xml " USERLINKOPT="-v " CXXLIBS="-Lblosc2/lib -L/blosc2/internal-complibs/zstd-1.5.2" objdump -p zipmat.mexw* | grep "DLL Name:" elif [[ "$RUNNER_OS" == "macOS" ]]; then make -C src mex CC=gcc-10 CXX=g++10 MEXLINKOPT="-static-libstdc++" otool -L zipmat.mex* upx -9 zipmat.mex* || true else make -C src mex CC=gcc CXX=g++ MEXLINKOPT="-static-libgcc -static-libstdc++ -Wl,-Bstatic -lz -Wl,-Bdynamic" MEX="mex -v" ldd zipmat.mex* fi strip -S zipmat.mex* upx -9 zipmat.mex* || true - name: Run MATLAB examples uses: matlab-actions/run-command@v1 with: command: addpath(pwd);cd test;run_zmat_test;cd ../example;demo_zmat_basic;zmat_speedbench - name: Create package folder run: | mkdir packages mkdir zmat mkdir -p zmat/private mkdir -p zmat/example cp *.{m,txt,rst} zmat/ cp -a example/*.m zmat/example cp -a test zmat/ - name: Create release tag run: perl -e "print 'RELEASE_TAG='. lc('${{ runner.os }}-${{ runner.arch }}-github-latest')" >> $GITHUB_ENV - name: Zip zmat run: | mv zipmat.mex* zmat/private if [[ "$RUNNER_OS" == "Windows" ]]; then 7z a -tzip packages/zmat-matlab-${{ env.RELEASE_TAG }}.zip zmat else zip -FSr --symlink packages/zmat-matlab-${{ env.RELEASE_TAG }}.zip zmat fi - name: Upload zmat package if: ${{ matrix.os == 'ubuntu-20.04' || matrix.os == 'macos-11' || matrix.os == 'windows-2019' }} uses: actions/upload-artifact@v3 with: name: all-zmat-packages path: packages/zmat-matlab-${{ env.RELEASE_TAG }}.zip upload_package: name: Upload Packages needs: [octave_test, matlab_test] runs-on: ubuntu-20.04 if: ${{ github.repository_owner == 'NeuroJSON' && github.event_name != 'pull_request'}} steps: - name: Download zmat uses: actions/download-artifact@v3 with: name: all-zmat-packages path: packages - name: Create all-in-one zmat package run: | cd packages unzip -n 'zmat*.zip' rm -rf zmat/example/c/testzmat.exe zmat/example/c/testzmat zmat/example/c/testzmat.dSYM tree . zip -FSr --symlink zmat-allinone-github-latest.zip zmat rm -rf zmat - name: Display structure of downloaded files run: ls -R packages - name: Copy package to server if: ${{ github.repository_owner == 'NeuroJSON' && github.event_name != 'pull_request'}} uses: wlixcc/SFTP-Deploy-Action@v1.2.4 with: server: ${{ secrets.MCX_SERVER }} username: ${{ secrets.MCX_SERVER_USER }} ssh_private_key: ${{ secrets.MCX_SERVER_SSH_KEY }} local_path: "packages/zmat-allinone-github-latest.zip" remote_path: ${{ secrets.MCX_CI_PATH }} zmat-0.9.9/ChangeLog.txt0000644000175200007730000002035014515254731015327 0ustar rlaboissrlaboiss= Change Log = == ZMAT 0.9.9 (Foxy the Fantastic Mr. Fox - RC1), FangQ == 2023-10-22 [c79f969] use valgrind to test memory error on Linux,run example scripts 2023-10-22 [48ba900] fix memory error using valgrind, run full tests on all OS 2023-10-13 [d5b2c40] revert blosc2 to v2.8.0 to remove new dependency libdl 2023-10-13 [f2eadb5] fix broken test 2023-10-13 [e6bbda1] update cmake file, update README 2023-10-13 [bc760d5] Merge branch 'master' of github.com:NeuroJSON/zmat 2023-10-13 [7472d29] update blosc2 to v2.10.5, update zstd to 1.5.5 2023-10-13 [437f300] rename zstd folder without version number 2023-10-13*[f4408f6] use miniz by default, rename zstd folder name 2023-10-13 [fcaa396] update lz4 to 1.9.4 2023-10-13 [57115dd] Update README.rst 2023-09-14 [e50dc6b] build octave mex with octave 8.3 on windows 2023-09-14 [edbeae9] update compilezmat.m without blosc2 and zstd support 2023-06-27 [906d422] run a simpler version of octave test on windows 2023-06-25 [d628473] run demo 2023-06-25 [8c97c17] run full test on windows 2023-06-25 [a28f9c7] use simple test on windows 2023-06-25 [f443ef7] skip error testing on windows and mac 2023-06-25 [2d6732c] add matlab mex build on ubuntu 16.04 2023-06-24 [68eede3] add new octave mex for windows and mac 2023-06-24 [7416b2c] double quote windows libloc 2023-06-24 [d9f239c] static link zlib again 2023-06-24 [cf8b9d4] fix windows matlab path 2023-06-24 [05be0d6] add additional path for libmx 2023-06-23 [30beb86] test ubuntu 20.04 build error 2023-06-23 [3383f74] fix windows mx/mex not found issue 2023-06-23 [910be8b] fix quotation mark issue 2023-06-23 [fb1a63d] upload all zip files 2023-06-22 [6f6b493] upload only the allinone package 2023-06-21 [2751923] strip mex file to reduce size 2023-06-21 [1461b9b] remove -lz from static linking on linux 2023-06-21 [df92d96] add LDFLAGS to mkoctfile 2023-06-10 [e3303c5] base64 no longer add new line by default, use level=3 to restore 2023-06-06*[ff150ae] support miniz-based gzip compression and decompression, update test 2023-06-06 [e098fd2] use miniz for gzip decompression 2023-06-03 [0acc9e3] allow using HAVE_ZLIB/HAVE_LZ4/HAVE_ZSTD=yes/no with make 2023-04-11*[2caae69] add github action based CI 2022-10-24*[c655c56] add miniz 2022-09-20*[e557c65] let zmatlib C function handle base64 encoding newline instead of zmat.m 2022-09-20 [afe5654] make flag compatible with zmat v0.9.8 convention 2022-09-20*[6f1e437] add speed benchmark 2022-09-18 [84892ca] update fortran90 example 2022-09-18 [edb768b] update octave mex using octave 5 2022-09-18 [e5107bb] fix zstd default clevel, force make rebuild, update octave mex 2022-09-18 [2be1661] add zstd tests, fix blosc condition, fix zstd error code 2022-09-18 [2526367] fix mac ci error 2022-09-17*[877f2d7] add zstd compression support, fix #5 2022-09-15 [ad4605d] statically link libblosc2 and libzstd to libzmat.a 2022-09-15 [b1b0ad7] add c example in CI, add blosc2 tests 2022-09-15 [a5c072d] adjust makefile for building so file 2022-09-14 [98ab1df] force CC as dll linker 2022-09-13 [b8bdc8d] change xcode version 2022-09-13 [d863adf] remove warning 2022-09-13 [ef9ace9] debug travis on windows and osx 2022-09-13 [269a972] use system MAKE variable 2022-09-13 [7431bfc] debug travis error 2022-09-13 [78b1fa9] fix typesize, fix composit flags, make pretty 2022-09-12 [552a007] add the missing blosc2 makefile 2022-09-12 [944fb12] support blosc2 compressors 2022-09-11 [3f9542e] update zstd makefile 2022-09-11 [f07c09a] revert c-blosc2 to stable release v2.3.1 2022-09-10 [65ae09f] add blosc2 code tree 2022-08-22 [a471522] revert ci setting 2022-08-22 [9d4293e] switch CI to NeuroJSON 2022-08-22 [290a46c] switch upstream git repo owner to NeuroJSON org 2022-08-22 [59d092d] run test on matlab R2010 2022-08-22 [63b3f3d] manually patch easylzma PR lloyd/easylzma#7 to prevent memory leak 2022-08-22 [16cb296] matlab/octave are different in exp(1) and mexErrMsgTxt format 2022-08-21*[3bc5d6f] allow zmat to return error status without throwing an error 2022-08-20 [138be0b] test return status, clear memory in c example 2022-08-20 [b914a67] zlib memory tweaks 2022-08-20 [ccf0546] reformat c example 2022-08-20*[eab19fb] add unit tests, zlib memory optimization 2022-08-18 [5fe4360] test older octave (3.8) 2022-08-18 [e542838] test built in portable octave mex file 2022-08-18 [2cdc1b4] revert demo script to print outputs 2022-08-18 [675c922] add ubuntu 22.04 in CI 2022-08-18*[11b858b] allow oct file built with 5.x or newer to run in 4.x and 3.x 2022-08-16 [8291ebe] fix a warning picked by fedora package building flags 2022-08-15 [af61014] remove CXX=mkoctfile to prevent infinite loop in octave 7, fix #9 2022-06-07 [617618d] complete reformat using miss_hit 2022-05-21 [ef858a5] reformat C source code using astyle 2022-05-19 [b7cf723] add ubuntu 20.04 in ci 2022-05-19 [4b6a87e] update PKG_ADD to use locally compiled mex file first 2022-04-28*[08bd437] autoload octave mex file when addpath, thanks to cdf and mmuetzel, https://octave.discourse.group/t/supporting-mexext/2578 2021-11-20 [5f31756] port debian package patch to respect CFLAGS and LDFLAGS, fix #7 == ZMAT 0.9.8 (Archie-the-goat - beta), FangQ == 2020-05-25 [b83efba] trim base64 newline, update compilezmat, update win64 mex, release 1.0 beta 2020-05-25 [ea83b12] move zmatlib.h header to the dedicated include folder 2020-05-25 [b333987] add fortran 90 function interface and demo code 2020-05-24 [f0cbcf5] add c example 2020-05-24 [18215a2] fix compilation instruction format 2020-05-24 [392d446] add doxygen documentation 2020-05-24 [6a3e038] fix crashing if input is empty 2020-05-24 [bc8349e] accept empty input, update help info, add cmake compilation 2020-05-24 [211cddb] change windows mex file permission 2020-05-23 [5261474] add cmake script 2020-05-23 [5fac274] add makefile in the top folder 2020-05-23 [cd45748] support compression level via iscompress, add code header 2020-05-23 [be5bba0] Merge branch 'master' of https://github.com/fangq/zmat 2020-05-23 [864e9d2] add cmake support 2020-05-11 [62ecc75] Add description for libzmat and interface 2020-05-11 [7252216] make compilezmat compatible with matlab 2010 2020-05-11 [6cedd38] fix all warnings for easylzma 2020-05-11 [1ac6464] include easylzma source tree directly for easy deployment and building 2019-10-07 [a477939] compile static and dynamic libraries == ZMAT 0.9.0 (Gus-the-duck), FangQ == 2019-09-17 [73c6257] update windows mex files 2019-09-16 [a6768b9] update mex files for mac os 2019-09-16*[a940d36] initial support for the fast lz4 compression method 2019-07-12 [2cd2ac6] Update formats 2019-07-12 [53485d9] Additional format updates 2019-07-12 [b3df542] Add compilation instructions == ZMAT 0.8.0 (Mox-the-fox), FangQ == 2019-07-11 [177ed52] move mex to private/, zmat.m can pre/post process, accept nd-array and logical, can restore array size/type 2019-07-11*[0412419] change makefile to compile both mex and library 2019-06-24 [274ce37] compile zmat on octave 5 2019-06-24 [a662701] update in-matlab compile script 2019-06-23 [68f35d0] place functions into a separate unit for libzmat.a 2019-06-03 [14a84a5] update changelog to add lzma support 2019-05-07 [27a8583] make lzma optional in compilation, use static library 2019-05-06*[3d8de61] support lzma compression via easylzma, close #1 == ZMAT 0.5.0 (Zac-the-rat), FangQ == 2019-05-04 [ ] tag and release v0.5.0 2019-05-04 [d8cd440] apply patch to compile on newer matlab and octave 2019-05-04 [bd099b9] handle large inflate output with dynamic buffer, compile with 1 command 2019-05-03 [86a0dea] return the zlib return value for debugging 2019-05-03 [f204eb4] add README 2019-05-02 [e3c2ae1] function is fully compatible with octave 2019-05-02 [d48a771] avoid windows mex error 2019-05-02 [ed388d7] zmat now supports base64 encoding and decoding 2019-05-01 [545876c] add help file 2019-05-01 [5ab9a67] compile zmat on mac 2019-05-01 [396cb6f] zmat now supports gnu octave on Linux 2019-05-01 [d6a48c6] now integrated with jsonlab 2019-05-01 [3c42350] first working version, both zipping and unzipping 2019-05-01 [9300e18] rename project to zmat 2019-04-30 [93b0a77] Initial commit zmat-0.9.9/fortran90/0000755000175200007730000000000014515254731014563 5ustar rlaboissrlaboisszmat-0.9.9/fortran90/zmatlib.f900000644000175200007730000001322014515254731016543 0ustar rlaboissrlaboiss!------------------------------------------------------------------------------ ! ZMat - A portable C-library and MATLAB/Octave toolbox for inline data compression !------------------------------------------------------------------------------ ! ! module: zmatlib ! !> @author Qianqian Fang ! !> @brief An interface to call the zmatlib C-library ! ! DESCRIPTION: !> ZMat provides an easy-to-use interface for stream compression and decompression. !> !> It can be compiled as a MATLAB/Octave mex function (zipmat.mex/zmat.m) and compresses !> arrays and strings in MATLAB/Octave. It can also be compiled as a lightweight !> C-library (libzmat.a/libzmat.so) that can be called in C/C++/FORTRAN etc to !> provide stream-level compression and decompression. !> !> Currently, zmat/libzmat supports 6 different compression algorthms, including !> - zlib and gzip : the most widely used algorithm algorithms for .zip and .gz files !> - lzma and lzip : high compression ratio LZMA based algorithms for .lzma and .lzip files !> - lz4 and lz4hc : real-time compression based on LZ4 and LZ4HC algorithms !> - base64 : base64 encoding and decoding ! !> @section slicense License !> GPL v3, see LICENSE.txt for details !------------------------------------------------------------------------------ module zmatlib use iso_c_binding, only: c_char,c_size_t,c_int,c_ptr, c_loc, c_f_pointer implicit none !------------------------------------------------------------------------------ !> @brief Compression/encoding methods ! ! DESCRIPTION: !> 0: zmZlib !> 1: zmGzip !> 2: zmBase64 !> 3: zmLzip !> 4: zmLzma !> 5: zmLz4 !> 6: zmLz4hc !------------------------------------------------------------------------------ integer(c_int), parameter :: zmZlib=0, zmGzip=1, zmBase64=2, zmLzip=3, zmLzma=4, zmLz4=5, zmLz4hc=6 interface !------------------------------------------------------------------------------ !> @brief Main interface to perform compression/decompression ! !> @param[in] inputsize: input stream buffer length !> @param[in] inputstr: input stream buffer pointer !> @param[out] outputsize: output stream buffer length !> @param[out] outputbuf: output stream buffer pointer !> @param[out] ret: encoder/decoder specific detailed error code (if error occurs) !> @param[in] iscompress: 0: decompression, 1: use default compression level; !> negative interger: set compression level (-1, less, to -9, more compression) !> @return return the coarse grained zmat error code; detailed error code is in ret. !------------------------------------------------------------------------------ integer(c_int) function zmat_run(inputsize, inputbuf, outputsize, outputbuf, zipid, ret, level) bind(C) use iso_c_binding, only: c_char,c_size_t,c_int,c_ptr integer(c_size_t), value :: inputsize integer(c_int), value :: zipid, level integer(c_size_t), intent(out) :: outputsize integer(c_int), intent(out) :: ret type(c_ptr), value, intent(in) :: inputbuf type(c_ptr),intent(out) :: outputbuf end function zmat_run !------------------------------------------------------------------------------ !> @brief Simplified interface to perform compression, same as zmat_run(...,1) ! !> @param[in] inputsize: input stream buffer length !> @param[in] inputstr: input stream buffer pointer !> @param[out] outputsize: output stream buffer length !> @param[out] outputbuf: output stream buffer pointer !> @param[out] ret: encoder/decoder specific detailed error code (if error occurs) !> @return return the coarse grained zmat error code; detailed error code is in ret. !------------------------------------------------------------------------------ integer(c_int) function zmat_encode(inputsize, inputbuf, outputsize, outputbuf, zipid, ret) bind(C) use iso_c_binding, only: c_char,c_size_t,c_int,c_ptr integer(c_size_t), value :: inputsize integer(c_int), value :: zipid integer(c_size_t), intent(out) :: outputsize integer(c_int), intent(out) :: ret type(c_ptr), value, intent(in) :: inputbuf type(c_ptr),intent(out) :: outputbuf end function zmat_encode !------------------------------------------------------------------------------ !> @brief Simplified interface to perform decompression, same as zmat_run(...,0) ! !> @param[in] inputsize: input stream buffer length !> @param[in] inputstr: input stream buffer pointer !> @param[out] outputsize: output stream buffer length !> @param[out] outputbuf: output stream buffer pointer !> @param[out] ret: encoder/decoder specific detailed error code (if error occurs) !> @return return the coarse grained zmat error code; detailed error code is in ret. !------------------------------------------------------------------------------ integer(c_int) function zmat_decode(inputsize, inputbuf, outputsize, outputbuf, zipid, ret) bind(C) use iso_c_binding, only: c_char,c_size_t,c_int,c_ptr integer(c_size_t), value :: inputsize integer(c_int), value :: zipid integer(c_size_t), intent(out) :: outputsize integer(c_int), intent(out) :: ret type(c_ptr), value, intent(in) :: inputbuf type(c_ptr),intent(out) :: outputbuf end function zmat_decode !------------------------------------------------------------------------------ !> @brief Deallocating the C-allocated output buffer, must be called after each zmat use ! !> @param[in,out] outputbuf: output stream buffer pointer !------------------------------------------------------------------------------ subroutine zmat_free(outputbuf) bind(C) use iso_c_binding, only: c_ptr type(c_ptr),intent(inout) :: outputbuf end subroutine zmat_free end interface end module zmat-0.9.9/PKG_ADD0000644000175200007730000000036314515254731013755 0ustar rlaboissrlaboissif(exist(file_in_loadpath('zipmat.mex'))) autoload('zipmat',file_in_loadpath('zipmat.mex')) else autoload('zipmat',file_in_loadpath(['octave' filesep regexprep(computer('arch'), 'darwin[0-9.]+-', 'darwin-') filesep 'zipmat.mex'])) end zmat-0.9.9/INDEX0000644000175200007730000000002314515254731013524 0ustar rlaboissrlaboisszmat >> ZMat zmat zmat-0.9.9/example/0000755000175200007730000000000014515254731014372 5ustar rlaboissrlaboisszmat-0.9.9/example/zmat_speedbench.m0000644000175200007730000000226014515254731017703 0ustar rlaboissrlaboissfunction zmat_speedbench(varargin) % % Usage: % zmat_speedbench % zmat_speedbench('nthread', 8, 'shuffle', 1, 'typesize', 8) % % Author: Qianqian Fang % codecs={'zlib', 'gzip', 'lzma', 'lz4', 'lz4hc', 'zstd', ... 'blosc2blosclz', 'blosc2lz4', 'blosc2lz4hc', ... 'blosc2zlib', 'blosc2zstd'}; runbench('1. eye(2000)', eye(2000), codecs, varargin{:}); runbench('2. rand(2000)', rand(2000), codecs, varargin{:}); runbench('3. magic(2000)', uint32(magic(2000)), codecs, varargin{:}); runbench('4. peaks(2000)', single(peaks(2000)), codecs, varargin{:}); %---------------------------------------------------------- function runbench(name, mat, codecs, varargin) disp(name) res=cellfun(@(x) benchmark(x, mat, varargin{:}), codecs, 'UniformOutput', false); if(exist('OCTAVE_VERSION','builtin')) disp(res) else res=sortrows(struct2table(cell2mat(res)),'total') end %---------------------------------------------------------- function res=benchmark(codec, x, varargin) tic; [a,info]=zmat(x, 1, codec, varargin{:}); res.codec=codec; res.save=toc; res.size=uint32(numel(a)); tic; b=zmat(a, info); res.load=toc; res.total=res.load+res.save; res.sum=sum(b(:)); zmat-0.9.9/example/demo_zmat_basic.m0000644000175200007730000000071014515254731017666 0ustar rlaboissrlaboiss% addpath('../') % compression [dzip,info]=zmat(uint8(eye(5,5))) % decompression orig=reshape(zmat(dzip,0),info.size) % base64 encoding and decoding base64=zmat('zmat toolbox',1,'base64'); char(base64) orig=zmat(base64,0,'base64'); char(orig) % encode ND numeric array orig=single(rand(5)); [Aencoded,info]=zmat(orig,1,'lzma') % decode compressed ND array and restore size/type Adecoded=zmat(Aencoded,info) all(all(Adecoded==orig)) class(Adecoded) zmat-0.9.9/example/f90/0000755000175200007730000000000014515254731014770 5ustar rlaboissrlaboisszmat-0.9.9/example/f90/Makefile0000644000175200007730000000030614515254731016427 0ustar rlaboissrlaboissF90 :=gfortran all: $(F90) -g -Wall -pedantic -c ../../fortran90/zmatlib.f90 $(F90) -g -Wall -pedantic testzmat.f90 -o testzmat -L../../lib -lzmat -lz -lpthread clean: -rm -f testzmat *.o *.mod zmat-0.9.9/example/f90/testzmat.f900000644000175200007730000000743714515254731017176 0ustar rlaboissrlaboiss!------------------------------------------------------------------------------ ! ZMatLib test program !------------------------------------------------------------------------------ ! !> @author Qianqian Fang !> @brief A demo program to call zmatlib functions to encode and decode ! ! DESCRIPTION: !> This demo program shows how to call zmat_run/zmat_encode/zmat_decode to !> perform buffer encoding and decoding !------------------------------------------------------------------------------ program zmatdemo !------------------------------------------------------------------------------ ! step 1: add the below line to use zmatlib unit !------------------------------------------------------------------------------ use zmatlib use iso_c_binding, only: c_double implicit none !------------------------------------------------------------------------------ ! step 2: define needed input and output variables !------------------------------------------------------------------------------ ! here are the fortran arrays/variables that you want to pass to zmat_run character (len=128), target :: inputstr real(kind=c_double), target :: dat(10) ! here are the interface variables, note that inputbuf/outputbuf are pointers (void*) integer(kind=c_int) :: ret, res, i integer(kind=c_size_t) :: inputlen, outputlen type(c_ptr) :: inputbuf, outputbuf ! zmat output is defined as a char pointer, one must free this after each call character(kind=c_char),pointer :: myout(:) !------------------------------------------------------------------------------ ! step 3: set whatever fortran variables !------------------------------------------------------------------------------ inputstr="__o000o__(o)(o)__o000o__ =^_^= __o000o__(o)(o)__o000o__" print *, trim(inputstr) !------------------------------------------------------------------------------ ! step 4: set inputbuf to the variable you want to encode, then call zmat_run !------------------------------------------------------------------------------ inputbuf=c_loc(inputstr) inputlen=len(trim(inputstr)) res=zmat_run(inputlen,inputbuf,outputlen, outputbuf, zmBase64, ret, 1); !------------------------------------------------------------------------------ ! step 5: once done, call c_f_pointer to transfer myout to outputbuf, print results !------------------------------------------------------------------------------ call c_f_pointer(outputbuf, myout, [outputlen]) print *, myout !------------------------------------------------------------------------------ ! step 6: free the c-allocated buffer !------------------------------------------------------------------------------ call zmat_free(outputbuf) !------------------------------------------------------------------------------ ! repeat step 3 for another variable: set whatever fortran variables !------------------------------------------------------------------------------ dat = (/ (i, i = 1, 10) /) !------------------------------------------------------------------------------ ! repeat step 4: set inputbuf to the variable you want to encode, then call zmat_run !------------------------------------------------------------------------------ inputlen=sizeof(dat) inputbuf=c_loc(dat) res=zmat_run(inputlen,inputbuf,outputlen, outputbuf, zmBase64, ret, 1); !------------------------------------------------------------------------------ ! repeat step 5: once done, call c_f_pointer to transfer myout to outputbuf, print results !------------------------------------------------------------------------------ call c_f_pointer(outputbuf, myout, [outputlen]) print *, myout !------------------------------------------------------------------------------ ! repeat step 6: free the c-allocated buffer !------------------------------------------------------------------------------ call zmat_free(outputbuf) end program zmat-0.9.9/example/c/0000755000175200007730000000000014515254731014614 5ustar rlaboissrlaboisszmat-0.9.9/example/c/Makefile0000644000175200007730000000023414515254731016253 0ustar rlaboissrlaboissLIBTYPE?=-static all: $(CC) -g -Wall -pedantic testzmat.c -o testzmat -I../../include -L../../lib $(LIBTYPE) -lzmat -lz -lpthread clean: -rm -f testzmat zmat-0.9.9/example/c/testzmat.c0000644000175200007730000000637014515254731016641 0ustar rlaboissrlaboiss/***************************************************************************//** ** \mainpage ZMat - A portable C-library and MATLAB/Octave toolbox for inline data compression ** ** \author Qianqian Fang ** \copyright Qianqian Fang, 2019,2020,2022 ** ** Demo on how to use zmatlib for simple compression and decompression in C ** ** \section slicense License ** GPL v3, see LICENSE.txt for details *******************************************************************************/ #include #include #include /** * to compile, one should add -I/path/to/zmatlib.h to the compilation command and * -L/path/to/libzmat.so(or .a) -lzmat -lz -lpthread to the linking command */ #include "zmatlib.h" int main(void) { char* test[] = {"__o000o__(o)(o)__o000o__ =^_^= __o000o__(o)(o)__o000o__"}; int ret = 0, status = 0; size_t compressedlen, encodedlen, decodelen, decompressedlen; union TZMatFlags flags = {-9}; /*output buffers will be allocated inside zmat functions, host is responsible to free after use*/ unsigned char* compressed = NULL, *encoded = NULL, *decoded = NULL, *decompressed = NULL; /*=====================================*/ /* compressing and encoding the string */ /*=====================================*/ /*first, perform zlib compression use the highest compression (-9); one can use zmat_encode as well*/ ret = zmat_run(strlen(test[0]), (unsigned char*)test[0], &compressedlen, &compressed, zmZlib, &status, flags.iscompress); if (ret == 0) { /*next, encode the compressed data using base64*/ ret = zmat_encode(compressedlen, compressed, &encodedlen, &encoded, zmBase64, &status); if (ret == 0) { printf("{\n\t\"original\":\"%s\",\n", test[0]); printf("\t\"encoded\":\"%s\",\n", encoded); } } /* error handling */ if (ret) { printf("encoding failed, error code: %d: encoder error code %d\n", ret, status); if (compressed) { free(compressed); } return ret; } /*==========================================================*/ /* decode and then decompress to restore the orginal string */ /*==========================================================*/ /*first, perform base64 decoding*/ ret = zmat_decode(encodedlen, encoded, &decodelen, &decoded, zmBase64, &status); if (ret == 0) { /*next, decompress using zlib (deflate) */ ret = zmat_decode(decodelen, decoded, &decompressedlen, &decompressed, zmZlib, &status); if (ret == 0) { printf("\t\"decompressed\":\"%s\",\n", decompressed); } } printf("}\n"); /* error handling */ if (ret) { printf("decoding failed, error code: %d: decoder error code %d\n", ret, status); if (decoded) { free(decoded); } return ret; } /*==================================*/ /* host must free buffers after use */ /*==================================*/ if (compressed) { free(compressed); } if (encoded) { free(encoded); } if (decoded) { free(decoded); } if (decompressed) { free(decompressed); } return 0; } zmat-0.9.9/DESCRIPTION0000644000175200007730000000140614515254731014446 0ustar rlaboissrlaboissName: zmat Version: 0.9.9 Date: 2023-22-10 Title: In-memory data compression with zlib/gzip/lzma/lz4/zstd/blosc2 for Octave Author: Qianqian Fang Maintainer: Qianqian Fang Description: ZMat is a portable mex function to enable zlib/gzip/lzma/lz4/zstd/blosc2 based data compression/decompression and base64 encoding/decoding support in MATLAB and GNU Octave. It is fast and compact, can process a large array within a fraction of a second. Among the supported compression methods, lz4 is the fastest for compression/decompression; lzma is the slowest but has the highest compression ratio; zlib/gzip have excellent balance between speed and compression time. URL: https://neurojson.org/zmat License: GPLv3+ Categories: Compression zmat-0.9.9/zmat.m0000644000175200007730000001334314515254731014074 0ustar rlaboissrlaboissfunction varargout = zmat(varargin) % % output=zmat(input) % or % [output, info]=zmat(input, iscompress, method) % [output, info]=zmat(input, iscompress, method, options ...) % output=zmat(input, info) % % A portable data compression/decompression toolbox for MATLAB/GNU Octave % % author: Qianqian Fang % initial version created on 04/30/2019 % % input: % input: a char, non-complex numeric or logical vector or array % iscompress: (optional) if iscompress is 1, zmat compresses/encodes the input, % if 0, it decompresses/decodes the input. Default value is 1. % % if iscompress is set to a negative integer, (-iscompress) specifies % the compression level. For zlib/gzip, default level is 6 (1-9); for % lzma/lzip, default level is 5 (1-9); for lz4hc, default level is 8 (1-16). % the default compression level is used if iscompress is set to 1. % % zmat removes the trailing newline when iscompress=2 and method='base64' % all newlines are kept when iscompress=3 and method='base64' % % if one defines iscompress as the info struct (2nd output of zmat), zmat % will perform a decoding/decompression operation and recover the original % input using the info stored in the info structure. % method: (optional) compression method, currently, zmat supports the below methods % 'zlib': zlib/zip based data compression (default) % 'gzip': gzip formatted data compression % 'lzip': lzip formatted data compression % 'lzma': lzma formatted data compression % 'lz4': lz4 formatted data compression % 'lz4hc':lz4hc (LZ4 with high-compression ratio) formatted data compression % 'zstd': zstd formatted data compression % 'blosc2blosclz': blosc2 meta-compressor with blosclz compression % 'blosc2lz4': blosc2 meta-compressor with lz4 compression % 'blosc2lz4hc': blosc2 meta-compressor with lz4hc compression % 'blosc2zlib: blosc2 meta-compressor with zlib/zip compression % 'blosc2zstd': blosc2 meta-compressor with zstd compression % 'base64': encode or decode use base64 format % options: a series of ('name', value) pairs, supported options include % 'nthread': followed by an integer specifying number of threads for blosc2 meta-compressors % 'typesize': followed by an integer specifying the number of bytes per data element (used for shuffle) % 'shuffle': shuffle methods in blosc2 meta-compressor, 0 disable, 1, byte-shuffle % % output: % output: a uint8 row vector, storing the compressed or decompressed data; % empty when an error is encountered % info: (optional) a struct storing additional info regarding the input data, may have % 'type': the class of the input array % 'size': the dimensions of the input array % 'byte': the number of bytes per element in the input array % 'method': a copy of the 3rd input indicating the encoding method % 'status': the zlib/lzma/lz4 compression/decompression function return value, % including potential error codes; see documentation of the respective % libraries for details % 'level': a copy of the iscompress flag; if non-zero, specifying compression % level, see above % % example: % % [ss, info]=zmat(eye(5)) % orig=zmat(ss,0) % orig=zmat(ss,info) % ss=char(zmat('zmat test',1,'base64')) % orig=char(zmat(ss,0,'base64')) % % -- this function is part of the ZMAT toolbox (https://github.com/NeuroJSON/zmat) % if (exist('zipmat') ~= 3 && exist('zipmat') ~= 2) error('zipmat mex file is not found. you must download the mex file or recompile'); end if (nargin == 0) fprintf(1, 'Usage:\n\t[output,info]=zmat(input,iscompress,method);\nPlease run "help zmat" for more details.\n'); return end input = varargin{1}; iscompress = 1; zipmethod = 'zlib'; if (~(ischar(input) || islogical(input) || (isnumeric(input) && isreal(input)))) error('input must be a char, non-complex numeric or logical vector or N-D array'); end inputinfo=whos('input'); if(~isempty(input)) typesize=inputinfo.bytes/numel(input); else typesize=0; end if (ischar(input)) input = uint8(input); end if (nargin > 1) iscompress = varargin{2}; if (isstruct(varargin{2})) inputinfo = varargin{2}; iscompress = 0; zipmethod = inputinfo.method; end end if (nargin > 2) zipmethod = varargin{3}; end opt=struct; if (nargin > 4 && ischar(varargin{4}) && bitand(length(varargin), 1)==1) opt=cell2struct(varargin(5:2:end), varargin(4:2:end), 2); end shuffle=0; if(strfind(zipmethod,'blosc2')) shuffle=1; end nthread=getoption('nthread', 1, opt); shuffle=getoption('shuffle', shuffle, opt); typesize=getoption('typesize', typesize, opt); iscompress = round(iscompress); if ((strcmp(zipmethod, 'zlib') || strcmp(zipmethod, 'gzip')) && iscompress <= -10) iscompress = -9; end [varargout{1:max(1, nargout)}] = zipmat(input, iscompress, zipmethod, nthread, shuffle, typesize); if (strcmp(zipmethod, 'base64') && iscompress > 1) varargout{1} = char(varargout{1}); end if (exist('inputinfo', 'var') && isfield(inputinfo, 'type')) if(strcmp(inputinfo.type,'logical')) varargout{1} = logical(varargout{1}); else varargout{1} = typecast(varargout{1}, inputinfo.type); end varargout{1} = reshape(varargout{1}, inputinfo.size); end %-------------------------------------------------------------------------------- function value=getoption(key, default, opt) value=default; if(isfield(opt, key)) value=opt.(key); endzmat-0.9.9/.miss_hit0000644000175200007730000000012314515254731014553 0ustar rlaboissrlaboissproject_root suppress_rule: "redundant_brackets" indent_function_file_body: false zmat-0.9.9/README.rst0000644000175200007730000004466514515254731014445 0ustar rlaboissrlaboiss.. image:: https://neurojson.org/wiki/upload/neurojson_banner_long.png ############################################################################## ZMAT: A portable C-library and MATLAB/Octave toolbox for zlib/gzip/lzma/lz4/zstd/blosc2 data compression ############################################################################## * Copyright (C) 2019-2023 Qianqian Fang * License: GNU General Public License version 3 (GPL v3), see License*.txt * Version: 0.9.9 (Foxy the Fantastic Mr. Fox - RC1) * URL: https://github.com/NeuroJSON/zmat * Acknowledgement: This project is part of the `NeuroJSON project `_ supported by US National Institute of Health (NIH) grant `U24-NS124027 `_ .. image:: https://github.com/NeuroJSON/zmat/actions/workflows/run_test.yml/badge.svg :target: https://github.com/NeuroJSON/zmat/actions/workflows/run_test.yml ################# Table of Contents ################# .. contents:: :local: :depth: 3 ============ Introduction ============ ZMat provides both an easy-to-use C-based data compression library - ``libzmat`` as well a portable mex function to enable ``zlib/gzip/lzma/lz4/zstd/blosc2`` based data compression/decompression and ``base64`` encoding/decoding support in MATLAB and GNU Octave. It is fast and compact, can process a large array within a fraction of a second. Among all the supported compression methods, or codecs, ``lz4`` is among the fastest for both compression/decompression; ``lzma`` is the slowest but offers the highest compression ratio; ``zlib/gzip`` have excellent balance between speed and compression time; ``zstd`` can be fast, although not as fast as ``lz4``, at the low-compression levels, and can offer higher compression ratios than ``zlib/gzip``, although not as high as ``lzma``, at high compression levels. We understand there are many other existing general purpose data compression algorithms. We prioritize the support of these compression algorithms because they have widespread use. Starting in v0.9.9, we added support to a high-performance meta-compressor, called ``blosc2`` (https://blosc.org). The ``blosc2`` compressor is not single compression method, but a container format that supports a diverse set of strategies to losslessly compress data, especially optimized for storing and retrieving N-D numerical data of uniform binary types. Users can choose between ``blosc2blosclz``, ``blosc2lz4``, ``blosc2lz4hc``, ``blosc2zlib`` and ``blosc2zstd``, to access these blosc2 codecs. The ``libzmat`` library, including the static library (``libzmat.a``) and the dynamic library ``libzmat.so`` or ``libzmat.dll``, provides a simple interface to conveniently compress or decompress a memory buffer: .. code:: c int zmat_run( const size_t inputsize, /* input buffer data length */ unsigned char *inputstr, /* input buffer */ size_t *outputsize, /* output buffer data length */ unsigned char **outputbuf, /* output buffer */ const int zipid, /* 0: zlib, 1: gzip, 2: base64, 3: lzma, 4: lzip, 5: lz4, 6: lz4hc 7: zstd, 8: blosc2blosclz, 9: blosc2lz4, 10: blosc2lz4hc, 11: blosc2zlib, 12: blosc2zstd */ int *status, /* return status for error handling */ const int clevel /* 1 to compress (default level); 0 to decompress, -1 to -9 (-22 for zstd): setting compression level */ ); When ``blosc2`` codes are used, users can set additional compression flags, including number of threads, byte-shuffling length, can be set via the ``flags.param`` interface in the following data structure, and pass on ``flags.iscompress`` as the last input to ``zmat_run``. .. code:: c union cflag { int iscompress; /* combined flag used to pass on to zmat_run */ struct settings { /* unpacked flags */ char clevel; /* compression level */ char nthread; /* number of compression/decompression threads */ char shuffle; /* byte shuffle length */ char typesize; /* for ND-array, the byte-size for each array element */ } param; } flags = {0}; The zmat library is highly portable and can be directly embedded in the source code to provide maximal portability. In the ``test`` folder, we provided sample codes to call ``zmat_run/zmat_encode/zmat_decode`` for stream-level compression and decompression in C and Fortran90. The Fortran90 C-binding module can be found in the ``fortran90`` folder. The ZMat MATLAB function accepts 3 types of inputs: char-based strings, numerical arrays or vectors, or logical arrays/vectors. Any other input format will result in an error unless you typecast the input into ``int8/uint8`` format. A multi-dimensional numerical array is accepted, and the original input's type/dimension info is stored in the 2nd output ``"info"``. If one calls ``zmat`` with both the encoded data (in byte vector) and the ``"info"`` structure, zmat will first decode the binary data and then restore the original input's type and size. The pre-compiled mex binaries for MATLAB are stored inside the subfolder named ``private``. Those precompiled for GNU Octave are stored in the subfolder named ``octave``, with one operating system per subfolder. The ``PKG_ADD`` script should automatically select the correct mex file when one types ``addpath`` in Octave. These precompiled mex files are expected to run out-of-box across a wide-range of MATLAB (tested as old as R2008) and Octave (tested as old as v3.8). If you do not want to compile zmat yourself, you can download the precompiled package by either clicking on the "Download ZIP" button on the above URL, or use the below git command: .. code:: shell git clone https://github.com/NeuroJSON/zmat.git ================ Installation ================ The installation of ZMat is no different from any other simple MATLAB toolboxes. You only need to download/unzip the package to a folder, and add the folder's path (that contains ``zmat.m`` and the ``"private"`` folder) to MATLAB's path list by using the following command: .. code:: matlab addpath('/path/to/zmat'); For Octave, one needs to copy the ``zipmat.mat`` file inside the "``octave``", from the subfolder matching the OS into the "``private``" subfolder. If you want to add this path permanently, you need to type "``pathtool``", browse to the zmat root folder and add to the list, then click "Save". Then, run "``rehash``" in MATLAB, and type "``which zmat``", if you see an output, that means ZMat is installed for MATLAB/Octave. If you use MATLAB in a shared environment such as a Linux server, the best way to add path is to type .. code:: shell mkdir ~/matlab/ nano ~/matlab/startup.m and type ``addpath('/path/to/zmat')`` in this file, save and quit the editor. MATLAB will execute this file every time it starts. For Octave, the file you need to edit is ``~/.octaverc`` , where "``~``" is your home directory. ================ Installing ZMat on Linux Distributions ================ One can directly install zmat on Fedora Linux 29 or later via the below shell command .. code:: shell sudo dnf install octave-zmat Similarly, the below command installs the ``libzmat`` library for developing software using this library: .. code:: shell sudo dnf install zmat zmat-devel zmat-static The above command installs the dynamic library, C/Fortran90 header files and static library, respectively Similarly, if one uses Debian (11) or Ubuntu 21.04 or newer, the command to install zmat toolbox for Octave (and optionally for MATLAB) is .. code:: shell sudo apt-get install octave-zmat matlab-zmat and that for installing the development environment is .. code:: shell sudo apt-get install libzmat1 libzmat1-dev A Ubuntu (16.04/18.04) user can use the same commands as Debian to install these packages but one must first run .. code:: shell sudo add-apt-repository ppa:fangq/ppa sudo apt-get update to enable the `relevant PPA `_ (personal package achieve) first. ================ Using ZMat in MATLAB ================ ZMat provides a single mex function, ``zipmat.mex*`` -- for both compressing/encoding or decompresing/decoding data streams. The help info of the function is shown below ---------- zmat.m ---------- .. code-block:: matlab output=zmat(input) or [output, info]=zmat(input, iscompress, method) [output, info]=zmat(input, iscompress, method, options ...) output=zmat(input, info) A portable data compression/decompression toolbox for MATLAB/GNU Octave author: Qianqian Fang initial version created on 04/30/2019 input: input: a char, non-complex numeric or logical vector or array iscompress: (optional) if iscompress is 1, zmat compresses/encodes the input, if 0, it decompresses/decodes the input. Default value is 1. if iscompress is set to a negative integer, (-iscompress) specifies the compression level. For zlib/gzip, default level is 6 (1-9); for lzma/lzip, default level is 5 (1-9); for lz4hc, default level is 8 (1-16). the default compression level is used if iscompress is set to 1. zmat removes the trailing newline when iscompress=2 and method='base64' all newlines are kept when iscompress=3 and method='base64' if one defines iscompress as the info struct (2nd output of zmat), zmat will perform a decoding/decompression operation and recover the original input using the info stored in the info structure. method: (optional) compression method, currently, zmat supports the below methods 'zlib': zlib/zip based data compression (default) 'gzip': gzip formatted data compression 'lzip': lzip formatted data compression 'lzma': lzma formatted data compression 'lz4': lz4 formatted data compression 'lz4hc':lz4hc (LZ4 with high-compression ratio) formatted data compression 'zstd': zstd formatted data compression 'blosc2blosclz': blosc2 meta-compressor with blosclz compression 'blosc2lz4': blosc2 meta-compressor with lz4 compression 'blosc2lz4hc': blosc2 meta-compressor with lz4hc compression 'blosc2zlib': blosc2 meta-compressor with zlib/zip compression 'blosc2zstd': blosc2 meta-compressor with zstd compression 'base64': encode or decode use base64 format options: a series of ('name', value) pairs, supported options include 'nthread': followed by an integer specifying number of threads for blosc2 meta-compressors 'typesize': followed by an integer specifying the number of bytes per data element (used for shuffle) 'shuffle': shuffle methods in blosc2 meta-compressor, 0 disable, 1, byte-shuffle output: output: a uint8 row vector, storing the compressed or decompressed data; empty when an error is encountered info: (optional) a struct storing additional info regarding the input data, may have 'type': the class of the input array 'size': the dimensions of the input array 'byte': the number of bytes per element in the input array 'method': a copy of the 3rd input indicating the encoding method 'status': the zlib/lzma/lz4 compression/decompression function return value, including potential error codes; see documentation of the respective libraries for details 'level': a copy of the iscompress flag; if non-zero, specifying compression level, see above example: [ss, info]=zmat(eye(5)) orig=zmat(ss,0) orig=zmat(ss,info) ss=char(zmat('zmat test',1,'base64')) orig=char(zmat(ss,0,'base64')) -- this function is part of the zmat toolbox (https://github.com/NeuroJSON/zmat) --------- examples --------- Under the ``"example"`` folder, you can find a demo script showing the basic utilities of ZMat. Running the ``"demo_zmat_basic.m"`` script, you can see how to compress/decompress a simple array, as well as apply base64 encoding/decoding to strings. Please run these examples and understand how ZMat works before you use it to process your data. Under the ``"c"`` and ``"f90"`` folders, sample C/Fortran90 units calling the compression/decompression APIs provided by zmat are also provided. You may run ``"make"`` in each of the folders to build the binary and execute the output program. --------- unit tests --------- Under the ``"test"`` folder, you can run ``"run_zmat_test.m"`` script to run unit tests on the key features provided by zmat. ========================== Compile ZMat ========================== To recompile ZMat, you first need to check out ZMat source code, along with the needed submodules from the Github repository using the below command .. code:: shell git clone https://github.com/NeuroJSON/zmat.git zmat Next, you need to make sure your system has ``gcc``, ``g++``, ``mex`` and ``mkoctfile`` (if compiling for Octave is needed). If not, please install gcc, MATLAB and GNU Octave and add the paths to these utilities to the system PATH environment variable. To compile zmat, you may choose one of the three methods: 1. Method 1: please open MATLAB or Octave, and run the below commands .. code-block:: matlab cd zmat/src compilezmat The above script utilizes the MinGW-w64 MATLAB Compiler plugin. To install the MinGW-w64 compiler plugin for MATLAB, please follow the below steps - If you have MATLAB R2017b or later, you may skip this step. To compile mcxlabcl in MATLAB R2017a or earlier on Windows, you must pre-install the MATLAB support for MinGW-w64 compiler https://www.mathworks.com/matlabcentral/fileexchange/52848-matlab-support-for-mingw-w64-c-c-compiler Note: it appears that installing the above Add On is no longer working and may give an error at the download stage. In this case, you should install MSYS2 from https://www.msys2.org/. Once you install MSYS2, run MSYS2.0 MinGW 64bit from Start menu, in the popup terminal window, type .. code-block:: shell pacman -Syu pacman -S base-devel gcc git zlib-devel Then, start MATLAB, and in the command window, run .. code-block:: matlab setenv('MW_MINGW64_LOC','C:\msys64\usr'); - After installation of MATLAB MinGW support, you must type ``mex -setup C`` in MATLAB and select "MinGW64 Compiler (C)". - Once you select the MingW C compiler, you should run ``mex -setup C++`` again in MATLAB and select "MinGW64 Compiler (C++)" to compile C++. 2. Method 2: Compile with cmake (3.3 or later) Please open a terminal, and run the below shall commands .. code-block:: shell cd zmat/src rm -rf build mkdir build && cd build cmake ../ make clean make if MATLAB was not installed in a standard path, you may change ``cmake ../`` to .. code-block:: shell cmake Matlab_ROOT_DIR=/path/to/matlab/root ../ by default, this will first compile ``libzmat.a`` and then create the ``.mex`` file that is statically linked with ``libzmat.a``. If one prefers to create a dynamic library ``libzmat.so`` and then a dynamically linked ``.mex`` file, this can be done by .. code-block:: shell cmake Matlab_ROOT_DIR=/path/to/matlab/root -DSTATIC_LIB=off ../ 3. Method 3: please open a terminal, and run the below shall commands .. code-block:: shell cd zmat/src make clean make mex to create the mex file for MATLAB, and run ``make clean oct`` to compile the mex file for Octave. The compilex mex files are named as ``zipmat.mex*`` under the zmat root folder. One may move those into the ``private`` folder to overwrite the existing files, or leave them in the root folder. MATLAB/Octave will use these files when ``zmat`` is called. ========================== Contribution and feedback ========================== ZMat is an open-source project. This means you can not only use it and modify it as you wish, but also you can contribute your changes back to ZMat so that everyone else can enjoy the improvement. For anyone who want to contribute, please download ZMat source code from its source code repositories by using the following command: .. code:: shell git clone https://github.com/NeuroJSON/zmat.git zmat or browsing the github site at .. code:: shell https://github.com/NeuroJSON/zmat You can make changes to the files as needed. Once you are satisfied with your changes, and ready to share it with others, please submit your changes as a "pull request" on github. The project maintainer, Dr. Qianqian Fang will review the changes and choose to accept the patch. We appreciate any suggestions and feedbacks from you. Please use the NeuroJSON forum to report any questions you may have regarding ZMat: `NeuroJSON forum `_ (Subscription to the mailing list is needed in order to post messages). ========================== Acknowledgement ========================== ZMat is linked against 4 open-source data compression libraries 1. ZLib library: https://www.zlib.net/ * Copyright (C) 1995-2017 Jean-loup Gailly and Mark Adler * License: Zlib license 2. Eazylzma: https://github.com/lloyd/easylzma * Author: Lloyd Hilaiel (lloyd) * License: public domain 3. Original LZMA library: * Author: Igor Pavlov * License: public domain 4. LZ4 library: https://lz4.github.io/lz4/ * Copyright (C) 2011-2019, Yann Collet. * License: BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) 5. C-blosc2: https://blosc.org/ * Copyright (c) 2009-2018 Francesc Alted * Copyright (c) 2019-present The Blosc Development Team * License: BSD 3-Clause License (http://www.opensource.org/licenses/bsd-license.php) 6. ZStd: https://facebook.github.io/zstd/ * Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved. * License: BSD 3-Clause License (http://www.opensource.org/licenses/bsd-license.php) 7. miniz: https://github.com/richgel999/miniz * Copyright (c) 2013-2014 RAD Game Tools and Valve Software * Copyright (c) 2010-2014 Rich Geldreich and Tenacious Software LLC * License: MIT License (https://github.com/richgel999/miniz/blob/master/LICENSE) zmat-0.9.9/test/0000755000175200007730000000000014515254731013716 5ustar rlaboissrlaboisszmat-0.9.9/test/test_zmat.m0000644000175200007730000000357514515254731016120 0ustar rlaboissrlaboissfunction test_zmat(testname, method, input, expected, varargin) opt=struct('level',1); if(length(varargin)>1 && rem(length(varargin),2)==0 && ischar(varargin{1})) for i=1:2:length(varargin) opt.(varargin{i})=varargin{i+1}; end end try [res, info] = zmat(input, opt.level, method); if(strcmp(method,'gzip') && opt.level && length(res)>=10) res(10)=0; % the 10th byte of the gzip header is OS ID end catch ME if(ischar(expected) && ~isempty(strfind(ME.message, expected))) fprintf(1, 'Testing exception %s: ok\n\toutput:''%s''\n', testname, ME.message); else warning('Test exception %s: failed: expected ''%s'', obtained ''%s''', testname, expected, ME.message); end return; end if(isfield(opt,'info')) res=info.(opt.info); end if(isfield(opt,'status')) if(info.status~=opt.status) warning('Test %s: failed: expected ''%s'', obtained ''%s''', testname, mat2str(expected), mat2str(res)); else fprintf(1, 'Testing %s error: ok\n\tstatus:''%d''\n', testname, info.status); end return; end if (~isequal(res, expected)) warning('Test %s: failed: expected ''%s'', obtained ''%s''', testname, mat2str(expected), mat2str(res)); else if(ischar(res)) fprintf(1, 'Testing %s: ok\n\toutput:''%s''\n', testname, res); else fprintf(1, 'Testing %s: ok\n\toutput:''%s''\n', testname, mat2str(res)); end if(isfield(opt,'info') || opt.level == 0) return; end newres = zmat(res, info); if (exist('isequaln')) try if (isequaln(newres, input)) fprintf(1, '\t%s successfully restored the input\n', method); end catch end else try if (newres == input) fprintf(1, '\t%s successfully restored the input\n', method); end catch end end end end zmat-0.9.9/test/run_zmat_test.m0000644000175200007730000002440314515254731016775 0ustar rlaboissrlaboissfunction run_zmat_test(tests) % % run_zmat_test % or % run_zmat_test(tests) % run_zmat_test({'c','d','err'}) % % Unit testing for ZMat toolbox % % authors:Qianqian Fang (q.fang neu.edu) % date: 2022/08/19 % % input: % tests: is a cell array of strings, possible elements include % 'c': compression % 'd': decompression % 'err': error handling % % license: % BSD or GPL version 3, see LICENSE_{BSD,GPLv3}.txt files for details % % -- this function is part of ZMat toolbox (https://github.com/NeuroJSON/zmat) % if (nargin == 0) tests = {'c', 'd', 'err'}; end %% if (ismember('c', tests)) fprintf(sprintf('%s\n', char(ones(1, 79) * 61))); fprintf('Test compression\n'); fprintf(sprintf('%s\n', char(ones(1, 79) * 61))); test_zmat('zlib (empty)', 'zlib', [], zeros(1,0)); test_zmat('gzip (empty)', 'gzip', '', zeros(1,0)); test_zmat('lzma (empty)', 'lzma', zeros(0,0), zeros(1,0)); test_zmat('lzip (empty)', 'lzip', [], zeros(1,0)); test_zmat('lz4 (empty)', 'lz4', '', zeros(1,0)); test_zmat('lz4hc (empty)', 'lz4hc', zeros(0,0), zeros(1,0)); test_zmat('base64 (empty)', 'base64', [], zeros(1,0)); isminiz=zmat('0',1,'gzip'); isminiz=(isminiz(10)==255); if(isminiz) test_zmat('zlib (scalar)', 'zlib', pi, [120 1 1 8 0 247 255 24 45 68 84 251 33 9 64 9 224 2 67]); test_zmat('gzip (scalar)', 'gzip', 'test gzip', [31 139 8 0 0 0 0 0 0 0 1 9 0 246 255 116 101 115 116 32 103 122 105 112 35 1 18 68 9 0 0 0]); else test_zmat('zlib (scalar)', 'zlib', pi, [120 156 147 208 117 9 249 173 200 233 0 0 9 224 2 67]); test_zmat('gzip (scalar)', 'gzip', 'test gzip', [31 139 8 0 0 0 0 0 0 0 43 73 45 46 81 72 175 202 44 0 0 35 1 18 68 9 0 0 0]); end test_zmat('lzma (scalar)', 'lzma', uint32(1902), [93 0 0 16 0 4 0 0 0 0 0 0 0 0 55 1 188 0 10 215 98 63 255 251 13 160 0]); test_zmat('lzip (scalar)', 'lzip', single(89.8901), [76 90 73 80 0 20 0 93 177 210 100 7 58 15 255 255 252 63 0 0 133 75 237 40 4 0 0 0 0 0 0 0]); test_zmat('lz4 (scalar)', 'lz4', 2.71828, [128 144 247 170 149 9 191 5 64]); test_zmat('lz4hc (scalar)', 'lz4hc', 0.0, [128 0 0 0 0 0 0 0 0]); test_zmat('zstd (scalar)', 'base64', zmat(uint8(198),1,'zstd'), 'KLUv/SABCQAAxg==', 'level', 2); test_zmat('blosc2blosclz (scalar)', 'base64', zmat(uint8(201),1,'blosc2blosclz'), 'BQEHAQEAAAABAAAAIQAAAAAAAAAAAAAAAAAAAAAAAADJ', 'level', 2); test_zmat('blosc2lz4 (scalar)', 'base64', zmat(single(202),1,'blosc2lz4'), 'BQEHBAQAAAAEAAAAJAAAAAAAAAAAAQEAAAAAAAAAAAAAAEpD', 'level', 2); test_zmat('blosc2lz4hc (scalar)', 'base64', zmat(uint32(58392),1,'blosc2lz4hc'), 'BQEHBAQAAAAEAAAAJAAAAAAAAAAAAQIAAAAAAAAAAAAY5AAA', 'level', 2); if(~isminiz) test_zmat('blosc2zlib (scalar)', 'base64', zmat(2.2,1,'blosc2zlib'), 'BQEHCAgAAAAIAAAAKAAAAAAAAAAAAQQAAAAAAAAAAACamZmZmZkBQA==', 'level', 2); end test_zmat('blosc2zstd (scalar)', 'base64', zmat(logical(0.1),1,'blosc2zstd'), 'BQEHAQEAAAABAAAAIQAAAAAAAAAAAAUAAAAAAAAAAAAB', 'level', 2); test_zmat('base64 (scalar)', 'base64', uint8(100), [90 65 61 61]); if(isminiz) test_zmat('zlib (array)', 'zlib', uint8([1,2,3]), [120 1 1 3 0 252 255 1 2 3 0 13 0 7]); test_zmat('gzip (array)', 'gzip', single([pi;exp(1)]), [31 139 8 0 0 0 0 0 0 0 1 8 0 247 255 219 15 73 64 84 248 45 64 197 103 247 17 8 0 0 0]); else test_zmat('zlib (array)', 'zlib', uint8([1,2,3]), [120 156 99 100 98 6 0 0 13 0 7]); test_zmat('gzip (array)', 'gzip', single([pi;exp(1)]), [31 139 8 0 0 0 0 0 0 0 187 205 239 233 16 242 67 215 1 0 197 103 247 17 8 0 0 0]); end test_zmat('lzma (array)', 'lzma', uint8(magic(3)), [93 0 0 16 0 9 0 0 0 0 0 0 0 0 4 0 207 17 232 198 252 139 53 45 235 13 99 255 249 133 192 0]); test_zmat('lzip (array)', 'lzip', uint8(reshape(1:(2*3*4), [3,2,4])), [76 90 73 80 0 20 0 0 128 157 97 211 13 93 174 25 62 219 132 40 29 52 41 93 234 35 61 128 60 72 152 87 41 88 255 253 203 224 0 163 16 142 146 24 0 0 0 0 0 0 0]); test_zmat('lz4 (array)', 'lz4', [1], [128 0 0 0 0 0 0 240 63]); test_zmat('lz4hc (array)', 'lz4hc', 'test zmat', [144 116 101 115 116 32 122 109 97 116]); test_zmat('zstd (array)', 'base64', zmat(uint8(magic(5)),1,'zstd'), 'KLUv/SAZyQAAERcECgsYBQYMEgEHDRMZCA4UFQIPEBYDCQ=='); test_zmat('blosc2blosclz (array)', 'base64', zmat(uint8(magic(4)),1,'blosc2blosclz'), 'BQEHARAAAAAQAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAQBQkEAgsHDgMKBg8NCAwB'); test_zmat('blosc2lz4 (array)', 'base64', zmat(uint16(magic(3)),1,'blosc2lz4'), 'BQEHAhIAAAASAAAAMgAAAAAAAAAAAQEAAAAAAAAAAAAIAAMABAABAAUACQAGAAcAAgA='); test_zmat('blosc2lz4hc (array)', 'base64', zmat([1.1,2.1,3.1],1,'blosc2lz4hc'), 'BQEHCBgAAAAYAAAAOAAAAAAAAAAAAQIAAAAAAAAAAACamZmZmZnxP83MzMzMzABAzczMzMzMCEA='); if(~isminiz) test_zmat('blosc2zlib (array)', 'base64', zmat(uint8(reshape(1:(2*3*4), [3,2,4])),1,'blosc2zlib'), 'BQEHARgAAAAYAAAAOAAAAAAAAAAAAAQAAAAAAAAAAAABAgMEBQYHCAkKCwwNDg8QERITFBUWFxg='); end test_zmat('blosc2zstd (array)', 'base64', zmat(uint8(ones(2,3,4)),1,'blosc2zstd'), 'BQEHARgAAAAYAAAAOAAAAAAAAAAAAAUAAAAAAAAAAAABAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQE='); test_zmat('base64 (array)', 'base64', ['test';'zmat'], [100 72 112 108 98 88 78 104 100 72 81 61]); if(isminiz) test_zmat('zlib (level=9)', 'zlib', 55, [120 1 1 8 0 247 255 0 0 0 0 0 128 75 64 2 94 1 12], 'level', -9); test_zmat('zlib (level=2.6)', 'zlib', 55, [120 1 1 8 0 247 255 0 0 0 0 0 128 75 64 2 94 1 12], 'level', -2.6); test_zmat('gzip (level)', 'gzip', 'level 9', [31 139 8 0 0 0 0 0 0 0 1 7 0 248 255 108 101 118 101 108 32 57 182 235 101 120 7 0 0 0], 'level', -9); else test_zmat('zlib (level=9)', 'zlib', 55, [120 218 99 96 0 130 6 111 7 0 2 94 1 12], 'level', -9); test_zmat('zlib (level=2.6)', 'zlib', 55, [120 94 99 96 0 130 6 111 7 0 2 94 1 12], 'level', -2.6); test_zmat('gzip (level)', 'gzip', 'level 9', [31 139 8 0 0 0 0 0 2 0 203 73 45 75 205 81 176 4 0 182 235 101 120 7 0 0 0], 'level', -9); end test_zmat('lzma (level)', 'lzma', uint8([1,2,3,4]), [93 0 0 16 0 4 0 0 0 0 0 0 0 0 0 128 157 97 229 167 24 31 255 247 52 128 0], 'level', -9); test_zmat('lzip (level)', 'lzip', logical([1,2,3,4]), [76 90 73 80 0 20 0 0 232 190 92 247 255 255 224 0 128 0 153 211 38 246 4 0 0 0 0 0 0 0],'level', -9); test_zmat('lz4 (level)', 'lz4', 'random data', [176 114 97 110 100 111 109 32 100 97 116 97],'level', -9); test_zmat('lz4hc (level)', 'lz4hc', 1.2, [128 51 51 51 51 51 51 243 63],'level', -9); test_zmat('zstd (level=1)', 'base64', zmat(eye(10),-1,'zstd'), 'KLUv/WAgAp0AAEgAAAAAAADwPwACAL+2UAGZwBE='); test_zmat('zstd (level=3)', 'base64', zmat(eye(10),-3,'zstd'), 'KLUv/WAgAo0AACgAAPA/AAMAv7ZQAQEzLIAF'); test_zmat('zstd (level=9)', 'base64', zmat(eye(10),-9,'zstd'), 'KLUv/WAgAo0AACgAAPA/AAMAv7ZQAQEzLIAF'); test_zmat('zstd (level=19)', 'base64', zmat(eye(10),-19,'zstd'), 'KLUv/WAgAp0AAEgAAAAAAADwPwACAMW2UAFGwRE='); test_zmat('blosc2blosclz (typesize=2)', 'base64', zmat(uint32(magic(4)),1,'blosc2blosclz','typesize',2), 'BQEFAkAAAABAAAAATAAAAAAAAAAAAQAAAAAAAAAAAAAkAAAAIAAAABAABQAJAAQAAgALAAcADgADAAoABgAPAA0ACAAMAAEAAAAAAA=='); test_zmat('blosc2blosclz (typesize=4)', 'base64', zmat(uint32(magic(4)),1,'blosc2blosclz','typesize',4), 'BQEXBEAAAABAAAAAYAAAAAAAAAAAAQAAAAAAAAAAAAAQAAAABQAAAAkAAAAEAAAAAgAAAAsAAAAHAAAADgAAAAMAAAAKAAAABgAAAA8AAAANAAAACAAAAAwAAAABAAAA'); test_zmat('blosc2blosclz (typesize=8)', 'base64', zmat(uint32(magic(4)),1,'blosc2blosclz','typesize',8), 'BQEXCEAAAABAAAAAYAAAAAAAAAAAAQAAAAAAAAAAAAAQAAAABQAAAAkAAAAEAAAAAgAAAAsAAAAHAAAADgAAAAMAAAAKAAAABgAAAA8AAAANAAAACAAAAAwAAAABAAAA'); test_zmat('blosc2zstd (typesize=2)', 'base64', zmat(single(magic(4)),1,'blosc2zstd','typesize',2), 'BQGXAkAAAABAAAAAYAAAAAAAAAAAAQUAAAAAAAAAAAAAAIBBAACgQAAAEEEAAIBAAAAAQAAAMEEAAOBAAABgQQAAQEAAACBBAADAQAAAcEEAAFBBAAAAQQAAQEEAAIA/'); test_zmat('blosc2zstd (typesize=4)', 'base64', zmat(single(magic(4)),1,'blosc2zstd','typesize',4), 'BQGVBEAAAABAAAAAVQAAAAAAAAAAAQUAAAAAAAAAAAAkAAAALQAAACi1L/0gQCUBAOAAAICgEIAAMOBgQCDAcFAAQIBBQEFAQEFBQUE/AgByQxMBFg=='); test_zmat('blosc2zstd (typesize=8)', 'base64', zmat(single(magic(4)),1,'blosc2zstd','typesize',8), 'BQGXCEAAAABAAAAAYAAAAAAAAAAAAQUAAAAAAAAAAAAAAIBBAACgQAAAEEEAAIBAAAAAQAAAMEEAAOBAAABgQQAAQEAAACBBAADAQAAAcEEAAFBBAAAAQQAAQEEAAIA/'); test_zmat('base64 (no newline)', 'base64', uint8(100), sprintf('ZA==\n'), 'level', 3); test_zmat('base64 (trailing newline)', 'base64', ones(7,1), sprintf('AAAAAAAA8D8AAAAAAADwPwAAAAAAAPA/AAAAAAAA8D8AAAAAAADwPwAAAAAAAPA/AAAAAAAA\n8D8='), 'level', 2); test_zmat('base64 (all newline)', 'base64', ones(7,1), sprintf('AAAAAAAA8D8AAAAAAADwPwAAAAAAAPA/AAAAAAAA8D8AAAAAAADwPwAAAAAAAPA/AAAAAAAA\n8D8=\n'), 'level', 3); end %% if (ismember('d', tests)) fprintf(sprintf('%s\n', char(ones(1, 79) * 61))); fprintf('Test decompression\n'); fprintf(sprintf('%s\n', char(ones(1, 79) * 61))); test_zmat('zlib (scalar)', 'zlib', uint8([120 156 147 208 117 9 249 173 200 233 0 0 9 224 2 67]), typecast(pi, 'uint8'), 'level', 0); end %% if (ismember('err', tests)) fprintf(sprintf('%s\n', char(ones(1, 79) * 61))); fprintf('Test error messages\n'); fprintf(sprintf('%s\n', char(ones(1, 79) * 61))); test_zmat('empty method', '', [], 'the ''method'' field must be a non-empty string'); test_zmat('unsupported method', 'ppp', [], 'the specified compression method is not supported'); test_zmat('unsupported input (cell)', 'zlib', {}, 'input must be a char, non-complex numeric or logical vector or N-D array'); test_zmat('unsupported input (handle)', 'zlib', @sin, 'input must be a char, non-complex numeric or logical vector or N-D array'); test_zmat('unsupported input (complex)', 'gzip', 1+3i, 'input must be a char, non-complex numeric or logical vector or N-D array'); if (exist('string')) if(~ischar(string('test'))) test_zmat('unsupported input (string)', 'zlib', string(sprintf('zmat\ntest')), 'input must be a char, non-complex numeric or logical vector or N-D array'); end end test_zmat('zlib wrong input format', 'zlib', [1, 2, 3, 4], [], 'level', 0, 'status', -3); test_zmat('blosc2zstd wrong input format', 'blosc2zstd', [1, 2, 3, 4], [], 'level', 0, 'status', -11); test_zmat('zstd wrong input format', 'zstd', [1, 2, 3, 4], [], 'level', 0, 'status', -9); end zmat-0.9.9/.travis.yml0000644000175200007730000000362714515254731015060 0ustar rlaboissrlaboisslanguage: c compiler: - gcc jobs: include: - os: linux name: Ubuntu 14.04 dist: trusty env: - BADGE=Ubuntu_14.04 - MAKE=make - os: linux name: Ubuntu 16.04 dist: xenial env: - BADGE=Ubuntu_16.04 - MAKE=make - os: linux name: Ubuntu 18.04 dist: bionic env: - BADGE=Ubuntu_18.04 - MAKE=make - os: linux name: Ubuntu 20.04 dist: focal env: - BADGE=Ubuntu_20.04 - MAKE=make - os: linux name: Ubuntu 22.04 dist: jammy env: - BADGE=Ubuntu_22.04 - MAKE=make - os: windows name: Windows env: - BADGE=Windows - MAKE=mingw32-make - os: osx name: OSX osx_image: xcode13.4 env: - BADGE=OSX - MAKE=make before_install: - if [ "$TRAVIS_OS_NAME" = "linux" ]; then sudo apt-get install liboctave-dev; fi # - if [ "$TRAVIS_OS_NAME" = "windows" ]; then # choco install octave.portable --version=4.2.1; # export PATH=/c/ProgramData/chocolatey/lib/octave.portable/tools/octave/bin:.:${PATH}; # fi addons: apt: packages: - liboctave-dev update: true script: - if [ "$TRAVIS_OS_NAME" = "linux" ]; then octave-cli --eval "addpath(pwd);cd test;run_zmat_test"; fi - ${MAKE} -C src lib - ${MAKE} -C example/c all LIBTYPE= - if [ "$TRAVIS_OS_NAME" = "osx" ]; then DYLD_LIBRARY_PATH=lib example/c/testzmat; else LD_LIBRARY_PATH=lib example/c/testzmat; fi - ${MAKE} -C src dll - ${MAKE} -C example/c all LIBTYPE= - if [ "$TRAVIS_OS_NAME" = "osx" ]; then DYLD_LIBRARY_PATH=lib example/c/testzmat; else LD_LIBRARY_PATH=lib example/c/testzmat; fi - if [ "$TRAVIS_OS_NAME" = "linux" ]; then ${MAKE} -C src oct; octave-cli --eval "addpath(pwd);cd test;run_zmat_test"; fi zmat-0.9.9/zmat.prj0000755000175200007730000001154614515254731014441 0ustar rlaboissrlaboiss zmat Qianqian Fang fangqq@gmail.com Northeastern University A portable data compression/decompression toolbox for MATLAB/Octave ZMat is a portable mex function to enable zlib/gzip/lzma/lz4/zstd/blosc2 based data compression/decompression and base64 encoding/decoding support in MATLAB and GNU Octave. It is fast and compact, can process a large array within a fraction of a second. Among the 6 supported compression methods, lz4 is the fastest for compression/decompression; lzma is the slowest but has the highest compression ratio; zlib/gzip have the best balance between speed and compression time. ${PROJECT_ROOT}\images\zmat_logo.png 0.9.9 ${PROJECT_ROOT}\zmat.mltbx f0c22463-24da-449c-99b5-f6b4dca39e3c fortran90 include octave src .git* Makefile true false false true true true true ${PROJECT_ROOT} ${PROJECT_ROOT}\AUTHORS.txt ${PROJECT_ROOT}\ChangeLog.txt ${PROJECT_ROOT}\example ${PROJECT_ROOT}\LICENSE.txt ${PROJECT_ROOT}\private ${PROJECT_ROOT}\README.rst ${PROJECT_ROOT}\test ${PROJECT_ROOT}\zmat.m zmat.mltbx C:\Program Files\MATLAB\R2020a false false true false false false false false 10.0 false true win64 true zmat-0.9.9/LICENSE.txt0000644000175200007730000010451414515254731014567 0ustar rlaboissrlaboiss GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: Copyright (C) This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read . zmat-0.9.9/src/0000755000175200007730000000000014515254731013526 5ustar rlaboissrlaboisszmat-0.9.9/src/Makefile0000644000175200007730000001236214515254731015172 0ustar rlaboissrlaboiss################################################################# # Makefile for ZMAT # Qianqian Fang # 2019/04/30 ################################################################# BACKEND ?= ROOTDIR ?=.. ZMATDIR ?=$(ROOTDIR) LIBDIR ?=$(ROOTDIR)/lib MKDIR :=mkdir HAVE_ZLIB ?=miniz HAVE_LZMA ?=yes HAVE_LZ4 ?=yes HAVE_ZSTD ?=yes HAVE_BLOSC2?=yes LIBZLIB ?=-lz export HAVE_ZLIB HAVE_LZ4 HAVE_ZSTD MEX?=mex AR=$(CC) ECHO :=echo SYMBLINK :=ln -s BINARY:=zipmat OUTPUT_DIR=$(ZMATDIR) DOXY := doxygen DOCDIR := $(ZMATDIR)/doc DOXYCFG=zmat.cfg INCLUDEDIRS=-I../include MEXLINKLIBS=-L"\$$MATLABROOT/extern/lib/\$$ARCH" -L"\$$MATLABROOT/bin/\$$ARCH" -lmx -lmex ARCH = $(shell uname -m) PLATFORM = $(shell uname -s) DLLFLAG=-fPIC OMP=-fopenmp CPPOPT=-g -Wall -O3 -fPIC # -g -Wall -std=c99 # -DUSE_OS_TIMER OUTPUTFLAG:=-o OBJSUFFIX=.o EXESUFFIX=.mex* FILES=zmatlib ifeq ($(findstring _NT-,$(PLATFORM)), _NT-) CPPOPT =-c EXESUFFIX= DLLFLAG= MEXLINKLIBS="\$$LINKLIBS" else ifeq ($(findstring Darwin,$(PLATFORM)), Darwin) else CPPOPT+= CUCCOPT+=-Xcompiler $(OMP) ifeq ($(findstring x86_64,$(ARCH)), x86_64) CPPOPT += CUCCOPT +=-m64 endif endif ifneq ($(HAVE_ZLIB),yes) CFLAGS+=-DNO_ZLIB -D_LARGEFILE64_SOURCE=1 INCLUDEDIRS+=-Iminiz FILES+=miniz/miniz LIBZLIB= endif ifeq ($(HAVE_LZMA),no) CFLAGS+=-DNO_LZMA else INCLUDEDIRS+=-Ieasylzma -Ieasylzma/pavlov FILES+=easylzma/compress easylzma/decompress \ easylzma/lzma_header easylzma/lzip_header easylzma/common_internal \ easylzma/pavlov/LzmaEnc easylzma/pavlov/LzmaDec easylzma/pavlov/LzmaLib \ easylzma/pavlov/LzFind easylzma/pavlov/Bra easylzma/pavlov/BraIA64 \ easylzma/pavlov/Alloc easylzma/pavlov/7zCrc endif ifeq ($(HAVE_BLOSC2),no) CFLAGS+=-DNO_BLOSC2 else ifeq ($(HAVE_LZ4),no) INCLUDEDIRS+=-Ilz4 FILES+= lz4/lz4 lz4/lz4hc endif ifeq ($(HAVE_ZSTD),no) INCLUDEDIRS+=-Iblosc2/internal-complibs/zstd LIBZLIB+=-Lblosc2/internal-complibs/zstd -lzstd endif LIBZLIB+=-Lblosc2/lib -lblosc2 -pthread ifeq ($(HAVE_ZLIB),yes) LIBZLIB+=-lz endif INCLUDEDIRS+=-Iblosc2/include endif ifeq ($(HAVE_LZ4),no) CFLAGS+=-DNO_LZ4 else INCLUDEDIRS+=-Ilz4 FILES+= lz4/lz4 lz4/lz4hc endif ifeq ($(HAVE_ZSTD),no) CFLAGS+=-DNO_ZSTD else INCLUDEDIRS+=-Iblosc2/internal-complibs/zstd LIBZLIB+=-Lblosc2/internal-complibs/zstd -lzstd endif ifeq ($(MAKECMDGOALS),lib) AR :=ar ARFLAGS :=cr BINARY :=libzmat.a AROUTPUT := LINKOPT :=blosc2/blosc/*$(OBJSUFFIX) blosc2/internal-complibs/zstd/obj/*/static/*$(OBJSUFFIX) OUTPUT_DIR :=$(LIBDIR) ifeq ($(findstring Darwin,$(PLATFORM)), Darwin) OUTPUTFLAG := endif endif ifeq ($(MAKECMDGOALS),dll) AR :=$(CC) OUTPUTFLAG :=-o BINARY :=libzmat.so.1 OUTPUT_DIR :=$(LIBDIR) LINKOPT +=$(LIBZLIB) ifeq ($(findstring Darwin,$(PLATFORM)), Darwin) ARFLAGS :=-shared -Wl,-install_name,$(BINARY) $(LIBZLIB) else ARFLAGS :=-shared -Wl,-soname,$(BINARY) $(LIBZLIB) endif endif dll: CPPOPT +=$(DLLFLAG) dll: ARFLAGS ?=-shared -Wl,-soname,$(BINARY).1 dll: LINKOPT +=$(LDFLAGS) dll: AROUTPUT :=-o oct mex: CPPOPT+= $(DLLFLAG) oct: OUTPUT_DIR=.. oct: AR= CXXFLAGS='-O3' LDFLAGS='$(MEXLINKOPT)' mkoctfile zmat.cpp oct: BINARY=zmat.mex oct: ARFLAGS := oct: LINKOPT+=--mex $(INCLUDEDIRS) $(LIBZLIB) mex: CXX=$(MEX) mex: OUTPUTFLAG:=-output mex: AR=$(MEX) zmat.cpp $(INCLUDEDIRS) mex: LINKOPT+=-cxx LINKLIBS="$(MEXLINKLIBS) $(MEXLINKOPT) $(LIBZLIB)" CXXLIBS="\$$CXXLIBS $(MEXLINKOPT) $(LIBZLIB)" -outdir $(ZMATDIR) mex: ARFLAGS := mex: OUTPUT_DIR=.. all: mex TARGETSUFFIX:=$(suffix $(BINARY)) doc: makedocdir $(DOXY) $(DOXYCFG) OBJS := $(addsuffix $(OBJSUFFIX), $(FILES)) all dll lib mex oct: $(OUTPUT_DIR)/$(BINARY) dll: linkdll blosc: @if [ $(HAVE_BLOSC2) = "yes" ]; then\ $(MAKE) -C blosc2 all;\ fi makedirs: @if test ! -d $(OUTPUT_DIR); then $(MKDIR) $(OUTPUT_DIR); fi linkdll: makedirs -$(SYMBLINK) $(BINARY) $(basename $(OUTPUT_DIR)/$(BINARY)) makedocdir: @if test ! -d $(DOCDIR); then $(MKDIR) $(DOCDIR); fi $(OUTPUT_DIR)/$(BINARY): makedirs $(OBJS) blosc $(OUTPUT_DIR)/$(BINARY): $(OBJS) @$(ECHO) Building $@ $(AR) $(ARFLAGS) $(OUTPUTFLAG) $@ $(OBJS) $(LINKOPT) $(USERLINKOPT) %$(OBJSUFFIX): %.cpp @$(ECHO) Building $@ $(CXX) $(INCLUDEDIRS) $(CPPOPT) -c -o $@ $< %$(OBJSUFFIX): %.c @$(ECHO) Building $@ $(CC) $(CPPFLAGS) $(CFLAGS) $(INCLUDEDIRS) $(CPPOPT) -c -o $@ $< %$(OBJSUFFIX): %.cu @$(ECHO) Building $@ $(CUDACC) -c $(CUCCOPT) -o $@ $< clean: -rm -f $(OBJS) $(OUTPUT_DIR)/$(BINARY)$(EXESUFFIX) zmat$(OBJSUFFIX) $(LIBDIR)/* -$(MAKE) -C blosc2 clean pretty: astyle \ --style=attach \ --indent=spaces=4 \ --indent-modifiers \ --indent-switches \ --indent-preproc-block \ --indent-preproc-define \ --indent-col1-comments \ --pad-oper \ --pad-header \ --align-pointer=type \ --align-reference=type \ --add-brackets \ --convert-tabs \ --lineend=linux \ --preserve-date \ --suffix=none \ --formatted \ --break-blocks \ "*.c" "../include/*.h" "*.cpp" "../example/c/*.c" .PHONY: all mex oct lib dll .DEFAULT_GOAL := all zmat-0.9.9/src/mexopts_msys2_gcc.xml0000644000175200007730000001067414515254731017730 0ustar rlaboissrlaboiss
zmat-0.9.9/src/compilezmat.m0000644000175200007730000000626114515254731016235 0ustar rlaboissrlaboiss%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Compilation script for zmat in MATLAB and GNU Octave % % author: Qianqian Fang % % Dependency (Windows only): % 1.If you have MATLAB R2017b or later, you may skip this step. % To compile mcxlabcl in MATLAB R2017a or earlier on Windows, you must % pre-install the MATLAB support for MinGW-w64 compiler % https://www.mathworks.com/matlabcentral/fileexchange/52848-matlab-support-for-mingw-w64-c-c-compiler % % Note: it appears that installing the above Add On is no longer working % and may give an error at the download stage. In this case, you should % install MSYS2 from https://www.msys2.org/. Once you install MSYS2, % run MSYS2.0 MinGW 64bit from Start menu, in the popup terminal window, % type % % pacman -Syu % pacman -S base-devel gcc git mingw-w64-x86_64-opencl-headers % % Then, start MATLAB, and in the command window, run % % setenv('MW_MINGW64_LOC','C:\msys64\usr'); % 2.After installation of MATLAB MinGW support, you must type % "mex -setup C" in MATLAB and select "MinGW64 Compiler (C)". % 3.Once you select the MingW C compiler, you should run "mex -setup C++" % again in MATLAB and select "MinGW64 Compiler (C++)" to compile C++. % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% filelist = {'miniz/miniz.c', 'lz4/lz4.c', 'lz4/lz4hc.c', 'easylzma/compress.c', 'easylzma/decompress.c', ... 'easylzma/lzma_header.c', 'easylzma/lzip_header.c', 'easylzma/common_internal.c', ... 'easylzma/pavlov/LzmaEnc.c', 'easylzma/pavlov/LzmaDec.c', 'easylzma/pavlov/LzmaLib.c' ... 'easylzma/pavlov/LzFind.c', 'easylzma/pavlov/Bra.c', 'easylzma/pavlov/BraIA64.c' ... 'easylzma/pavlov/Alloc.c', 'easylzma/pavlov/7zCrc.c', 'zmatlib.c'}; mexfile = 'zmat.cpp'; suffix = '.o'; includdir = '-I../include -Ieasylzma -Ieasylzma/pavlov -Ilz4 -Iminiz'; if (ispc) suffix = '.obj'; end if (~exist('OCTAVE_VERSION', 'builtin')) delete(['*', suffix]); CCFLAG = ['CFLAGS=''-O3 -g -DNO_BLOSC2 -DNO_ZSTD -DNO_ZLIB -D_LARGEFILE64_SOURCE=1 ' includdir ' -fPIC'' -c']; LINKFLAG = 'CXXLIBS=''$CXXLIBS'' -output ../zipmat -outdir ../'; for i = 1:length(filelist) fprintf(1, 'mex %s %s\n', CCFLAG, filelist{i}); eval(sprintf('mex %s %s', CCFLAG, filelist{i})); end filelist = dir(['*' suffix]); filelist = {filelist.name}; cmd = sprintf('mex %s %s %s %s', mexfile, LINKFLAG, includdir, sprintf('%s ', filelist{:})); fprintf(1, '%s\n', cmd); eval(cmd); else delete('*.o'); CCFLAG = ['-O3 -g -DNO_BLOSC2 -DNO_ZSTD -DNO_ZLIB -D_LARGEFILE64_SOURCE=1 -c ' includdir]; LINKFLAG = '-o ../zipmat'; for i = 1:length(filelist) fprintf(stdout, 'mex %s %s\n', CCFLAG, filelist{i}); fflush(stdout); eval(sprintf('mex %s %s', CCFLAG, filelist{i})); end if (ispc) filelist = dir(['*.o']); filelist = {filelist.name}; end cmd = sprintf('mex %s -I../include -Ieasylzma %s %s', mexfile, LINKFLAG, regexprep(sprintf('%s ', filelist{:}), '\.c[p]*', '\.o')); fprintf(stdout, '%s\n', cmd); fflush(stdout); eval(cmd); end zmat-0.9.9/src/lz4/0000755000175200007730000000000014515254731014237 5ustar rlaboissrlaboisszmat-0.9.9/src/lz4/lz4.c0000644000175200007730000033535614515254731015133 0ustar rlaboissrlaboiss/* LZ4 - Fast LZ compression algorithm Copyright (C) 2011-2020, Yann Collet. BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You can contact the author at : - LZ4 homepage : http://www.lz4.org - LZ4 source repository : https://github.com/lz4/lz4 */ /*-************************************ * Tuning parameters **************************************/ /* * LZ4_HEAPMODE : * Select how default compression functions will allocate memory for their hash table, * in memory stack (0:default, fastest), or in memory heap (1:requires malloc()). */ #ifndef LZ4_HEAPMODE # define LZ4_HEAPMODE 0 #endif /* * LZ4_ACCELERATION_DEFAULT : * Select "acceleration" for LZ4_compress_fast() when parameter value <= 0 */ #define LZ4_ACCELERATION_DEFAULT 1 /* * LZ4_ACCELERATION_MAX : * Any "acceleration" value higher than this threshold * get treated as LZ4_ACCELERATION_MAX instead (fix #876) */ #define LZ4_ACCELERATION_MAX 65537 /*-************************************ * CPU Feature Detection **************************************/ /* LZ4_FORCE_MEMORY_ACCESS * By default, access to unaligned memory is controlled by `memcpy()`, which is safe and portable. * Unfortunately, on some target/compiler combinations, the generated assembly is sub-optimal. * The below switch allow to select different access method for improved performance. * Method 0 (default) : use `memcpy()`. Safe and portable. * Method 1 : `__packed` statement. It depends on compiler extension (ie, not portable). * This method is safe if your compiler supports it, and *generally* as fast or faster than `memcpy`. * Method 2 : direct access. This method is portable but violate C standard. * It can generate buggy code on targets which assembly generation depends on alignment. * But in some circumstances, it's the only known way to get the most performance (ie GCC + ARMv6) * See https://fastcompression.blogspot.fr/2015/08/accessing-unaligned-memory.html for details. * Prefer these methods in priority order (0 > 1 > 2) */ #ifndef LZ4_FORCE_MEMORY_ACCESS /* can be defined externally */ # if defined(__GNUC__) && \ ( defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) || defined(__ARM_ARCH_6K__) \ || defined(__ARM_ARCH_6Z__) || defined(__ARM_ARCH_6ZK__) || defined(__ARM_ARCH_6T2__) ) # define LZ4_FORCE_MEMORY_ACCESS 2 # elif (defined(__INTEL_COMPILER) && !defined(_WIN32)) || defined(__GNUC__) # define LZ4_FORCE_MEMORY_ACCESS 1 # endif #endif /* * LZ4_FORCE_SW_BITCOUNT * Define this parameter if your target system or compiler does not support hardware bit count */ #if defined(_MSC_VER) && defined(_WIN32_WCE) /* Visual Studio for WinCE doesn't support Hardware bit count */ # undef LZ4_FORCE_SW_BITCOUNT /* avoid double def */ # define LZ4_FORCE_SW_BITCOUNT #endif /*-************************************ * Dependency **************************************/ /* * LZ4_SRC_INCLUDED: * Amalgamation flag, whether lz4.c is included */ #ifndef LZ4_SRC_INCLUDED # define LZ4_SRC_INCLUDED 1 #endif #ifndef LZ4_STATIC_LINKING_ONLY #define LZ4_STATIC_LINKING_ONLY #endif #ifndef LZ4_DISABLE_DEPRECATE_WARNINGS #define LZ4_DISABLE_DEPRECATE_WARNINGS /* due to LZ4_decompress_safe_withPrefix64k */ #endif #define LZ4_STATIC_LINKING_ONLY /* LZ4_DISTANCE_MAX */ #include "lz4.h" /* see also "memory routines" below */ /*-************************************ * Compiler Options **************************************/ #if defined(_MSC_VER) && (_MSC_VER >= 1400) /* Visual Studio 2005+ */ # include /* only present in VS2005+ */ # pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ # pragma warning(disable : 6237) /* disable: C6237: conditional expression is always 0 */ #endif /* _MSC_VER */ #ifndef LZ4_FORCE_INLINE # ifdef _MSC_VER /* Visual Studio */ # define LZ4_FORCE_INLINE static __forceinline # else # if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */ # ifdef __GNUC__ # define LZ4_FORCE_INLINE static inline __attribute__((always_inline)) # else # define LZ4_FORCE_INLINE static inline # endif # else # define LZ4_FORCE_INLINE static # endif /* __STDC_VERSION__ */ # endif /* _MSC_VER */ #endif /* LZ4_FORCE_INLINE */ /* LZ4_FORCE_O2 and LZ4_FORCE_INLINE * gcc on ppc64le generates an unrolled SIMDized loop for LZ4_wildCopy8, * together with a simple 8-byte copy loop as a fall-back path. * However, this optimization hurts the decompression speed by >30%, * because the execution does not go to the optimized loop * for typical compressible data, and all of the preamble checks * before going to the fall-back path become useless overhead. * This optimization happens only with the -O3 flag, and -O2 generates * a simple 8-byte copy loop. * With gcc on ppc64le, all of the LZ4_decompress_* and LZ4_wildCopy8 * functions are annotated with __attribute__((optimize("O2"))), * and also LZ4_wildCopy8 is forcibly inlined, so that the O2 attribute * of LZ4_wildCopy8 does not affect the compression speed. */ #if defined(__PPC64__) && defined(__LITTLE_ENDIAN__) && defined(__GNUC__) && !defined(__clang__) # define LZ4_FORCE_O2 __attribute__((optimize("O2"))) # undef LZ4_FORCE_INLINE # define LZ4_FORCE_INLINE static __inline __attribute__((optimize("O2"),always_inline)) #else # define LZ4_FORCE_O2 #endif #if (defined(__GNUC__) && (__GNUC__ >= 3)) || (defined(__INTEL_COMPILER) && (__INTEL_COMPILER >= 800)) || defined(__clang__) # define expect(expr,value) (__builtin_expect ((expr),(value)) ) #else # define expect(expr,value) (expr) #endif #ifndef likely #define likely(expr) expect((expr) != 0, 1) #endif #ifndef unlikely #define unlikely(expr) expect((expr) != 0, 0) #endif /* Should the alignment test prove unreliable, for some reason, * it can be disabled by setting LZ4_ALIGN_TEST to 0 */ #ifndef LZ4_ALIGN_TEST /* can be externally provided */ # define LZ4_ALIGN_TEST 1 #endif /*-************************************ * Memory routines **************************************/ /*! LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION : * Disable relatively high-level LZ4/HC functions that use dynamic memory * allocation functions (malloc(), calloc(), free()). * * Note that this is a compile-time switch. And since it disables * public/stable LZ4 v1 API functions, we don't recommend using this * symbol to generate a library for distribution. * * The following public functions are removed when this symbol is defined. * - lz4 : LZ4_createStream, LZ4_freeStream, * LZ4_createStreamDecode, LZ4_freeStreamDecode, LZ4_create (deprecated) * - lz4hc : LZ4_createStreamHC, LZ4_freeStreamHC, * LZ4_createHC (deprecated), LZ4_freeHC (deprecated) * - lz4frame, lz4file : All LZ4F_* functions */ #if defined(LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION) # define ALLOC(s) lz4_error_memory_allocation_is_disabled # define ALLOC_AND_ZERO(s) lz4_error_memory_allocation_is_disabled # define FREEMEM(p) lz4_error_memory_allocation_is_disabled #elif defined(LZ4_USER_MEMORY_FUNCTIONS) /* memory management functions can be customized by user project. * Below functions must exist somewhere in the Project * and be available at link time */ void* LZ4_malloc(size_t s); void* LZ4_calloc(size_t n, size_t s); void LZ4_free(void* p); # define ALLOC(s) LZ4_malloc(s) # define ALLOC_AND_ZERO(s) LZ4_calloc(1,s) # define FREEMEM(p) LZ4_free(p) #else # include /* malloc, calloc, free */ # define ALLOC(s) malloc(s) # define ALLOC_AND_ZERO(s) calloc(1,s) # define FREEMEM(p) free(p) #endif #if ! LZ4_FREESTANDING # include /* memset, memcpy */ #endif #if !defined(LZ4_memset) # define LZ4_memset(p,v,s) memset((p),(v),(s)) #endif #define MEM_INIT(p,v,s) LZ4_memset((p),(v),(s)) /*-************************************ * Common Constants **************************************/ #define MINMATCH 4 #define WILDCOPYLENGTH 8 #define LASTLITERALS 5 /* see ../doc/lz4_Block_format.md#parsing-restrictions */ #define MFLIMIT 12 /* see ../doc/lz4_Block_format.md#parsing-restrictions */ #define MATCH_SAFEGUARD_DISTANCE ((2*WILDCOPYLENGTH) - MINMATCH) /* ensure it's possible to write 2 x wildcopyLength without overflowing output buffer */ #define FASTLOOP_SAFE_DISTANCE 64 static const int LZ4_minLength = (MFLIMIT+1); #define KB *(1 <<10) #define MB *(1 <<20) #define GB *(1U<<30) #define LZ4_DISTANCE_ABSOLUTE_MAX 65535 #if (LZ4_DISTANCE_MAX > LZ4_DISTANCE_ABSOLUTE_MAX) /* max supported by LZ4 format */ # error "LZ4_DISTANCE_MAX is too big : must be <= 65535" #endif #define ML_BITS 4 #define ML_MASK ((1U<=1) # include #else # ifndef assert # define assert(condition) ((void)0) # endif #endif #define LZ4_STATIC_ASSERT(c) { enum { LZ4_static_assert = 1/(int)(!!(c)) }; } /* use after variable declarations */ #if defined(LZ4_DEBUG) && (LZ4_DEBUG>=2) # include static int g_debuglog_enable = 1; # define DEBUGLOG(l, ...) { \ if ((g_debuglog_enable) && (l<=LZ4_DEBUG)) { \ fprintf(stderr, __FILE__ ": "); \ fprintf(stderr, __VA_ARGS__); \ fprintf(stderr, " \n"); \ } } #else # define DEBUGLOG(l, ...) {} /* disabled */ #endif static int LZ4_isAligned(const void* ptr, size_t alignment) { return ((size_t)ptr & (alignment -1)) == 0; } /*-************************************ * Types **************************************/ #include #if defined(__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) # include typedef uint8_t BYTE; typedef uint16_t U16; typedef uint32_t U32; typedef int32_t S32; typedef uint64_t U64; typedef uintptr_t uptrval; #else # if UINT_MAX != 4294967295UL # error "LZ4 code (when not C++ or C99) assumes that sizeof(int) == 4" # endif typedef unsigned char BYTE; typedef unsigned short U16; typedef unsigned int U32; typedef signed int S32; typedef unsigned long long U64; typedef size_t uptrval; /* generally true, except OpenVMS-64 */ #endif #if defined(__x86_64__) typedef U64 reg_t; /* 64-bits in x32 mode */ #else typedef size_t reg_t; /* 32-bits in x32 mode */ #endif typedef enum { notLimited = 0, limitedOutput = 1, fillOutput = 2 } limitedOutput_directive; /*-************************************ * Reading and writing into memory **************************************/ /** * LZ4 relies on memcpy with a constant size being inlined. In freestanding * environments, the compiler can't assume the implementation of memcpy() is * standard compliant, so it can't apply its specialized memcpy() inlining * logic. When possible, use __builtin_memcpy() to tell the compiler to analyze * memcpy() as if it were standard compliant, so it can inline it in freestanding * environments. This is needed when decompressing the Linux Kernel, for example. */ #if !defined(LZ4_memcpy) # if defined(__GNUC__) && (__GNUC__ >= 4) # define LZ4_memcpy(dst, src, size) __builtin_memcpy(dst, src, size) # else # define LZ4_memcpy(dst, src, size) memcpy(dst, src, size) # endif #endif #if !defined(LZ4_memmove) # if defined(__GNUC__) && (__GNUC__ >= 4) # define LZ4_memmove __builtin_memmove # else # define LZ4_memmove memmove # endif #endif static unsigned LZ4_isLittleEndian(void) { const union { U32 u; BYTE c[4]; } one = { 1 }; /* don't use static : performance detrimental */ return one.c[0]; } #if defined(LZ4_FORCE_MEMORY_ACCESS) && (LZ4_FORCE_MEMORY_ACCESS==2) /* lie to the compiler about data alignment; use with caution */ static U16 LZ4_read16(const void* memPtr) { return *(const U16*) memPtr; } static U32 LZ4_read32(const void* memPtr) { return *(const U32*) memPtr; } static reg_t LZ4_read_ARCH(const void* memPtr) { return *(const reg_t*) memPtr; } static void LZ4_write16(void* memPtr, U16 value) { *(U16*)memPtr = value; } static void LZ4_write32(void* memPtr, U32 value) { *(U32*)memPtr = value; } #elif defined(LZ4_FORCE_MEMORY_ACCESS) && (LZ4_FORCE_MEMORY_ACCESS==1) /* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */ /* currently only defined for gcc and icc */ typedef union { U16 u16; U32 u32; reg_t uArch; } __attribute__((packed)) LZ4_unalign; static U16 LZ4_read16(const void* ptr) { return ((const LZ4_unalign*)ptr)->u16; } static U32 LZ4_read32(const void* ptr) { return ((const LZ4_unalign*)ptr)->u32; } static reg_t LZ4_read_ARCH(const void* ptr) { return ((const LZ4_unalign*)ptr)->uArch; } static void LZ4_write16(void* memPtr, U16 value) { ((LZ4_unalign*)memPtr)->u16 = value; } static void LZ4_write32(void* memPtr, U32 value) { ((LZ4_unalign*)memPtr)->u32 = value; } #else /* safe and portable access using memcpy() */ static U16 LZ4_read16(const void* memPtr) { U16 val; LZ4_memcpy(&val, memPtr, sizeof(val)); return val; } static U32 LZ4_read32(const void* memPtr) { U32 val; LZ4_memcpy(&val, memPtr, sizeof(val)); return val; } static reg_t LZ4_read_ARCH(const void* memPtr) { reg_t val; LZ4_memcpy(&val, memPtr, sizeof(val)); return val; } static void LZ4_write16(void* memPtr, U16 value) { LZ4_memcpy(memPtr, &value, sizeof(value)); } static void LZ4_write32(void* memPtr, U32 value) { LZ4_memcpy(memPtr, &value, sizeof(value)); } #endif /* LZ4_FORCE_MEMORY_ACCESS */ static U16 LZ4_readLE16(const void* memPtr) { if (LZ4_isLittleEndian()) { return LZ4_read16(memPtr); } else { const BYTE* p = (const BYTE*)memPtr; return (U16)((U16)p[0] + (p[1]<<8)); } } static void LZ4_writeLE16(void* memPtr, U16 value) { if (LZ4_isLittleEndian()) { LZ4_write16(memPtr, value); } else { BYTE* p = (BYTE*)memPtr; p[0] = (BYTE) value; p[1] = (BYTE)(value>>8); } } /* customized variant of memcpy, which can overwrite up to 8 bytes beyond dstEnd */ LZ4_FORCE_INLINE void LZ4_wildCopy8(void* dstPtr, const void* srcPtr, void* dstEnd) { BYTE* d = (BYTE*)dstPtr; const BYTE* s = (const BYTE*)srcPtr; BYTE* const e = (BYTE*)dstEnd; do { LZ4_memcpy(d,s,8); d+=8; s+=8; } while (d= 16. */ LZ4_FORCE_INLINE void LZ4_wildCopy32(void* dstPtr, const void* srcPtr, void* dstEnd) { BYTE* d = (BYTE*)dstPtr; const BYTE* s = (const BYTE*)srcPtr; BYTE* const e = (BYTE*)dstEnd; do { LZ4_memcpy(d,s,16); LZ4_memcpy(d+16,s+16,16); d+=32; s+=32; } while (d= dstPtr + MINMATCH * - there is at least 8 bytes available to write after dstEnd */ LZ4_FORCE_INLINE void LZ4_memcpy_using_offset(BYTE* dstPtr, const BYTE* srcPtr, BYTE* dstEnd, const size_t offset) { BYTE v[8]; assert(dstEnd >= dstPtr + MINMATCH); switch(offset) { case 1: MEM_INIT(v, *srcPtr, 8); break; case 2: LZ4_memcpy(v, srcPtr, 2); LZ4_memcpy(&v[2], srcPtr, 2); #if defined(_MSC_VER) && (_MSC_VER <= 1933) /* MSVC 2022 ver 17.3 or earlier */ # pragma warning(push) # pragma warning(disable : 6385) /* warning C6385: Reading invalid data from 'v'. */ #endif LZ4_memcpy(&v[4], v, 4); #if defined(_MSC_VER) && (_MSC_VER <= 1933) /* MSVC 2022 ver 17.3 or earlier */ # pragma warning(pop) #endif break; case 4: LZ4_memcpy(v, srcPtr, 4); LZ4_memcpy(&v[4], srcPtr, 4); break; default: LZ4_memcpy_using_offset_base(dstPtr, srcPtr, dstEnd, offset); return; } LZ4_memcpy(dstPtr, v, 8); dstPtr += 8; while (dstPtr < dstEnd) { LZ4_memcpy(dstPtr, v, 8); dstPtr += 8; } } #endif /*-************************************ * Common functions **************************************/ static unsigned LZ4_NbCommonBytes (reg_t val) { assert(val != 0); if (LZ4_isLittleEndian()) { if (sizeof(val) == 8) { # if defined(_MSC_VER) && (_MSC_VER >= 1800) && (defined(_M_AMD64) && !defined(_M_ARM64EC)) && !defined(LZ4_FORCE_SW_BITCOUNT) /*-************************************************************************************************* * ARM64EC is a Microsoft-designed ARM64 ABI compatible with AMD64 applications on ARM64 Windows 11. * The ARM64EC ABI does not support AVX/AVX2/AVX512 instructions, nor their relevant intrinsics * including _tzcnt_u64. Therefore, we need to neuter the _tzcnt_u64 code path for ARM64EC. ****************************************************************************************************/ # if defined(__clang__) && (__clang_major__ < 10) /* Avoid undefined clang-cl intrinsics issue. * See https://github.com/lz4/lz4/pull/1017 for details. */ return (unsigned)__builtin_ia32_tzcnt_u64(val) >> 3; # else /* x64 CPUS without BMI support interpret `TZCNT` as `REP BSF` */ return (unsigned)_tzcnt_u64(val) >> 3; # endif # elif defined(_MSC_VER) && defined(_WIN64) && !defined(LZ4_FORCE_SW_BITCOUNT) unsigned long r = 0; _BitScanForward64(&r, (U64)val); return (unsigned)r >> 3; # elif (defined(__clang__) || (defined(__GNUC__) && ((__GNUC__ > 3) || \ ((__GNUC__ == 3) && (__GNUC_MINOR__ >= 4))))) && \ !defined(LZ4_FORCE_SW_BITCOUNT) return (unsigned)__builtin_ctzll((U64)val) >> 3; # else const U64 m = 0x0101010101010101ULL; val ^= val - 1; return (unsigned)(((U64)((val & (m - 1)) * m)) >> 56); # endif } else /* 32 bits */ { # if defined(_MSC_VER) && (_MSC_VER >= 1400) && !defined(LZ4_FORCE_SW_BITCOUNT) unsigned long r; _BitScanForward(&r, (U32)val); return (unsigned)r >> 3; # elif (defined(__clang__) || (defined(__GNUC__) && ((__GNUC__ > 3) || \ ((__GNUC__ == 3) && (__GNUC_MINOR__ >= 4))))) && \ !defined(__TINYC__) && !defined(LZ4_FORCE_SW_BITCOUNT) return (unsigned)__builtin_ctz((U32)val) >> 3; # else const U32 m = 0x01010101; return (unsigned)((((val - 1) ^ val) & (m - 1)) * m) >> 24; # endif } } else /* Big Endian CPU */ { if (sizeof(val)==8) { # if (defined(__clang__) || (defined(__GNUC__) && ((__GNUC__ > 3) || \ ((__GNUC__ == 3) && (__GNUC_MINOR__ >= 4))))) && \ !defined(__TINYC__) && !defined(LZ4_FORCE_SW_BITCOUNT) return (unsigned)__builtin_clzll((U64)val) >> 3; # else #if 1 /* this method is probably faster, * but adds a 128 bytes lookup table */ static const unsigned char ctz7_tab[128] = { 7, 0, 1, 0, 2, 0, 1, 0, 3, 0, 1, 0, 2, 0, 1, 0, 4, 0, 1, 0, 2, 0, 1, 0, 3, 0, 1, 0, 2, 0, 1, 0, 5, 0, 1, 0, 2, 0, 1, 0, 3, 0, 1, 0, 2, 0, 1, 0, 4, 0, 1, 0, 2, 0, 1, 0, 3, 0, 1, 0, 2, 0, 1, 0, 6, 0, 1, 0, 2, 0, 1, 0, 3, 0, 1, 0, 2, 0, 1, 0, 4, 0, 1, 0, 2, 0, 1, 0, 3, 0, 1, 0, 2, 0, 1, 0, 5, 0, 1, 0, 2, 0, 1, 0, 3, 0, 1, 0, 2, 0, 1, 0, 4, 0, 1, 0, 2, 0, 1, 0, 3, 0, 1, 0, 2, 0, 1, 0, }; U64 const mask = 0x0101010101010101ULL; U64 const t = (((val >> 8) - mask) | val) & mask; return ctz7_tab[(t * 0x0080402010080402ULL) >> 57]; #else /* this method doesn't consume memory space like the previous one, * but it contains several branches, * that may end up slowing execution */ static const U32 by32 = sizeof(val)*4; /* 32 on 64 bits (goal), 16 on 32 bits. Just to avoid some static analyzer complaining about shift by 32 on 32-bits target. Note that this code path is never triggered in 32-bits mode. */ unsigned r; if (!(val>>by32)) { r=4; } else { r=0; val>>=by32; } if (!(val>>16)) { r+=2; val>>=8; } else { val>>=24; } r += (!val); return r; #endif # endif } else /* 32 bits */ { # if (defined(__clang__) || (defined(__GNUC__) && ((__GNUC__ > 3) || \ ((__GNUC__ == 3) && (__GNUC_MINOR__ >= 4))))) && \ !defined(LZ4_FORCE_SW_BITCOUNT) return (unsigned)__builtin_clz((U32)val) >> 3; # else val >>= 8; val = ((((val + 0x00FFFF00) | 0x00FFFFFF) + val) | (val + 0x00FF0000)) >> 24; return (unsigned)val ^ 3; # endif } } } #define STEPSIZE sizeof(reg_t) LZ4_FORCE_INLINE unsigned LZ4_count(const BYTE* pIn, const BYTE* pMatch, const BYTE* pInLimit) { const BYTE* const pStart = pIn; if (likely(pIn < pInLimit-(STEPSIZE-1))) { reg_t const diff = LZ4_read_ARCH(pMatch) ^ LZ4_read_ARCH(pIn); if (!diff) { pIn+=STEPSIZE; pMatch+=STEPSIZE; } else { return LZ4_NbCommonBytes(diff); } } while (likely(pIn < pInLimit-(STEPSIZE-1))) { reg_t const diff = LZ4_read_ARCH(pMatch) ^ LZ4_read_ARCH(pIn); if (!diff) { pIn+=STEPSIZE; pMatch+=STEPSIZE; continue; } pIn += LZ4_NbCommonBytes(diff); return (unsigned)(pIn - pStart); } if ((STEPSIZE==8) && (pIn<(pInLimit-3)) && (LZ4_read32(pMatch) == LZ4_read32(pIn))) { pIn+=4; pMatch+=4; } if ((pIn<(pInLimit-1)) && (LZ4_read16(pMatch) == LZ4_read16(pIn))) { pIn+=2; pMatch+=2; } if ((pIn compression run slower on incompressible data */ /*-************************************ * Local Structures and types **************************************/ typedef enum { clearedTable = 0, byPtr, byU32, byU16 } tableType_t; /** * This enum distinguishes several different modes of accessing previous * content in the stream. * * - noDict : There is no preceding content. * - withPrefix64k : Table entries up to ctx->dictSize before the current blob * blob being compressed are valid and refer to the preceding * content (of length ctx->dictSize), which is available * contiguously preceding in memory the content currently * being compressed. * - usingExtDict : Like withPrefix64k, but the preceding content is somewhere * else in memory, starting at ctx->dictionary with length * ctx->dictSize. * - usingDictCtx : Everything concerning the preceding content is * in a separate context, pointed to by ctx->dictCtx. * ctx->dictionary, ctx->dictSize, and table entries * in the current context that refer to positions * preceding the beginning of the current compression are * ignored. Instead, ctx->dictCtx->dictionary and ctx->dictCtx * ->dictSize describe the location and size of the preceding * content, and matches are found by looking in the ctx * ->dictCtx->hashTable. */ typedef enum { noDict = 0, withPrefix64k, usingExtDict, usingDictCtx } dict_directive; typedef enum { noDictIssue = 0, dictSmall } dictIssue_directive; /*-************************************ * Local Utils **************************************/ int LZ4_versionNumber (void) { return LZ4_VERSION_NUMBER; } const char* LZ4_versionString(void) { return LZ4_VERSION_STRING; } int LZ4_compressBound(int isize) { return LZ4_COMPRESSBOUND(isize); } int LZ4_sizeofState(void) { return sizeof(LZ4_stream_t); } /*-**************************************** * Internal Definitions, used only in Tests *******************************************/ #if defined (__cplusplus) extern "C" { #endif int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_dict, const char* source, char* dest, int srcSize); int LZ4_decompress_safe_forceExtDict(const char* source, char* dest, int compressedSize, int maxOutputSize, const void* dictStart, size_t dictSize); int LZ4_decompress_safe_partial_forceExtDict(const char* source, char* dest, int compressedSize, int targetOutputSize, int dstCapacity, const void* dictStart, size_t dictSize); #if defined (__cplusplus) } #endif /*-****************************** * Compression functions ********************************/ LZ4_FORCE_INLINE U32 LZ4_hash4(U32 sequence, tableType_t const tableType) { if (tableType == byU16) return ((sequence * 2654435761U) >> ((MINMATCH*8)-(LZ4_HASHLOG+1))); else return ((sequence * 2654435761U) >> ((MINMATCH*8)-LZ4_HASHLOG)); } LZ4_FORCE_INLINE U32 LZ4_hash5(U64 sequence, tableType_t const tableType) { const U32 hashLog = (tableType == byU16) ? LZ4_HASHLOG+1 : LZ4_HASHLOG; if (LZ4_isLittleEndian()) { const U64 prime5bytes = 889523592379ULL; return (U32)(((sequence << 24) * prime5bytes) >> (64 - hashLog)); } else { const U64 prime8bytes = 11400714785074694791ULL; return (U32)(((sequence >> 24) * prime8bytes) >> (64 - hashLog)); } } LZ4_FORCE_INLINE U32 LZ4_hashPosition(const void* const p, tableType_t const tableType) { if ((sizeof(reg_t)==8) && (tableType != byU16)) return LZ4_hash5(LZ4_read_ARCH(p), tableType); return LZ4_hash4(LZ4_read32(p), tableType); } LZ4_FORCE_INLINE void LZ4_clearHash(U32 h, void* tableBase, tableType_t const tableType) { switch (tableType) { default: /* fallthrough */ case clearedTable: { /* illegal! */ assert(0); return; } case byPtr: { const BYTE** hashTable = (const BYTE**)tableBase; hashTable[h] = NULL; return; } case byU32: { U32* hashTable = (U32*) tableBase; hashTable[h] = 0; return; } case byU16: { U16* hashTable = (U16*) tableBase; hashTable[h] = 0; return; } } } LZ4_FORCE_INLINE void LZ4_putIndexOnHash(U32 idx, U32 h, void* tableBase, tableType_t const tableType) { switch (tableType) { default: /* fallthrough */ case clearedTable: /* fallthrough */ case byPtr: { /* illegal! */ assert(0); return; } case byU32: { U32* hashTable = (U32*) tableBase; hashTable[h] = idx; return; } case byU16: { U16* hashTable = (U16*) tableBase; assert(idx < 65536); hashTable[h] = (U16)idx; return; } } } LZ4_FORCE_INLINE void LZ4_putPositionOnHash(const BYTE* p, U32 h, void* tableBase, tableType_t const tableType, const BYTE* srcBase) { switch (tableType) { case clearedTable: { /* illegal! */ assert(0); return; } case byPtr: { const BYTE** hashTable = (const BYTE**)tableBase; hashTable[h] = p; return; } case byU32: { U32* hashTable = (U32*) tableBase; hashTable[h] = (U32)(p-srcBase); return; } case byU16: { U16* hashTable = (U16*) tableBase; hashTable[h] = (U16)(p-srcBase); return; } } } LZ4_FORCE_INLINE void LZ4_putPosition(const BYTE* p, void* tableBase, tableType_t tableType, const BYTE* srcBase) { U32 const h = LZ4_hashPosition(p, tableType); LZ4_putPositionOnHash(p, h, tableBase, tableType, srcBase); } /* LZ4_getIndexOnHash() : * Index of match position registered in hash table. * hash position must be calculated by using base+index, or dictBase+index. * Assumption 1 : only valid if tableType == byU32 or byU16. * Assumption 2 : h is presumed valid (within limits of hash table) */ LZ4_FORCE_INLINE U32 LZ4_getIndexOnHash(U32 h, const void* tableBase, tableType_t tableType) { LZ4_STATIC_ASSERT(LZ4_MEMORY_USAGE > 2); if (tableType == byU32) { const U32* const hashTable = (const U32*) tableBase; assert(h < (1U << (LZ4_MEMORY_USAGE-2))); return hashTable[h]; } if (tableType == byU16) { const U16* const hashTable = (const U16*) tableBase; assert(h < (1U << (LZ4_MEMORY_USAGE-1))); return hashTable[h]; } assert(0); return 0; /* forbidden case */ } static const BYTE* LZ4_getPositionOnHash(U32 h, const void* tableBase, tableType_t tableType, const BYTE* srcBase) { if (tableType == byPtr) { const BYTE* const* hashTable = (const BYTE* const*) tableBase; return hashTable[h]; } if (tableType == byU32) { const U32* const hashTable = (const U32*) tableBase; return hashTable[h] + srcBase; } { const U16* const hashTable = (const U16*) tableBase; return hashTable[h] + srcBase; } /* default, to ensure a return */ } LZ4_FORCE_INLINE const BYTE* LZ4_getPosition(const BYTE* p, const void* tableBase, tableType_t tableType, const BYTE* srcBase) { U32 const h = LZ4_hashPosition(p, tableType); return LZ4_getPositionOnHash(h, tableBase, tableType, srcBase); } LZ4_FORCE_INLINE void LZ4_prepareTable(LZ4_stream_t_internal* const cctx, const int inputSize, const tableType_t tableType) { /* If the table hasn't been used, it's guaranteed to be zeroed out, and is * therefore safe to use no matter what mode we're in. Otherwise, we figure * out if it's safe to leave as is or whether it needs to be reset. */ if ((tableType_t)cctx->tableType != clearedTable) { assert(inputSize >= 0); if ((tableType_t)cctx->tableType != tableType || ((tableType == byU16) && cctx->currentOffset + (unsigned)inputSize >= 0xFFFFU) || ((tableType == byU32) && cctx->currentOffset > 1 GB) || tableType == byPtr || inputSize >= 4 KB) { DEBUGLOG(4, "LZ4_prepareTable: Resetting table in %p", cctx); MEM_INIT(cctx->hashTable, 0, LZ4_HASHTABLESIZE); cctx->currentOffset = 0; cctx->tableType = (U32)clearedTable; } else { DEBUGLOG(4, "LZ4_prepareTable: Re-use hash table (no reset)"); } } /* Adding a gap, so all previous entries are > LZ4_DISTANCE_MAX back, * is faster than compressing without a gap. * However, compressing with currentOffset == 0 is faster still, * so we preserve that case. */ if (cctx->currentOffset != 0 && tableType == byU32) { DEBUGLOG(5, "LZ4_prepareTable: adding 64KB to currentOffset"); cctx->currentOffset += 64 KB; } /* Finally, clear history */ cctx->dictCtx = NULL; cctx->dictionary = NULL; cctx->dictSize = 0; } /** LZ4_compress_generic() : * inlined, to ensure branches are decided at compilation time. * Presumed already validated at this stage: * - source != NULL * - inputSize > 0 */ LZ4_FORCE_INLINE int LZ4_compress_generic_validated( LZ4_stream_t_internal* const cctx, const char* const source, char* const dest, const int inputSize, int* inputConsumed, /* only written when outputDirective == fillOutput */ const int maxOutputSize, const limitedOutput_directive outputDirective, const tableType_t tableType, const dict_directive dictDirective, const dictIssue_directive dictIssue, const int acceleration) { int result; const BYTE* ip = (const BYTE*) source; U32 const startIndex = cctx->currentOffset; const BYTE* base = (const BYTE*) source - startIndex; const BYTE* lowLimit; const LZ4_stream_t_internal* dictCtx = (const LZ4_stream_t_internal*) cctx->dictCtx; const BYTE* const dictionary = dictDirective == usingDictCtx ? dictCtx->dictionary : cctx->dictionary; const U32 dictSize = dictDirective == usingDictCtx ? dictCtx->dictSize : cctx->dictSize; const U32 dictDelta = (dictDirective == usingDictCtx) ? startIndex - dictCtx->currentOffset : 0; /* make indexes in dictCtx comparable with index in current context */ int const maybe_extMem = (dictDirective == usingExtDict) || (dictDirective == usingDictCtx); U32 const prefixIdxLimit = startIndex - dictSize; /* used when dictDirective == dictSmall */ const BYTE* const dictEnd = dictionary ? dictionary + dictSize : dictionary; const BYTE* anchor = (const BYTE*) source; const BYTE* const iend = ip + inputSize; const BYTE* const mflimitPlusOne = iend - MFLIMIT + 1; const BYTE* const matchlimit = iend - LASTLITERALS; /* the dictCtx currentOffset is indexed on the start of the dictionary, * while a dictionary in the current context precedes the currentOffset */ const BYTE* dictBase = (dictionary == NULL) ? NULL : (dictDirective == usingDictCtx) ? dictionary + dictSize - dictCtx->currentOffset : dictionary + dictSize - startIndex; BYTE* op = (BYTE*) dest; BYTE* const olimit = op + maxOutputSize; U32 offset = 0; U32 forwardH; DEBUGLOG(5, "LZ4_compress_generic_validated: srcSize=%i, tableType=%u", inputSize, tableType); assert(ip != NULL); /* If init conditions are not met, we don't have to mark stream * as having dirty context, since no action was taken yet */ if (outputDirective == fillOutput && maxOutputSize < 1) { return 0; } /* Impossible to store anything */ if ((tableType == byU16) && (inputSize>=LZ4_64Klimit)) { return 0; } /* Size too large (not within 64K limit) */ if (tableType==byPtr) assert(dictDirective==noDict); /* only supported use case with byPtr */ assert(acceleration >= 1); lowLimit = (const BYTE*)source - (dictDirective == withPrefix64k ? dictSize : 0); /* Update context state */ if (dictDirective == usingDictCtx) { /* Subsequent linked blocks can't use the dictionary. */ /* Instead, they use the block we just compressed. */ cctx->dictCtx = NULL; cctx->dictSize = (U32)inputSize; } else { cctx->dictSize += (U32)inputSize; } cctx->currentOffset += (U32)inputSize; cctx->tableType = (U32)tableType; if (inputSizehashTable, tableType, base); ip++; forwardH = LZ4_hashPosition(ip, tableType); /* Main Loop */ for ( ; ; ) { const BYTE* match; BYTE* token; const BYTE* filledIp; /* Find a match */ if (tableType == byPtr) { const BYTE* forwardIp = ip; int step = 1; int searchMatchNb = acceleration << LZ4_skipTrigger; do { U32 const h = forwardH; ip = forwardIp; forwardIp += step; step = (searchMatchNb++ >> LZ4_skipTrigger); if (unlikely(forwardIp > mflimitPlusOne)) goto _last_literals; assert(ip < mflimitPlusOne); match = LZ4_getPositionOnHash(h, cctx->hashTable, tableType, base); forwardH = LZ4_hashPosition(forwardIp, tableType); LZ4_putPositionOnHash(ip, h, cctx->hashTable, tableType, base); } while ( (match+LZ4_DISTANCE_MAX < ip) || (LZ4_read32(match) != LZ4_read32(ip)) ); } else { /* byU32, byU16 */ const BYTE* forwardIp = ip; int step = 1; int searchMatchNb = acceleration << LZ4_skipTrigger; do { U32 const h = forwardH; U32 const current = (U32)(forwardIp - base); U32 matchIndex = LZ4_getIndexOnHash(h, cctx->hashTable, tableType); assert(matchIndex <= current); assert(forwardIp - base < (ptrdiff_t)(2 GB - 1)); ip = forwardIp; forwardIp += step; step = (searchMatchNb++ >> LZ4_skipTrigger); if (unlikely(forwardIp > mflimitPlusOne)) goto _last_literals; assert(ip < mflimitPlusOne); if (dictDirective == usingDictCtx) { if (matchIndex < startIndex) { /* there was no match, try the dictionary */ assert(tableType == byU32); matchIndex = LZ4_getIndexOnHash(h, dictCtx->hashTable, byU32); match = dictBase + matchIndex; matchIndex += dictDelta; /* make dictCtx index comparable with current context */ lowLimit = dictionary; } else { match = base + matchIndex; lowLimit = (const BYTE*)source; } } else if (dictDirective == usingExtDict) { if (matchIndex < startIndex) { DEBUGLOG(7, "extDict candidate: matchIndex=%5u < startIndex=%5u", matchIndex, startIndex); assert(startIndex - matchIndex >= MINMATCH); assert(dictBase); match = dictBase + matchIndex; lowLimit = dictionary; } else { match = base + matchIndex; lowLimit = (const BYTE*)source; } } else { /* single continuous memory segment */ match = base + matchIndex; } forwardH = LZ4_hashPosition(forwardIp, tableType); LZ4_putIndexOnHash(current, h, cctx->hashTable, tableType); DEBUGLOG(7, "candidate at pos=%u (offset=%u \n", matchIndex, current - matchIndex); if ((dictIssue == dictSmall) && (matchIndex < prefixIdxLimit)) { continue; } /* match outside of valid area */ assert(matchIndex < current); if ( ((tableType != byU16) || (LZ4_DISTANCE_MAX < LZ4_DISTANCE_ABSOLUTE_MAX)) && (matchIndex+LZ4_DISTANCE_MAX < current)) { continue; } /* too far */ assert((current - matchIndex) <= LZ4_DISTANCE_MAX); /* match now expected within distance */ if (LZ4_read32(match) == LZ4_read32(ip)) { if (maybe_extMem) offset = current - matchIndex; break; /* match found */ } } while(1); } /* Catch up */ filledIp = ip; while (((ip>anchor) & (match > lowLimit)) && (unlikely(ip[-1]==match[-1]))) { ip--; match--; } /* Encode Literals */ { unsigned const litLength = (unsigned)(ip - anchor); token = op++; if ((outputDirective == limitedOutput) && /* Check output buffer overflow */ (unlikely(op + litLength + (2 + 1 + LASTLITERALS) + (litLength/255) > olimit)) ) { return 0; /* cannot compress within `dst` budget. Stored indexes in hash table are nonetheless fine */ } if ((outputDirective == fillOutput) && (unlikely(op + (litLength+240)/255 /* litlen */ + litLength /* literals */ + 2 /* offset */ + 1 /* token */ + MFLIMIT - MINMATCH /* min last literals so last match is <= end - MFLIMIT */ > olimit))) { op--; goto _last_literals; } if (litLength >= RUN_MASK) { int len = (int)(litLength - RUN_MASK); *token = (RUN_MASK<= 255 ; len-=255) *op++ = 255; *op++ = (BYTE)len; } else *token = (BYTE)(litLength< olimit)) { /* the match was too close to the end, rewind and go to last literals */ op = token; goto _last_literals; } /* Encode Offset */ if (maybe_extMem) { /* static test */ DEBUGLOG(6, " with offset=%u (ext if > %i)", offset, (int)(ip - (const BYTE*)source)); assert(offset <= LZ4_DISTANCE_MAX && offset > 0); LZ4_writeLE16(op, (U16)offset); op+=2; } else { DEBUGLOG(6, " with offset=%u (same segment)", (U32)(ip - match)); assert(ip-match <= LZ4_DISTANCE_MAX); LZ4_writeLE16(op, (U16)(ip - match)); op+=2; } /* Encode MatchLength */ { unsigned matchCode; if ( (dictDirective==usingExtDict || dictDirective==usingDictCtx) && (lowLimit==dictionary) /* match within extDict */ ) { const BYTE* limit = ip + (dictEnd-match); assert(dictEnd > match); if (limit > matchlimit) limit = matchlimit; matchCode = LZ4_count(ip+MINMATCH, match+MINMATCH, limit); ip += (size_t)matchCode + MINMATCH; if (ip==limit) { unsigned const more = LZ4_count(limit, (const BYTE*)source, matchlimit); matchCode += more; ip += more; } DEBUGLOG(6, " with matchLength=%u starting in extDict", matchCode+MINMATCH); } else { matchCode = LZ4_count(ip+MINMATCH, match+MINMATCH, matchlimit); ip += (size_t)matchCode + MINMATCH; DEBUGLOG(6, " with matchLength=%u", matchCode+MINMATCH); } if ((outputDirective) && /* Check output buffer overflow */ (unlikely(op + (1 + LASTLITERALS) + (matchCode+240)/255 > olimit)) ) { if (outputDirective == fillOutput) { /* Match description too long : reduce it */ U32 newMatchCode = 15 /* in token */ - 1 /* to avoid needing a zero byte */ + ((U32)(olimit - op) - 1 - LASTLITERALS) * 255; ip -= matchCode - newMatchCode; assert(newMatchCode < matchCode); matchCode = newMatchCode; if (unlikely(ip <= filledIp)) { /* We have already filled up to filledIp so if ip ends up less than filledIp * we have positions in the hash table beyond the current position. This is * a problem if we reuse the hash table. So we have to remove these positions * from the hash table. */ const BYTE* ptr; DEBUGLOG(5, "Clearing %u positions", (U32)(filledIp - ip)); for (ptr = ip; ptr <= filledIp; ++ptr) { U32 const h = LZ4_hashPosition(ptr, tableType); LZ4_clearHash(h, cctx->hashTable, tableType); } } } else { assert(outputDirective == limitedOutput); return 0; /* cannot compress within `dst` budget. Stored indexes in hash table are nonetheless fine */ } } if (matchCode >= ML_MASK) { *token += ML_MASK; matchCode -= ML_MASK; LZ4_write32(op, 0xFFFFFFFF); while (matchCode >= 4*255) { op+=4; LZ4_write32(op, 0xFFFFFFFF); matchCode -= 4*255; } op += matchCode / 255; *op++ = (BYTE)(matchCode % 255); } else *token += (BYTE)(matchCode); } /* Ensure we have enough space for the last literals. */ assert(!(outputDirective == fillOutput && op + 1 + LASTLITERALS > olimit)); anchor = ip; /* Test end of chunk */ if (ip >= mflimitPlusOne) break; /* Fill table */ LZ4_putPosition(ip-2, cctx->hashTable, tableType, base); /* Test next position */ if (tableType == byPtr) { match = LZ4_getPosition(ip, cctx->hashTable, tableType, base); LZ4_putPosition(ip, cctx->hashTable, tableType, base); if ( (match+LZ4_DISTANCE_MAX >= ip) && (LZ4_read32(match) == LZ4_read32(ip)) ) { token=op++; *token=0; goto _next_match; } } else { /* byU32, byU16 */ U32 const h = LZ4_hashPosition(ip, tableType); U32 const current = (U32)(ip-base); U32 matchIndex = LZ4_getIndexOnHash(h, cctx->hashTable, tableType); assert(matchIndex < current); if (dictDirective == usingDictCtx) { if (matchIndex < startIndex) { /* there was no match, try the dictionary */ matchIndex = LZ4_getIndexOnHash(h, dictCtx->hashTable, byU32); match = dictBase + matchIndex; lowLimit = dictionary; /* required for match length counter */ matchIndex += dictDelta; } else { match = base + matchIndex; lowLimit = (const BYTE*)source; /* required for match length counter */ } } else if (dictDirective==usingExtDict) { if (matchIndex < startIndex) { assert(dictBase); match = dictBase + matchIndex; lowLimit = dictionary; /* required for match length counter */ } else { match = base + matchIndex; lowLimit = (const BYTE*)source; /* required for match length counter */ } } else { /* single memory segment */ match = base + matchIndex; } LZ4_putIndexOnHash(current, h, cctx->hashTable, tableType); assert(matchIndex < current); if ( ((dictIssue==dictSmall) ? (matchIndex >= prefixIdxLimit) : 1) && (((tableType==byU16) && (LZ4_DISTANCE_MAX == LZ4_DISTANCE_ABSOLUTE_MAX)) ? 1 : (matchIndex+LZ4_DISTANCE_MAX >= current)) && (LZ4_read32(match) == LZ4_read32(ip)) ) { token=op++; *token=0; if (maybe_extMem) offset = current - matchIndex; DEBUGLOG(6, "seq.start:%i, literals=%u, match.start:%i", (int)(anchor-(const BYTE*)source), 0, (int)(ip-(const BYTE*)source)); goto _next_match; } } /* Prepare next loop */ forwardH = LZ4_hashPosition(++ip, tableType); } _last_literals: /* Encode Last Literals */ { size_t lastRun = (size_t)(iend - anchor); if ( (outputDirective) && /* Check output buffer overflow */ (op + lastRun + 1 + ((lastRun+255-RUN_MASK)/255) > olimit)) { if (outputDirective == fillOutput) { /* adapt lastRun to fill 'dst' */ assert(olimit >= op); lastRun = (size_t)(olimit-op) - 1/*token*/; lastRun -= (lastRun + 256 - RUN_MASK) / 256; /*additional length tokens*/ } else { assert(outputDirective == limitedOutput); return 0; /* cannot compress within `dst` budget. Stored indexes in hash table are nonetheless fine */ } } DEBUGLOG(6, "Final literal run : %i literals", (int)lastRun); if (lastRun >= RUN_MASK) { size_t accumulator = lastRun - RUN_MASK; *op++ = RUN_MASK << ML_BITS; for(; accumulator >= 255 ; accumulator-=255) *op++ = 255; *op++ = (BYTE) accumulator; } else { *op++ = (BYTE)(lastRun< 0); DEBUGLOG(5, "LZ4_compress_generic: compressed %i bytes into %i bytes", inputSize, result); return result; } /** LZ4_compress_generic() : * inlined, to ensure branches are decided at compilation time; * takes care of src == (NULL, 0) * and forward the rest to LZ4_compress_generic_validated */ LZ4_FORCE_INLINE int LZ4_compress_generic( LZ4_stream_t_internal* const cctx, const char* const src, char* const dst, const int srcSize, int *inputConsumed, /* only written when outputDirective == fillOutput */ const int dstCapacity, const limitedOutput_directive outputDirective, const tableType_t tableType, const dict_directive dictDirective, const dictIssue_directive dictIssue, const int acceleration) { DEBUGLOG(5, "LZ4_compress_generic: srcSize=%i, dstCapacity=%i", srcSize, dstCapacity); if ((U32)srcSize > (U32)LZ4_MAX_INPUT_SIZE) { return 0; } /* Unsupported srcSize, too large (or negative) */ if (srcSize == 0) { /* src == NULL supported if srcSize == 0 */ if (outputDirective != notLimited && dstCapacity <= 0) return 0; /* no output, can't write anything */ DEBUGLOG(5, "Generating an empty block"); assert(outputDirective == notLimited || dstCapacity >= 1); assert(dst != NULL); dst[0] = 0; if (outputDirective == fillOutput) { assert (inputConsumed != NULL); *inputConsumed = 0; } return 1; } assert(src != NULL); return LZ4_compress_generic_validated(cctx, src, dst, srcSize, inputConsumed, /* only written into if outputDirective == fillOutput */ dstCapacity, outputDirective, tableType, dictDirective, dictIssue, acceleration); } int LZ4_compress_fast_extState(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) { LZ4_stream_t_internal* const ctx = & LZ4_initStream(state, sizeof(LZ4_stream_t)) -> internal_donotuse; assert(ctx != NULL); if (acceleration < 1) acceleration = LZ4_ACCELERATION_DEFAULT; if (acceleration > LZ4_ACCELERATION_MAX) acceleration = LZ4_ACCELERATION_MAX; if (maxOutputSize >= LZ4_compressBound(inputSize)) { if (inputSize < LZ4_64Klimit) { return LZ4_compress_generic(ctx, source, dest, inputSize, NULL, 0, notLimited, byU16, noDict, noDictIssue, acceleration); } else { const tableType_t tableType = ((sizeof(void*)==4) && ((uptrval)source > LZ4_DISTANCE_MAX)) ? byPtr : byU32; return LZ4_compress_generic(ctx, source, dest, inputSize, NULL, 0, notLimited, tableType, noDict, noDictIssue, acceleration); } } else { if (inputSize < LZ4_64Klimit) { return LZ4_compress_generic(ctx, source, dest, inputSize, NULL, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, acceleration); } else { const tableType_t tableType = ((sizeof(void*)==4) && ((uptrval)source > LZ4_DISTANCE_MAX)) ? byPtr : byU32; return LZ4_compress_generic(ctx, source, dest, inputSize, NULL, maxOutputSize, limitedOutput, tableType, noDict, noDictIssue, acceleration); } } } /** * LZ4_compress_fast_extState_fastReset() : * A variant of LZ4_compress_fast_extState(). * * Using this variant avoids an expensive initialization step. It is only safe * to call if the state buffer is known to be correctly initialized already * (see comment in lz4.h on LZ4_resetStream_fast() for a definition of * "correctly initialized"). */ int LZ4_compress_fast_extState_fastReset(void* state, const char* src, char* dst, int srcSize, int dstCapacity, int acceleration) { LZ4_stream_t_internal* ctx = &((LZ4_stream_t*)state)->internal_donotuse; if (acceleration < 1) acceleration = LZ4_ACCELERATION_DEFAULT; if (acceleration > LZ4_ACCELERATION_MAX) acceleration = LZ4_ACCELERATION_MAX; if (dstCapacity >= LZ4_compressBound(srcSize)) { if (srcSize < LZ4_64Klimit) { const tableType_t tableType = byU16; LZ4_prepareTable(ctx, srcSize, tableType); if (ctx->currentOffset) { return LZ4_compress_generic(ctx, src, dst, srcSize, NULL, 0, notLimited, tableType, noDict, dictSmall, acceleration); } else { return LZ4_compress_generic(ctx, src, dst, srcSize, NULL, 0, notLimited, tableType, noDict, noDictIssue, acceleration); } } else { const tableType_t tableType = ((sizeof(void*)==4) && ((uptrval)src > LZ4_DISTANCE_MAX)) ? byPtr : byU32; LZ4_prepareTable(ctx, srcSize, tableType); return LZ4_compress_generic(ctx, src, dst, srcSize, NULL, 0, notLimited, tableType, noDict, noDictIssue, acceleration); } } else { if (srcSize < LZ4_64Klimit) { const tableType_t tableType = byU16; LZ4_prepareTable(ctx, srcSize, tableType); if (ctx->currentOffset) { return LZ4_compress_generic(ctx, src, dst, srcSize, NULL, dstCapacity, limitedOutput, tableType, noDict, dictSmall, acceleration); } else { return LZ4_compress_generic(ctx, src, dst, srcSize, NULL, dstCapacity, limitedOutput, tableType, noDict, noDictIssue, acceleration); } } else { const tableType_t tableType = ((sizeof(void*)==4) && ((uptrval)src > LZ4_DISTANCE_MAX)) ? byPtr : byU32; LZ4_prepareTable(ctx, srcSize, tableType); return LZ4_compress_generic(ctx, src, dst, srcSize, NULL, dstCapacity, limitedOutput, tableType, noDict, noDictIssue, acceleration); } } } int LZ4_compress_fast(const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) { int result; #if (LZ4_HEAPMODE) LZ4_stream_t* ctxPtr = (LZ4_stream_t*)ALLOC(sizeof(LZ4_stream_t)); /* malloc-calloc always properly aligned */ if (ctxPtr == NULL) return 0; #else LZ4_stream_t ctx; LZ4_stream_t* const ctxPtr = &ctx; #endif result = LZ4_compress_fast_extState(ctxPtr, source, dest, inputSize, maxOutputSize, acceleration); #if (LZ4_HEAPMODE) FREEMEM(ctxPtr); #endif return result; } int LZ4_compress_default(const char* src, char* dst, int srcSize, int maxOutputSize) { return LZ4_compress_fast(src, dst, srcSize, maxOutputSize, 1); } /* Note!: This function leaves the stream in an unclean/broken state! * It is not safe to subsequently use the same state with a _fastReset() or * _continue() call without resetting it. */ static int LZ4_compress_destSize_extState (LZ4_stream_t* state, const char* src, char* dst, int* srcSizePtr, int targetDstSize) { void* const s = LZ4_initStream(state, sizeof (*state)); assert(s != NULL); (void)s; if (targetDstSize >= LZ4_compressBound(*srcSizePtr)) { /* compression success is guaranteed */ return LZ4_compress_fast_extState(state, src, dst, *srcSizePtr, targetDstSize, 1); } else { if (*srcSizePtr < LZ4_64Klimit) { return LZ4_compress_generic(&state->internal_donotuse, src, dst, *srcSizePtr, srcSizePtr, targetDstSize, fillOutput, byU16, noDict, noDictIssue, 1); } else { tableType_t const addrMode = ((sizeof(void*)==4) && ((uptrval)src > LZ4_DISTANCE_MAX)) ? byPtr : byU32; return LZ4_compress_generic(&state->internal_donotuse, src, dst, *srcSizePtr, srcSizePtr, targetDstSize, fillOutput, addrMode, noDict, noDictIssue, 1); } } } int LZ4_compress_destSize(const char* src, char* dst, int* srcSizePtr, int targetDstSize) { #if (LZ4_HEAPMODE) LZ4_stream_t* ctx = (LZ4_stream_t*)ALLOC(sizeof(LZ4_stream_t)); /* malloc-calloc always properly aligned */ if (ctx == NULL) return 0; #else LZ4_stream_t ctxBody; LZ4_stream_t* ctx = &ctxBody; #endif int result = LZ4_compress_destSize_extState(ctx, src, dst, srcSizePtr, targetDstSize); #if (LZ4_HEAPMODE) FREEMEM(ctx); #endif return result; } /*-****************************** * Streaming functions ********************************/ #if !defined(LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION) LZ4_stream_t* LZ4_createStream(void) { LZ4_stream_t* const lz4s = (LZ4_stream_t*)ALLOC(sizeof(LZ4_stream_t)); LZ4_STATIC_ASSERT(sizeof(LZ4_stream_t) >= sizeof(LZ4_stream_t_internal)); DEBUGLOG(4, "LZ4_createStream %p", lz4s); if (lz4s == NULL) return NULL; LZ4_initStream(lz4s, sizeof(*lz4s)); return lz4s; } #endif static size_t LZ4_stream_t_alignment(void) { #if LZ4_ALIGN_TEST typedef struct { char c; LZ4_stream_t t; } t_a; return sizeof(t_a) - sizeof(LZ4_stream_t); #else return 1; /* effectively disabled */ #endif } LZ4_stream_t* LZ4_initStream (void* buffer, size_t size) { DEBUGLOG(5, "LZ4_initStream"); if (buffer == NULL) { return NULL; } if (size < sizeof(LZ4_stream_t)) { return NULL; } if (!LZ4_isAligned(buffer, LZ4_stream_t_alignment())) return NULL; MEM_INIT(buffer, 0, sizeof(LZ4_stream_t_internal)); return (LZ4_stream_t*)buffer; } /* resetStream is now deprecated, * prefer initStream() which is more general */ void LZ4_resetStream (LZ4_stream_t* LZ4_stream) { DEBUGLOG(5, "LZ4_resetStream (ctx:%p)", LZ4_stream); MEM_INIT(LZ4_stream, 0, sizeof(LZ4_stream_t_internal)); } void LZ4_resetStream_fast(LZ4_stream_t* ctx) { LZ4_prepareTable(&(ctx->internal_donotuse), 0, byU32); } #if !defined(LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION) int LZ4_freeStream (LZ4_stream_t* LZ4_stream) { if (!LZ4_stream) return 0; /* support free on NULL */ DEBUGLOG(5, "LZ4_freeStream %p", LZ4_stream); FREEMEM(LZ4_stream); return (0); } #endif #define HASH_UNIT sizeof(reg_t) int LZ4_loadDict (LZ4_stream_t* LZ4_dict, const char* dictionary, int dictSize) { LZ4_stream_t_internal* dict = &LZ4_dict->internal_donotuse; const tableType_t tableType = byU32; const BYTE* p = (const BYTE*)dictionary; const BYTE* const dictEnd = p + dictSize; const BYTE* base; DEBUGLOG(4, "LZ4_loadDict (%i bytes from %p into %p)", dictSize, dictionary, LZ4_dict); /* It's necessary to reset the context, * and not just continue it with prepareTable() * to avoid any risk of generating overflowing matchIndex * when compressing using this dictionary */ LZ4_resetStream(LZ4_dict); /* We always increment the offset by 64 KB, since, if the dict is longer, * we truncate it to the last 64k, and if it's shorter, we still want to * advance by a whole window length so we can provide the guarantee that * there are only valid offsets in the window, which allows an optimization * in LZ4_compress_fast_continue() where it uses noDictIssue even when the * dictionary isn't a full 64k. */ dict->currentOffset += 64 KB; if (dictSize < (int)HASH_UNIT) { return 0; } if ((dictEnd - p) > 64 KB) p = dictEnd - 64 KB; base = dictEnd - dict->currentOffset; dict->dictionary = p; dict->dictSize = (U32)(dictEnd - p); dict->tableType = (U32)tableType; while (p <= dictEnd-HASH_UNIT) { LZ4_putPosition(p, dict->hashTable, tableType, base); p+=3; } return (int)dict->dictSize; } void LZ4_attach_dictionary(LZ4_stream_t* workingStream, const LZ4_stream_t* dictionaryStream) { const LZ4_stream_t_internal* dictCtx = (dictionaryStream == NULL) ? NULL : &(dictionaryStream->internal_donotuse); DEBUGLOG(4, "LZ4_attach_dictionary (%p, %p, size %u)", workingStream, dictionaryStream, dictCtx != NULL ? dictCtx->dictSize : 0); if (dictCtx != NULL) { /* If the current offset is zero, we will never look in the * external dictionary context, since there is no value a table * entry can take that indicate a miss. In that case, we need * to bump the offset to something non-zero. */ if (workingStream->internal_donotuse.currentOffset == 0) { workingStream->internal_donotuse.currentOffset = 64 KB; } /* Don't actually attach an empty dictionary. */ if (dictCtx->dictSize == 0) { dictCtx = NULL; } } workingStream->internal_donotuse.dictCtx = dictCtx; } static void LZ4_renormDictT(LZ4_stream_t_internal* LZ4_dict, int nextSize) { assert(nextSize >= 0); if (LZ4_dict->currentOffset + (unsigned)nextSize > 0x80000000) { /* potential ptrdiff_t overflow (32-bits mode) */ /* rescale hash table */ U32 const delta = LZ4_dict->currentOffset - 64 KB; const BYTE* dictEnd = LZ4_dict->dictionary + LZ4_dict->dictSize; int i; DEBUGLOG(4, "LZ4_renormDictT"); for (i=0; ihashTable[i] < delta) LZ4_dict->hashTable[i]=0; else LZ4_dict->hashTable[i] -= delta; } LZ4_dict->currentOffset = 64 KB; if (LZ4_dict->dictSize > 64 KB) LZ4_dict->dictSize = 64 KB; LZ4_dict->dictionary = dictEnd - LZ4_dict->dictSize; } } int LZ4_compress_fast_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) { const tableType_t tableType = byU32; LZ4_stream_t_internal* const streamPtr = &LZ4_stream->internal_donotuse; const char* dictEnd = streamPtr->dictSize ? (const char*)streamPtr->dictionary + streamPtr->dictSize : NULL; DEBUGLOG(5, "LZ4_compress_fast_continue (inputSize=%i, dictSize=%u)", inputSize, streamPtr->dictSize); LZ4_renormDictT(streamPtr, inputSize); /* fix index overflow */ if (acceleration < 1) acceleration = LZ4_ACCELERATION_DEFAULT; if (acceleration > LZ4_ACCELERATION_MAX) acceleration = LZ4_ACCELERATION_MAX; /* invalidate tiny dictionaries */ if ( (streamPtr->dictSize < 4) /* tiny dictionary : not enough for a hash */ && (dictEnd != source) /* prefix mode */ && (inputSize > 0) /* tolerance : don't lose history, in case next invocation would use prefix mode */ && (streamPtr->dictCtx == NULL) /* usingDictCtx */ ) { DEBUGLOG(5, "LZ4_compress_fast_continue: dictSize(%u) at addr:%p is too small", streamPtr->dictSize, streamPtr->dictionary); /* remove dictionary existence from history, to employ faster prefix mode */ streamPtr->dictSize = 0; streamPtr->dictionary = (const BYTE*)source; dictEnd = source; } /* Check overlapping input/dictionary space */ { const char* const sourceEnd = source + inputSize; if ((sourceEnd > (const char*)streamPtr->dictionary) && (sourceEnd < dictEnd)) { streamPtr->dictSize = (U32)(dictEnd - sourceEnd); if (streamPtr->dictSize > 64 KB) streamPtr->dictSize = 64 KB; if (streamPtr->dictSize < 4) streamPtr->dictSize = 0; streamPtr->dictionary = (const BYTE*)dictEnd - streamPtr->dictSize; } } /* prefix mode : source data follows dictionary */ if (dictEnd == source) { if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) return LZ4_compress_generic(streamPtr, source, dest, inputSize, NULL, maxOutputSize, limitedOutput, tableType, withPrefix64k, dictSmall, acceleration); else return LZ4_compress_generic(streamPtr, source, dest, inputSize, NULL, maxOutputSize, limitedOutput, tableType, withPrefix64k, noDictIssue, acceleration); } /* external dictionary mode */ { int result; if (streamPtr->dictCtx) { /* We depend here on the fact that dictCtx'es (produced by * LZ4_loadDict) guarantee that their tables contain no references * to offsets between dictCtx->currentOffset - 64 KB and * dictCtx->currentOffset - dictCtx->dictSize. This makes it safe * to use noDictIssue even when the dict isn't a full 64 KB. */ if (inputSize > 4 KB) { /* For compressing large blobs, it is faster to pay the setup * cost to copy the dictionary's tables into the active context, * so that the compression loop is only looking into one table. */ LZ4_memcpy(streamPtr, streamPtr->dictCtx, sizeof(*streamPtr)); result = LZ4_compress_generic(streamPtr, source, dest, inputSize, NULL, maxOutputSize, limitedOutput, tableType, usingExtDict, noDictIssue, acceleration); } else { result = LZ4_compress_generic(streamPtr, source, dest, inputSize, NULL, maxOutputSize, limitedOutput, tableType, usingDictCtx, noDictIssue, acceleration); } } else { /* small data <= 4 KB */ if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) { result = LZ4_compress_generic(streamPtr, source, dest, inputSize, NULL, maxOutputSize, limitedOutput, tableType, usingExtDict, dictSmall, acceleration); } else { result = LZ4_compress_generic(streamPtr, source, dest, inputSize, NULL, maxOutputSize, limitedOutput, tableType, usingExtDict, noDictIssue, acceleration); } } streamPtr->dictionary = (const BYTE*)source; streamPtr->dictSize = (U32)inputSize; return result; } } /* Hidden debug function, to force-test external dictionary mode */ int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_dict, const char* source, char* dest, int srcSize) { LZ4_stream_t_internal* streamPtr = &LZ4_dict->internal_donotuse; int result; LZ4_renormDictT(streamPtr, srcSize); if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) { result = LZ4_compress_generic(streamPtr, source, dest, srcSize, NULL, 0, notLimited, byU32, usingExtDict, dictSmall, 1); } else { result = LZ4_compress_generic(streamPtr, source, dest, srcSize, NULL, 0, notLimited, byU32, usingExtDict, noDictIssue, 1); } streamPtr->dictionary = (const BYTE*)source; streamPtr->dictSize = (U32)srcSize; return result; } /*! LZ4_saveDict() : * If previously compressed data block is not guaranteed to remain available at its memory location, * save it into a safer place (char* safeBuffer). * Note : no need to call LZ4_loadDict() afterwards, dictionary is immediately usable, * one can therefore call LZ4_compress_fast_continue() right after. * @return : saved dictionary size in bytes (necessarily <= dictSize), or 0 if error. */ int LZ4_saveDict (LZ4_stream_t* LZ4_dict, char* safeBuffer, int dictSize) { LZ4_stream_t_internal* const dict = &LZ4_dict->internal_donotuse; DEBUGLOG(5, "LZ4_saveDict : dictSize=%i, safeBuffer=%p", dictSize, safeBuffer); if ((U32)dictSize > 64 KB) { dictSize = 64 KB; } /* useless to define a dictionary > 64 KB */ if ((U32)dictSize > dict->dictSize) { dictSize = (int)dict->dictSize; } if (safeBuffer == NULL) assert(dictSize == 0); if (dictSize > 0) { const BYTE* const previousDictEnd = dict->dictionary + dict->dictSize; assert(dict->dictionary); LZ4_memmove(safeBuffer, previousDictEnd - dictSize, (size_t)dictSize); } dict->dictionary = (const BYTE*)safeBuffer; dict->dictSize = (U32)dictSize; return dictSize; } /*-******************************* * Decompression functions ********************************/ typedef enum { decode_full_block = 0, partial_decode = 1 } earlyEnd_directive; #undef MIN #define MIN(a,b) ( (a) < (b) ? (a) : (b) ) /* variant for decompress_unsafe() * does not know end of input * presumes input is well formed * note : will consume at least one byte */ size_t read_long_length_no_check(const BYTE** pp) { size_t b, l = 0; do { b = **pp; (*pp)++; l += b; } while (b==255); DEBUGLOG(6, "read_long_length_no_check: +length=%zu using %zu input bytes", l, l/255 + 1) return l; } /* core decoder variant for LZ4_decompress_fast*() * for legacy support only : these entry points are deprecated. * - Presumes input is correctly formed (no defense vs malformed inputs) * - Does not know input size (presume input buffer is "large enough") * - Decompress a full block (only) * @return : nb of bytes read from input. * Note : this variant is not optimized for speed, just for maintenance. * the goal is to remove support of decompress_fast*() variants by v2.0 **/ LZ4_FORCE_INLINE int LZ4_decompress_unsafe_generic( const BYTE* const istart, BYTE* const ostart, int decompressedSize, size_t prefixSize, const BYTE* const dictStart, /* only if dict==usingExtDict */ const size_t dictSize /* note: =0 if dictStart==NULL */ ) { const BYTE* ip = istart; BYTE* op = (BYTE*)ostart; BYTE* const oend = ostart + decompressedSize; const BYTE* const prefixStart = ostart - prefixSize; DEBUGLOG(5, "LZ4_decompress_unsafe_generic"); if (dictStart == NULL) assert(dictSize == 0); while (1) { /* start new sequence */ unsigned token = *ip++; /* literals */ { size_t ll = token >> ML_BITS; if (ll==15) { /* long literal length */ ll += read_long_length_no_check(&ip); } if ((size_t)(oend-op) < ll) return -1; /* output buffer overflow */ LZ4_memmove(op, ip, ll); /* support in-place decompression */ op += ll; ip += ll; if ((size_t)(oend-op) < MFLIMIT) { if (op==oend) break; /* end of block */ DEBUGLOG(5, "invalid: literals end at distance %zi from end of block", oend-op); /* incorrect end of block : * last match must start at least MFLIMIT==12 bytes before end of output block */ return -1; } } /* match */ { size_t ml = token & 15; size_t const offset = LZ4_readLE16(ip); ip+=2; if (ml==15) { /* long literal length */ ml += read_long_length_no_check(&ip); } ml += MINMATCH; if ((size_t)(oend-op) < ml) return -1; /* output buffer overflow */ { const BYTE* match = op - offset; /* out of range */ if (offset > (size_t)(op - prefixStart) + dictSize) { DEBUGLOG(6, "offset out of range"); return -1; } /* check special case : extDict */ if (offset > (size_t)(op - prefixStart)) { /* extDict scenario */ const BYTE* const dictEnd = dictStart + dictSize; const BYTE* extMatch = dictEnd - (offset - (size_t)(op-prefixStart)); size_t const extml = (size_t)(dictEnd - extMatch); if (extml > ml) { /* match entirely within extDict */ LZ4_memmove(op, extMatch, ml); op += ml; ml = 0; } else { /* match split between extDict & prefix */ LZ4_memmove(op, extMatch, extml); op += extml; ml -= extml; } match = prefixStart; } /* match copy - slow variant, supporting overlap copy */ { size_t u; for (u=0; u= ipmax before start of loop. Returns initial_error if so. * @error (output) - error code. Must be set to 0 before call. **/ typedef size_t Rvl_t; static const Rvl_t rvl_error = (Rvl_t)(-1); LZ4_FORCE_INLINE Rvl_t read_variable_length(const BYTE** ip, const BYTE* ilimit, int initial_check) { Rvl_t s, length = 0; assert(ip != NULL); assert(*ip != NULL); assert(ilimit != NULL); if (initial_check && unlikely((*ip) >= ilimit)) { /* read limit reached */ return rvl_error; } do { s = **ip; (*ip)++; length += s; if (unlikely((*ip) > ilimit)) { /* read limit reached */ return rvl_error; } /* accumulator overflow detection (32-bit mode only) */ if ((sizeof(length)<8) && unlikely(length > ((Rvl_t)(-1)/2)) ) { return rvl_error; } } while (s==255); return length; } /*! LZ4_decompress_generic() : * This generic decompression function covers all use cases. * It shall be instantiated several times, using different sets of directives. * Note that it is important for performance that this function really get inlined, * in order to remove useless branches during compilation optimization. */ LZ4_FORCE_INLINE int LZ4_decompress_generic( const char* const src, char* const dst, int srcSize, int outputSize, /* If endOnInput==endOnInputSize, this value is `dstCapacity` */ earlyEnd_directive partialDecoding, /* full, partial */ dict_directive dict, /* noDict, withPrefix64k, usingExtDict */ const BYTE* const lowPrefix, /* always <= dst, == dst when no prefix */ const BYTE* const dictStart, /* only if dict==usingExtDict */ const size_t dictSize /* note : = 0 if noDict */ ) { if ((src == NULL) || (outputSize < 0)) { return -1; } { const BYTE* ip = (const BYTE*) src; const BYTE* const iend = ip + srcSize; BYTE* op = (BYTE*) dst; BYTE* const oend = op + outputSize; BYTE* cpy; const BYTE* const dictEnd = (dictStart == NULL) ? NULL : dictStart + dictSize; const int checkOffset = (dictSize < (int)(64 KB)); /* Set up the "end" pointers for the shortcut. */ const BYTE* const shortiend = iend - 14 /*maxLL*/ - 2 /*offset*/; const BYTE* const shortoend = oend - 14 /*maxLL*/ - 18 /*maxML*/; const BYTE* match; size_t offset; unsigned token; size_t length; DEBUGLOG(5, "LZ4_decompress_generic (srcSize:%i, dstSize:%i)", srcSize, outputSize); /* Special cases */ assert(lowPrefix <= op); if (unlikely(outputSize==0)) { /* Empty output buffer */ if (partialDecoding) return 0; return ((srcSize==1) && (*ip==0)) ? 0 : -1; } if (unlikely(srcSize==0)) { return -1; } /* LZ4_FAST_DEC_LOOP: * designed for modern OoO performance cpus, * where copying reliably 32-bytes is preferable to an unpredictable branch. * note : fast loop may show a regression for some client arm chips. */ #if LZ4_FAST_DEC_LOOP if ((oend - op) < FASTLOOP_SAFE_DISTANCE) { DEBUGLOG(6, "skip fast decode loop"); goto safe_decode; } /* Fast loop : decode sequences as long as output < oend-FASTLOOP_SAFE_DISTANCE */ while (1) { /* Main fastloop assertion: We can always wildcopy FASTLOOP_SAFE_DISTANCE */ assert(oend - op >= FASTLOOP_SAFE_DISTANCE); assert(ip < iend); token = *ip++; length = token >> ML_BITS; /* literal length */ /* decode literal length */ if (length == RUN_MASK) { size_t const addl = read_variable_length(&ip, iend-RUN_MASK, 1); if (addl == rvl_error) { goto _output_error; } length += addl; if (unlikely((uptrval)(op)+length<(uptrval)(op))) { goto _output_error; } /* overflow detection */ if (unlikely((uptrval)(ip)+length<(uptrval)(ip))) { goto _output_error; } /* overflow detection */ /* copy literals */ cpy = op+length; LZ4_STATIC_ASSERT(MFLIMIT >= WILDCOPYLENGTH); if ((cpy>oend-32) || (ip+length>iend-32)) { goto safe_literal_copy; } LZ4_wildCopy32(op, ip, cpy); ip += length; op = cpy; } else { cpy = op+length; DEBUGLOG(7, "copy %u bytes in a 16-bytes stripe", (unsigned)length); /* We don't need to check oend, since we check it once for each loop below */ if (ip > iend-(16 + 1/*max lit + offset + nextToken*/)) { goto safe_literal_copy; } /* Literals can only be <= 14, but hope compilers optimize better when copy by a register size */ LZ4_memcpy(op, ip, 16); ip += length; op = cpy; } /* get offset */ offset = LZ4_readLE16(ip); ip+=2; match = op - offset; assert(match <= op); /* overflow check */ /* get matchlength */ length = token & ML_MASK; if (length == ML_MASK) { size_t const addl = read_variable_length(&ip, iend - LASTLITERALS + 1, 0); if (addl == rvl_error) { goto _output_error; } length += addl; length += MINMATCH; if (unlikely((uptrval)(op)+length<(uptrval)op)) { goto _output_error; } /* overflow detection */ if ((checkOffset) && (unlikely(match + dictSize < lowPrefix))) { goto _output_error; } /* Error : offset outside buffers */ if (op + length >= oend - FASTLOOP_SAFE_DISTANCE) { goto safe_match_copy; } } else { length += MINMATCH; if (op + length >= oend - FASTLOOP_SAFE_DISTANCE) { goto safe_match_copy; } /* Fastpath check: skip LZ4_wildCopy32 when true */ if ((dict == withPrefix64k) || (match >= lowPrefix)) { if (offset >= 8) { assert(match >= lowPrefix); assert(match <= op); assert(op + 18 <= oend); LZ4_memcpy(op, match, 8); LZ4_memcpy(op+8, match+8, 8); LZ4_memcpy(op+16, match+16, 2); op += length; continue; } } } if (checkOffset && (unlikely(match + dictSize < lowPrefix))) { goto _output_error; } /* Error : offset outside buffers */ /* match starting within external dictionary */ if ((dict==usingExtDict) && (match < lowPrefix)) { assert(dictEnd != NULL); if (unlikely(op+length > oend-LASTLITERALS)) { if (partialDecoding) { DEBUGLOG(7, "partialDecoding: dictionary match, close to dstEnd"); length = MIN(length, (size_t)(oend-op)); } else { goto _output_error; /* end-of-block condition violated */ } } if (length <= (size_t)(lowPrefix-match)) { /* match fits entirely within external dictionary : just copy */ LZ4_memmove(op, dictEnd - (lowPrefix-match), length); op += length; } else { /* match stretches into both external dictionary and current block */ size_t const copySize = (size_t)(lowPrefix - match); size_t const restSize = length - copySize; LZ4_memcpy(op, dictEnd - copySize, copySize); op += copySize; if (restSize > (size_t)(op - lowPrefix)) { /* overlap copy */ BYTE* const endOfMatch = op + restSize; const BYTE* copyFrom = lowPrefix; while (op < endOfMatch) { *op++ = *copyFrom++; } } else { LZ4_memcpy(op, lowPrefix, restSize); op += restSize; } } continue; } /* copy match within block */ cpy = op + length; assert((op <= oend) && (oend-op >= 32)); if (unlikely(offset<16)) { LZ4_memcpy_using_offset(op, match, cpy, offset); } else { LZ4_wildCopy32(op, match, cpy); } op = cpy; /* wildcopy correction */ } safe_decode: #endif /* Main Loop : decode remaining sequences where output < FASTLOOP_SAFE_DISTANCE */ while (1) { assert(ip < iend); token = *ip++; length = token >> ML_BITS; /* literal length */ /* A two-stage shortcut for the most common case: * 1) If the literal length is 0..14, and there is enough space, * enter the shortcut and copy 16 bytes on behalf of the literals * (in the fast mode, only 8 bytes can be safely copied this way). * 2) Further if the match length is 4..18, copy 18 bytes in a similar * manner; but we ensure that there's enough space in the output for * those 18 bytes earlier, upon entering the shortcut (in other words, * there is a combined check for both stages). */ if ( (length != RUN_MASK) /* strictly "less than" on input, to re-enter the loop with at least one byte */ && likely((ip < shortiend) & (op <= shortoend)) ) { /* Copy the literals */ LZ4_memcpy(op, ip, 16); op += length; ip += length; /* The second stage: prepare for match copying, decode full info. * If it doesn't work out, the info won't be wasted. */ length = token & ML_MASK; /* match length */ offset = LZ4_readLE16(ip); ip += 2; match = op - offset; assert(match <= op); /* check overflow */ /* Do not deal with overlapping matches. */ if ( (length != ML_MASK) && (offset >= 8) && (dict==withPrefix64k || match >= lowPrefix) ) { /* Copy the match. */ LZ4_memcpy(op + 0, match + 0, 8); LZ4_memcpy(op + 8, match + 8, 8); LZ4_memcpy(op +16, match +16, 2); op += length + MINMATCH; /* Both stages worked, load the next token. */ continue; } /* The second stage didn't work out, but the info is ready. * Propel it right to the point of match copying. */ goto _copy_match; } /* decode literal length */ if (length == RUN_MASK) { size_t const addl = read_variable_length(&ip, iend-RUN_MASK, 1); if (addl == rvl_error) { goto _output_error; } length += addl; if (unlikely((uptrval)(op)+length<(uptrval)(op))) { goto _output_error; } /* overflow detection */ if (unlikely((uptrval)(ip)+length<(uptrval)(ip))) { goto _output_error; } /* overflow detection */ } /* copy literals */ cpy = op+length; #if LZ4_FAST_DEC_LOOP safe_literal_copy: #endif LZ4_STATIC_ASSERT(MFLIMIT >= WILDCOPYLENGTH); if ((cpy>oend-MFLIMIT) || (ip+length>iend-(2+1+LASTLITERALS))) { /* We've either hit the input parsing restriction or the output parsing restriction. * In the normal scenario, decoding a full block, it must be the last sequence, * otherwise it's an error (invalid input or dimensions). * In partialDecoding scenario, it's necessary to ensure there is no buffer overflow. */ if (partialDecoding) { /* Since we are partial decoding we may be in this block because of the output parsing * restriction, which is not valid since the output buffer is allowed to be undersized. */ DEBUGLOG(7, "partialDecoding: copying literals, close to input or output end") DEBUGLOG(7, "partialDecoding: literal length = %u", (unsigned)length); DEBUGLOG(7, "partialDecoding: remaining space in dstBuffer : %i", (int)(oend - op)); DEBUGLOG(7, "partialDecoding: remaining space in srcBuffer : %i", (int)(iend - ip)); /* Finishing in the middle of a literals segment, * due to lack of input. */ if (ip+length > iend) { length = (size_t)(iend-ip); cpy = op + length; } /* Finishing in the middle of a literals segment, * due to lack of output space. */ if (cpy > oend) { cpy = oend; assert(op<=oend); length = (size_t)(oend-op); } } else { /* We must be on the last sequence (or invalid) because of the parsing limitations * so check that we exactly consume the input and don't overrun the output buffer. */ if ((ip+length != iend) || (cpy > oend)) { DEBUGLOG(6, "should have been last run of literals") DEBUGLOG(6, "ip(%p) + length(%i) = %p != iend (%p)", ip, (int)length, ip+length, iend); DEBUGLOG(6, "or cpy(%p) > oend(%p)", cpy, oend); goto _output_error; } } LZ4_memmove(op, ip, length); /* supports overlapping memory regions, for in-place decompression scenarios */ ip += length; op += length; /* Necessarily EOF when !partialDecoding. * When partialDecoding, it is EOF if we've either * filled the output buffer or * can't proceed with reading an offset for following match. */ if (!partialDecoding || (cpy == oend) || (ip >= (iend-2))) { break; } } else { LZ4_wildCopy8(op, ip, cpy); /* can overwrite up to 8 bytes beyond cpy */ ip += length; op = cpy; } /* get offset */ offset = LZ4_readLE16(ip); ip+=2; match = op - offset; /* get matchlength */ length = token & ML_MASK; _copy_match: if (length == ML_MASK) { size_t const addl = read_variable_length(&ip, iend - LASTLITERALS + 1, 0); if (addl == rvl_error) { goto _output_error; } length += addl; if (unlikely((uptrval)(op)+length<(uptrval)op)) goto _output_error; /* overflow detection */ } length += MINMATCH; #if LZ4_FAST_DEC_LOOP safe_match_copy: #endif if ((checkOffset) && (unlikely(match + dictSize < lowPrefix))) goto _output_error; /* Error : offset outside buffers */ /* match starting within external dictionary */ if ((dict==usingExtDict) && (match < lowPrefix)) { assert(dictEnd != NULL); if (unlikely(op+length > oend-LASTLITERALS)) { if (partialDecoding) length = MIN(length, (size_t)(oend-op)); else goto _output_error; /* doesn't respect parsing restriction */ } if (length <= (size_t)(lowPrefix-match)) { /* match fits entirely within external dictionary : just copy */ LZ4_memmove(op, dictEnd - (lowPrefix-match), length); op += length; } else { /* match stretches into both external dictionary and current block */ size_t const copySize = (size_t)(lowPrefix - match); size_t const restSize = length - copySize; LZ4_memcpy(op, dictEnd - copySize, copySize); op += copySize; if (restSize > (size_t)(op - lowPrefix)) { /* overlap copy */ BYTE* const endOfMatch = op + restSize; const BYTE* copyFrom = lowPrefix; while (op < endOfMatch) *op++ = *copyFrom++; } else { LZ4_memcpy(op, lowPrefix, restSize); op += restSize; } } continue; } assert(match >= lowPrefix); /* copy match within block */ cpy = op + length; /* partialDecoding : may end anywhere within the block */ assert(op<=oend); if (partialDecoding && (cpy > oend-MATCH_SAFEGUARD_DISTANCE)) { size_t const mlen = MIN(length, (size_t)(oend-op)); const BYTE* const matchEnd = match + mlen; BYTE* const copyEnd = op + mlen; if (matchEnd > op) { /* overlap copy */ while (op < copyEnd) { *op++ = *match++; } } else { LZ4_memcpy(op, match, mlen); } op = copyEnd; if (op == oend) { break; } continue; } if (unlikely(offset<8)) { LZ4_write32(op, 0); /* silence msan warning when offset==0 */ op[0] = match[0]; op[1] = match[1]; op[2] = match[2]; op[3] = match[3]; match += inc32table[offset]; LZ4_memcpy(op+4, match, 4); match -= dec64table[offset]; } else { LZ4_memcpy(op, match, 8); match += 8; } op += 8; if (unlikely(cpy > oend-MATCH_SAFEGUARD_DISTANCE)) { BYTE* const oCopyLimit = oend - (WILDCOPYLENGTH-1); if (cpy > oend-LASTLITERALS) { goto _output_error; } /* Error : last LASTLITERALS bytes must be literals (uncompressed) */ if (op < oCopyLimit) { LZ4_wildCopy8(op, match, oCopyLimit); match += oCopyLimit - op; op = oCopyLimit; } while (op < cpy) { *op++ = *match++; } } else { LZ4_memcpy(op, match, 8); if (length > 16) { LZ4_wildCopy8(op+8, match+8, cpy); } } op = cpy; /* wildcopy correction */ } /* end of decoding */ DEBUGLOG(5, "decoded %i bytes", (int) (((char*)op)-dst)); return (int) (((char*)op)-dst); /* Nb of output bytes decoded */ /* Overflow error detected */ _output_error: return (int) (-(((const char*)ip)-src))-1; } } /*===== Instantiate the API decoding functions. =====*/ LZ4_FORCE_O2 int LZ4_decompress_safe(const char* source, char* dest, int compressedSize, int maxDecompressedSize) { return LZ4_decompress_generic(source, dest, compressedSize, maxDecompressedSize, decode_full_block, noDict, (BYTE*)dest, NULL, 0); } LZ4_FORCE_O2 int LZ4_decompress_safe_partial(const char* src, char* dst, int compressedSize, int targetOutputSize, int dstCapacity) { dstCapacity = MIN(targetOutputSize, dstCapacity); return LZ4_decompress_generic(src, dst, compressedSize, dstCapacity, partial_decode, noDict, (BYTE*)dst, NULL, 0); } LZ4_FORCE_O2 int LZ4_decompress_fast(const char* source, char* dest, int originalSize) { DEBUGLOG(5, "LZ4_decompress_fast"); return LZ4_decompress_unsafe_generic( (const BYTE*)source, (BYTE*)dest, originalSize, 0, NULL, 0); } /*===== Instantiate a few more decoding cases, used more than once. =====*/ LZ4_FORCE_O2 /* Exported, an obsolete API function. */ int LZ4_decompress_safe_withPrefix64k(const char* source, char* dest, int compressedSize, int maxOutputSize) { return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, decode_full_block, withPrefix64k, (BYTE*)dest - 64 KB, NULL, 0); } LZ4_FORCE_O2 static int LZ4_decompress_safe_partial_withPrefix64k(const char* source, char* dest, int compressedSize, int targetOutputSize, int dstCapacity) { dstCapacity = MIN(targetOutputSize, dstCapacity); return LZ4_decompress_generic(source, dest, compressedSize, dstCapacity, partial_decode, withPrefix64k, (BYTE*)dest - 64 KB, NULL, 0); } /* Another obsolete API function, paired with the previous one. */ int LZ4_decompress_fast_withPrefix64k(const char* source, char* dest, int originalSize) { return LZ4_decompress_unsafe_generic( (const BYTE*)source, (BYTE*)dest, originalSize, 64 KB, NULL, 0); } LZ4_FORCE_O2 static int LZ4_decompress_safe_withSmallPrefix(const char* source, char* dest, int compressedSize, int maxOutputSize, size_t prefixSize) { return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, decode_full_block, noDict, (BYTE*)dest-prefixSize, NULL, 0); } LZ4_FORCE_O2 static int LZ4_decompress_safe_partial_withSmallPrefix(const char* source, char* dest, int compressedSize, int targetOutputSize, int dstCapacity, size_t prefixSize) { dstCapacity = MIN(targetOutputSize, dstCapacity); return LZ4_decompress_generic(source, dest, compressedSize, dstCapacity, partial_decode, noDict, (BYTE*)dest-prefixSize, NULL, 0); } LZ4_FORCE_O2 int LZ4_decompress_safe_forceExtDict(const char* source, char* dest, int compressedSize, int maxOutputSize, const void* dictStart, size_t dictSize) { return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, decode_full_block, usingExtDict, (BYTE*)dest, (const BYTE*)dictStart, dictSize); } LZ4_FORCE_O2 int LZ4_decompress_safe_partial_forceExtDict(const char* source, char* dest, int compressedSize, int targetOutputSize, int dstCapacity, const void* dictStart, size_t dictSize) { dstCapacity = MIN(targetOutputSize, dstCapacity); return LZ4_decompress_generic(source, dest, compressedSize, dstCapacity, partial_decode, usingExtDict, (BYTE*)dest, (const BYTE*)dictStart, dictSize); } LZ4_FORCE_O2 static int LZ4_decompress_fast_extDict(const char* source, char* dest, int originalSize, const void* dictStart, size_t dictSize) { return LZ4_decompress_unsafe_generic( (const BYTE*)source, (BYTE*)dest, originalSize, 0, (const BYTE*)dictStart, dictSize); } /* The "double dictionary" mode, for use with e.g. ring buffers: the first part * of the dictionary is passed as prefix, and the second via dictStart + dictSize. * These routines are used only once, in LZ4_decompress_*_continue(). */ LZ4_FORCE_INLINE int LZ4_decompress_safe_doubleDict(const char* source, char* dest, int compressedSize, int maxOutputSize, size_t prefixSize, const void* dictStart, size_t dictSize) { return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, decode_full_block, usingExtDict, (BYTE*)dest-prefixSize, (const BYTE*)dictStart, dictSize); } /*===== streaming decompression functions =====*/ #if !defined(LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION) LZ4_streamDecode_t* LZ4_createStreamDecode(void) { LZ4_STATIC_ASSERT(sizeof(LZ4_streamDecode_t) >= sizeof(LZ4_streamDecode_t_internal)); return (LZ4_streamDecode_t*) ALLOC_AND_ZERO(sizeof(LZ4_streamDecode_t)); } int LZ4_freeStreamDecode (LZ4_streamDecode_t* LZ4_stream) { if (LZ4_stream == NULL) { return 0; } /* support free on NULL */ FREEMEM(LZ4_stream); return 0; } #endif /*! LZ4_setStreamDecode() : * Use this function to instruct where to find the dictionary. * This function is not necessary if previous data is still available where it was decoded. * Loading a size of 0 is allowed (same effect as no dictionary). * @return : 1 if OK, 0 if error */ int LZ4_setStreamDecode (LZ4_streamDecode_t* LZ4_streamDecode, const char* dictionary, int dictSize) { LZ4_streamDecode_t_internal* lz4sd = &LZ4_streamDecode->internal_donotuse; lz4sd->prefixSize = (size_t)dictSize; if (dictSize) { assert(dictionary != NULL); lz4sd->prefixEnd = (const BYTE*) dictionary + dictSize; } else { lz4sd->prefixEnd = (const BYTE*) dictionary; } lz4sd->externalDict = NULL; lz4sd->extDictSize = 0; return 1; } /*! LZ4_decoderRingBufferSize() : * when setting a ring buffer for streaming decompression (optional scenario), * provides the minimum size of this ring buffer * to be compatible with any source respecting maxBlockSize condition. * Note : in a ring buffer scenario, * blocks are presumed decompressed next to each other. * When not enough space remains for next block (remainingSize < maxBlockSize), * decoding resumes from beginning of ring buffer. * @return : minimum ring buffer size, * or 0 if there is an error (invalid maxBlockSize). */ int LZ4_decoderRingBufferSize(int maxBlockSize) { if (maxBlockSize < 0) return 0; if (maxBlockSize > LZ4_MAX_INPUT_SIZE) return 0; if (maxBlockSize < 16) maxBlockSize = 16; return LZ4_DECODER_RING_BUFFER_SIZE(maxBlockSize); } /* *_continue() : These decoding functions allow decompression of multiple blocks in "streaming" mode. Previously decoded blocks must still be available at the memory position where they were decoded. If it's not possible, save the relevant part of decoded data into a safe buffer, and indicate where it stands using LZ4_setStreamDecode() */ LZ4_FORCE_O2 int LZ4_decompress_safe_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int compressedSize, int maxOutputSize) { LZ4_streamDecode_t_internal* lz4sd = &LZ4_streamDecode->internal_donotuse; int result; if (lz4sd->prefixSize == 0) { /* The first call, no dictionary yet. */ assert(lz4sd->extDictSize == 0); result = LZ4_decompress_safe(source, dest, compressedSize, maxOutputSize); if (result <= 0) return result; lz4sd->prefixSize = (size_t)result; lz4sd->prefixEnd = (BYTE*)dest + result; } else if (lz4sd->prefixEnd == (BYTE*)dest) { /* They're rolling the current segment. */ if (lz4sd->prefixSize >= 64 KB - 1) result = LZ4_decompress_safe_withPrefix64k(source, dest, compressedSize, maxOutputSize); else if (lz4sd->extDictSize == 0) result = LZ4_decompress_safe_withSmallPrefix(source, dest, compressedSize, maxOutputSize, lz4sd->prefixSize); else result = LZ4_decompress_safe_doubleDict(source, dest, compressedSize, maxOutputSize, lz4sd->prefixSize, lz4sd->externalDict, lz4sd->extDictSize); if (result <= 0) return result; lz4sd->prefixSize += (size_t)result; lz4sd->prefixEnd += result; } else { /* The buffer wraps around, or they're switching to another buffer. */ lz4sd->extDictSize = lz4sd->prefixSize; lz4sd->externalDict = lz4sd->prefixEnd - lz4sd->extDictSize; result = LZ4_decompress_safe_forceExtDict(source, dest, compressedSize, maxOutputSize, lz4sd->externalDict, lz4sd->extDictSize); if (result <= 0) return result; lz4sd->prefixSize = (size_t)result; lz4sd->prefixEnd = (BYTE*)dest + result; } return result; } LZ4_FORCE_O2 int LZ4_decompress_fast_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int originalSize) { LZ4_streamDecode_t_internal* const lz4sd = (assert(LZ4_streamDecode!=NULL), &LZ4_streamDecode->internal_donotuse); int result; DEBUGLOG(5, "LZ4_decompress_fast_continue (toDecodeSize=%i)", originalSize); assert(originalSize >= 0); if (lz4sd->prefixSize == 0) { DEBUGLOG(5, "first invocation : no prefix nor extDict"); assert(lz4sd->extDictSize == 0); result = LZ4_decompress_fast(source, dest, originalSize); if (result <= 0) return result; lz4sd->prefixSize = (size_t)originalSize; lz4sd->prefixEnd = (BYTE*)dest + originalSize; } else if (lz4sd->prefixEnd == (BYTE*)dest) { DEBUGLOG(5, "continue using existing prefix"); result = LZ4_decompress_unsafe_generic( (const BYTE*)source, (BYTE*)dest, originalSize, lz4sd->prefixSize, lz4sd->externalDict, lz4sd->extDictSize); if (result <= 0) return result; lz4sd->prefixSize += (size_t)originalSize; lz4sd->prefixEnd += originalSize; } else { DEBUGLOG(5, "prefix becomes extDict"); lz4sd->extDictSize = lz4sd->prefixSize; lz4sd->externalDict = lz4sd->prefixEnd - lz4sd->extDictSize; result = LZ4_decompress_fast_extDict(source, dest, originalSize, lz4sd->externalDict, lz4sd->extDictSize); if (result <= 0) return result; lz4sd->prefixSize = (size_t)originalSize; lz4sd->prefixEnd = (BYTE*)dest + originalSize; } return result; } /* Advanced decoding functions : *_usingDict() : These decoding functions work the same as "_continue" ones, the dictionary must be explicitly provided within parameters */ int LZ4_decompress_safe_usingDict(const char* source, char* dest, int compressedSize, int maxOutputSize, const char* dictStart, int dictSize) { if (dictSize==0) return LZ4_decompress_safe(source, dest, compressedSize, maxOutputSize); if (dictStart+dictSize == dest) { if (dictSize >= 64 KB - 1) { return LZ4_decompress_safe_withPrefix64k(source, dest, compressedSize, maxOutputSize); } assert(dictSize >= 0); return LZ4_decompress_safe_withSmallPrefix(source, dest, compressedSize, maxOutputSize, (size_t)dictSize); } assert(dictSize >= 0); return LZ4_decompress_safe_forceExtDict(source, dest, compressedSize, maxOutputSize, dictStart, (size_t)dictSize); } int LZ4_decompress_safe_partial_usingDict(const char* source, char* dest, int compressedSize, int targetOutputSize, int dstCapacity, const char* dictStart, int dictSize) { if (dictSize==0) return LZ4_decompress_safe_partial(source, dest, compressedSize, targetOutputSize, dstCapacity); if (dictStart+dictSize == dest) { if (dictSize >= 64 KB - 1) { return LZ4_decompress_safe_partial_withPrefix64k(source, dest, compressedSize, targetOutputSize, dstCapacity); } assert(dictSize >= 0); return LZ4_decompress_safe_partial_withSmallPrefix(source, dest, compressedSize, targetOutputSize, dstCapacity, (size_t)dictSize); } assert(dictSize >= 0); return LZ4_decompress_safe_partial_forceExtDict(source, dest, compressedSize, targetOutputSize, dstCapacity, dictStart, (size_t)dictSize); } int LZ4_decompress_fast_usingDict(const char* source, char* dest, int originalSize, const char* dictStart, int dictSize) { if (dictSize==0 || dictStart+dictSize == dest) return LZ4_decompress_unsafe_generic( (const BYTE*)source, (BYTE*)dest, originalSize, (size_t)dictSize, NULL, 0); assert(dictSize >= 0); return LZ4_decompress_fast_extDict(source, dest, originalSize, dictStart, (size_t)dictSize); } /*=************************************************* * Obsolete Functions ***************************************************/ /* obsolete compression functions */ int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize) { return LZ4_compress_default(source, dest, inputSize, maxOutputSize); } int LZ4_compress(const char* src, char* dest, int srcSize) { return LZ4_compress_default(src, dest, srcSize, LZ4_compressBound(srcSize)); } int LZ4_compress_limitedOutput_withState (void* state, const char* src, char* dst, int srcSize, int dstSize) { return LZ4_compress_fast_extState(state, src, dst, srcSize, dstSize, 1); } int LZ4_compress_withState (void* state, const char* src, char* dst, int srcSize) { return LZ4_compress_fast_extState(state, src, dst, srcSize, LZ4_compressBound(srcSize), 1); } int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* src, char* dst, int srcSize, int dstCapacity) { return LZ4_compress_fast_continue(LZ4_stream, src, dst, srcSize, dstCapacity, 1); } int LZ4_compress_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize) { return LZ4_compress_fast_continue(LZ4_stream, source, dest, inputSize, LZ4_compressBound(inputSize), 1); } /* These decompression functions are deprecated and should no longer be used. They are only provided here for compatibility with older user programs. - LZ4_uncompress is totally equivalent to LZ4_decompress_fast - LZ4_uncompress_unknownOutputSize is totally equivalent to LZ4_decompress_safe */ int LZ4_uncompress (const char* source, char* dest, int outputSize) { return LZ4_decompress_fast(source, dest, outputSize); } int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int maxOutputSize) { return LZ4_decompress_safe(source, dest, isize, maxOutputSize); } /* Obsolete Streaming functions */ int LZ4_sizeofStreamState(void) { return sizeof(LZ4_stream_t); } int LZ4_resetStreamState(void* state, char* inputBuffer) { (void)inputBuffer; LZ4_resetStream((LZ4_stream_t*)state); return 0; } #if !defined(LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION) void* LZ4_create (char* inputBuffer) { (void)inputBuffer; return LZ4_createStream(); } #endif char* LZ4_slideInputBuffer (void* state) { /* avoid const char * -> char * conversion warning */ return (char *)(uptrval)((LZ4_stream_t*)state)->internal_donotuse.dictionary; } #endif /* LZ4_COMMONDEFS_ONLY */ zmat-0.9.9/src/lz4/lz4hc.c0000644000175200007730000021076114515254731015436 0ustar rlaboissrlaboiss/* LZ4 HC - High Compression Mode of LZ4 Copyright (C) 2011-2020, Yann Collet. BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You can contact the author at : - LZ4 source repository : https://github.com/lz4/lz4 - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c */ /* note : lz4hc is not an independent module, it requires lz4.h/lz4.c for proper compilation */ /* ************************************* * Tuning Parameter ***************************************/ /*! HEAPMODE : * Select how default compression function will allocate workplace memory, * in stack (0:fastest), or in heap (1:requires malloc()). * Since workplace is rather large, heap mode is recommended. **/ #ifndef LZ4HC_HEAPMODE # define LZ4HC_HEAPMODE 1 #endif /*=== Dependency ===*/ #define LZ4_HC_STATIC_LINKING_ONLY #include "lz4hc.h" /*=== Common definitions ===*/ #if defined(__GNUC__) # pragma GCC diagnostic ignored "-Wunused-function" #endif #if defined (__clang__) # pragma clang diagnostic ignored "-Wunused-function" #endif #define LZ4_COMMONDEFS_ONLY #ifndef LZ4_SRC_INCLUDED #include "lz4.c" /* LZ4_count, constants, mem */ #endif /*=== Enums ===*/ typedef enum { noDictCtx, usingDictCtxHc } dictCtx_directive; /*=== Constants ===*/ #define OPTIMAL_ML (int)((ML_MASK-1)+MINMATCH) #define LZ4_OPT_NUM (1<<12) /*=== Macros ===*/ #define MIN(a,b) ( (a) < (b) ? (a) : (b) ) #define MAX(a,b) ( (a) > (b) ? (a) : (b) ) #define HASH_FUNCTION(i) (((i) * 2654435761U) >> ((MINMATCH*8)-LZ4HC_HASH_LOG)) #define DELTANEXTMAXD(p) chainTable[(p) & LZ4HC_MAXD_MASK] /* flexible, LZ4HC_MAXD dependent */ #define DELTANEXTU16(table, pos) table[(U16)(pos)] /* faster */ /* Make fields passed to, and updated by LZ4HC_encodeSequence explicit */ #define UPDATABLE(ip, op, anchor) &ip, &op, &anchor static U32 LZ4HC_hashPtr(const void* ptr) { return HASH_FUNCTION(LZ4_read32(ptr)); } /************************************** * HC Compression **************************************/ static void LZ4HC_clearTables (LZ4HC_CCtx_internal* hc4) { MEM_INIT(hc4->hashTable, 0, sizeof(hc4->hashTable)); MEM_INIT(hc4->chainTable, 0xFF, sizeof(hc4->chainTable)); } static void LZ4HC_init_internal (LZ4HC_CCtx_internal* hc4, const BYTE* start) { size_t const bufferSize = (size_t)(hc4->end - hc4->prefixStart); size_t newStartingOffset = bufferSize + hc4->dictLimit; assert(newStartingOffset >= bufferSize); /* check overflow */ if (newStartingOffset > 1 GB) { LZ4HC_clearTables(hc4); newStartingOffset = 0; } newStartingOffset += 64 KB; hc4->nextToUpdate = (U32)newStartingOffset; hc4->prefixStart = start; hc4->end = start; hc4->dictStart = start; hc4->dictLimit = (U32)newStartingOffset; hc4->lowLimit = (U32)newStartingOffset; } /* Update chains up to ip (excluded) */ LZ4_FORCE_INLINE void LZ4HC_Insert (LZ4HC_CCtx_internal* hc4, const BYTE* ip) { U16* const chainTable = hc4->chainTable; U32* const hashTable = hc4->hashTable; const BYTE* const prefixPtr = hc4->prefixStart; U32 const prefixIdx = hc4->dictLimit; U32 const target = (U32)(ip - prefixPtr) + prefixIdx; U32 idx = hc4->nextToUpdate; assert(ip >= prefixPtr); assert(target >= prefixIdx); while (idx < target) { U32 const h = LZ4HC_hashPtr(prefixPtr+idx-prefixIdx); size_t delta = idx - hashTable[h]; if (delta>LZ4_DISTANCE_MAX) delta = LZ4_DISTANCE_MAX; DELTANEXTU16(chainTable, idx) = (U16)delta; hashTable[h] = idx; idx++; } hc4->nextToUpdate = target; } /** LZ4HC_countBack() : * @return : negative value, nb of common bytes before ip/match */ LZ4_FORCE_INLINE int LZ4HC_countBack(const BYTE* const ip, const BYTE* const match, const BYTE* const iMin, const BYTE* const mMin) { int back = 0; int const min = (int)MAX(iMin - ip, mMin - match); assert(min <= 0); assert(ip >= iMin); assert((size_t)(ip-iMin) < (1U<<31)); assert(match >= mMin); assert((size_t)(match - mMin) < (1U<<31)); while ( (back > min) && (ip[back-1] == match[back-1]) ) back--; return back; } #if defined(_MSC_VER) # define LZ4HC_rotl32(x,r) _rotl(x,r) #else # define LZ4HC_rotl32(x,r) ((x << r) | (x >> (32 - r))) #endif static U32 LZ4HC_rotatePattern(size_t const rotate, U32 const pattern) { size_t const bitsToRotate = (rotate & (sizeof(pattern) - 1)) << 3; if (bitsToRotate == 0) return pattern; return LZ4HC_rotl32(pattern, (int)bitsToRotate); } /* LZ4HC_countPattern() : * pattern32 must be a sample of repetitive pattern of length 1, 2 or 4 (but not 3!) */ static unsigned LZ4HC_countPattern(const BYTE* ip, const BYTE* const iEnd, U32 const pattern32) { const BYTE* const iStart = ip; reg_t const pattern = (sizeof(pattern)==8) ? (reg_t)pattern32 + (((reg_t)pattern32) << (sizeof(pattern)*4)) : pattern32; while (likely(ip < iEnd-(sizeof(pattern)-1))) { reg_t const diff = LZ4_read_ARCH(ip) ^ pattern; if (!diff) { ip+=sizeof(pattern); continue; } ip += LZ4_NbCommonBytes(diff); return (unsigned)(ip - iStart); } if (LZ4_isLittleEndian()) { reg_t patternByte = pattern; while ((ip>= 8; } } else { /* big endian */ U32 bitOffset = (sizeof(pattern)*8) - 8; while (ip < iEnd) { BYTE const byte = (BYTE)(pattern >> bitOffset); if (*ip != byte) break; ip ++; bitOffset -= 8; } } return (unsigned)(ip - iStart); } /* LZ4HC_reverseCountPattern() : * pattern must be a sample of repetitive pattern of length 1, 2 or 4 (but not 3!) * read using natural platform endianness */ static unsigned LZ4HC_reverseCountPattern(const BYTE* ip, const BYTE* const iLow, U32 pattern) { const BYTE* const iStart = ip; while (likely(ip >= iLow+4)) { if (LZ4_read32(ip-4) != pattern) break; ip -= 4; } { const BYTE* bytePtr = (const BYTE*)(&pattern) + 3; /* works for any endianness */ while (likely(ip>iLow)) { if (ip[-1] != *bytePtr) break; ip--; bytePtr--; } } return (unsigned)(iStart - ip); } /* LZ4HC_protectDictEnd() : * Checks if the match is in the last 3 bytes of the dictionary, so reading the * 4 byte MINMATCH would overflow. * @returns true if the match index is okay. */ static int LZ4HC_protectDictEnd(U32 const dictLimit, U32 const matchIndex) { return ((U32)((dictLimit - 1) - matchIndex) >= 3); } typedef enum { rep_untested, rep_not, rep_confirmed } repeat_state_e; typedef enum { favorCompressionRatio=0, favorDecompressionSpeed } HCfavor_e; LZ4_FORCE_INLINE int LZ4HC_InsertAndGetWiderMatch ( LZ4HC_CCtx_internal* const hc4, const BYTE* const ip, const BYTE* const iLowLimit, const BYTE* const iHighLimit, int longest, const BYTE** matchpos, const BYTE** startpos, const int maxNbAttempts, const int patternAnalysis, const int chainSwap, const dictCtx_directive dict, const HCfavor_e favorDecSpeed) { U16* const chainTable = hc4->chainTable; U32* const HashTable = hc4->hashTable; const LZ4HC_CCtx_internal * const dictCtx = hc4->dictCtx; const BYTE* const prefixPtr = hc4->prefixStart; const U32 prefixIdx = hc4->dictLimit; const U32 ipIndex = (U32)(ip - prefixPtr) + prefixIdx; const int withinStartDistance = (hc4->lowLimit + (LZ4_DISTANCE_MAX + 1) > ipIndex); const U32 lowestMatchIndex = (withinStartDistance) ? hc4->lowLimit : ipIndex - LZ4_DISTANCE_MAX; const BYTE* const dictStart = hc4->dictStart; const U32 dictIdx = hc4->lowLimit; const BYTE* const dictEnd = dictStart + prefixIdx - dictIdx; int const lookBackLength = (int)(ip-iLowLimit); int nbAttempts = maxNbAttempts; U32 matchChainPos = 0; U32 const pattern = LZ4_read32(ip); U32 matchIndex; repeat_state_e repeat = rep_untested; size_t srcPatternLength = 0; DEBUGLOG(7, "LZ4HC_InsertAndGetWiderMatch"); /* First Match */ LZ4HC_Insert(hc4, ip); matchIndex = HashTable[LZ4HC_hashPtr(ip)]; DEBUGLOG(7, "First match at index %u / %u (lowestMatchIndex)", matchIndex, lowestMatchIndex); while ((matchIndex>=lowestMatchIndex) && (nbAttempts>0)) { int matchLength=0; nbAttempts--; assert(matchIndex < ipIndex); if (favorDecSpeed && (ipIndex - matchIndex < 8)) { /* do nothing */ } else if (matchIndex >= prefixIdx) { /* within current Prefix */ const BYTE* const matchPtr = prefixPtr + matchIndex - prefixIdx; assert(matchPtr < ip); assert(longest >= 1); if (LZ4_read16(iLowLimit + longest - 1) == LZ4_read16(matchPtr - lookBackLength + longest - 1)) { if (LZ4_read32(matchPtr) == pattern) { int const back = lookBackLength ? LZ4HC_countBack(ip, matchPtr, iLowLimit, prefixPtr) : 0; matchLength = MINMATCH + (int)LZ4_count(ip+MINMATCH, matchPtr+MINMATCH, iHighLimit); matchLength -= back; if (matchLength > longest) { longest = matchLength; *matchpos = matchPtr + back; *startpos = ip + back; } } } } else { /* lowestMatchIndex <= matchIndex < dictLimit */ const BYTE* const matchPtr = dictStart + (matchIndex - dictIdx); assert(matchIndex >= dictIdx); if ( likely(matchIndex <= prefixIdx - 4) && (LZ4_read32(matchPtr) == pattern) ) { int back = 0; const BYTE* vLimit = ip + (prefixIdx - matchIndex); if (vLimit > iHighLimit) vLimit = iHighLimit; matchLength = (int)LZ4_count(ip+MINMATCH, matchPtr+MINMATCH, vLimit) + MINMATCH; if ((ip+matchLength == vLimit) && (vLimit < iHighLimit)) matchLength += LZ4_count(ip+matchLength, prefixPtr, iHighLimit); back = lookBackLength ? LZ4HC_countBack(ip, matchPtr, iLowLimit, dictStart) : 0; matchLength -= back; if (matchLength > longest) { longest = matchLength; *matchpos = prefixPtr - prefixIdx + matchIndex + back; /* virtual pos, relative to ip, to retrieve offset */ *startpos = ip + back; } } } if (chainSwap && matchLength==longest) { /* better match => select a better chain */ assert(lookBackLength==0); /* search forward only */ if (matchIndex + (U32)longest <= ipIndex) { int const kTrigger = 4; U32 distanceToNextMatch = 1; int const end = longest - MINMATCH + 1; int step = 1; int accel = 1 << kTrigger; int pos; for (pos = 0; pos < end; pos += step) { U32 const candidateDist = DELTANEXTU16(chainTable, matchIndex + (U32)pos); step = (accel++ >> kTrigger); if (candidateDist > distanceToNextMatch) { distanceToNextMatch = candidateDist; matchChainPos = (U32)pos; accel = 1 << kTrigger; } } if (distanceToNextMatch > 1) { if (distanceToNextMatch > matchIndex) break; /* avoid overflow */ matchIndex -= distanceToNextMatch; continue; } } } { U32 const distNextMatch = DELTANEXTU16(chainTable, matchIndex); if (patternAnalysis && distNextMatch==1 && matchChainPos==0) { U32 const matchCandidateIdx = matchIndex-1; /* may be a repeated pattern */ if (repeat == rep_untested) { if ( ((pattern & 0xFFFF) == (pattern >> 16)) & ((pattern & 0xFF) == (pattern >> 24)) ) { repeat = rep_confirmed; srcPatternLength = LZ4HC_countPattern(ip+sizeof(pattern), iHighLimit, pattern) + sizeof(pattern); } else { repeat = rep_not; } } if ( (repeat == rep_confirmed) && (matchCandidateIdx >= lowestMatchIndex) && LZ4HC_protectDictEnd(prefixIdx, matchCandidateIdx) ) { const int extDict = matchCandidateIdx < prefixIdx; const BYTE* const matchPtr = (extDict ? dictStart - dictIdx : prefixPtr - prefixIdx) + matchCandidateIdx; if (LZ4_read32(matchPtr) == pattern) { /* good candidate */ const BYTE* const iLimit = extDict ? dictEnd : iHighLimit; size_t forwardPatternLength = LZ4HC_countPattern(matchPtr+sizeof(pattern), iLimit, pattern) + sizeof(pattern); if (extDict && matchPtr + forwardPatternLength == iLimit) { U32 const rotatedPattern = LZ4HC_rotatePattern(forwardPatternLength, pattern); forwardPatternLength += LZ4HC_countPattern(prefixPtr, iHighLimit, rotatedPattern); } { const BYTE* const lowestMatchPtr = extDict ? dictStart : prefixPtr; size_t backLength = LZ4HC_reverseCountPattern(matchPtr, lowestMatchPtr, pattern); size_t currentSegmentLength; if (!extDict && matchPtr - backLength == prefixPtr && dictIdx < prefixIdx) { U32 const rotatedPattern = LZ4HC_rotatePattern((U32)(-(int)backLength), pattern); backLength += LZ4HC_reverseCountPattern(dictEnd, dictStart, rotatedPattern); } /* Limit backLength not go further than lowestMatchIndex */ backLength = matchCandidateIdx - MAX(matchCandidateIdx - (U32)backLength, lowestMatchIndex); assert(matchCandidateIdx - backLength >= lowestMatchIndex); currentSegmentLength = backLength + forwardPatternLength; /* Adjust to end of pattern if the source pattern fits, otherwise the beginning of the pattern */ if ( (currentSegmentLength >= srcPatternLength) /* current pattern segment large enough to contain full srcPatternLength */ && (forwardPatternLength <= srcPatternLength) ) { /* haven't reached this position yet */ U32 const newMatchIndex = matchCandidateIdx + (U32)forwardPatternLength - (U32)srcPatternLength; /* best position, full pattern, might be followed by more match */ if (LZ4HC_protectDictEnd(prefixIdx, newMatchIndex)) matchIndex = newMatchIndex; else { /* Can only happen if started in the prefix */ assert(newMatchIndex >= prefixIdx - 3 && newMatchIndex < prefixIdx && !extDict); matchIndex = prefixIdx; } } else { U32 const newMatchIndex = matchCandidateIdx - (U32)backLength; /* farthest position in current segment, will find a match of length currentSegmentLength + maybe some back */ if (!LZ4HC_protectDictEnd(prefixIdx, newMatchIndex)) { assert(newMatchIndex >= prefixIdx - 3 && newMatchIndex < prefixIdx && !extDict); matchIndex = prefixIdx; } else { matchIndex = newMatchIndex; if (lookBackLength==0) { /* no back possible */ size_t const maxML = MIN(currentSegmentLength, srcPatternLength); if ((size_t)longest < maxML) { assert(prefixPtr - prefixIdx + matchIndex != ip); if ((size_t)(ip - prefixPtr) + prefixIdx - matchIndex > LZ4_DISTANCE_MAX) break; assert(maxML < 2 GB); longest = (int)maxML; *matchpos = prefixPtr - prefixIdx + matchIndex; /* virtual pos, relative to ip, to retrieve offset */ *startpos = ip; } { U32 const distToNextPattern = DELTANEXTU16(chainTable, matchIndex); if (distToNextPattern > matchIndex) break; /* avoid overflow */ matchIndex -= distToNextPattern; } } } } } continue; } } } } /* PA optimization */ /* follow current chain */ matchIndex -= DELTANEXTU16(chainTable, matchIndex + matchChainPos); } /* while ((matchIndex>=lowestMatchIndex) && (nbAttempts)) */ if ( dict == usingDictCtxHc && nbAttempts > 0 && ipIndex - lowestMatchIndex < LZ4_DISTANCE_MAX) { size_t const dictEndOffset = (size_t)(dictCtx->end - dictCtx->prefixStart) + dictCtx->dictLimit; U32 dictMatchIndex = dictCtx->hashTable[LZ4HC_hashPtr(ip)]; assert(dictEndOffset <= 1 GB); matchIndex = dictMatchIndex + lowestMatchIndex - (U32)dictEndOffset; while (ipIndex - matchIndex <= LZ4_DISTANCE_MAX && nbAttempts--) { const BYTE* const matchPtr = dictCtx->prefixStart - dictCtx->dictLimit + dictMatchIndex; if (LZ4_read32(matchPtr) == pattern) { int mlt; int back = 0; const BYTE* vLimit = ip + (dictEndOffset - dictMatchIndex); if (vLimit > iHighLimit) vLimit = iHighLimit; mlt = (int)LZ4_count(ip+MINMATCH, matchPtr+MINMATCH, vLimit) + MINMATCH; back = lookBackLength ? LZ4HC_countBack(ip, matchPtr, iLowLimit, dictCtx->prefixStart) : 0; mlt -= back; if (mlt > longest) { longest = mlt; *matchpos = prefixPtr - prefixIdx + matchIndex + back; *startpos = ip + back; } } { U32 const nextOffset = DELTANEXTU16(dictCtx->chainTable, dictMatchIndex); dictMatchIndex -= nextOffset; matchIndex -= nextOffset; } } } return longest; } LZ4_FORCE_INLINE int LZ4HC_InsertAndFindBestMatch(LZ4HC_CCtx_internal* const hc4, /* Index table will be updated */ const BYTE* const ip, const BYTE* const iLimit, const BYTE** matchpos, const int maxNbAttempts, const int patternAnalysis, const dictCtx_directive dict) { const BYTE* uselessPtr = ip; /* note : LZ4HC_InsertAndGetWiderMatch() is able to modify the starting position of a match (*startpos), * but this won't be the case here, as we define iLowLimit==ip, * so LZ4HC_InsertAndGetWiderMatch() won't be allowed to search past ip */ return LZ4HC_InsertAndGetWiderMatch(hc4, ip, ip, iLimit, MINMATCH-1, matchpos, &uselessPtr, maxNbAttempts, patternAnalysis, 0 /*chainSwap*/, dict, favorCompressionRatio); } /* LZ4HC_encodeSequence() : * @return : 0 if ok, * 1 if buffer issue detected */ LZ4_FORCE_INLINE int LZ4HC_encodeSequence ( const BYTE** _ip, BYTE** _op, const BYTE** _anchor, int matchLength, const BYTE* const match, limitedOutput_directive limit, BYTE* oend) { #define ip (*_ip) #define op (*_op) #define anchor (*_anchor) size_t length; BYTE* const token = op++; #if defined(LZ4_DEBUG) && (LZ4_DEBUG >= 6) static const BYTE* start = NULL; static U32 totalCost = 0; U32 const pos = (start==NULL) ? 0 : (U32)(anchor - start); U32 const ll = (U32)(ip - anchor); U32 const llAdd = (ll>=15) ? ((ll-15) / 255) + 1 : 0; U32 const mlAdd = (matchLength>=19) ? ((matchLength-19) / 255) + 1 : 0; U32 const cost = 1 + llAdd + ll + 2 + mlAdd; if (start==NULL) start = anchor; /* only works for single segment */ /* g_debuglog_enable = (pos >= 2228) & (pos <= 2262); */ DEBUGLOG(6, "pos:%7u -- literals:%4u, match:%4i, offset:%5u, cost:%4u + %5u", pos, (U32)(ip - anchor), matchLength, (U32)(ip-match), cost, totalCost); totalCost += cost; #endif /* Encode Literal length */ length = (size_t)(ip - anchor); LZ4_STATIC_ASSERT(notLimited == 0); /* Check output limit */ if (limit && ((op + (length / 255) + length + (2 + 1 + LASTLITERALS)) > oend)) { DEBUGLOG(6, "Not enough room to write %i literals (%i bytes remaining)", (int)length, (int)(oend - op)); return 1; } if (length >= RUN_MASK) { size_t len = length - RUN_MASK; *token = (RUN_MASK << ML_BITS); for(; len >= 255 ; len -= 255) *op++ = 255; *op++ = (BYTE)len; } else { *token = (BYTE)(length << ML_BITS); } /* Copy Literals */ LZ4_wildCopy8(op, anchor, op + length); op += length; /* Encode Offset */ assert( (ip - match) <= LZ4_DISTANCE_MAX ); /* note : consider providing offset as a value, rather than as a pointer difference */ LZ4_writeLE16(op, (U16)(ip - match)); op += 2; /* Encode MatchLength */ assert(matchLength >= MINMATCH); length = (size_t)matchLength - MINMATCH; if (limit && (op + (length / 255) + (1 + LASTLITERALS) > oend)) { DEBUGLOG(6, "Not enough room to write match length"); return 1; /* Check output limit */ } if (length >= ML_MASK) { *token += ML_MASK; length -= ML_MASK; for(; length >= 510 ; length -= 510) { *op++ = 255; *op++ = 255; } if (length >= 255) { length -= 255; *op++ = 255; } *op++ = (BYTE)length; } else { *token += (BYTE)(length); } /* Prepare next loop */ ip += matchLength; anchor = ip; return 0; } #undef ip #undef op #undef anchor LZ4_FORCE_INLINE int LZ4HC_compress_hashChain ( LZ4HC_CCtx_internal* const ctx, const char* const source, char* const dest, int* srcSizePtr, int const maxOutputSize, int maxNbAttempts, const limitedOutput_directive limit, const dictCtx_directive dict ) { const int inputSize = *srcSizePtr; const int patternAnalysis = (maxNbAttempts > 128); /* levels 9+ */ const BYTE* ip = (const BYTE*) source; const BYTE* anchor = ip; const BYTE* const iend = ip + inputSize; const BYTE* const mflimit = iend - MFLIMIT; const BYTE* const matchlimit = (iend - LASTLITERALS); BYTE* optr = (BYTE*) dest; BYTE* op = (BYTE*) dest; BYTE* oend = op + maxOutputSize; int ml0, ml, ml2, ml3; const BYTE* start0; const BYTE* ref0; const BYTE* ref = NULL; const BYTE* start2 = NULL; const BYTE* ref2 = NULL; const BYTE* start3 = NULL; const BYTE* ref3 = NULL; /* init */ *srcSizePtr = 0; if (limit == fillOutput) oend -= LASTLITERALS; /* Hack for support LZ4 format restriction */ if (inputSize < LZ4_minLength) goto _last_literals; /* Input too small, no compression (all literals) */ /* Main Loop */ while (ip <= mflimit) { ml = LZ4HC_InsertAndFindBestMatch(ctx, ip, matchlimit, &ref, maxNbAttempts, patternAnalysis, dict); if (ml encode ML1 */ optr = op; if (LZ4HC_encodeSequence(UPDATABLE(ip, op, anchor), ml, ref, limit, oend)) goto _dest_overflow; continue; } if (start0 < ip) { /* first match was skipped at least once */ if (start2 < ip + ml0) { /* squeezing ML1 between ML0(original ML1) and ML2 */ ip = start0; ref = ref0; ml = ml0; /* restore initial ML1 */ } } /* Here, start0==ip */ if ((start2 - ip) < 3) { /* First Match too small : removed */ ml = ml2; ip = start2; ref =ref2; goto _Search2; } _Search3: /* At this stage, we have : * ml2 > ml1, and * ip1+3 <= ip2 (usually < ip1+ml1) */ if ((start2 - ip) < OPTIMAL_ML) { int correction; int new_ml = ml; if (new_ml > OPTIMAL_ML) new_ml = OPTIMAL_ML; if (ip+new_ml > start2 + ml2 - MINMATCH) new_ml = (int)(start2 - ip) + ml2 - MINMATCH; correction = new_ml - (int)(start2 - ip); if (correction > 0) { start2 += correction; ref2 += correction; ml2 -= correction; } } /* Now, we have start2 = ip+new_ml, with new_ml = min(ml, OPTIMAL_ML=18) */ if (start2 + ml2 <= mflimit) { ml3 = LZ4HC_InsertAndGetWiderMatch(ctx, start2 + ml2 - 3, start2, matchlimit, ml2, &ref3, &start3, maxNbAttempts, patternAnalysis, 0, dict, favorCompressionRatio); } else { ml3 = ml2; } if (ml3 == ml2) { /* No better match => encode ML1 and ML2 */ /* ip & ref are known; Now for ml */ if (start2 < ip+ml) ml = (int)(start2 - ip); /* Now, encode 2 sequences */ optr = op; if (LZ4HC_encodeSequence(UPDATABLE(ip, op, anchor), ml, ref, limit, oend)) goto _dest_overflow; ip = start2; optr = op; if (LZ4HC_encodeSequence(UPDATABLE(ip, op, anchor), ml2, ref2, limit, oend)) { ml = ml2; ref = ref2; goto _dest_overflow; } continue; } if (start3 < ip+ml+3) { /* Not enough space for match 2 : remove it */ if (start3 >= (ip+ml)) { /* can write Seq1 immediately ==> Seq2 is removed, so Seq3 becomes Seq1 */ if (start2 < ip+ml) { int correction = (int)(ip+ml - start2); start2 += correction; ref2 += correction; ml2 -= correction; if (ml2 < MINMATCH) { start2 = start3; ref2 = ref3; ml2 = ml3; } } optr = op; if (LZ4HC_encodeSequence(UPDATABLE(ip, op, anchor), ml, ref, limit, oend)) goto _dest_overflow; ip = start3; ref = ref3; ml = ml3; start0 = start2; ref0 = ref2; ml0 = ml2; goto _Search2; } start2 = start3; ref2 = ref3; ml2 = ml3; goto _Search3; } /* * OK, now we have 3 ascending matches; * let's write the first one ML1. * ip & ref are known; Now decide ml. */ if (start2 < ip+ml) { if ((start2 - ip) < OPTIMAL_ML) { int correction; if (ml > OPTIMAL_ML) ml = OPTIMAL_ML; if (ip + ml > start2 + ml2 - MINMATCH) ml = (int)(start2 - ip) + ml2 - MINMATCH; correction = ml - (int)(start2 - ip); if (correction > 0) { start2 += correction; ref2 += correction; ml2 -= correction; } } else { ml = (int)(start2 - ip); } } optr = op; if (LZ4HC_encodeSequence(UPDATABLE(ip, op, anchor), ml, ref, limit, oend)) goto _dest_overflow; /* ML2 becomes ML1 */ ip = start2; ref = ref2; ml = ml2; /* ML3 becomes ML2 */ start2 = start3; ref2 = ref3; ml2 = ml3; /* let's find a new ML3 */ goto _Search3; } _last_literals: /* Encode Last Literals */ { size_t lastRunSize = (size_t)(iend - anchor); /* literals */ size_t llAdd = (lastRunSize + 255 - RUN_MASK) / 255; size_t const totalSize = 1 + llAdd + lastRunSize; if (limit == fillOutput) oend += LASTLITERALS; /* restore correct value */ if (limit && (op + totalSize > oend)) { if (limit == limitedOutput) return 0; /* adapt lastRunSize to fill 'dest' */ lastRunSize = (size_t)(oend - op) - 1 /*token*/; llAdd = (lastRunSize + 256 - RUN_MASK) / 256; lastRunSize -= llAdd; } DEBUGLOG(6, "Final literal run : %i literals", (int)lastRunSize); ip = anchor + lastRunSize; /* can be != iend if limit==fillOutput */ if (lastRunSize >= RUN_MASK) { size_t accumulator = lastRunSize - RUN_MASK; *op++ = (RUN_MASK << ML_BITS); for(; accumulator >= 255 ; accumulator -= 255) *op++ = 255; *op++ = (BYTE) accumulator; } else { *op++ = (BYTE)(lastRunSize << ML_BITS); } LZ4_memcpy(op, anchor, lastRunSize); op += lastRunSize; } /* End */ *srcSizePtr = (int) (((const char*)ip) - source); return (int) (((char*)op)-dest); _dest_overflow: if (limit == fillOutput) { /* Assumption : ip, anchor, ml and ref must be set correctly */ size_t const ll = (size_t)(ip - anchor); size_t const ll_addbytes = (ll + 240) / 255; size_t const ll_totalCost = 1 + ll_addbytes + ll; BYTE* const maxLitPos = oend - 3; /* 2 for offset, 1 for token */ DEBUGLOG(6, "Last sequence overflowing"); op = optr; /* restore correct out pointer */ if (op + ll_totalCost <= maxLitPos) { /* ll validated; now adjust match length */ size_t const bytesLeftForMl = (size_t)(maxLitPos - (op+ll_totalCost)); size_t const maxMlSize = MINMATCH + (ML_MASK-1) + (bytesLeftForMl * 255); assert(maxMlSize < INT_MAX); assert(ml >= 0); if ((size_t)ml > maxMlSize) ml = (int)maxMlSize; if ((oend + LASTLITERALS) - (op + ll_totalCost + 2) - 1 + ml >= MFLIMIT) { LZ4HC_encodeSequence(UPDATABLE(ip, op, anchor), ml, ref, notLimited, oend); } } goto _last_literals; } /* compression failed */ return 0; } static int LZ4HC_compress_optimal( LZ4HC_CCtx_internal* ctx, const char* const source, char* dst, int* srcSizePtr, int dstCapacity, int const nbSearches, size_t sufficient_len, const limitedOutput_directive limit, int const fullUpdate, const dictCtx_directive dict, const HCfavor_e favorDecSpeed); LZ4_FORCE_INLINE int LZ4HC_compress_generic_internal ( LZ4HC_CCtx_internal* const ctx, const char* const src, char* const dst, int* const srcSizePtr, int const dstCapacity, int cLevel, const limitedOutput_directive limit, const dictCtx_directive dict ) { typedef enum { lz4hc, lz4opt } lz4hc_strat_e; typedef struct { lz4hc_strat_e strat; int nbSearches; U32 targetLength; } cParams_t; static const cParams_t clTable[LZ4HC_CLEVEL_MAX+1] = { { lz4hc, 2, 16 }, /* 0, unused */ { lz4hc, 2, 16 }, /* 1, unused */ { lz4hc, 2, 16 }, /* 2, unused */ { lz4hc, 4, 16 }, /* 3 */ { lz4hc, 8, 16 }, /* 4 */ { lz4hc, 16, 16 }, /* 5 */ { lz4hc, 32, 16 }, /* 6 */ { lz4hc, 64, 16 }, /* 7 */ { lz4hc, 128, 16 }, /* 8 */ { lz4hc, 256, 16 }, /* 9 */ { lz4opt, 96, 64 }, /*10==LZ4HC_CLEVEL_OPT_MIN*/ { lz4opt, 512,128 }, /*11 */ { lz4opt,16384,LZ4_OPT_NUM }, /* 12==LZ4HC_CLEVEL_MAX */ }; DEBUGLOG(4, "LZ4HC_compress_generic(ctx=%p, src=%p, srcSize=%d, limit=%d)", ctx, src, *srcSizePtr, limit); if (limit == fillOutput && dstCapacity < 1) return 0; /* Impossible to store anything */ if ((U32)*srcSizePtr > (U32)LZ4_MAX_INPUT_SIZE) return 0; /* Unsupported input size (too large or negative) */ ctx->end += *srcSizePtr; if (cLevel < 1) cLevel = LZ4HC_CLEVEL_DEFAULT; /* note : convention is different from lz4frame, maybe something to review */ cLevel = MIN(LZ4HC_CLEVEL_MAX, cLevel); { cParams_t const cParam = clTable[cLevel]; HCfavor_e const favor = ctx->favorDecSpeed ? favorDecompressionSpeed : favorCompressionRatio; int result; if (cParam.strat == lz4hc) { result = LZ4HC_compress_hashChain(ctx, src, dst, srcSizePtr, dstCapacity, cParam.nbSearches, limit, dict); } else { assert(cParam.strat == lz4opt); result = LZ4HC_compress_optimal(ctx, src, dst, srcSizePtr, dstCapacity, cParam.nbSearches, cParam.targetLength, limit, cLevel == LZ4HC_CLEVEL_MAX, /* ultra mode */ dict, favor); } if (result <= 0) ctx->dirty = 1; return result; } } static void LZ4HC_setExternalDict(LZ4HC_CCtx_internal* ctxPtr, const BYTE* newBlock); static int LZ4HC_compress_generic_noDictCtx ( LZ4HC_CCtx_internal* const ctx, const char* const src, char* const dst, int* const srcSizePtr, int const dstCapacity, int cLevel, limitedOutput_directive limit ) { assert(ctx->dictCtx == NULL); return LZ4HC_compress_generic_internal(ctx, src, dst, srcSizePtr, dstCapacity, cLevel, limit, noDictCtx); } static int LZ4HC_compress_generic_dictCtx ( LZ4HC_CCtx_internal* const ctx, const char* const src, char* const dst, int* const srcSizePtr, int const dstCapacity, int cLevel, limitedOutput_directive limit ) { const size_t position = (size_t)(ctx->end - ctx->prefixStart) + (ctx->dictLimit - ctx->lowLimit); assert(ctx->dictCtx != NULL); if (position >= 64 KB) { ctx->dictCtx = NULL; return LZ4HC_compress_generic_noDictCtx(ctx, src, dst, srcSizePtr, dstCapacity, cLevel, limit); } else if (position == 0 && *srcSizePtr > 4 KB) { LZ4_memcpy(ctx, ctx->dictCtx, sizeof(LZ4HC_CCtx_internal)); LZ4HC_setExternalDict(ctx, (const BYTE *)src); ctx->compressionLevel = (short)cLevel; return LZ4HC_compress_generic_noDictCtx(ctx, src, dst, srcSizePtr, dstCapacity, cLevel, limit); } else { return LZ4HC_compress_generic_internal(ctx, src, dst, srcSizePtr, dstCapacity, cLevel, limit, usingDictCtxHc); } } static int LZ4HC_compress_generic ( LZ4HC_CCtx_internal* const ctx, const char* const src, char* const dst, int* const srcSizePtr, int const dstCapacity, int cLevel, limitedOutput_directive limit ) { if (ctx->dictCtx == NULL) { return LZ4HC_compress_generic_noDictCtx(ctx, src, dst, srcSizePtr, dstCapacity, cLevel, limit); } else { return LZ4HC_compress_generic_dictCtx(ctx, src, dst, srcSizePtr, dstCapacity, cLevel, limit); } } int LZ4_sizeofStateHC(void) { return (int)sizeof(LZ4_streamHC_t); } static size_t LZ4_streamHC_t_alignment(void) { #if LZ4_ALIGN_TEST typedef struct { char c; LZ4_streamHC_t t; } t_a; return sizeof(t_a) - sizeof(LZ4_streamHC_t); #else return 1; /* effectively disabled */ #endif } /* state is presumed correctly initialized, * in which case its size and alignment have already been validate */ int LZ4_compress_HC_extStateHC_fastReset (void* state, const char* src, char* dst, int srcSize, int dstCapacity, int compressionLevel) { LZ4HC_CCtx_internal* const ctx = &((LZ4_streamHC_t*)state)->internal_donotuse; if (!LZ4_isAligned(state, LZ4_streamHC_t_alignment())) return 0; LZ4_resetStreamHC_fast((LZ4_streamHC_t*)state, compressionLevel); LZ4HC_init_internal (ctx, (const BYTE*)src); if (dstCapacity < LZ4_compressBound(srcSize)) return LZ4HC_compress_generic (ctx, src, dst, &srcSize, dstCapacity, compressionLevel, limitedOutput); else return LZ4HC_compress_generic (ctx, src, dst, &srcSize, dstCapacity, compressionLevel, notLimited); } int LZ4_compress_HC_extStateHC (void* state, const char* src, char* dst, int srcSize, int dstCapacity, int compressionLevel) { LZ4_streamHC_t* const ctx = LZ4_initStreamHC(state, sizeof(*ctx)); if (ctx==NULL) return 0; /* init failure */ return LZ4_compress_HC_extStateHC_fastReset(state, src, dst, srcSize, dstCapacity, compressionLevel); } int LZ4_compress_HC(const char* src, char* dst, int srcSize, int dstCapacity, int compressionLevel) { int cSize; #if defined(LZ4HC_HEAPMODE) && LZ4HC_HEAPMODE==1 LZ4_streamHC_t* const statePtr = (LZ4_streamHC_t*)ALLOC(sizeof(LZ4_streamHC_t)); if (statePtr==NULL) return 0; #else LZ4_streamHC_t state; LZ4_streamHC_t* const statePtr = &state; #endif cSize = LZ4_compress_HC_extStateHC(statePtr, src, dst, srcSize, dstCapacity, compressionLevel); #if defined(LZ4HC_HEAPMODE) && LZ4HC_HEAPMODE==1 FREEMEM(statePtr); #endif return cSize; } /* state is presumed sized correctly (>= sizeof(LZ4_streamHC_t)) */ int LZ4_compress_HC_destSize(void* state, const char* source, char* dest, int* sourceSizePtr, int targetDestSize, int cLevel) { LZ4_streamHC_t* const ctx = LZ4_initStreamHC(state, sizeof(*ctx)); if (ctx==NULL) return 0; /* init failure */ LZ4HC_init_internal(&ctx->internal_donotuse, (const BYTE*) source); LZ4_setCompressionLevel(ctx, cLevel); return LZ4HC_compress_generic(&ctx->internal_donotuse, source, dest, sourceSizePtr, targetDestSize, cLevel, fillOutput); } /************************************** * Streaming Functions **************************************/ /* allocation */ #if !defined(LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION) LZ4_streamHC_t* LZ4_createStreamHC(void) { LZ4_streamHC_t* const state = (LZ4_streamHC_t*)ALLOC_AND_ZERO(sizeof(LZ4_streamHC_t)); if (state == NULL) return NULL; LZ4_setCompressionLevel(state, LZ4HC_CLEVEL_DEFAULT); return state; } int LZ4_freeStreamHC (LZ4_streamHC_t* LZ4_streamHCPtr) { DEBUGLOG(4, "LZ4_freeStreamHC(%p)", LZ4_streamHCPtr); if (!LZ4_streamHCPtr) return 0; /* support free on NULL */ FREEMEM(LZ4_streamHCPtr); return 0; } #endif LZ4_streamHC_t* LZ4_initStreamHC (void* buffer, size_t size) { LZ4_streamHC_t* const LZ4_streamHCPtr = (LZ4_streamHC_t*)buffer; DEBUGLOG(4, "LZ4_initStreamHC(%p, %u)", buffer, (unsigned)size); /* check conditions */ if (buffer == NULL) return NULL; if (size < sizeof(LZ4_streamHC_t)) return NULL; if (!LZ4_isAligned(buffer, LZ4_streamHC_t_alignment())) return NULL; /* init */ { LZ4HC_CCtx_internal* const hcstate = &(LZ4_streamHCPtr->internal_donotuse); MEM_INIT(hcstate, 0, sizeof(*hcstate)); } LZ4_setCompressionLevel(LZ4_streamHCPtr, LZ4HC_CLEVEL_DEFAULT); return LZ4_streamHCPtr; } /* just a stub */ void LZ4_resetStreamHC (LZ4_streamHC_t* LZ4_streamHCPtr, int compressionLevel) { LZ4_initStreamHC(LZ4_streamHCPtr, sizeof(*LZ4_streamHCPtr)); LZ4_setCompressionLevel(LZ4_streamHCPtr, compressionLevel); } void LZ4_resetStreamHC_fast (LZ4_streamHC_t* LZ4_streamHCPtr, int compressionLevel) { DEBUGLOG(4, "LZ4_resetStreamHC_fast(%p, %d)", LZ4_streamHCPtr, compressionLevel); if (LZ4_streamHCPtr->internal_donotuse.dirty) { LZ4_initStreamHC(LZ4_streamHCPtr, sizeof(*LZ4_streamHCPtr)); } else { /* preserve end - prefixStart : can trigger clearTable's threshold */ if (LZ4_streamHCPtr->internal_donotuse.end != NULL) { LZ4_streamHCPtr->internal_donotuse.end -= (uptrval)LZ4_streamHCPtr->internal_donotuse.prefixStart; } else { assert(LZ4_streamHCPtr->internal_donotuse.prefixStart == NULL); } LZ4_streamHCPtr->internal_donotuse.prefixStart = NULL; LZ4_streamHCPtr->internal_donotuse.dictCtx = NULL; } LZ4_setCompressionLevel(LZ4_streamHCPtr, compressionLevel); } void LZ4_setCompressionLevel(LZ4_streamHC_t* LZ4_streamHCPtr, int compressionLevel) { DEBUGLOG(5, "LZ4_setCompressionLevel(%p, %d)", LZ4_streamHCPtr, compressionLevel); if (compressionLevel < 1) compressionLevel = LZ4HC_CLEVEL_DEFAULT; if (compressionLevel > LZ4HC_CLEVEL_MAX) compressionLevel = LZ4HC_CLEVEL_MAX; LZ4_streamHCPtr->internal_donotuse.compressionLevel = (short)compressionLevel; } void LZ4_favorDecompressionSpeed(LZ4_streamHC_t* LZ4_streamHCPtr, int favor) { LZ4_streamHCPtr->internal_donotuse.favorDecSpeed = (favor!=0); } /* LZ4_loadDictHC() : * LZ4_streamHCPtr is presumed properly initialized */ int LZ4_loadDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, const char* dictionary, int dictSize) { LZ4HC_CCtx_internal* const ctxPtr = &LZ4_streamHCPtr->internal_donotuse; DEBUGLOG(4, "LZ4_loadDictHC(ctx:%p, dict:%p, dictSize:%d)", LZ4_streamHCPtr, dictionary, dictSize); assert(LZ4_streamHCPtr != NULL); if (dictSize > 64 KB) { dictionary += (size_t)dictSize - 64 KB; dictSize = 64 KB; } /* need a full initialization, there are bad side-effects when using resetFast() */ { int const cLevel = ctxPtr->compressionLevel; LZ4_initStreamHC(LZ4_streamHCPtr, sizeof(*LZ4_streamHCPtr)); LZ4_setCompressionLevel(LZ4_streamHCPtr, cLevel); } LZ4HC_init_internal (ctxPtr, (const BYTE*)dictionary); ctxPtr->end = (const BYTE*)dictionary + dictSize; if (dictSize >= 4) LZ4HC_Insert (ctxPtr, ctxPtr->end-3); return dictSize; } void LZ4_attach_HC_dictionary(LZ4_streamHC_t *working_stream, const LZ4_streamHC_t *dictionary_stream) { working_stream->internal_donotuse.dictCtx = dictionary_stream != NULL ? &(dictionary_stream->internal_donotuse) : NULL; } /* compression */ static void LZ4HC_setExternalDict(LZ4HC_CCtx_internal* ctxPtr, const BYTE* newBlock) { DEBUGLOG(4, "LZ4HC_setExternalDict(%p, %p)", ctxPtr, newBlock); if (ctxPtr->end >= ctxPtr->prefixStart + 4) LZ4HC_Insert (ctxPtr, ctxPtr->end-3); /* Referencing remaining dictionary content */ /* Only one memory segment for extDict, so any previous extDict is lost at this stage */ ctxPtr->lowLimit = ctxPtr->dictLimit; ctxPtr->dictStart = ctxPtr->prefixStart; ctxPtr->dictLimit += (U32)(ctxPtr->end - ctxPtr->prefixStart); ctxPtr->prefixStart = newBlock; ctxPtr->end = newBlock; ctxPtr->nextToUpdate = ctxPtr->dictLimit; /* match referencing will resume from there */ /* cannot reference an extDict and a dictCtx at the same time */ ctxPtr->dictCtx = NULL; } static int LZ4_compressHC_continue_generic (LZ4_streamHC_t* LZ4_streamHCPtr, const char* src, char* dst, int* srcSizePtr, int dstCapacity, limitedOutput_directive limit) { LZ4HC_CCtx_internal* const ctxPtr = &LZ4_streamHCPtr->internal_donotuse; DEBUGLOG(5, "LZ4_compressHC_continue_generic(ctx=%p, src=%p, srcSize=%d, limit=%d)", LZ4_streamHCPtr, src, *srcSizePtr, limit); assert(ctxPtr != NULL); /* auto-init if forgotten */ if (ctxPtr->prefixStart == NULL) LZ4HC_init_internal (ctxPtr, (const BYTE*) src); /* Check overflow */ if ((size_t)(ctxPtr->end - ctxPtr->prefixStart) + ctxPtr->dictLimit > 2 GB) { size_t dictSize = (size_t)(ctxPtr->end - ctxPtr->prefixStart); if (dictSize > 64 KB) dictSize = 64 KB; LZ4_loadDictHC(LZ4_streamHCPtr, (const char*)(ctxPtr->end) - dictSize, (int)dictSize); } /* Check if blocks follow each other */ if ((const BYTE*)src != ctxPtr->end) LZ4HC_setExternalDict(ctxPtr, (const BYTE*)src); /* Check overlapping input/dictionary space */ { const BYTE* sourceEnd = (const BYTE*) src + *srcSizePtr; const BYTE* const dictBegin = ctxPtr->dictStart; const BYTE* const dictEnd = ctxPtr->dictStart + (ctxPtr->dictLimit - ctxPtr->lowLimit); if ((sourceEnd > dictBegin) && ((const BYTE*)src < dictEnd)) { if (sourceEnd > dictEnd) sourceEnd = dictEnd; ctxPtr->lowLimit += (U32)(sourceEnd - ctxPtr->dictStart); ctxPtr->dictStart += (U32)(sourceEnd - ctxPtr->dictStart); if (ctxPtr->dictLimit - ctxPtr->lowLimit < 4) { ctxPtr->lowLimit = ctxPtr->dictLimit; ctxPtr->dictStart = ctxPtr->prefixStart; } } } return LZ4HC_compress_generic (ctxPtr, src, dst, srcSizePtr, dstCapacity, ctxPtr->compressionLevel, limit); } int LZ4_compress_HC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* src, char* dst, int srcSize, int dstCapacity) { if (dstCapacity < LZ4_compressBound(srcSize)) return LZ4_compressHC_continue_generic (LZ4_streamHCPtr, src, dst, &srcSize, dstCapacity, limitedOutput); else return LZ4_compressHC_continue_generic (LZ4_streamHCPtr, src, dst, &srcSize, dstCapacity, notLimited); } int LZ4_compress_HC_continue_destSize (LZ4_streamHC_t* LZ4_streamHCPtr, const char* src, char* dst, int* srcSizePtr, int targetDestSize) { return LZ4_compressHC_continue_generic(LZ4_streamHCPtr, src, dst, srcSizePtr, targetDestSize, fillOutput); } /* LZ4_saveDictHC : * save history content * into a user-provided buffer * which is then used to continue compression */ int LZ4_saveDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, char* safeBuffer, int dictSize) { LZ4HC_CCtx_internal* const streamPtr = &LZ4_streamHCPtr->internal_donotuse; int const prefixSize = (int)(streamPtr->end - streamPtr->prefixStart); DEBUGLOG(5, "LZ4_saveDictHC(%p, %p, %d)", LZ4_streamHCPtr, safeBuffer, dictSize); assert(prefixSize >= 0); if (dictSize > 64 KB) dictSize = 64 KB; if (dictSize < 4) dictSize = 0; if (dictSize > prefixSize) dictSize = prefixSize; if (safeBuffer == NULL) assert(dictSize == 0); if (dictSize > 0) LZ4_memmove(safeBuffer, streamPtr->end - dictSize, dictSize); { U32 const endIndex = (U32)(streamPtr->end - streamPtr->prefixStart) + streamPtr->dictLimit; streamPtr->end = (const BYTE*)safeBuffer + dictSize; streamPtr->prefixStart = streamPtr->end - dictSize; streamPtr->dictLimit = endIndex - (U32)dictSize; streamPtr->lowLimit = endIndex - (U32)dictSize; streamPtr->dictStart = streamPtr->prefixStart; if (streamPtr->nextToUpdate < streamPtr->dictLimit) streamPtr->nextToUpdate = streamPtr->dictLimit; } return dictSize; } /*************************************************** * Deprecated Functions ***************************************************/ /* These functions currently generate deprecation warnings */ /* Wrappers for deprecated compression functions */ int LZ4_compressHC(const char* src, char* dst, int srcSize) { return LZ4_compress_HC (src, dst, srcSize, LZ4_compressBound(srcSize), 0); } int LZ4_compressHC_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_HC(src, dst, srcSize, maxDstSize, 0); } int LZ4_compressHC2(const char* src, char* dst, int srcSize, int cLevel) { return LZ4_compress_HC (src, dst, srcSize, LZ4_compressBound(srcSize), cLevel); } int LZ4_compressHC2_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize, int cLevel) { return LZ4_compress_HC(src, dst, srcSize, maxDstSize, cLevel); } int LZ4_compressHC_withStateHC (void* state, const char* src, char* dst, int srcSize) { return LZ4_compress_HC_extStateHC (state, src, dst, srcSize, LZ4_compressBound(srcSize), 0); } int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_HC_extStateHC (state, src, dst, srcSize, maxDstSize, 0); } int LZ4_compressHC2_withStateHC (void* state, const char* src, char* dst, int srcSize, int cLevel) { return LZ4_compress_HC_extStateHC(state, src, dst, srcSize, LZ4_compressBound(srcSize), cLevel); } int LZ4_compressHC2_limitedOutput_withStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize, int cLevel) { return LZ4_compress_HC_extStateHC(state, src, dst, srcSize, maxDstSize, cLevel); } int LZ4_compressHC_continue (LZ4_streamHC_t* ctx, const char* src, char* dst, int srcSize) { return LZ4_compress_HC_continue (ctx, src, dst, srcSize, LZ4_compressBound(srcSize)); } int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* ctx, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_HC_continue (ctx, src, dst, srcSize, maxDstSize); } /* Deprecated streaming functions */ int LZ4_sizeofStreamStateHC(void) { return sizeof(LZ4_streamHC_t); } /* state is presumed correctly sized, aka >= sizeof(LZ4_streamHC_t) * @return : 0 on success, !=0 if error */ int LZ4_resetStreamStateHC(void* state, char* inputBuffer) { LZ4_streamHC_t* const hc4 = LZ4_initStreamHC(state, sizeof(*hc4)); if (hc4 == NULL) return 1; /* init failed */ LZ4HC_init_internal (&hc4->internal_donotuse, (const BYTE*)inputBuffer); return 0; } #if !defined(LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION) void* LZ4_createHC (const char* inputBuffer) { LZ4_streamHC_t* const hc4 = LZ4_createStreamHC(); if (hc4 == NULL) return NULL; /* not enough memory */ LZ4HC_init_internal (&hc4->internal_donotuse, (const BYTE*)inputBuffer); return hc4; } int LZ4_freeHC (void* LZ4HC_Data) { if (!LZ4HC_Data) return 0; /* support free on NULL */ FREEMEM(LZ4HC_Data); return 0; } #endif int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* src, char* dst, int srcSize, int cLevel) { return LZ4HC_compress_generic (&((LZ4_streamHC_t*)LZ4HC_Data)->internal_donotuse, src, dst, &srcSize, 0, cLevel, notLimited); } int LZ4_compressHC2_limitedOutput_continue (void* LZ4HC_Data, const char* src, char* dst, int srcSize, int dstCapacity, int cLevel) { return LZ4HC_compress_generic (&((LZ4_streamHC_t*)LZ4HC_Data)->internal_donotuse, src, dst, &srcSize, dstCapacity, cLevel, limitedOutput); } char* LZ4_slideInputBufferHC(void* LZ4HC_Data) { LZ4_streamHC_t* const ctx = (LZ4_streamHC_t*)LZ4HC_Data; const BYTE* bufferStart = ctx->internal_donotuse.prefixStart - ctx->internal_donotuse.dictLimit + ctx->internal_donotuse.lowLimit; LZ4_resetStreamHC_fast(ctx, ctx->internal_donotuse.compressionLevel); /* avoid const char * -> char * conversion warning :( */ return (char*)(uptrval)bufferStart; } /* ================================================ * LZ4 Optimal parser (levels [LZ4HC_CLEVEL_OPT_MIN - LZ4HC_CLEVEL_MAX]) * ===============================================*/ typedef struct { int price; int off; int mlen; int litlen; } LZ4HC_optimal_t; /* price in bytes */ LZ4_FORCE_INLINE int LZ4HC_literalsPrice(int const litlen) { int price = litlen; assert(litlen >= 0); if (litlen >= (int)RUN_MASK) price += 1 + ((litlen-(int)RUN_MASK) / 255); return price; } /* requires mlen >= MINMATCH */ LZ4_FORCE_INLINE int LZ4HC_sequencePrice(int litlen, int mlen) { int price = 1 + 2 ; /* token + 16-bit offset */ assert(litlen >= 0); assert(mlen >= MINMATCH); price += LZ4HC_literalsPrice(litlen); if (mlen >= (int)(ML_MASK+MINMATCH)) price += 1 + ((mlen-(int)(ML_MASK+MINMATCH)) / 255); return price; } typedef struct { int off; int len; } LZ4HC_match_t; LZ4_FORCE_INLINE LZ4HC_match_t LZ4HC_FindLongerMatch(LZ4HC_CCtx_internal* const ctx, const BYTE* ip, const BYTE* const iHighLimit, int minLen, int nbSearches, const dictCtx_directive dict, const HCfavor_e favorDecSpeed) { LZ4HC_match_t match = { 0 , 0 }; const BYTE* matchPtr = NULL; /* note : LZ4HC_InsertAndGetWiderMatch() is able to modify the starting position of a match (*startpos), * but this won't be the case here, as we define iLowLimit==ip, * so LZ4HC_InsertAndGetWiderMatch() won't be allowed to search past ip */ int matchLength = LZ4HC_InsertAndGetWiderMatch(ctx, ip, ip, iHighLimit, minLen, &matchPtr, &ip, nbSearches, 1 /*patternAnalysis*/, 1 /*chainSwap*/, dict, favorDecSpeed); if (matchLength <= minLen) return match; if (favorDecSpeed) { if ((matchLength>18) & (matchLength<=36)) matchLength=18; /* favor shortcut */ } match.len = matchLength; match.off = (int)(ip-matchPtr); return match; } static int LZ4HC_compress_optimal ( LZ4HC_CCtx_internal* ctx, const char* const source, char* dst, int* srcSizePtr, int dstCapacity, int const nbSearches, size_t sufficient_len, const limitedOutput_directive limit, int const fullUpdate, const dictCtx_directive dict, const HCfavor_e favorDecSpeed) { int retval = 0; #define TRAILING_LITERALS 3 #if defined(LZ4HC_HEAPMODE) && LZ4HC_HEAPMODE==1 LZ4HC_optimal_t* const opt = (LZ4HC_optimal_t*)ALLOC(sizeof(LZ4HC_optimal_t) * (LZ4_OPT_NUM + TRAILING_LITERALS)); #else LZ4HC_optimal_t opt[LZ4_OPT_NUM + TRAILING_LITERALS]; /* ~64 KB, which is a bit large for stack... */ #endif const BYTE* ip = (const BYTE*) source; const BYTE* anchor = ip; const BYTE* const iend = ip + *srcSizePtr; const BYTE* const mflimit = iend - MFLIMIT; const BYTE* const matchlimit = iend - LASTLITERALS; BYTE* op = (BYTE*) dst; BYTE* opSaved = (BYTE*) dst; BYTE* oend = op + dstCapacity; int ovml = MINMATCH; /* overflow - last sequence */ const BYTE* ovref = NULL; /* init */ #if defined(LZ4HC_HEAPMODE) && LZ4HC_HEAPMODE==1 if (opt == NULL) goto _return_label; #endif DEBUGLOG(5, "LZ4HC_compress_optimal(dst=%p, dstCapa=%u)", dst, (unsigned)dstCapacity); *srcSizePtr = 0; if (limit == fillOutput) oend -= LASTLITERALS; /* Hack for support LZ4 format restriction */ if (sufficient_len >= LZ4_OPT_NUM) sufficient_len = LZ4_OPT_NUM-1; /* Main Loop */ while (ip <= mflimit) { int const llen = (int)(ip - anchor); int best_mlen, best_off; int cur, last_match_pos = 0; LZ4HC_match_t const firstMatch = LZ4HC_FindLongerMatch(ctx, ip, matchlimit, MINMATCH-1, nbSearches, dict, favorDecSpeed); if (firstMatch.len==0) { ip++; continue; } if ((size_t)firstMatch.len > sufficient_len) { /* good enough solution : immediate encoding */ int const firstML = firstMatch.len; const BYTE* const matchPos = ip - firstMatch.off; opSaved = op; if ( LZ4HC_encodeSequence(UPDATABLE(ip, op, anchor), firstML, matchPos, limit, oend) ) { /* updates ip, op and anchor */ ovml = firstML; ovref = matchPos; goto _dest_overflow; } continue; } /* set prices for first positions (literals) */ { int rPos; for (rPos = 0 ; rPos < MINMATCH ; rPos++) { int const cost = LZ4HC_literalsPrice(llen + rPos); opt[rPos].mlen = 1; opt[rPos].off = 0; opt[rPos].litlen = llen + rPos; opt[rPos].price = cost; DEBUGLOG(7, "rPos:%3i => price:%3i (litlen=%i) -- initial setup", rPos, cost, opt[rPos].litlen); } } /* set prices using initial match */ { int mlen = MINMATCH; int const matchML = firstMatch.len; /* necessarily < sufficient_len < LZ4_OPT_NUM */ int const offset = firstMatch.off; assert(matchML < LZ4_OPT_NUM); for ( ; mlen <= matchML ; mlen++) { int const cost = LZ4HC_sequencePrice(llen, mlen); opt[mlen].mlen = mlen; opt[mlen].off = offset; opt[mlen].litlen = llen; opt[mlen].price = cost; DEBUGLOG(7, "rPos:%3i => price:%3i (matchlen=%i) -- initial setup", mlen, cost, mlen); } } last_match_pos = firstMatch.len; { int addLit; for (addLit = 1; addLit <= TRAILING_LITERALS; addLit ++) { opt[last_match_pos+addLit].mlen = 1; /* literal */ opt[last_match_pos+addLit].off = 0; opt[last_match_pos+addLit].litlen = addLit; opt[last_match_pos+addLit].price = opt[last_match_pos].price + LZ4HC_literalsPrice(addLit); DEBUGLOG(7, "rPos:%3i => price:%3i (litlen=%i) -- initial setup", last_match_pos+addLit, opt[last_match_pos+addLit].price, addLit); } } /* check further positions */ for (cur = 1; cur < last_match_pos; cur++) { const BYTE* const curPtr = ip + cur; LZ4HC_match_t newMatch; if (curPtr > mflimit) break; DEBUGLOG(7, "rPos:%u[%u] vs [%u]%u", cur, opt[cur].price, opt[cur+1].price, cur+1); if (fullUpdate) { /* not useful to search here if next position has same (or lower) cost */ if ( (opt[cur+1].price <= opt[cur].price) /* in some cases, next position has same cost, but cost rises sharply after, so a small match would still be beneficial */ && (opt[cur+MINMATCH].price < opt[cur].price + 3/*min seq price*/) ) continue; } else { /* not useful to search here if next position has same (or lower) cost */ if (opt[cur+1].price <= opt[cur].price) continue; } DEBUGLOG(7, "search at rPos:%u", cur); if (fullUpdate) newMatch = LZ4HC_FindLongerMatch(ctx, curPtr, matchlimit, MINMATCH-1, nbSearches, dict, favorDecSpeed); else /* only test matches of minimum length; slightly faster, but misses a few bytes */ newMatch = LZ4HC_FindLongerMatch(ctx, curPtr, matchlimit, last_match_pos - cur, nbSearches, dict, favorDecSpeed); if (!newMatch.len) continue; if ( ((size_t)newMatch.len > sufficient_len) || (newMatch.len + cur >= LZ4_OPT_NUM) ) { /* immediate encoding */ best_mlen = newMatch.len; best_off = newMatch.off; last_match_pos = cur + 1; goto encode; } /* before match : set price with literals at beginning */ { int const baseLitlen = opt[cur].litlen; int litlen; for (litlen = 1; litlen < MINMATCH; litlen++) { int const price = opt[cur].price - LZ4HC_literalsPrice(baseLitlen) + LZ4HC_literalsPrice(baseLitlen+litlen); int const pos = cur + litlen; if (price < opt[pos].price) { opt[pos].mlen = 1; /* literal */ opt[pos].off = 0; opt[pos].litlen = baseLitlen+litlen; opt[pos].price = price; DEBUGLOG(7, "rPos:%3i => price:%3i (litlen=%i)", pos, price, opt[pos].litlen); } } } /* set prices using match at position = cur */ { int const matchML = newMatch.len; int ml = MINMATCH; assert(cur + newMatch.len < LZ4_OPT_NUM); for ( ; ml <= matchML ; ml++) { int const pos = cur + ml; int const offset = newMatch.off; int price; int ll; DEBUGLOG(7, "testing price rPos %i (last_match_pos=%i)", pos, last_match_pos); if (opt[cur].mlen == 1) { ll = opt[cur].litlen; price = ((cur > ll) ? opt[cur - ll].price : 0) + LZ4HC_sequencePrice(ll, ml); } else { ll = 0; price = opt[cur].price + LZ4HC_sequencePrice(0, ml); } assert((U32)favorDecSpeed <= 1); if (pos > last_match_pos+TRAILING_LITERALS || price <= opt[pos].price - (int)favorDecSpeed) { DEBUGLOG(7, "rPos:%3i => price:%3i (matchlen=%i)", pos, price, ml); assert(pos < LZ4_OPT_NUM); if ( (ml == matchML) /* last pos of last match */ && (last_match_pos < pos) ) last_match_pos = pos; opt[pos].mlen = ml; opt[pos].off = offset; opt[pos].litlen = ll; opt[pos].price = price; } } } /* complete following positions with literals */ { int addLit; for (addLit = 1; addLit <= TRAILING_LITERALS; addLit ++) { opt[last_match_pos+addLit].mlen = 1; /* literal */ opt[last_match_pos+addLit].off = 0; opt[last_match_pos+addLit].litlen = addLit; opt[last_match_pos+addLit].price = opt[last_match_pos].price + LZ4HC_literalsPrice(addLit); DEBUGLOG(7, "rPos:%3i => price:%3i (litlen=%i)", last_match_pos+addLit, opt[last_match_pos+addLit].price, addLit); } } } /* for (cur = 1; cur <= last_match_pos; cur++) */ assert(last_match_pos < LZ4_OPT_NUM + TRAILING_LITERALS); best_mlen = opt[last_match_pos].mlen; best_off = opt[last_match_pos].off; cur = last_match_pos - best_mlen; encode: /* cur, last_match_pos, best_mlen, best_off must be set */ assert(cur < LZ4_OPT_NUM); assert(last_match_pos >= 1); /* == 1 when only one candidate */ DEBUGLOG(6, "reverse traversal, looking for shortest path (last_match_pos=%i)", last_match_pos); { int candidate_pos = cur; int selected_matchLength = best_mlen; int selected_offset = best_off; while (1) { /* from end to beginning */ int const next_matchLength = opt[candidate_pos].mlen; /* can be 1, means literal */ int const next_offset = opt[candidate_pos].off; DEBUGLOG(7, "pos %i: sequence length %i", candidate_pos, selected_matchLength); opt[candidate_pos].mlen = selected_matchLength; opt[candidate_pos].off = selected_offset; selected_matchLength = next_matchLength; selected_offset = next_offset; if (next_matchLength > candidate_pos) break; /* last match elected, first match to encode */ assert(next_matchLength > 0); /* can be 1, means literal */ candidate_pos -= next_matchLength; } } /* encode all recorded sequences in order */ { int rPos = 0; /* relative position (to ip) */ while (rPos < last_match_pos) { int const ml = opt[rPos].mlen; int const offset = opt[rPos].off; if (ml == 1) { ip++; rPos++; continue; } /* literal; note: can end up with several literals, in which case, skip them */ rPos += ml; assert(ml >= MINMATCH); assert((offset >= 1) && (offset <= LZ4_DISTANCE_MAX)); opSaved = op; if ( LZ4HC_encodeSequence(UPDATABLE(ip, op, anchor), ml, ip - offset, limit, oend) ) { /* updates ip, op and anchor */ ovml = ml; ovref = ip - offset; goto _dest_overflow; } } } } /* while (ip <= mflimit) */ _last_literals: /* Encode Last Literals */ { size_t lastRunSize = (size_t)(iend - anchor); /* literals */ size_t llAdd = (lastRunSize + 255 - RUN_MASK) / 255; size_t const totalSize = 1 + llAdd + lastRunSize; if (limit == fillOutput) oend += LASTLITERALS; /* restore correct value */ if (limit && (op + totalSize > oend)) { if (limit == limitedOutput) { /* Check output limit */ retval = 0; goto _return_label; } /* adapt lastRunSize to fill 'dst' */ lastRunSize = (size_t)(oend - op) - 1 /*token*/; llAdd = (lastRunSize + 256 - RUN_MASK) / 256; lastRunSize -= llAdd; } DEBUGLOG(6, "Final literal run : %i literals", (int)lastRunSize); ip = anchor + lastRunSize; /* can be != iend if limit==fillOutput */ if (lastRunSize >= RUN_MASK) { size_t accumulator = lastRunSize - RUN_MASK; *op++ = (RUN_MASK << ML_BITS); for(; accumulator >= 255 ; accumulator -= 255) *op++ = 255; *op++ = (BYTE) accumulator; } else { *op++ = (BYTE)(lastRunSize << ML_BITS); } LZ4_memcpy(op, anchor, lastRunSize); op += lastRunSize; } /* End */ *srcSizePtr = (int) (((const char*)ip) - source); retval = (int) ((char*)op-dst); goto _return_label; _dest_overflow: if (limit == fillOutput) { /* Assumption : ip, anchor, ovml and ovref must be set correctly */ size_t const ll = (size_t)(ip - anchor); size_t const ll_addbytes = (ll + 240) / 255; size_t const ll_totalCost = 1 + ll_addbytes + ll; BYTE* const maxLitPos = oend - 3; /* 2 for offset, 1 for token */ DEBUGLOG(6, "Last sequence overflowing (only %i bytes remaining)", (int)(oend-1-opSaved)); op = opSaved; /* restore correct out pointer */ if (op + ll_totalCost <= maxLitPos) { /* ll validated; now adjust match length */ size_t const bytesLeftForMl = (size_t)(maxLitPos - (op+ll_totalCost)); size_t const maxMlSize = MINMATCH + (ML_MASK-1) + (bytesLeftForMl * 255); assert(maxMlSize < INT_MAX); assert(ovml >= 0); if ((size_t)ovml > maxMlSize) ovml = (int)maxMlSize; if ((oend + LASTLITERALS) - (op + ll_totalCost + 2) - 1 + ovml >= MFLIMIT) { DEBUGLOG(6, "Space to end : %i + ml (%i)", (int)((oend + LASTLITERALS) - (op + ll_totalCost + 2) - 1), ovml); DEBUGLOG(6, "Before : ip = %p, anchor = %p", ip, anchor); LZ4HC_encodeSequence(UPDATABLE(ip, op, anchor), ovml, ovref, notLimited, oend); DEBUGLOG(6, "After : ip = %p, anchor = %p", ip, anchor); } } goto _last_literals; } _return_label: #if defined(LZ4HC_HEAPMODE) && LZ4HC_HEAPMODE==1 FREEMEM(opt); #endif return retval; } zmat-0.9.9/src/lz4/LICENSE0000644000175200007730000000243714515254731015252 0ustar rlaboissrlaboissLZ4 Library Copyright (c) 2011-2020, Yann Collet All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. zmat-0.9.9/src/lz4/lz4.h0000644000175200007730000012437714515254731015137 0ustar rlaboissrlaboiss/* * LZ4 - Fast LZ compression algorithm * Header File * Copyright (C) 2011-2020, Yann Collet. BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You can contact the author at : - LZ4 homepage : http://www.lz4.org - LZ4 source repository : https://github.com/lz4/lz4 */ #if defined (__cplusplus) extern "C" { #endif #ifndef LZ4_H_2983827168210 #define LZ4_H_2983827168210 /* --- Dependency --- */ #include /* size_t */ /** Introduction LZ4 is lossless compression algorithm, providing compression speed >500 MB/s per core, scalable with multi-cores CPU. It features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems. The LZ4 compression library provides in-memory compression and decompression functions. It gives full buffer control to user. Compression can be done in: - a single step (described as Simple Functions) - a single step, reusing a context (described in Advanced Functions) - unbounded multiple steps (described as Streaming compression) lz4.h generates and decodes LZ4-compressed blocks (doc/lz4_Block_format.md). Decompressing such a compressed block requires additional metadata. Exact metadata depends on exact decompression function. For the typical case of LZ4_decompress_safe(), metadata includes block's compressed size, and maximum bound of decompressed size. Each application is free to encode and pass such metadata in whichever way it wants. lz4.h only handle blocks, it can not generate Frames. Blocks are different from Frames (doc/lz4_Frame_format.md). Frames bundle both blocks and metadata in a specified manner. Embedding metadata is required for compressed data to be self-contained and portable. Frame format is delivered through a companion API, declared in lz4frame.h. The `lz4` CLI can only manage frames. */ /*^*************************************************************** * Export parameters *****************************************************************/ /* * LZ4_DLL_EXPORT : * Enable exporting of functions when building a Windows DLL * LZ4LIB_VISIBILITY : * Control library symbols visibility. */ #ifndef LZ4LIB_VISIBILITY # if defined(__GNUC__) && (__GNUC__ >= 4) # define LZ4LIB_VISIBILITY __attribute__ ((visibility ("default"))) # else # define LZ4LIB_VISIBILITY # endif #endif #if defined(LZ4_DLL_EXPORT) && (LZ4_DLL_EXPORT==1) # define LZ4LIB_API __declspec(dllexport) LZ4LIB_VISIBILITY #elif defined(LZ4_DLL_IMPORT) && (LZ4_DLL_IMPORT==1) # define LZ4LIB_API __declspec(dllimport) LZ4LIB_VISIBILITY /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/ #else # define LZ4LIB_API LZ4LIB_VISIBILITY #endif /*! LZ4_FREESTANDING : * When this macro is set to 1, it enables "freestanding mode" that is * suitable for typical freestanding environment which doesn't support * standard C library. * * - LZ4_FREESTANDING is a compile-time switch. * - It requires the following macros to be defined: * LZ4_memcpy, LZ4_memmove, LZ4_memset. * - It only enables LZ4/HC functions which don't use heap. * All LZ4F_* functions are not supported. * - See tests/freestanding.c to check its basic setup. */ #if defined(LZ4_FREESTANDING) && (LZ4_FREESTANDING == 1) # define LZ4_HEAPMODE 0 # define LZ4HC_HEAPMODE 0 # define LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION 1 # if !defined(LZ4_memcpy) # error "LZ4_FREESTANDING requires macro 'LZ4_memcpy'." # endif # if !defined(LZ4_memset) # error "LZ4_FREESTANDING requires macro 'LZ4_memset'." # endif # if !defined(LZ4_memmove) # error "LZ4_FREESTANDING requires macro 'LZ4_memmove'." # endif #elif ! defined(LZ4_FREESTANDING) # define LZ4_FREESTANDING 0 #endif /*------ Version ------*/ #define LZ4_VERSION_MAJOR 1 /* for breaking interface changes */ #define LZ4_VERSION_MINOR 9 /* for new (non-breaking) interface capabilities */ #define LZ4_VERSION_RELEASE 4 /* for tweaks, bug-fixes, or development */ #define LZ4_VERSION_NUMBER (LZ4_VERSION_MAJOR *100*100 + LZ4_VERSION_MINOR *100 + LZ4_VERSION_RELEASE) #define LZ4_LIB_VERSION LZ4_VERSION_MAJOR.LZ4_VERSION_MINOR.LZ4_VERSION_RELEASE #define LZ4_QUOTE(str) #str #define LZ4_EXPAND_AND_QUOTE(str) LZ4_QUOTE(str) #define LZ4_VERSION_STRING LZ4_EXPAND_AND_QUOTE(LZ4_LIB_VERSION) /* requires v1.7.3+ */ LZ4LIB_API int LZ4_versionNumber (void); /**< library version number; useful to check dll version; requires v1.3.0+ */ LZ4LIB_API const char* LZ4_versionString (void); /**< library version string; useful to check dll version; requires v1.7.5+ */ /*-************************************ * Tuning parameter **************************************/ #define LZ4_MEMORY_USAGE_MIN 10 #define LZ4_MEMORY_USAGE_DEFAULT 14 #define LZ4_MEMORY_USAGE_MAX 20 /*! * LZ4_MEMORY_USAGE : * Memory usage formula : N->2^N Bytes (examples : 10 -> 1KB; 12 -> 4KB ; 16 -> 64KB; 20 -> 1MB; ) * Increasing memory usage improves compression ratio, at the cost of speed. * Reduced memory usage may improve speed at the cost of ratio, thanks to better cache locality. * Default value is 14, for 16KB, which nicely fits into Intel x86 L1 cache */ #ifndef LZ4_MEMORY_USAGE # define LZ4_MEMORY_USAGE LZ4_MEMORY_USAGE_DEFAULT #endif #if (LZ4_MEMORY_USAGE < LZ4_MEMORY_USAGE_MIN) # error "LZ4_MEMORY_USAGE is too small !" #endif #if (LZ4_MEMORY_USAGE > LZ4_MEMORY_USAGE_MAX) # error "LZ4_MEMORY_USAGE is too large !" #endif /*-************************************ * Simple Functions **************************************/ /*! LZ4_compress_default() : * Compresses 'srcSize' bytes from buffer 'src' * into already allocated 'dst' buffer of size 'dstCapacity'. * Compression is guaranteed to succeed if 'dstCapacity' >= LZ4_compressBound(srcSize). * It also runs faster, so it's a recommended setting. * If the function cannot compress 'src' into a more limited 'dst' budget, * compression stops *immediately*, and the function result is zero. * In which case, 'dst' content is undefined (invalid). * srcSize : max supported value is LZ4_MAX_INPUT_SIZE. * dstCapacity : size of buffer 'dst' (which must be already allocated) * @return : the number of bytes written into buffer 'dst' (necessarily <= dstCapacity) * or 0 if compression fails * Note : This function is protected against buffer overflow scenarios (never writes outside 'dst' buffer, nor read outside 'source' buffer). */ LZ4LIB_API int LZ4_compress_default(const char* src, char* dst, int srcSize, int dstCapacity); /*! LZ4_decompress_safe() : * compressedSize : is the exact complete size of the compressed block. * dstCapacity : is the size of destination buffer (which must be already allocated), presumed an upper bound of decompressed size. * @return : the number of bytes decompressed into destination buffer (necessarily <= dstCapacity) * If destination buffer is not large enough, decoding will stop and output an error code (negative value). * If the source stream is detected malformed, the function will stop decoding and return a negative result. * Note 1 : This function is protected against malicious data packets : * it will never writes outside 'dst' buffer, nor read outside 'source' buffer, * even if the compressed block is maliciously modified to order the decoder to do these actions. * In such case, the decoder stops immediately, and considers the compressed block malformed. * Note 2 : compressedSize and dstCapacity must be provided to the function, the compressed block does not contain them. * The implementation is free to send / store / derive this information in whichever way is most beneficial. * If there is a need for a different format which bundles together both compressed data and its metadata, consider looking at lz4frame.h instead. */ LZ4LIB_API int LZ4_decompress_safe (const char* src, char* dst, int compressedSize, int dstCapacity); /*-************************************ * Advanced Functions **************************************/ #define LZ4_MAX_INPUT_SIZE 0x7E000000 /* 2 113 929 216 bytes */ #define LZ4_COMPRESSBOUND(isize) ((unsigned)(isize) > (unsigned)LZ4_MAX_INPUT_SIZE ? 0 : (isize) + ((isize)/255) + 16) /*! LZ4_compressBound() : Provides the maximum size that LZ4 compression may output in a "worst case" scenario (input data not compressible) This function is primarily useful for memory allocation purposes (destination buffer size). Macro LZ4_COMPRESSBOUND() is also provided for compilation-time evaluation (stack memory allocation for example). Note that LZ4_compress_default() compresses faster when dstCapacity is >= LZ4_compressBound(srcSize) inputSize : max supported value is LZ4_MAX_INPUT_SIZE return : maximum output size in a "worst case" scenario or 0, if input size is incorrect (too large or negative) */ LZ4LIB_API int LZ4_compressBound(int inputSize); /*! LZ4_compress_fast() : Same as LZ4_compress_default(), but allows selection of "acceleration" factor. The larger the acceleration value, the faster the algorithm, but also the lesser the compression. It's a trade-off. It can be fine tuned, with each successive value providing roughly +~3% to speed. An acceleration value of "1" is the same as regular LZ4_compress_default() Values <= 0 will be replaced by LZ4_ACCELERATION_DEFAULT (currently == 1, see lz4.c). Values > LZ4_ACCELERATION_MAX will be replaced by LZ4_ACCELERATION_MAX (currently == 65537, see lz4.c). */ LZ4LIB_API int LZ4_compress_fast (const char* src, char* dst, int srcSize, int dstCapacity, int acceleration); /*! LZ4_compress_fast_extState() : * Same as LZ4_compress_fast(), using an externally allocated memory space for its state. * Use LZ4_sizeofState() to know how much memory must be allocated, * and allocate it on 8-bytes boundaries (using `malloc()` typically). * Then, provide this buffer as `void* state` to compression function. */ LZ4LIB_API int LZ4_sizeofState(void); LZ4LIB_API int LZ4_compress_fast_extState (void* state, const char* src, char* dst, int srcSize, int dstCapacity, int acceleration); /*! LZ4_compress_destSize() : * Reverse the logic : compresses as much data as possible from 'src' buffer * into already allocated buffer 'dst', of size >= 'targetDestSize'. * This function either compresses the entire 'src' content into 'dst' if it's large enough, * or fill 'dst' buffer completely with as much data as possible from 'src'. * note: acceleration parameter is fixed to "default". * * *srcSizePtr : will be modified to indicate how many bytes where read from 'src' to fill 'dst'. * New value is necessarily <= input value. * @return : Nb bytes written into 'dst' (necessarily <= targetDestSize) * or 0 if compression fails. * * Note : from v1.8.2 to v1.9.1, this function had a bug (fixed un v1.9.2+): * the produced compressed content could, in specific circumstances, * require to be decompressed into a destination buffer larger * by at least 1 byte than the content to decompress. * If an application uses `LZ4_compress_destSize()`, * it's highly recommended to update liblz4 to v1.9.2 or better. * If this can't be done or ensured, * the receiving decompression function should provide * a dstCapacity which is > decompressedSize, by at least 1 byte. * See https://github.com/lz4/lz4/issues/859 for details */ LZ4LIB_API int LZ4_compress_destSize (const char* src, char* dst, int* srcSizePtr, int targetDstSize); /*! LZ4_decompress_safe_partial() : * Decompress an LZ4 compressed block, of size 'srcSize' at position 'src', * into destination buffer 'dst' of size 'dstCapacity'. * Up to 'targetOutputSize' bytes will be decoded. * The function stops decoding on reaching this objective. * This can be useful to boost performance * whenever only the beginning of a block is required. * * @return : the number of bytes decoded in `dst` (necessarily <= targetOutputSize) * If source stream is detected malformed, function returns a negative result. * * Note 1 : @return can be < targetOutputSize, if compressed block contains less data. * * Note 2 : targetOutputSize must be <= dstCapacity * * Note 3 : this function effectively stops decoding on reaching targetOutputSize, * so dstCapacity is kind of redundant. * This is because in older versions of this function, * decoding operation would still write complete sequences. * Therefore, there was no guarantee that it would stop writing at exactly targetOutputSize, * it could write more bytes, though only up to dstCapacity. * Some "margin" used to be required for this operation to work properly. * Thankfully, this is no longer necessary. * The function nonetheless keeps the same signature, in an effort to preserve API compatibility. * * Note 4 : If srcSize is the exact size of the block, * then targetOutputSize can be any value, * including larger than the block's decompressed size. * The function will, at most, generate block's decompressed size. * * Note 5 : If srcSize is _larger_ than block's compressed size, * then targetOutputSize **MUST** be <= block's decompressed size. * Otherwise, *silent corruption will occur*. */ LZ4LIB_API int LZ4_decompress_safe_partial (const char* src, char* dst, int srcSize, int targetOutputSize, int dstCapacity); /*-********************************************* * Streaming Compression Functions ***********************************************/ typedef union LZ4_stream_u LZ4_stream_t; /* incomplete type (defined later) */ /** Note about RC_INVOKED - RC_INVOKED is predefined symbol of rc.exe (the resource compiler which is part of MSVC/Visual Studio). https://docs.microsoft.com/en-us/windows/win32/menurc/predefined-macros - Since rc.exe is a legacy compiler, it truncates long symbol (> 30 chars) and reports warning "RC4011: identifier truncated". - To eliminate the warning, we surround long preprocessor symbol with "#if !defined(RC_INVOKED) ... #endif" block that means "skip this block when rc.exe is trying to read it". */ #if !defined(RC_INVOKED) /* https://docs.microsoft.com/en-us/windows/win32/menurc/predefined-macros */ #if !defined(LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION) LZ4LIB_API LZ4_stream_t* LZ4_createStream(void); LZ4LIB_API int LZ4_freeStream (LZ4_stream_t* streamPtr); #endif /* !defined(LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION) */ #endif /*! LZ4_resetStream_fast() : v1.9.0+ * Use this to prepare an LZ4_stream_t for a new chain of dependent blocks * (e.g., LZ4_compress_fast_continue()). * * An LZ4_stream_t must be initialized once before usage. * This is automatically done when created by LZ4_createStream(). * However, should the LZ4_stream_t be simply declared on stack (for example), * it's necessary to initialize it first, using LZ4_initStream(). * * After init, start any new stream with LZ4_resetStream_fast(). * A same LZ4_stream_t can be re-used multiple times consecutively * and compress multiple streams, * provided that it starts each new stream with LZ4_resetStream_fast(). * * LZ4_resetStream_fast() is much faster than LZ4_initStream(), * but is not compatible with memory regions containing garbage data. * * Note: it's only useful to call LZ4_resetStream_fast() * in the context of streaming compression. * The *extState* functions perform their own resets. * Invoking LZ4_resetStream_fast() before is redundant, and even counterproductive. */ LZ4LIB_API void LZ4_resetStream_fast (LZ4_stream_t* streamPtr); /*! LZ4_loadDict() : * Use this function to reference a static dictionary into LZ4_stream_t. * The dictionary must remain available during compression. * LZ4_loadDict() triggers a reset, so any previous data will be forgotten. * The same dictionary will have to be loaded on decompression side for successful decoding. * Dictionary are useful for better compression of small data (KB range). * While LZ4 accept any input as dictionary, * results are generally better when using Zstandard's Dictionary Builder. * Loading a size of 0 is allowed, and is the same as reset. * @return : loaded dictionary size, in bytes (necessarily <= 64 KB) */ LZ4LIB_API int LZ4_loadDict (LZ4_stream_t* streamPtr, const char* dictionary, int dictSize); /*! LZ4_compress_fast_continue() : * Compress 'src' content using data from previously compressed blocks, for better compression ratio. * 'dst' buffer must be already allocated. * If dstCapacity >= LZ4_compressBound(srcSize), compression is guaranteed to succeed, and runs faster. * * @return : size of compressed block * or 0 if there is an error (typically, cannot fit into 'dst'). * * Note 1 : Each invocation to LZ4_compress_fast_continue() generates a new block. * Each block has precise boundaries. * Each block must be decompressed separately, calling LZ4_decompress_*() with relevant metadata. * It's not possible to append blocks together and expect a single invocation of LZ4_decompress_*() to decompress them together. * * Note 2 : The previous 64KB of source data is __assumed__ to remain present, unmodified, at same address in memory ! * * Note 3 : When input is structured as a double-buffer, each buffer can have any size, including < 64 KB. * Make sure that buffers are separated, by at least one byte. * This construction ensures that each block only depends on previous block. * * Note 4 : If input buffer is a ring-buffer, it can have any size, including < 64 KB. * * Note 5 : After an error, the stream status is undefined (invalid), it can only be reset or freed. */ LZ4LIB_API int LZ4_compress_fast_continue (LZ4_stream_t* streamPtr, const char* src, char* dst, int srcSize, int dstCapacity, int acceleration); /*! LZ4_saveDict() : * If last 64KB data cannot be guaranteed to remain available at its current memory location, * save it into a safer place (char* safeBuffer). * This is schematically equivalent to a memcpy() followed by LZ4_loadDict(), * but is much faster, because LZ4_saveDict() doesn't need to rebuild tables. * @return : saved dictionary size in bytes (necessarily <= maxDictSize), or 0 if error. */ LZ4LIB_API int LZ4_saveDict (LZ4_stream_t* streamPtr, char* safeBuffer, int maxDictSize); /*-********************************************** * Streaming Decompression Functions * Bufferless synchronous API ************************************************/ typedef union LZ4_streamDecode_u LZ4_streamDecode_t; /* tracking context */ /*! LZ4_createStreamDecode() and LZ4_freeStreamDecode() : * creation / destruction of streaming decompression tracking context. * A tracking context can be re-used multiple times. */ #if !defined(RC_INVOKED) /* https://docs.microsoft.com/en-us/windows/win32/menurc/predefined-macros */ #if !defined(LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION) LZ4LIB_API LZ4_streamDecode_t* LZ4_createStreamDecode(void); LZ4LIB_API int LZ4_freeStreamDecode (LZ4_streamDecode_t* LZ4_stream); #endif /* !defined(LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION) */ #endif /*! LZ4_setStreamDecode() : * An LZ4_streamDecode_t context can be allocated once and re-used multiple times. * Use this function to start decompression of a new stream of blocks. * A dictionary can optionally be set. Use NULL or size 0 for a reset order. * Dictionary is presumed stable : it must remain accessible and unmodified during next decompression. * @return : 1 if OK, 0 if error */ LZ4LIB_API int LZ4_setStreamDecode (LZ4_streamDecode_t* LZ4_streamDecode, const char* dictionary, int dictSize); /*! LZ4_decoderRingBufferSize() : v1.8.2+ * Note : in a ring buffer scenario (optional), * blocks are presumed decompressed next to each other * up to the moment there is not enough remaining space for next block (remainingSize < maxBlockSize), * at which stage it resumes from beginning of ring buffer. * When setting such a ring buffer for streaming decompression, * provides the minimum size of this ring buffer * to be compatible with any source respecting maxBlockSize condition. * @return : minimum ring buffer size, * or 0 if there is an error (invalid maxBlockSize). */ LZ4LIB_API int LZ4_decoderRingBufferSize(int maxBlockSize); #define LZ4_DECODER_RING_BUFFER_SIZE(maxBlockSize) (65536 + 14 + (maxBlockSize)) /* for static allocation; maxBlockSize presumed valid */ /*! LZ4_decompress_*_continue() : * These decoding functions allow decompression of consecutive blocks in "streaming" mode. * A block is an unsplittable entity, it must be presented entirely to a decompression function. * Decompression functions only accepts one block at a time. * The last 64KB of previously decoded data *must* remain available and unmodified at the memory position where they were decoded. * If less than 64KB of data has been decoded, all the data must be present. * * Special : if decompression side sets a ring buffer, it must respect one of the following conditions : * - Decompression buffer size is _at least_ LZ4_decoderRingBufferSize(maxBlockSize). * maxBlockSize is the maximum size of any single block. It can have any value > 16 bytes. * In which case, encoding and decoding buffers do not need to be synchronized. * Actually, data can be produced by any source compliant with LZ4 format specification, and respecting maxBlockSize. * - Synchronized mode : * Decompression buffer size is _exactly_ the same as compression buffer size, * and follows exactly same update rule (block boundaries at same positions), * and decoding function is provided with exact decompressed size of each block (exception for last block of the stream), * _then_ decoding & encoding ring buffer can have any size, including small ones ( < 64 KB). * - Decompression buffer is larger than encoding buffer, by a minimum of maxBlockSize more bytes. * In which case, encoding and decoding buffers do not need to be synchronized, * and encoding ring buffer can have any size, including small ones ( < 64 KB). * * Whenever these conditions are not possible, * save the last 64KB of decoded data into a safe buffer where it can't be modified during decompression, * then indicate where this data is saved using LZ4_setStreamDecode(), before decompressing next block. */ LZ4LIB_API int LZ4_decompress_safe_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* src, char* dst, int srcSize, int dstCapacity); /*! LZ4_decompress_*_usingDict() : * These decoding functions work the same as * a combination of LZ4_setStreamDecode() followed by LZ4_decompress_*_continue() * They are stand-alone, and don't need an LZ4_streamDecode_t structure. * Dictionary is presumed stable : it must remain accessible and unmodified during decompression. * Performance tip : Decompression speed can be substantially increased * when dst == dictStart + dictSize. */ LZ4LIB_API int LZ4_decompress_safe_usingDict(const char* src, char* dst, int srcSize, int dstCapacity, const char* dictStart, int dictSize); LZ4LIB_API int LZ4_decompress_safe_partial_usingDict(const char* src, char* dst, int compressedSize, int targetOutputSize, int maxOutputSize, const char* dictStart, int dictSize); #endif /* LZ4_H_2983827168210 */ /*^************************************* * !!!!!! STATIC LINKING ONLY !!!!!! ***************************************/ /*-**************************************************************************** * Experimental section * * Symbols declared in this section must be considered unstable. Their * signatures or semantics may change, or they may be removed altogether in the * future. They are therefore only safe to depend on when the caller is * statically linked against the library. * * To protect against unsafe usage, not only are the declarations guarded, * the definitions are hidden by default * when building LZ4 as a shared/dynamic library. * * In order to access these declarations, * define LZ4_STATIC_LINKING_ONLY in your application * before including LZ4's headers. * * In order to make their implementations accessible dynamically, you must * define LZ4_PUBLISH_STATIC_FUNCTIONS when building the LZ4 library. ******************************************************************************/ #ifdef LZ4_STATIC_LINKING_ONLY #ifndef LZ4_STATIC_3504398509 #define LZ4_STATIC_3504398509 #ifdef LZ4_PUBLISH_STATIC_FUNCTIONS #define LZ4LIB_STATIC_API LZ4LIB_API #else #define LZ4LIB_STATIC_API #endif /*! LZ4_compress_fast_extState_fastReset() : * A variant of LZ4_compress_fast_extState(). * * Using this variant avoids an expensive initialization step. * It is only safe to call if the state buffer is known to be correctly initialized already * (see above comment on LZ4_resetStream_fast() for a definition of "correctly initialized"). * From a high level, the difference is that * this function initializes the provided state with a call to something like LZ4_resetStream_fast() * while LZ4_compress_fast_extState() starts with a call to LZ4_resetStream(). */ LZ4LIB_STATIC_API int LZ4_compress_fast_extState_fastReset (void* state, const char* src, char* dst, int srcSize, int dstCapacity, int acceleration); /*! LZ4_attach_dictionary() : * This is an experimental API that allows * efficient use of a static dictionary many times. * * Rather than re-loading the dictionary buffer into a working context before * each compression, or copying a pre-loaded dictionary's LZ4_stream_t into a * working LZ4_stream_t, this function introduces a no-copy setup mechanism, * in which the working stream references the dictionary stream in-place. * * Several assumptions are made about the state of the dictionary stream. * Currently, only streams which have been prepared by LZ4_loadDict() should * be expected to work. * * Alternatively, the provided dictionaryStream may be NULL, * in which case any existing dictionary stream is unset. * * If a dictionary is provided, it replaces any pre-existing stream history. * The dictionary contents are the only history that can be referenced and * logically immediately precede the data compressed in the first subsequent * compression call. * * The dictionary will only remain attached to the working stream through the * first compression call, at the end of which it is cleared. The dictionary * stream (and source buffer) must remain in-place / accessible / unchanged * through the completion of the first compression call on the stream. */ LZ4LIB_STATIC_API void LZ4_attach_dictionary(LZ4_stream_t* workingStream, const LZ4_stream_t* dictionaryStream); /*! In-place compression and decompression * * It's possible to have input and output sharing the same buffer, * for highly constrained memory environments. * In both cases, it requires input to lay at the end of the buffer, * and decompression to start at beginning of the buffer. * Buffer size must feature some margin, hence be larger than final size. * * |<------------------------buffer--------------------------------->| * |<-----------compressed data--------->| * |<-----------decompressed size------------------>| * |<----margin---->| * * This technique is more useful for decompression, * since decompressed size is typically larger, * and margin is short. * * In-place decompression will work inside any buffer * which size is >= LZ4_DECOMPRESS_INPLACE_BUFFER_SIZE(decompressedSize). * This presumes that decompressedSize > compressedSize. * Otherwise, it means compression actually expanded data, * and it would be more efficient to store such data with a flag indicating it's not compressed. * This can happen when data is not compressible (already compressed, or encrypted). * * For in-place compression, margin is larger, as it must be able to cope with both * history preservation, requiring input data to remain unmodified up to LZ4_DISTANCE_MAX, * and data expansion, which can happen when input is not compressible. * As a consequence, buffer size requirements are much higher, * and memory savings offered by in-place compression are more limited. * * There are ways to limit this cost for compression : * - Reduce history size, by modifying LZ4_DISTANCE_MAX. * Note that it is a compile-time constant, so all compressions will apply this limit. * Lower values will reduce compression ratio, except when input_size < LZ4_DISTANCE_MAX, * so it's a reasonable trick when inputs are known to be small. * - Require the compressor to deliver a "maximum compressed size". * This is the `dstCapacity` parameter in `LZ4_compress*()`. * When this size is < LZ4_COMPRESSBOUND(inputSize), then compression can fail, * in which case, the return code will be 0 (zero). * The caller must be ready for these cases to happen, * and typically design a backup scheme to send data uncompressed. * The combination of both techniques can significantly reduce * the amount of margin required for in-place compression. * * In-place compression can work in any buffer * which size is >= (maxCompressedSize) * with maxCompressedSize == LZ4_COMPRESSBOUND(srcSize) for guaranteed compression success. * LZ4_COMPRESS_INPLACE_BUFFER_SIZE() depends on both maxCompressedSize and LZ4_DISTANCE_MAX, * so it's possible to reduce memory requirements by playing with them. */ #define LZ4_DECOMPRESS_INPLACE_MARGIN(compressedSize) (((compressedSize) >> 8) + 32) #define LZ4_DECOMPRESS_INPLACE_BUFFER_SIZE(decompressedSize) ((decompressedSize) + LZ4_DECOMPRESS_INPLACE_MARGIN(decompressedSize)) /**< note: presumes that compressedSize < decompressedSize. note2: margin is overestimated a bit, since it could use compressedSize instead */ #ifndef LZ4_DISTANCE_MAX /* history window size; can be user-defined at compile time */ # define LZ4_DISTANCE_MAX 65535 /* set to maximum value by default */ #endif #define LZ4_COMPRESS_INPLACE_MARGIN (LZ4_DISTANCE_MAX + 32) /* LZ4_DISTANCE_MAX can be safely replaced by srcSize when it's smaller */ #define LZ4_COMPRESS_INPLACE_BUFFER_SIZE(maxCompressedSize) ((maxCompressedSize) + LZ4_COMPRESS_INPLACE_MARGIN) /**< maxCompressedSize is generally LZ4_COMPRESSBOUND(inputSize), but can be set to any lower value, with the risk that compression can fail (return code 0(zero)) */ #endif /* LZ4_STATIC_3504398509 */ #endif /* LZ4_STATIC_LINKING_ONLY */ #ifndef LZ4_H_98237428734687 #define LZ4_H_98237428734687 /*-************************************************************ * Private Definitions ************************************************************** * Do not use these definitions directly. * They are only exposed to allow static allocation of `LZ4_stream_t` and `LZ4_streamDecode_t`. * Accessing members will expose user code to API and/or ABI break in future versions of the library. **************************************************************/ #define LZ4_HASHLOG (LZ4_MEMORY_USAGE-2) #define LZ4_HASHTABLESIZE (1 << LZ4_MEMORY_USAGE) #define LZ4_HASH_SIZE_U32 (1 << LZ4_HASHLOG) /* required as macro for static allocation */ #if defined(__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) # include typedef int8_t LZ4_i8; typedef uint8_t LZ4_byte; typedef uint16_t LZ4_u16; typedef uint32_t LZ4_u32; #else typedef signed char LZ4_i8; typedef unsigned char LZ4_byte; typedef unsigned short LZ4_u16; typedef unsigned int LZ4_u32; #endif /*! LZ4_stream_t : * Never ever use below internal definitions directly ! * These definitions are not API/ABI safe, and may change in future versions. * If you need static allocation, declare or allocate an LZ4_stream_t object. **/ typedef struct LZ4_stream_t_internal LZ4_stream_t_internal; struct LZ4_stream_t_internal { LZ4_u32 hashTable[LZ4_HASH_SIZE_U32]; const LZ4_byte* dictionary; const LZ4_stream_t_internal* dictCtx; LZ4_u32 currentOffset; LZ4_u32 tableType; LZ4_u32 dictSize; /* Implicit padding to ensure structure is aligned */ }; #define LZ4_STREAM_MINSIZE ((1UL << LZ4_MEMORY_USAGE) + 32) /* static size, for inter-version compatibility */ union LZ4_stream_u { char minStateSize[LZ4_STREAM_MINSIZE]; LZ4_stream_t_internal internal_donotuse; }; /* previously typedef'd to LZ4_stream_t */ /*! LZ4_initStream() : v1.9.0+ * An LZ4_stream_t structure must be initialized at least once. * This is automatically done when invoking LZ4_createStream(), * but it's not when the structure is simply declared on stack (for example). * * Use LZ4_initStream() to properly initialize a newly declared LZ4_stream_t. * It can also initialize any arbitrary buffer of sufficient size, * and will @return a pointer of proper type upon initialization. * * Note : initialization fails if size and alignment conditions are not respected. * In which case, the function will @return NULL. * Note2: An LZ4_stream_t structure guarantees correct alignment and size. * Note3: Before v1.9.0, use LZ4_resetStream() instead **/ LZ4LIB_API LZ4_stream_t* LZ4_initStream (void* buffer, size_t size); /*! LZ4_streamDecode_t : * Never ever use below internal definitions directly ! * These definitions are not API/ABI safe, and may change in future versions. * If you need static allocation, declare or allocate an LZ4_streamDecode_t object. **/ typedef struct { const LZ4_byte* externalDict; const LZ4_byte* prefixEnd; size_t extDictSize; size_t prefixSize; } LZ4_streamDecode_t_internal; #define LZ4_STREAMDECODE_MINSIZE 32 union LZ4_streamDecode_u { char minStateSize[LZ4_STREAMDECODE_MINSIZE]; LZ4_streamDecode_t_internal internal_donotuse; } ; /* previously typedef'd to LZ4_streamDecode_t */ /*-************************************ * Obsolete Functions **************************************/ /*! Deprecation warnings * * Deprecated functions make the compiler generate a warning when invoked. * This is meant to invite users to update their source code. * Should deprecation warnings be a problem, it is generally possible to disable them, * typically with -Wno-deprecated-declarations for gcc * or _CRT_SECURE_NO_WARNINGS in Visual. * * Another method is to define LZ4_DISABLE_DEPRECATE_WARNINGS * before including the header file. */ #ifdef LZ4_DISABLE_DEPRECATE_WARNINGS # define LZ4_DEPRECATED(message) /* disable deprecation warnings */ #else # if defined (__cplusplus) && (__cplusplus >= 201402) /* C++14 or greater */ # define LZ4_DEPRECATED(message) [[deprecated(message)]] # elif defined(_MSC_VER) # define LZ4_DEPRECATED(message) __declspec(deprecated(message)) # elif defined(__clang__) || (defined(__GNUC__) && (__GNUC__ * 10 + __GNUC_MINOR__ >= 45)) # define LZ4_DEPRECATED(message) __attribute__((deprecated(message))) # elif defined(__GNUC__) && (__GNUC__ * 10 + __GNUC_MINOR__ >= 31) # define LZ4_DEPRECATED(message) __attribute__((deprecated)) # else # pragma message("WARNING: LZ4_DEPRECATED needs custom implementation for this compiler") # define LZ4_DEPRECATED(message) /* disabled */ # endif #endif /* LZ4_DISABLE_DEPRECATE_WARNINGS */ /*! Obsolete compression functions (since v1.7.3) */ LZ4_DEPRECATED("use LZ4_compress_default() instead") LZ4LIB_API int LZ4_compress (const char* src, char* dest, int srcSize); LZ4_DEPRECATED("use LZ4_compress_default() instead") LZ4LIB_API int LZ4_compress_limitedOutput (const char* src, char* dest, int srcSize, int maxOutputSize); LZ4_DEPRECATED("use LZ4_compress_fast_extState() instead") LZ4LIB_API int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize); LZ4_DEPRECATED("use LZ4_compress_fast_extState() instead") LZ4LIB_API int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize); LZ4_DEPRECATED("use LZ4_compress_fast_continue() instead") LZ4LIB_API int LZ4_compress_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize); LZ4_DEPRECATED("use LZ4_compress_fast_continue() instead") LZ4LIB_API int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize, int maxOutputSize); /*! Obsolete decompression functions (since v1.8.0) */ LZ4_DEPRECATED("use LZ4_decompress_fast() instead") LZ4LIB_API int LZ4_uncompress (const char* source, char* dest, int outputSize); LZ4_DEPRECATED("use LZ4_decompress_safe() instead") LZ4LIB_API int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int maxOutputSize); /* Obsolete streaming functions (since v1.7.0) * degraded functionality; do not use! * * In order to perform streaming compression, these functions depended on data * that is no longer tracked in the state. They have been preserved as well as * possible: using them will still produce a correct output. However, they don't * actually retain any history between compression calls. The compression ratio * achieved will therefore be no better than compressing each chunk * independently. */ LZ4_DEPRECATED("Use LZ4_createStream() instead") LZ4LIB_API void* LZ4_create (char* inputBuffer); LZ4_DEPRECATED("Use LZ4_createStream() instead") LZ4LIB_API int LZ4_sizeofStreamState(void); LZ4_DEPRECATED("Use LZ4_resetStream() instead") LZ4LIB_API int LZ4_resetStreamState(void* state, char* inputBuffer); LZ4_DEPRECATED("Use LZ4_saveDict() instead") LZ4LIB_API char* LZ4_slideInputBuffer (void* state); /*! Obsolete streaming decoding functions (since v1.7.0) */ LZ4_DEPRECATED("use LZ4_decompress_safe_usingDict() instead") LZ4LIB_API int LZ4_decompress_safe_withPrefix64k (const char* src, char* dst, int compressedSize, int maxDstSize); LZ4_DEPRECATED("use LZ4_decompress_fast_usingDict() instead") LZ4LIB_API int LZ4_decompress_fast_withPrefix64k (const char* src, char* dst, int originalSize); /*! Obsolete LZ4_decompress_fast variants (since v1.9.0) : * These functions used to be faster than LZ4_decompress_safe(), * but this is no longer the case. They are now slower. * This is because LZ4_decompress_fast() doesn't know the input size, * and therefore must progress more cautiously into the input buffer to not read beyond the end of block. * On top of that `LZ4_decompress_fast()` is not protected vs malformed or malicious inputs, making it a security liability. * As a consequence, LZ4_decompress_fast() is strongly discouraged, and deprecated. * * The last remaining LZ4_decompress_fast() specificity is that * it can decompress a block without knowing its compressed size. * Such functionality can be achieved in a more secure manner * by employing LZ4_decompress_safe_partial(). * * Parameters: * originalSize : is the uncompressed size to regenerate. * `dst` must be already allocated, its size must be >= 'originalSize' bytes. * @return : number of bytes read from source buffer (== compressed size). * The function expects to finish at block's end exactly. * If the source stream is detected malformed, the function stops decoding and returns a negative result. * note : LZ4_decompress_fast*() requires originalSize. Thanks to this information, it never writes past the output buffer. * However, since it doesn't know its 'src' size, it may read an unknown amount of input, past input buffer bounds. * Also, since match offsets are not validated, match reads from 'src' may underflow too. * These issues never happen if input (compressed) data is correct. * But they may happen if input data is invalid (error or intentional tampering). * As a consequence, use these functions in trusted environments with trusted data **only**. */ LZ4_DEPRECATED("This function is deprecated and unsafe. Consider using LZ4_decompress_safe() instead") LZ4LIB_API int LZ4_decompress_fast (const char* src, char* dst, int originalSize); LZ4_DEPRECATED("This function is deprecated and unsafe. Consider using LZ4_decompress_safe_continue() instead") LZ4LIB_API int LZ4_decompress_fast_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* src, char* dst, int originalSize); LZ4_DEPRECATED("This function is deprecated and unsafe. Consider using LZ4_decompress_safe_usingDict() instead") LZ4LIB_API int LZ4_decompress_fast_usingDict (const char* src, char* dst, int originalSize, const char* dictStart, int dictSize); /*! LZ4_resetStream() : * An LZ4_stream_t structure must be initialized at least once. * This is done with LZ4_initStream(), or LZ4_resetStream(). * Consider switching to LZ4_initStream(), * invoking LZ4_resetStream() will trigger deprecation warnings in the future. */ LZ4LIB_API void LZ4_resetStream (LZ4_stream_t* streamPtr); #endif /* LZ4_H_98237428734687 */ #if defined (__cplusplus) } #endif zmat-0.9.9/src/lz4/lz4hc.h0000644000175200007730000004732314515254731015445 0ustar rlaboissrlaboiss/* LZ4 HC - High Compression Mode of LZ4 Header File Copyright (C) 2011-2020, Yann Collet. BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You can contact the author at : - LZ4 source repository : https://github.com/lz4/lz4 - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c */ #ifndef LZ4_HC_H_19834876238432 #define LZ4_HC_H_19834876238432 #if defined (__cplusplus) extern "C" { #endif /* --- Dependency --- */ /* note : lz4hc requires lz4.h/lz4.c for compilation */ #include "lz4.h" /* stddef, LZ4LIB_API, LZ4_DEPRECATED */ /* --- Useful constants --- */ #define LZ4HC_CLEVEL_MIN 3 #define LZ4HC_CLEVEL_DEFAULT 9 #define LZ4HC_CLEVEL_OPT_MIN 10 #define LZ4HC_CLEVEL_MAX 12 /*-************************************ * Block Compression **************************************/ /*! LZ4_compress_HC() : * Compress data from `src` into `dst`, using the powerful but slower "HC" algorithm. * `dst` must be already allocated. * Compression is guaranteed to succeed if `dstCapacity >= LZ4_compressBound(srcSize)` (see "lz4.h") * Max supported `srcSize` value is LZ4_MAX_INPUT_SIZE (see "lz4.h") * `compressionLevel` : any value between 1 and LZ4HC_CLEVEL_MAX will work. * Values > LZ4HC_CLEVEL_MAX behave the same as LZ4HC_CLEVEL_MAX. * @return : the number of bytes written into 'dst' * or 0 if compression fails. */ LZ4LIB_API int LZ4_compress_HC (const char* src, char* dst, int srcSize, int dstCapacity, int compressionLevel); /* Note : * Decompression functions are provided within "lz4.h" (BSD license) */ /*! LZ4_compress_HC_extStateHC() : * Same as LZ4_compress_HC(), but using an externally allocated memory segment for `state`. * `state` size is provided by LZ4_sizeofStateHC(). * Memory segment must be aligned on 8-bytes boundaries (which a normal malloc() should do properly). */ LZ4LIB_API int LZ4_sizeofStateHC(void); LZ4LIB_API int LZ4_compress_HC_extStateHC(void* stateHC, const char* src, char* dst, int srcSize, int maxDstSize, int compressionLevel); /*! LZ4_compress_HC_destSize() : v1.9.0+ * Will compress as much data as possible from `src` * to fit into `targetDstSize` budget. * Result is provided in 2 parts : * @return : the number of bytes written into 'dst' (necessarily <= targetDstSize) * or 0 if compression fails. * `srcSizePtr` : on success, *srcSizePtr is updated to indicate how much bytes were read from `src` */ LZ4LIB_API int LZ4_compress_HC_destSize(void* stateHC, const char* src, char* dst, int* srcSizePtr, int targetDstSize, int compressionLevel); /*-************************************ * Streaming Compression * Bufferless synchronous API **************************************/ typedef union LZ4_streamHC_u LZ4_streamHC_t; /* incomplete type (defined later) */ /*! LZ4_createStreamHC() and LZ4_freeStreamHC() : * These functions create and release memory for LZ4 HC streaming state. * Newly created states are automatically initialized. * A same state can be used multiple times consecutively, * starting with LZ4_resetStreamHC_fast() to start a new stream of blocks. */ LZ4LIB_API LZ4_streamHC_t* LZ4_createStreamHC(void); LZ4LIB_API int LZ4_freeStreamHC (LZ4_streamHC_t* streamHCPtr); /* These functions compress data in successive blocks of any size, using previous blocks as dictionary, to improve compression ratio. One key assumption is that previous blocks (up to 64 KB) remain read-accessible while compressing next blocks. There is an exception for ring buffers, which can be smaller than 64 KB. Ring-buffer scenario is automatically detected and handled within LZ4_compress_HC_continue(). Before starting compression, state must be allocated and properly initialized. LZ4_createStreamHC() does both, though compression level is set to LZ4HC_CLEVEL_DEFAULT. Selecting the compression level can be done with LZ4_resetStreamHC_fast() (starts a new stream) or LZ4_setCompressionLevel() (anytime, between blocks in the same stream) (experimental). LZ4_resetStreamHC_fast() only works on states which have been properly initialized at least once, which is automatically the case when state is created using LZ4_createStreamHC(). After reset, a first "fictional block" can be designated as initial dictionary, using LZ4_loadDictHC() (Optional). Invoke LZ4_compress_HC_continue() to compress each successive block. The number of blocks is unlimited. Previous input blocks, including initial dictionary when present, must remain accessible and unmodified during compression. It's allowed to update compression level anytime between blocks, using LZ4_setCompressionLevel() (experimental). 'dst' buffer should be sized to handle worst case scenarios (see LZ4_compressBound(), it ensures compression success). In case of failure, the API does not guarantee recovery, so the state _must_ be reset. To ensure compression success whenever `dst` buffer size cannot be made >= LZ4_compressBound(), consider using LZ4_compress_HC_continue_destSize(). Whenever previous input blocks can't be preserved unmodified in-place during compression of next blocks, it's possible to copy the last blocks into a more stable memory space, using LZ4_saveDictHC(). Return value of LZ4_saveDictHC() is the size of dictionary effectively saved into 'safeBuffer' (<= 64 KB) After completing a streaming compression, it's possible to start a new stream of blocks, using the same LZ4_streamHC_t state, just by resetting it, using LZ4_resetStreamHC_fast(). */ LZ4LIB_API void LZ4_resetStreamHC_fast(LZ4_streamHC_t* streamHCPtr, int compressionLevel); /* v1.9.0+ */ LZ4LIB_API int LZ4_loadDictHC (LZ4_streamHC_t* streamHCPtr, const char* dictionary, int dictSize); LZ4LIB_API int LZ4_compress_HC_continue (LZ4_streamHC_t* streamHCPtr, const char* src, char* dst, int srcSize, int maxDstSize); /*! LZ4_compress_HC_continue_destSize() : v1.9.0+ * Similar to LZ4_compress_HC_continue(), * but will read as much data as possible from `src` * to fit into `targetDstSize` budget. * Result is provided into 2 parts : * @return : the number of bytes written into 'dst' (necessarily <= targetDstSize) * or 0 if compression fails. * `srcSizePtr` : on success, *srcSizePtr will be updated to indicate how much bytes were read from `src`. * Note that this function may not consume the entire input. */ LZ4LIB_API int LZ4_compress_HC_continue_destSize(LZ4_streamHC_t* LZ4_streamHCPtr, const char* src, char* dst, int* srcSizePtr, int targetDstSize); LZ4LIB_API int LZ4_saveDictHC (LZ4_streamHC_t* streamHCPtr, char* safeBuffer, int maxDictSize); /*^********************************************** * !!!!!! STATIC LINKING ONLY !!!!!! ***********************************************/ /*-****************************************************************** * PRIVATE DEFINITIONS : * Do not use these definitions directly. * They are merely exposed to allow static allocation of `LZ4_streamHC_t`. * Declare an `LZ4_streamHC_t` directly, rather than any type below. * Even then, only do so in the context of static linking, as definitions may change between versions. ********************************************************************/ #define LZ4HC_DICTIONARY_LOGSIZE 16 #define LZ4HC_MAXD (1<= LZ4HC_CLEVEL_OPT_MIN. */ LZ4LIB_STATIC_API void LZ4_favorDecompressionSpeed( LZ4_streamHC_t* LZ4_streamHCPtr, int favor); /*! LZ4_resetStreamHC_fast() : v1.9.0+ * When an LZ4_streamHC_t is known to be in a internally coherent state, * it can often be prepared for a new compression with almost no work, only * sometimes falling back to the full, expensive reset that is always required * when the stream is in an indeterminate state (i.e., the reset performed by * LZ4_resetStreamHC()). * * LZ4_streamHCs are guaranteed to be in a valid state when: * - returned from LZ4_createStreamHC() * - reset by LZ4_resetStreamHC() * - memset(stream, 0, sizeof(LZ4_streamHC_t)) * - the stream was in a valid state and was reset by LZ4_resetStreamHC_fast() * - the stream was in a valid state and was then used in any compression call * that returned success * - the stream was in an indeterminate state and was used in a compression * call that fully reset the state (LZ4_compress_HC_extStateHC()) and that * returned success * * Note: * A stream that was last used in a compression call that returned an error * may be passed to this function. However, it will be fully reset, which will * clear any existing history and settings from the context. */ LZ4LIB_STATIC_API void LZ4_resetStreamHC_fast( LZ4_streamHC_t* LZ4_streamHCPtr, int compressionLevel); /*! LZ4_compress_HC_extStateHC_fastReset() : * A variant of LZ4_compress_HC_extStateHC(). * * Using this variant avoids an expensive initialization step. It is only safe * to call if the state buffer is known to be correctly initialized already * (see above comment on LZ4_resetStreamHC_fast() for a definition of * "correctly initialized"). From a high level, the difference is that this * function initializes the provided state with a call to * LZ4_resetStreamHC_fast() while LZ4_compress_HC_extStateHC() starts with a * call to LZ4_resetStreamHC(). */ LZ4LIB_STATIC_API int LZ4_compress_HC_extStateHC_fastReset ( void* state, const char* src, char* dst, int srcSize, int dstCapacity, int compressionLevel); /*! LZ4_attach_HC_dictionary() : * This is an experimental API that allows for the efficient use of a * static dictionary many times. * * Rather than re-loading the dictionary buffer into a working context before * each compression, or copying a pre-loaded dictionary's LZ4_streamHC_t into a * working LZ4_streamHC_t, this function introduces a no-copy setup mechanism, * in which the working stream references the dictionary stream in-place. * * Several assumptions are made about the state of the dictionary stream. * Currently, only streams which have been prepared by LZ4_loadDictHC() should * be expected to work. * * Alternatively, the provided dictionary stream pointer may be NULL, in which * case any existing dictionary stream is unset. * * A dictionary should only be attached to a stream without any history (i.e., * a stream that has just been reset). * * The dictionary will remain attached to the working stream only for the * current stream session. Calls to LZ4_resetStreamHC(_fast) will remove the * dictionary context association from the working stream. The dictionary * stream (and source buffer) must remain in-place / accessible / unchanged * through the lifetime of the stream session. */ LZ4LIB_STATIC_API void LZ4_attach_HC_dictionary( LZ4_streamHC_t *working_stream, const LZ4_streamHC_t *dictionary_stream); #if defined (__cplusplus) } #endif #endif /* LZ4_HC_SLO_098092834 */ #endif /* LZ4_HC_STATIC_LINKING_ONLY */ zmat-0.9.9/src/CMakeLists.txt0000644000175200007730000000532714515254731016275 0ustar rlaboissrlaboiss################################################################# # CMake configure file for ZMat # Qianqian Fang # 2020/05/23 ################################################################# cmake_minimum_required(VERSION 3.3) project(zmat) option(USE_ZLIB "Use zlib instead of miniz" OFF) option(USE_LZ4 "Use lz4" ON) option(USE_BLOSC2 "Use blosc2" ON) option(USE_ZSTD "Use ZStd" OFF) option(USE_LZMA "Use lzma" ON) find_package(Matlab) option(STATIC_LIB "Build static library" ON) # C Options set(CMAKE_C_FLAGS "-g -Wall -O3 -fPIC") set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/../) if(USE_ZLIB) find_package(ZLIB REQUIRED) else() set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DNO_ZLIB -D_LARGEFILE64_SOURCE=1") include_directories(miniz) endif() # Add include directories include_directories(../include) if(USE_LZ4) include_directories(lz4) else() set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DNO_LZ4") endif() if(USE_LZMA) include_directories(easylzma) include_directories(easylzma/pavlov) endif() if(USE_BLOSC2) include_directories(blosc2/include) add_subdirectory(blosc2/blosc) else() set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DNO_BLOSC2") endif() if(USE_ZSTD) include_directories(blosc2/internal-complibs/zstd) else() set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DNO_ZSTD") endif() # Add all project units if(STATIC_LIB) add_library(zmat STATIC zmatlib.c miniz/miniz.c lz4/lz4.c lz4/lz4hc.c easylzma/compress.c easylzma/decompress.c easylzma/lzma_header.c easylzma/lzip_header.c easylzma/common_internal.c easylzma/pavlov/LzmaEnc.c easylzma/pavlov/LzmaDec.c easylzma/pavlov/LzmaLib.c easylzma/pavlov/LzFind.c easylzma/pavlov/Bra.c easylzma/pavlov/BraIA64.c easylzma/pavlov/Alloc.c easylzma/pavlov/7zCrc.c ) else() # Add all project units add_library(zmat SHARED zmatlib.c miniz/miniz.c lz4/lz4.c lz4/lz4hc.c easylzma/compress.c easylzma/decompress.c easylzma/lzma_header.c easylzma/lzip_header.c easylzma/common_internal.c easylzma/pavlov/LzmaEnc.c easylzma/pavlov/LzmaDec.c easylzma/pavlov/LzmaLib.c easylzma/pavlov/LzFind.c easylzma/pavlov/Bra.c easylzma/pavlov/BraIA64.c easylzma/pavlov/Alloc.c easylzma/pavlov/7zCrc.c ) endif() # Link options target_link_libraries( zmat ) if(Matlab_FOUND) if(${CMAKE_VERSION} VERSION_LESS "3.24.0") matlab_add_mex( NAME zipmat SRC zmat.cpp LINK_TO zmat ) else() matlab_add_mex( NAME zipmat SRC zmat.cpp NO_IMPLICIT_LINK_TO_MATLAB_LIBRARIES LINK_TO ${Matlab_MEX_LIBRARY} ${Matlab_MX_LIBRARY} zmat ) endif() endif() zmat-0.9.9/src/runcmake.sh0000755000175200007730000000011514515254731015667 0ustar rlaboissrlaboiss#!/bin/sh rm -rf build mkdir build && cd build cmake $@ ../ make clean make zmat-0.9.9/src/blosc2/0000755000175200007730000000000014515254731014712 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/internal-complibs/0000755000175200007730000000000014515254731020334 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/internal-complibs/zstd/0000755000175200007730000000000014515254731021320 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/internal-complibs/zstd/zdict.h0000644000175200007730000006350114515254731022613 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #if defined (__cplusplus) extern "C" { #endif #ifndef ZSTD_ZDICT_H #define ZSTD_ZDICT_H /*====== Dependencies ======*/ #include /* size_t */ /* ===== ZDICTLIB_API : control library symbols visibility ===== */ #ifndef ZDICTLIB_VISIBLE /* Backwards compatibility with old macro name */ # ifdef ZDICTLIB_VISIBILITY # define ZDICTLIB_VISIBLE ZDICTLIB_VISIBILITY # elif defined(__GNUC__) && (__GNUC__ >= 4) && !defined(__MINGW32__) # define ZDICTLIB_VISIBLE __attribute__ ((visibility ("default"))) # else # define ZDICTLIB_VISIBLE # endif #endif #ifndef ZDICTLIB_HIDDEN # if defined(__GNUC__) && (__GNUC__ >= 4) && !defined(__MINGW32__) # define ZDICTLIB_HIDDEN __attribute__ ((visibility ("hidden"))) # else # define ZDICTLIB_HIDDEN # endif #endif #if defined(ZSTD_DLL_EXPORT) && (ZSTD_DLL_EXPORT==1) # define ZDICTLIB_API __declspec(dllexport) ZDICTLIB_VISIBLE #elif defined(ZSTD_DLL_IMPORT) && (ZSTD_DLL_IMPORT==1) # define ZDICTLIB_API __declspec(dllimport) ZDICTLIB_VISIBLE /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/ #else # define ZDICTLIB_API ZDICTLIB_VISIBLE #endif /******************************************************************************* * Zstd dictionary builder * * FAQ * === * Why should I use a dictionary? * ------------------------------ * * Zstd can use dictionaries to improve compression ratio of small data. * Traditionally small files don't compress well because there is very little * repetition in a single sample, since it is small. But, if you are compressing * many similar files, like a bunch of JSON records that share the same * structure, you can train a dictionary on ahead of time on some samples of * these files. Then, zstd can use the dictionary to find repetitions that are * present across samples. This can vastly improve compression ratio. * * When is a dictionary useful? * ---------------------------- * * Dictionaries are useful when compressing many small files that are similar. * The larger a file is, the less benefit a dictionary will have. Generally, * we don't expect dictionary compression to be effective past 100KB. And the * smaller a file is, the more we would expect the dictionary to help. * * How do I use a dictionary? * -------------------------- * * Simply pass the dictionary to the zstd compressor with * `ZSTD_CCtx_loadDictionary()`. The same dictionary must then be passed to * the decompressor, using `ZSTD_DCtx_loadDictionary()`. There are other * more advanced functions that allow selecting some options, see zstd.h for * complete documentation. * * What is a zstd dictionary? * -------------------------- * * A zstd dictionary has two pieces: Its header, and its content. The header * contains a magic number, the dictionary ID, and entropy tables. These * entropy tables allow zstd to save on header costs in the compressed file, * which really matters for small data. The content is just bytes, which are * repeated content that is common across many samples. * * What is a raw content dictionary? * --------------------------------- * * A raw content dictionary is just bytes. It doesn't have a zstd dictionary * header, a dictionary ID, or entropy tables. Any buffer is a valid raw * content dictionary. * * How do I train a dictionary? * ---------------------------- * * Gather samples from your use case. These samples should be similar to each * other. If you have several use cases, you could try to train one dictionary * per use case. * * Pass those samples to `ZDICT_trainFromBuffer()` and that will train your * dictionary. There are a few advanced versions of this function, but this * is a great starting point. If you want to further tune your dictionary * you could try `ZDICT_optimizeTrainFromBuffer_cover()`. If that is too slow * you can try `ZDICT_optimizeTrainFromBuffer_fastCover()`. * * If the dictionary training function fails, that is likely because you * either passed too few samples, or a dictionary would not be effective * for your data. Look at the messages that the dictionary trainer printed, * if it doesn't say too few samples, then a dictionary would not be effective. * * How large should my dictionary be? * ---------------------------------- * * A reasonable dictionary size, the `dictBufferCapacity`, is about 100KB. * The zstd CLI defaults to a 110KB dictionary. You likely don't need a * dictionary larger than that. But, most use cases can get away with a * smaller dictionary. The advanced dictionary builders can automatically * shrink the dictionary for you, and select the smallest size that doesn't * hurt compression ratio too much. See the `shrinkDict` parameter. * A smaller dictionary can save memory, and potentially speed up * compression. * * How many samples should I provide to the dictionary builder? * ------------------------------------------------------------ * * We generally recommend passing ~100x the size of the dictionary * in samples. A few thousand should suffice. Having too few samples * can hurt the dictionaries effectiveness. Having more samples will * only improve the dictionaries effectiveness. But having too many * samples can slow down the dictionary builder. * * How do I determine if a dictionary will be effective? * ----------------------------------------------------- * * Simply train a dictionary and try it out. You can use zstd's built in * benchmarking tool to test the dictionary effectiveness. * * # Benchmark levels 1-3 without a dictionary * zstd -b1e3 -r /path/to/my/files * # Benchmark levels 1-3 with a dictionary * zstd -b1e3 -r /path/to/my/files -D /path/to/my/dictionary * * When should I retrain a dictionary? * ----------------------------------- * * You should retrain a dictionary when its effectiveness drops. Dictionary * effectiveness drops as the data you are compressing changes. Generally, we do * expect dictionaries to "decay" over time, as your data changes, but the rate * at which they decay depends on your use case. Internally, we regularly * retrain dictionaries, and if the new dictionary performs significantly * better than the old dictionary, we will ship the new dictionary. * * I have a raw content dictionary, how do I turn it into a zstd dictionary? * ------------------------------------------------------------------------- * * If you have a raw content dictionary, e.g. by manually constructing it, or * using a third-party dictionary builder, you can turn it into a zstd * dictionary by using `ZDICT_finalizeDictionary()`. You'll also have to * provide some samples of the data. It will add the zstd header to the * raw content, which contains a dictionary ID and entropy tables, which * will improve compression ratio, and allow zstd to write the dictionary ID * into the frame, if you so choose. * * Do I have to use zstd's dictionary builder? * ------------------------------------------- * * No! You can construct dictionary content however you please, it is just * bytes. It will always be valid as a raw content dictionary. If you want * a zstd dictionary, which can improve compression ratio, use * `ZDICT_finalizeDictionary()`. * * What is the attack surface of a zstd dictionary? * ------------------------------------------------ * * Zstd is heavily fuzz tested, including loading fuzzed dictionaries, so * zstd should never crash, or access out-of-bounds memory no matter what * the dictionary is. However, if an attacker can control the dictionary * during decompression, they can cause zstd to generate arbitrary bytes, * just like if they controlled the compressed data. * ******************************************************************************/ /*! ZDICT_trainFromBuffer(): * Train a dictionary from an array of samples. * Redirect towards ZDICT_optimizeTrainFromBuffer_fastCover() single-threaded, with d=8, steps=4, * f=20, and accel=1. * Samples must be stored concatenated in a single flat buffer `samplesBuffer`, * supplied with an array of sizes `samplesSizes`, providing the size of each sample, in order. * The resulting dictionary will be saved into `dictBuffer`. * @return: size of dictionary stored into `dictBuffer` (<= `dictBufferCapacity`) * or an error code, which can be tested with ZDICT_isError(). * Note: Dictionary training will fail if there are not enough samples to construct a * dictionary, or if most of the samples are too small (< 8 bytes being the lower limit). * If dictionary training fails, you should use zstd without a dictionary, as the dictionary * would've been ineffective anyways. If you believe your samples would benefit from a dictionary * please open an issue with details, and we can look into it. * Note: ZDICT_trainFromBuffer()'s memory usage is about 6 MB. * Tips: In general, a reasonable dictionary has a size of ~ 100 KB. * It's possible to select smaller or larger size, just by specifying `dictBufferCapacity`. * In general, it's recommended to provide a few thousands samples, though this can vary a lot. * It's recommended that total size of all samples be about ~x100 times the target size of dictionary. */ ZDICTLIB_API size_t ZDICT_trainFromBuffer(void* dictBuffer, size_t dictBufferCapacity, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples); typedef struct { int compressionLevel; /**< optimize for a specific zstd compression level; 0 means default */ unsigned notificationLevel; /**< Write log to stderr; 0 = none (default); 1 = errors; 2 = progression; 3 = details; 4 = debug; */ unsigned dictID; /**< force dictID value; 0 means auto mode (32-bits random value) * NOTE: The zstd format reserves some dictionary IDs for future use. * You may use them in private settings, but be warned that they * may be used by zstd in a public dictionary registry in the future. * These dictionary IDs are: * - low range : <= 32767 * - high range : >= (2^31) */ } ZDICT_params_t; /*! ZDICT_finalizeDictionary(): * Given a custom content as a basis for dictionary, and a set of samples, * finalize dictionary by adding headers and statistics according to the zstd * dictionary format. * * Samples must be stored concatenated in a flat buffer `samplesBuffer`, * supplied with an array of sizes `samplesSizes`, providing the size of each * sample in order. The samples are used to construct the statistics, so they * should be representative of what you will compress with this dictionary. * * The compression level can be set in `parameters`. You should pass the * compression level you expect to use in production. The statistics for each * compression level differ, so tuning the dictionary for the compression level * can help quite a bit. * * You can set an explicit dictionary ID in `parameters`, or allow us to pick * a random dictionary ID for you, but we can't guarantee no collisions. * * The dstDictBuffer and the dictContent may overlap, and the content will be * appended to the end of the header. If the header + the content doesn't fit in * maxDictSize the beginning of the content is truncated to make room, since it * is presumed that the most profitable content is at the end of the dictionary, * since that is the cheapest to reference. * * `maxDictSize` must be >= max(dictContentSize, ZSTD_DICTSIZE_MIN). * * @return: size of dictionary stored into `dstDictBuffer` (<= `maxDictSize`), * or an error code, which can be tested by ZDICT_isError(). * Note: ZDICT_finalizeDictionary() will push notifications into stderr if * instructed to, using notificationLevel>0. * NOTE: This function currently may fail in several edge cases including: * * Not enough samples * * Samples are uncompressible * * Samples are all exactly the same */ ZDICTLIB_API size_t ZDICT_finalizeDictionary(void* dstDictBuffer, size_t maxDictSize, const void* dictContent, size_t dictContentSize, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples, ZDICT_params_t parameters); /*====== Helper functions ======*/ ZDICTLIB_API unsigned ZDICT_getDictID(const void* dictBuffer, size_t dictSize); /**< extracts dictID; @return zero if error (not a valid dictionary) */ ZDICTLIB_API size_t ZDICT_getDictHeaderSize(const void* dictBuffer, size_t dictSize); /* returns dict header size; returns a ZSTD error code on failure */ ZDICTLIB_API unsigned ZDICT_isError(size_t errorCode); ZDICTLIB_API const char* ZDICT_getErrorName(size_t errorCode); #endif /* ZSTD_ZDICT_H */ #if defined(ZDICT_STATIC_LINKING_ONLY) && !defined(ZSTD_ZDICT_H_STATIC) #define ZSTD_ZDICT_H_STATIC /* This can be overridden externally to hide static symbols. */ #ifndef ZDICTLIB_STATIC_API # if defined(ZSTD_DLL_EXPORT) && (ZSTD_DLL_EXPORT==1) # define ZDICTLIB_STATIC_API __declspec(dllexport) ZDICTLIB_VISIBLE # elif defined(ZSTD_DLL_IMPORT) && (ZSTD_DLL_IMPORT==1) # define ZDICTLIB_STATIC_API __declspec(dllimport) ZDICTLIB_VISIBLE # else # define ZDICTLIB_STATIC_API ZDICTLIB_VISIBLE # endif #endif /* ==================================================================================== * The definitions in this section are considered experimental. * They should never be used with a dynamic library, as they may change in the future. * They are provided for advanced usages. * Use them only in association with static linking. * ==================================================================================== */ #define ZDICT_DICTSIZE_MIN 256 /* Deprecated: Remove in v1.6.0 */ #define ZDICT_CONTENTSIZE_MIN 128 /*! ZDICT_cover_params_t: * k and d are the only required parameters. * For others, value 0 means default. */ typedef struct { unsigned k; /* Segment size : constraint: 0 < k : Reasonable range [16, 2048+] */ unsigned d; /* dmer size : constraint: 0 < d <= k : Reasonable range [6, 16] */ unsigned steps; /* Number of steps : Only used for optimization : 0 means default (40) : Higher means more parameters checked */ unsigned nbThreads; /* Number of threads : constraint: 0 < nbThreads : 1 means single-threaded : Only used for optimization : Ignored if ZSTD_MULTITHREAD is not defined */ double splitPoint; /* Percentage of samples used for training: Only used for optimization : the first nbSamples * splitPoint samples will be used to training, the last nbSamples * (1 - splitPoint) samples will be used for testing, 0 means default (1.0), 1.0 when all samples are used for both training and testing */ unsigned shrinkDict; /* Train dictionaries to shrink in size starting from the minimum size and selects the smallest dictionary that is shrinkDictMaxRegression% worse than the largest dictionary. 0 means no shrinking and 1 means shrinking */ unsigned shrinkDictMaxRegression; /* Sets shrinkDictMaxRegression so that a smaller dictionary can be at worse shrinkDictMaxRegression% worse than the max dict size dictionary. */ ZDICT_params_t zParams; } ZDICT_cover_params_t; typedef struct { unsigned k; /* Segment size : constraint: 0 < k : Reasonable range [16, 2048+] */ unsigned d; /* dmer size : constraint: 0 < d <= k : Reasonable range [6, 16] */ unsigned f; /* log of size of frequency array : constraint: 0 < f <= 31 : 1 means default(20)*/ unsigned steps; /* Number of steps : Only used for optimization : 0 means default (40) : Higher means more parameters checked */ unsigned nbThreads; /* Number of threads : constraint: 0 < nbThreads : 1 means single-threaded : Only used for optimization : Ignored if ZSTD_MULTITHREAD is not defined */ double splitPoint; /* Percentage of samples used for training: Only used for optimization : the first nbSamples * splitPoint samples will be used to training, the last nbSamples * (1 - splitPoint) samples will be used for testing, 0 means default (0.75), 1.0 when all samples are used for both training and testing */ unsigned accel; /* Acceleration level: constraint: 0 < accel <= 10, higher means faster and less accurate, 0 means default(1) */ unsigned shrinkDict; /* Train dictionaries to shrink in size starting from the minimum size and selects the smallest dictionary that is shrinkDictMaxRegression% worse than the largest dictionary. 0 means no shrinking and 1 means shrinking */ unsigned shrinkDictMaxRegression; /* Sets shrinkDictMaxRegression so that a smaller dictionary can be at worse shrinkDictMaxRegression% worse than the max dict size dictionary. */ ZDICT_params_t zParams; } ZDICT_fastCover_params_t; /*! ZDICT_trainFromBuffer_cover(): * Train a dictionary from an array of samples using the COVER algorithm. * Samples must be stored concatenated in a single flat buffer `samplesBuffer`, * supplied with an array of sizes `samplesSizes`, providing the size of each sample, in order. * The resulting dictionary will be saved into `dictBuffer`. * @return: size of dictionary stored into `dictBuffer` (<= `dictBufferCapacity`) * or an error code, which can be tested with ZDICT_isError(). * See ZDICT_trainFromBuffer() for details on failure modes. * Note: ZDICT_trainFromBuffer_cover() requires about 9 bytes of memory for each input byte. * Tips: In general, a reasonable dictionary has a size of ~ 100 KB. * It's possible to select smaller or larger size, just by specifying `dictBufferCapacity`. * In general, it's recommended to provide a few thousands samples, though this can vary a lot. * It's recommended that total size of all samples be about ~x100 times the target size of dictionary. */ ZDICTLIB_STATIC_API size_t ZDICT_trainFromBuffer_cover( void *dictBuffer, size_t dictBufferCapacity, const void *samplesBuffer, const size_t *samplesSizes, unsigned nbSamples, ZDICT_cover_params_t parameters); /*! ZDICT_optimizeTrainFromBuffer_cover(): * The same requirements as above hold for all the parameters except `parameters`. * This function tries many parameter combinations and picks the best parameters. * `*parameters` is filled with the best parameters found, * dictionary constructed with those parameters is stored in `dictBuffer`. * * All of the parameters d, k, steps are optional. * If d is non-zero then we don't check multiple values of d, otherwise we check d = {6, 8}. * if steps is zero it defaults to its default value. * If k is non-zero then we don't check multiple values of k, otherwise we check steps values in [50, 2000]. * * @return: size of dictionary stored into `dictBuffer` (<= `dictBufferCapacity`) * or an error code, which can be tested with ZDICT_isError(). * On success `*parameters` contains the parameters selected. * See ZDICT_trainFromBuffer() for details on failure modes. * Note: ZDICT_optimizeTrainFromBuffer_cover() requires about 8 bytes of memory for each input byte and additionally another 5 bytes of memory for each byte of memory for each thread. */ ZDICTLIB_STATIC_API size_t ZDICT_optimizeTrainFromBuffer_cover( void* dictBuffer, size_t dictBufferCapacity, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples, ZDICT_cover_params_t* parameters); /*! ZDICT_trainFromBuffer_fastCover(): * Train a dictionary from an array of samples using a modified version of COVER algorithm. * Samples must be stored concatenated in a single flat buffer `samplesBuffer`, * supplied with an array of sizes `samplesSizes`, providing the size of each sample, in order. * d and k are required. * All other parameters are optional, will use default values if not provided * The resulting dictionary will be saved into `dictBuffer`. * @return: size of dictionary stored into `dictBuffer` (<= `dictBufferCapacity`) * or an error code, which can be tested with ZDICT_isError(). * See ZDICT_trainFromBuffer() for details on failure modes. * Note: ZDICT_trainFromBuffer_fastCover() requires 6 * 2^f bytes of memory. * Tips: In general, a reasonable dictionary has a size of ~ 100 KB. * It's possible to select smaller or larger size, just by specifying `dictBufferCapacity`. * In general, it's recommended to provide a few thousands samples, though this can vary a lot. * It's recommended that total size of all samples be about ~x100 times the target size of dictionary. */ ZDICTLIB_STATIC_API size_t ZDICT_trainFromBuffer_fastCover(void *dictBuffer, size_t dictBufferCapacity, const void *samplesBuffer, const size_t *samplesSizes, unsigned nbSamples, ZDICT_fastCover_params_t parameters); /*! ZDICT_optimizeTrainFromBuffer_fastCover(): * The same requirements as above hold for all the parameters except `parameters`. * This function tries many parameter combinations (specifically, k and d combinations) * and picks the best parameters. `*parameters` is filled with the best parameters found, * dictionary constructed with those parameters is stored in `dictBuffer`. * All of the parameters d, k, steps, f, and accel are optional. * If d is non-zero then we don't check multiple values of d, otherwise we check d = {6, 8}. * if steps is zero it defaults to its default value. * If k is non-zero then we don't check multiple values of k, otherwise we check steps values in [50, 2000]. * If f is zero, default value of 20 is used. * If accel is zero, default value of 1 is used. * * @return: size of dictionary stored into `dictBuffer` (<= `dictBufferCapacity`) * or an error code, which can be tested with ZDICT_isError(). * On success `*parameters` contains the parameters selected. * See ZDICT_trainFromBuffer() for details on failure modes. * Note: ZDICT_optimizeTrainFromBuffer_fastCover() requires about 6 * 2^f bytes of memory for each thread. */ ZDICTLIB_STATIC_API size_t ZDICT_optimizeTrainFromBuffer_fastCover(void* dictBuffer, size_t dictBufferCapacity, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples, ZDICT_fastCover_params_t* parameters); typedef struct { unsigned selectivityLevel; /* 0 means default; larger => select more => larger dictionary */ ZDICT_params_t zParams; } ZDICT_legacy_params_t; /*! ZDICT_trainFromBuffer_legacy(): * Train a dictionary from an array of samples. * Samples must be stored concatenated in a single flat buffer `samplesBuffer`, * supplied with an array of sizes `samplesSizes`, providing the size of each sample, in order. * The resulting dictionary will be saved into `dictBuffer`. * `parameters` is optional and can be provided with values set to 0 to mean "default". * @return: size of dictionary stored into `dictBuffer` (<= `dictBufferCapacity`) * or an error code, which can be tested with ZDICT_isError(). * See ZDICT_trainFromBuffer() for details on failure modes. * Tips: In general, a reasonable dictionary has a size of ~ 100 KB. * It's possible to select smaller or larger size, just by specifying `dictBufferCapacity`. * In general, it's recommended to provide a few thousands samples, though this can vary a lot. * It's recommended that total size of all samples be about ~x100 times the target size of dictionary. * Note: ZDICT_trainFromBuffer_legacy() will send notifications into stderr if instructed to, using notificationLevel>0. */ ZDICTLIB_STATIC_API size_t ZDICT_trainFromBuffer_legacy( void* dictBuffer, size_t dictBufferCapacity, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples, ZDICT_legacy_params_t parameters); /* Deprecation warnings */ /* It is generally possible to disable deprecation warnings from compiler, for example with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual. Otherwise, it's also possible to manually define ZDICT_DISABLE_DEPRECATE_WARNINGS */ #ifdef ZDICT_DISABLE_DEPRECATE_WARNINGS # define ZDICT_DEPRECATED(message) /* disable deprecation warnings */ #else # define ZDICT_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) # if defined (__cplusplus) && (__cplusplus >= 201402) /* C++14 or greater */ # define ZDICT_DEPRECATED(message) [[deprecated(message)]] # elif defined(__clang__) || (ZDICT_GCC_VERSION >= 405) # define ZDICT_DEPRECATED(message) __attribute__((deprecated(message))) # elif (ZDICT_GCC_VERSION >= 301) # define ZDICT_DEPRECATED(message) __attribute__((deprecated)) # elif defined(_MSC_VER) # define ZDICT_DEPRECATED(message) __declspec(deprecated(message)) # else # pragma message("WARNING: You need to implement ZDICT_DEPRECATED for this compiler") # define ZDICT_DEPRECATED(message) # endif #endif /* ZDICT_DISABLE_DEPRECATE_WARNINGS */ ZDICT_DEPRECATED("use ZDICT_finalizeDictionary() instead") ZDICTLIB_STATIC_API size_t ZDICT_addEntropyTablesFromBuffer(void* dictBuffer, size_t dictContentSize, size_t dictBufferCapacity, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples); #endif /* ZSTD_ZDICT_H_STATIC */ #if defined (__cplusplus) } #endif zmat-0.9.9/src/blosc2/internal-complibs/zstd/zstd.h0000644000175200007730000051656214515254731022474 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #if defined (__cplusplus) extern "C" { #endif #ifndef ZSTD_H_235446 #define ZSTD_H_235446 /* ====== Dependencies ======*/ #include /* INT_MAX */ #include /* size_t */ /* ===== ZSTDLIB_API : control library symbols visibility ===== */ #ifndef ZSTDLIB_VISIBLE /* Backwards compatibility with old macro name */ # ifdef ZSTDLIB_VISIBILITY # define ZSTDLIB_VISIBLE ZSTDLIB_VISIBILITY # elif defined(__GNUC__) && (__GNUC__ >= 4) && !defined(__MINGW32__) # define ZSTDLIB_VISIBLE __attribute__ ((visibility ("default"))) # else # define ZSTDLIB_VISIBLE # endif #endif #ifndef ZSTDLIB_HIDDEN # if defined(__GNUC__) && (__GNUC__ >= 4) && !defined(__MINGW32__) # define ZSTDLIB_HIDDEN __attribute__ ((visibility ("hidden"))) # else # define ZSTDLIB_HIDDEN # endif #endif #if defined(ZSTD_DLL_EXPORT) && (ZSTD_DLL_EXPORT==1) # define ZSTDLIB_API __declspec(dllexport) ZSTDLIB_VISIBLE #elif defined(ZSTD_DLL_IMPORT) && (ZSTD_DLL_IMPORT==1) # define ZSTDLIB_API __declspec(dllimport) ZSTDLIB_VISIBLE /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/ #else # define ZSTDLIB_API ZSTDLIB_VISIBLE #endif /* Deprecation warnings : * Should these warnings be a problem, it is generally possible to disable them, * typically with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual. * Otherwise, it's also possible to define ZSTD_DISABLE_DEPRECATE_WARNINGS. */ #ifdef ZSTD_DISABLE_DEPRECATE_WARNINGS # define ZSTD_DEPRECATED(message) /* disable deprecation warnings */ #else # if defined (__cplusplus) && (__cplusplus >= 201402) /* C++14 or greater */ # define ZSTD_DEPRECATED(message) [[deprecated(message)]] # elif (defined(GNUC) && (GNUC > 4 || (GNUC == 4 && GNUC_MINOR >= 5))) || defined(__clang__) # define ZSTD_DEPRECATED(message) __attribute__((deprecated(message))) # elif defined(__GNUC__) && (__GNUC__ >= 3) # define ZSTD_DEPRECATED(message) __attribute__((deprecated)) # elif defined(_MSC_VER) # define ZSTD_DEPRECATED(message) __declspec(deprecated(message)) # else # pragma message("WARNING: You need to implement ZSTD_DEPRECATED for this compiler") # define ZSTD_DEPRECATED(message) # endif #endif /* ZSTD_DISABLE_DEPRECATE_WARNINGS */ /******************************************************************************* Introduction zstd, short for Zstandard, is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios. The zstd compression library provides in-memory compression and decompression functions. The library supports regular compression levels from 1 up to ZSTD_maxCLevel(), which is currently 22. Levels >= 20, labeled `--ultra`, should be used with caution, as they require more memory. The library also offers negative compression levels, which extend the range of speed vs. ratio preferences. The lower the level, the faster the speed (at the cost of compression). Compression can be done in: - a single step (described as Simple API) - a single step, reusing a context (described as Explicit context) - unbounded multiple steps (described as Streaming compression) The compression ratio achievable on small data can be highly improved using a dictionary. Dictionary compression can be performed in: - a single step (described as Simple dictionary API) - a single step, reusing a dictionary (described as Bulk-processing dictionary API) Advanced experimental functions can be accessed using `#define ZSTD_STATIC_LINKING_ONLY` before including zstd.h. Advanced experimental APIs should never be used with a dynamically-linked library. They are not "stable"; their definitions or signatures may change in the future. Only static linking is allowed. *******************************************************************************/ /*------ Version ------*/ #define ZSTD_VERSION_MAJOR 1 #define ZSTD_VERSION_MINOR 5 #define ZSTD_VERSION_RELEASE 5 #define ZSTD_VERSION_NUMBER (ZSTD_VERSION_MAJOR *100*100 + ZSTD_VERSION_MINOR *100 + ZSTD_VERSION_RELEASE) /*! ZSTD_versionNumber() : * Return runtime library version, the value is (MAJOR*100*100 + MINOR*100 + RELEASE). */ ZSTDLIB_API unsigned ZSTD_versionNumber(void); #define ZSTD_LIB_VERSION ZSTD_VERSION_MAJOR.ZSTD_VERSION_MINOR.ZSTD_VERSION_RELEASE #define ZSTD_QUOTE(str) #str #define ZSTD_EXPAND_AND_QUOTE(str) ZSTD_QUOTE(str) #define ZSTD_VERSION_STRING ZSTD_EXPAND_AND_QUOTE(ZSTD_LIB_VERSION) /*! ZSTD_versionString() : * Return runtime library version, like "1.4.5". Requires v1.3.0+. */ ZSTDLIB_API const char* ZSTD_versionString(void); /* ************************************* * Default constant ***************************************/ #ifndef ZSTD_CLEVEL_DEFAULT # define ZSTD_CLEVEL_DEFAULT 3 #endif /* ************************************* * Constants ***************************************/ /* All magic numbers are supposed read/written to/from files/memory using little-endian convention */ #define ZSTD_MAGICNUMBER 0xFD2FB528 /* valid since v0.8.0 */ #define ZSTD_MAGIC_DICTIONARY 0xEC30A437 /* valid since v0.7.0 */ #define ZSTD_MAGIC_SKIPPABLE_START 0x184D2A50 /* all 16 values, from 0x184D2A50 to 0x184D2A5F, signal the beginning of a skippable frame */ #define ZSTD_MAGIC_SKIPPABLE_MASK 0xFFFFFFF0 #define ZSTD_BLOCKSIZELOG_MAX 17 #define ZSTD_BLOCKSIZE_MAX (1<= ZSTD_compressBound(srcSize)` guarantees that zstd will have * enough space to successfully compress the data. * @return : compressed size written into `dst` (<= `dstCapacity), * or an error code if it fails (which can be tested using ZSTD_isError()). */ ZSTDLIB_API size_t ZSTD_compress( void* dst, size_t dstCapacity, const void* src, size_t srcSize, int compressionLevel); /*! ZSTD_decompress() : * `compressedSize` : must be the _exact_ size of some number of compressed and/or skippable frames. * `dstCapacity` is an upper bound of originalSize to regenerate. * If user cannot imply a maximum upper bound, it's better to use streaming mode to decompress data. * @return : the number of bytes decompressed into `dst` (<= `dstCapacity`), * or an errorCode if it fails (which can be tested using ZSTD_isError()). */ ZSTDLIB_API size_t ZSTD_decompress( void* dst, size_t dstCapacity, const void* src, size_t compressedSize); /*! ZSTD_getFrameContentSize() : requires v1.3.0+ * `src` should point to the start of a ZSTD encoded frame. * `srcSize` must be at least as large as the frame header. * hint : any size >= `ZSTD_frameHeaderSize_max` is large enough. * @return : - decompressed size of `src` frame content, if known * - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined * - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small) * note 1 : a 0 return value means the frame is valid but "empty". * note 2 : decompressed size is an optional field, it may not be present, typically in streaming mode. * When `return==ZSTD_CONTENTSIZE_UNKNOWN`, data to decompress could be any size. * In which case, it's necessary to use streaming mode to decompress data. * Optionally, application can rely on some implicit limit, * as ZSTD_decompress() only needs an upper bound of decompressed size. * (For example, data could be necessarily cut into blocks <= 16 KB). * note 3 : decompressed size is always present when compression is completed using single-pass functions, * such as ZSTD_compress(), ZSTD_compressCCtx() ZSTD_compress_usingDict() or ZSTD_compress_usingCDict(). * note 4 : decompressed size can be very large (64-bits value), * potentially larger than what local system can handle as a single memory segment. * In which case, it's necessary to use streaming mode to decompress data. * note 5 : If source is untrusted, decompressed size could be wrong or intentionally modified. * Always ensure return value fits within application's authorized limits. * Each application can set its own limits. * note 6 : This function replaces ZSTD_getDecompressedSize() */ #define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1) #define ZSTD_CONTENTSIZE_ERROR (0ULL - 2) ZSTDLIB_API unsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize); /*! ZSTD_getDecompressedSize() : * NOTE: This function is now obsolete, in favor of ZSTD_getFrameContentSize(). * Both functions work the same way, but ZSTD_getDecompressedSize() blends * "empty", "unknown" and "error" results to the same return value (0), * while ZSTD_getFrameContentSize() gives them separate return values. * @return : decompressed size of `src` frame content _if known and not empty_, 0 otherwise. */ ZSTD_DEPRECATED("Replaced by ZSTD_getFrameContentSize") ZSTDLIB_API unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize); /*! ZSTD_findFrameCompressedSize() : Requires v1.4.0+ * `src` should point to the start of a ZSTD frame or skippable frame. * `srcSize` must be >= first frame size * @return : the compressed size of the first frame starting at `src`, * suitable to pass as `srcSize` to `ZSTD_decompress` or similar, * or an error code if input is invalid */ ZSTDLIB_API size_t ZSTD_findFrameCompressedSize(const void* src, size_t srcSize); /*====== Helper functions ======*/ /* ZSTD_compressBound() : * maximum compressed size in worst case single-pass scenario. * When invoking `ZSTD_compress()` or any other one-pass compression function, * it's recommended to provide @dstCapacity >= ZSTD_compressBound(srcSize) * as it eliminates one potential failure scenario, * aka not enough room in dst buffer to write the compressed frame. * Note : ZSTD_compressBound() itself can fail, if @srcSize > ZSTD_MAX_INPUT_SIZE . * In which case, ZSTD_compressBound() will return an error code * which can be tested using ZSTD_isError(). * * ZSTD_COMPRESSBOUND() : * same as ZSTD_compressBound(), but as a macro. * It can be used to produce constants, which can be useful for static allocation, * for example to size a static array on stack. * Will produce constant value 0 if srcSize too large. */ #define ZSTD_MAX_INPUT_SIZE ((sizeof(size_t)==8) ? 0xFF00FF00FF00FF00LLU : 0xFF00FF00U) #define ZSTD_COMPRESSBOUND(srcSize) (((size_t)(srcSize) >= ZSTD_MAX_INPUT_SIZE) ? 0 : (srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /* margin, from 64 to 0 */ : 0)) /* this formula ensures that bound(A) + bound(B) <= bound(A+B) as long as A and B >= 128 KB */ ZSTDLIB_API size_t ZSTD_compressBound(size_t srcSize); /*!< maximum compressed size in worst case single-pass scenario */ /* ZSTD_isError() : * Most ZSTD_* functions returning a size_t value can be tested for error, * using ZSTD_isError(). * @return 1 if error, 0 otherwise */ ZSTDLIB_API unsigned ZSTD_isError(size_t code); /*!< tells if a `size_t` function result is an error code */ ZSTDLIB_API const char* ZSTD_getErrorName(size_t code); /*!< provides readable string from an error code */ ZSTDLIB_API int ZSTD_minCLevel(void); /*!< minimum negative compression level allowed, requires v1.4.0+ */ ZSTDLIB_API int ZSTD_maxCLevel(void); /*!< maximum compression level available */ ZSTDLIB_API int ZSTD_defaultCLevel(void); /*!< default compression level, specified by ZSTD_CLEVEL_DEFAULT, requires v1.5.0+ */ /*************************************** * Explicit context ***************************************/ /*= Compression context * When compressing many times, * it is recommended to allocate a context just once, * and re-use it for each successive compression operation. * This will make workload friendlier for system's memory. * Note : re-using context is just a speed / resource optimization. * It doesn't change the compression ratio, which remains identical. * Note 2 : In multi-threaded environments, * use one different context per thread for parallel execution. */ typedef struct ZSTD_CCtx_s ZSTD_CCtx; ZSTDLIB_API ZSTD_CCtx* ZSTD_createCCtx(void); ZSTDLIB_API size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx); /* accept NULL pointer */ /*! ZSTD_compressCCtx() : * Same as ZSTD_compress(), using an explicit ZSTD_CCtx. * Important : in order to behave similarly to `ZSTD_compress()`, * this function compresses at requested compression level, * __ignoring any other parameter__ . * If any advanced parameter was set using the advanced API, * they will all be reset. Only `compressionLevel` remains. */ ZSTDLIB_API size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, int compressionLevel); /*= Decompression context * When decompressing many times, * it is recommended to allocate a context only once, * and re-use it for each successive compression operation. * This will make workload friendlier for system's memory. * Use one context per thread for parallel execution. */ typedef struct ZSTD_DCtx_s ZSTD_DCtx; ZSTDLIB_API ZSTD_DCtx* ZSTD_createDCtx(void); ZSTDLIB_API size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); /* accept NULL pointer */ /*! ZSTD_decompressDCtx() : * Same as ZSTD_decompress(), * requires an allocated ZSTD_DCtx. * Compatible with sticky parameters. */ ZSTDLIB_API size_t ZSTD_decompressDCtx(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); /********************************************* * Advanced compression API (Requires v1.4.0+) **********************************************/ /* API design : * Parameters are pushed one by one into an existing context, * using ZSTD_CCtx_set*() functions. * Pushed parameters are sticky : they are valid for next compressed frame, and any subsequent frame. * "sticky" parameters are applicable to `ZSTD_compress2()` and `ZSTD_compressStream*()` ! * __They do not apply to "simple" one-shot variants such as ZSTD_compressCCtx()__ . * * It's possible to reset all parameters to "default" using ZSTD_CCtx_reset(). * * This API supersedes all other "advanced" API entry points in the experimental section. * In the future, we expect to remove from experimental API entry points which are redundant with this API. */ /* Compression strategies, listed from fastest to strongest */ typedef enum { ZSTD_fast=1, ZSTD_dfast=2, ZSTD_greedy=3, ZSTD_lazy=4, ZSTD_lazy2=5, ZSTD_btlazy2=6, ZSTD_btopt=7, ZSTD_btultra=8, ZSTD_btultra2=9 /* note : new strategies _might_ be added in the future. Only the order (from fast to strong) is guaranteed */ } ZSTD_strategy; typedef enum { /* compression parameters * Note: When compressing with a ZSTD_CDict these parameters are superseded * by the parameters used to construct the ZSTD_CDict. * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */ ZSTD_c_compressionLevel=100, /* Set compression parameters according to pre-defined cLevel table. * Note that exact compression parameters are dynamically determined, * depending on both compression level and srcSize (when known). * Default level is ZSTD_CLEVEL_DEFAULT==3. * Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT. * Note 1 : it's possible to pass a negative compression level. * Note 2 : setting a level does not automatically set all other compression parameters * to default. Setting this will however eventually dynamically impact the compression * parameters which have not been manually set. The manually set * ones will 'stick'. */ /* Advanced compression parameters : * It's possible to pin down compression parameters to some specific values. * In which case, these values are no longer dynamically selected by the compressor */ ZSTD_c_windowLog=101, /* Maximum allowed back-reference distance, expressed as power of 2. * This will set a memory budget for streaming decompression, * with larger values requiring more memory * and typically compressing more. * Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX. * Special: value 0 means "use default windowLog". * Note: Using a windowLog greater than ZSTD_WINDOWLOG_LIMIT_DEFAULT * requires explicitly allowing such size at streaming decompression stage. */ ZSTD_c_hashLog=102, /* Size of the initial probe table, as a power of 2. * Resulting memory usage is (1 << (hashLog+2)). * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX. * Larger tables improve compression ratio of strategies <= dFast, * and improve speed of strategies > dFast. * Special: value 0 means "use default hashLog". */ ZSTD_c_chainLog=103, /* Size of the multi-probe search table, as a power of 2. * Resulting memory usage is (1 << (chainLog+2)). * Must be clamped between ZSTD_CHAINLOG_MIN and ZSTD_CHAINLOG_MAX. * Larger tables result in better and slower compression. * This parameter is useless for "fast" strategy. * It's still useful when using "dfast" strategy, * in which case it defines a secondary probe table. * Special: value 0 means "use default chainLog". */ ZSTD_c_searchLog=104, /* Number of search attempts, as a power of 2. * More attempts result in better and slower compression. * This parameter is useless for "fast" and "dFast" strategies. * Special: value 0 means "use default searchLog". */ ZSTD_c_minMatch=105, /* Minimum size of searched matches. * Note that Zstandard can still find matches of smaller size, * it just tweaks its search algorithm to look for this size and larger. * Larger values increase compression and decompression speed, but decrease ratio. * Must be clamped between ZSTD_MINMATCH_MIN and ZSTD_MINMATCH_MAX. * Note that currently, for all strategies < btopt, effective minimum is 4. * , for all strategies > fast, effective maximum is 6. * Special: value 0 means "use default minMatchLength". */ ZSTD_c_targetLength=106, /* Impact of this field depends on strategy. * For strategies btopt, btultra & btultra2: * Length of Match considered "good enough" to stop search. * Larger values make compression stronger, and slower. * For strategy fast: * Distance between match sampling. * Larger values make compression faster, and weaker. * Special: value 0 means "use default targetLength". */ ZSTD_c_strategy=107, /* See ZSTD_strategy enum definition. * The higher the value of selected strategy, the more complex it is, * resulting in stronger and slower compression. * Special: value 0 means "use default strategy". */ /* LDM mode parameters */ ZSTD_c_enableLongDistanceMatching=160, /* Enable long distance matching. * This parameter is designed to improve compression ratio * for large inputs, by finding large matches at long distance. * It increases memory usage and window size. * Note: enabling this parameter increases default ZSTD_c_windowLog to 128 MB * except when expressly set to a different value. * Note: will be enabled by default if ZSTD_c_windowLog >= 128 MB and * compression strategy >= ZSTD_btopt (== compression level 16+) */ ZSTD_c_ldmHashLog=161, /* Size of the table for long distance matching, as a power of 2. * Larger values increase memory usage and compression ratio, * but decrease compression speed. * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX * default: windowlog - 7. * Special: value 0 means "automatically determine hashlog". */ ZSTD_c_ldmMinMatch=162, /* Minimum match size for long distance matcher. * Larger/too small values usually decrease compression ratio. * Must be clamped between ZSTD_LDM_MINMATCH_MIN and ZSTD_LDM_MINMATCH_MAX. * Special: value 0 means "use default value" (default: 64). */ ZSTD_c_ldmBucketSizeLog=163, /* Log size of each bucket in the LDM hash table for collision resolution. * Larger values improve collision resolution but decrease compression speed. * The maximum value is ZSTD_LDM_BUCKETSIZELOG_MAX. * Special: value 0 means "use default value" (default: 3). */ ZSTD_c_ldmHashRateLog=164, /* Frequency of inserting/looking up entries into the LDM hash table. * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN). * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage. * Larger values improve compression speed. * Deviating far from default value will likely result in a compression ratio decrease. * Special: value 0 means "automatically determine hashRateLog". */ /* frame parameters */ ZSTD_c_contentSizeFlag=200, /* Content size will be written into frame header _whenever known_ (default:1) * Content size must be known at the beginning of compression. * This is automatically the case when using ZSTD_compress2(), * For streaming scenarios, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */ ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */ ZSTD_c_dictIDFlag=202, /* When applicable, dictionary's ID is written into frame header (default:1) */ /* multi-threading parameters */ /* These parameters are only active if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD). * Otherwise, trying to set any other value than default (0) will be a no-op and return an error. * In a situation where it's unknown if the linked library supports multi-threading or not, * setting ZSTD_c_nbWorkers to any value >= 1 and consulting the return value provides a quick way to check this property. */ ZSTD_c_nbWorkers=400, /* Select how many threads will be spawned to compress in parallel. * When nbWorkers >= 1, triggers asynchronous mode when invoking ZSTD_compressStream*() : * ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back control to caller, * while compression is performed in parallel, within worker thread(s). * (note : a strong exception to this rule is when first invocation of ZSTD_compressStream2() sets ZSTD_e_end : * in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call). * More workers improve speed, but also increase memory usage. * Default value is `0`, aka "single-threaded mode" : no worker is spawned, * compression is performed inside Caller's thread, and all invocations are blocking */ ZSTD_c_jobSize=401, /* Size of a compression job. This value is enforced only when nbWorkers >= 1. * Each compression job is completed in parallel, so this value can indirectly impact the nb of active threads. * 0 means default, which is dynamically determined based on compression parameters. * Job size must be a minimum of overlap size, or ZSTDMT_JOBSIZE_MIN (= 512 KB), whichever is largest. * The minimum size is automatically and transparently enforced. */ ZSTD_c_overlapLog=402, /* Control the overlap size, as a fraction of window size. * The overlap size is an amount of data reloaded from previous job at the beginning of a new job. * It helps preserve compression ratio, while each job is compressed in parallel. * This value is enforced only when nbWorkers >= 1. * Larger values increase compression ratio, but decrease speed. * Possible values range from 0 to 9 : * - 0 means "default" : value will be determined by the library, depending on strategy * - 1 means "no overlap" * - 9 means "full overlap", using a full window size. * Each intermediate rank increases/decreases load size by a factor 2 : * 9: full window; 8: w/2; 7: w/4; 6: w/8; 5:w/16; 4: w/32; 3:w/64; 2:w/128; 1:no overlap; 0:default * default value varies between 6 and 9, depending on strategy */ /* note : additional experimental parameters are also available * within the experimental section of the API. * At the time of this writing, they include : * ZSTD_c_rsyncable * ZSTD_c_format * ZSTD_c_forceMaxWindow * ZSTD_c_forceAttachDict * ZSTD_c_literalCompressionMode * ZSTD_c_targetCBlockSize * ZSTD_c_srcSizeHint * ZSTD_c_enableDedicatedDictSearch * ZSTD_c_stableInBuffer * ZSTD_c_stableOutBuffer * ZSTD_c_blockDelimiters * ZSTD_c_validateSequences * ZSTD_c_useBlockSplitter * ZSTD_c_useRowMatchFinder * ZSTD_c_prefetchCDictTables * ZSTD_c_enableSeqProducerFallback * ZSTD_c_maxBlockSize * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them. * note : never ever use experimentalParam? names directly; * also, the enums values themselves are unstable and can still change. */ ZSTD_c_experimentalParam1=500, ZSTD_c_experimentalParam2=10, ZSTD_c_experimentalParam3=1000, ZSTD_c_experimentalParam4=1001, ZSTD_c_experimentalParam5=1002, ZSTD_c_experimentalParam6=1003, ZSTD_c_experimentalParam7=1004, ZSTD_c_experimentalParam8=1005, ZSTD_c_experimentalParam9=1006, ZSTD_c_experimentalParam10=1007, ZSTD_c_experimentalParam11=1008, ZSTD_c_experimentalParam12=1009, ZSTD_c_experimentalParam13=1010, ZSTD_c_experimentalParam14=1011, ZSTD_c_experimentalParam15=1012, ZSTD_c_experimentalParam16=1013, ZSTD_c_experimentalParam17=1014, ZSTD_c_experimentalParam18=1015, ZSTD_c_experimentalParam19=1016 } ZSTD_cParameter; typedef struct { size_t error; int lowerBound; int upperBound; } ZSTD_bounds; /*! ZSTD_cParam_getBounds() : * All parameters must belong to an interval with lower and upper bounds, * otherwise they will either trigger an error or be automatically clamped. * @return : a structure, ZSTD_bounds, which contains * - an error status field, which must be tested using ZSTD_isError() * - lower and upper bounds, both inclusive */ ZSTDLIB_API ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter cParam); /*! ZSTD_CCtx_setParameter() : * Set one compression parameter, selected by enum ZSTD_cParameter. * All parameters have valid bounds. Bounds can be queried using ZSTD_cParam_getBounds(). * Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter). * Setting a parameter is generally only possible during frame initialization (before starting compression). * Exception : when using multi-threading mode (nbWorkers >= 1), * the following parameters can be updated _during_ compression (within same frame): * => compressionLevel, hashLog, chainLog, searchLog, minMatch, targetLength and strategy. * new parameters will be active for next job only (after a flush()). * @return : an error code (which can be tested using ZSTD_isError()). */ ZSTDLIB_API size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value); /*! ZSTD_CCtx_setPledgedSrcSize() : * Total input data size to be compressed as a single frame. * Value will be written in frame header, unless if explicitly forbidden using ZSTD_c_contentSizeFlag. * This value will also be controlled at end of frame, and trigger an error if not respected. * @result : 0, or an error code (which can be tested with ZSTD_isError()). * Note 1 : pledgedSrcSize==0 actually means zero, aka an empty frame. * In order to mean "unknown content size", pass constant ZSTD_CONTENTSIZE_UNKNOWN. * ZSTD_CONTENTSIZE_UNKNOWN is default value for any new frame. * Note 2 : pledgedSrcSize is only valid once, for the next frame. * It's discarded at the end of the frame, and replaced by ZSTD_CONTENTSIZE_UNKNOWN. * Note 3 : Whenever all input data is provided and consumed in a single round, * for example with ZSTD_compress2(), * or invoking immediately ZSTD_compressStream2(,,,ZSTD_e_end), * this value is automatically overridden by srcSize instead. */ ZSTDLIB_API size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long pledgedSrcSize); typedef enum { ZSTD_reset_session_only = 1, ZSTD_reset_parameters = 2, ZSTD_reset_session_and_parameters = 3 } ZSTD_ResetDirective; /*! ZSTD_CCtx_reset() : * There are 2 different things that can be reset, independently or jointly : * - The session : will stop compressing current frame, and make CCtx ready to start a new one. * Useful after an error, or to interrupt any ongoing compression. * Any internal data not yet flushed is cancelled. * Compression parameters and dictionary remain unchanged. * They will be used to compress next frame. * Resetting session never fails. * - The parameters : changes all parameters back to "default". * This also removes any reference to any dictionary or external sequence producer. * Parameters can only be changed between 2 sessions (i.e. no compression is currently ongoing) * otherwise the reset fails, and function returns an error value (which can be tested using ZSTD_isError()) * - Both : similar to resetting the session, followed by resetting parameters. */ ZSTDLIB_API size_t ZSTD_CCtx_reset(ZSTD_CCtx* cctx, ZSTD_ResetDirective reset); /*! ZSTD_compress2() : * Behave the same as ZSTD_compressCCtx(), but compression parameters are set using the advanced API. * ZSTD_compress2() always starts a new frame. * Should cctx hold data from a previously unfinished frame, everything about it is forgotten. * - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*() * - The function is always blocking, returns when compression is completed. * NOTE: Providing `dstCapacity >= ZSTD_compressBound(srcSize)` guarantees that zstd will have * enough space to successfully compress the data, though it is possible it fails for other reasons. * @return : compressed size written into `dst` (<= `dstCapacity), * or an error code if it fails (which can be tested using ZSTD_isError()). */ ZSTDLIB_API size_t ZSTD_compress2( ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); /*********************************************** * Advanced decompression API (Requires v1.4.0+) ************************************************/ /* The advanced API pushes parameters one by one into an existing DCtx context. * Parameters are sticky, and remain valid for all following frames * using the same DCtx context. * It's possible to reset parameters to default values using ZSTD_DCtx_reset(). * Note : This API is compatible with existing ZSTD_decompressDCtx() and ZSTD_decompressStream(). * Therefore, no new decompression function is necessary. */ typedef enum { ZSTD_d_windowLogMax=100, /* Select a size limit (in power of 2) beyond which * the streaming API will refuse to allocate memory buffer * in order to protect the host from unreasonable memory requirements. * This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode. * By default, a decompression context accepts window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT). * Special: value 0 means "use default maximum windowLog". */ /* note : additional experimental parameters are also available * within the experimental section of the API. * At the time of this writing, they include : * ZSTD_d_format * ZSTD_d_stableOutBuffer * ZSTD_d_forceIgnoreChecksum * ZSTD_d_refMultipleDDicts * ZSTD_d_disableHuffmanAssembly * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them. * note : never ever use experimentalParam? names directly */ ZSTD_d_experimentalParam1=1000, ZSTD_d_experimentalParam2=1001, ZSTD_d_experimentalParam3=1002, ZSTD_d_experimentalParam4=1003, ZSTD_d_experimentalParam5=1004 } ZSTD_dParameter; /*! ZSTD_dParam_getBounds() : * All parameters must belong to an interval with lower and upper bounds, * otherwise they will either trigger an error or be automatically clamped. * @return : a structure, ZSTD_bounds, which contains * - an error status field, which must be tested using ZSTD_isError() * - both lower and upper bounds, inclusive */ ZSTDLIB_API ZSTD_bounds ZSTD_dParam_getBounds(ZSTD_dParameter dParam); /*! ZSTD_DCtx_setParameter() : * Set one compression parameter, selected by enum ZSTD_dParameter. * All parameters have valid bounds. Bounds can be queried using ZSTD_dParam_getBounds(). * Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter). * Setting a parameter is only possible during frame initialization (before starting decompression). * @return : 0, or an error code (which can be tested using ZSTD_isError()). */ ZSTDLIB_API size_t ZSTD_DCtx_setParameter(ZSTD_DCtx* dctx, ZSTD_dParameter param, int value); /*! ZSTD_DCtx_reset() : * Return a DCtx to clean state. * Session and parameters can be reset jointly or separately. * Parameters can only be reset when no active frame is being decompressed. * @return : 0, or an error code, which can be tested with ZSTD_isError() */ ZSTDLIB_API size_t ZSTD_DCtx_reset(ZSTD_DCtx* dctx, ZSTD_ResetDirective reset); /**************************** * Streaming ****************************/ typedef struct ZSTD_inBuffer_s { const void* src; /**< start of input buffer */ size_t size; /**< size of input buffer */ size_t pos; /**< position where reading stopped. Will be updated. Necessarily 0 <= pos <= size */ } ZSTD_inBuffer; typedef struct ZSTD_outBuffer_s { void* dst; /**< start of output buffer */ size_t size; /**< size of output buffer */ size_t pos; /**< position where writing stopped. Will be updated. Necessarily 0 <= pos <= size */ } ZSTD_outBuffer; /*-*********************************************************************** * Streaming compression - HowTo * * A ZSTD_CStream object is required to track streaming operation. * Use ZSTD_createCStream() and ZSTD_freeCStream() to create/release resources. * ZSTD_CStream objects can be reused multiple times on consecutive compression operations. * It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-using already allocated memory. * * For parallel execution, use one separate ZSTD_CStream per thread. * * note : since v1.3.0, ZSTD_CStream and ZSTD_CCtx are the same thing. * * Parameters are sticky : when starting a new compression on the same context, * it will re-use the same sticky parameters as previous compression session. * When in doubt, it's recommended to fully initialize the context before usage. * Use ZSTD_CCtx_reset() to reset the context and ZSTD_CCtx_setParameter(), * ZSTD_CCtx_setPledgedSrcSize(), or ZSTD_CCtx_loadDictionary() and friends to * set more specific parameters, the pledged source size, or load a dictionary. * * Use ZSTD_compressStream2() with ZSTD_e_continue as many times as necessary to * consume input stream. The function will automatically update both `pos` * fields within `input` and `output`. * Note that the function may not consume the entire input, for example, because * the output buffer is already full, in which case `input.pos < input.size`. * The caller must check if input has been entirely consumed. * If not, the caller must make some room to receive more compressed data, * and then present again remaining input data. * note: ZSTD_e_continue is guaranteed to make some forward progress when called, * but doesn't guarantee maximal forward progress. This is especially relevant * when compressing with multiple threads. The call won't block if it can * consume some input, but if it can't it will wait for some, but not all, * output to be flushed. * @return : provides a minimum amount of data remaining to be flushed from internal buffers * or an error code, which can be tested using ZSTD_isError(). * * At any moment, it's possible to flush whatever data might remain stuck within internal buffer, * using ZSTD_compressStream2() with ZSTD_e_flush. `output->pos` will be updated. * Note that, if `output->size` is too small, a single invocation with ZSTD_e_flush might not be enough (return code > 0). * In which case, make some room to receive more compressed data, and call again ZSTD_compressStream2() with ZSTD_e_flush. * You must continue calling ZSTD_compressStream2() with ZSTD_e_flush until it returns 0, at which point you can change the * operation. * note: ZSTD_e_flush will flush as much output as possible, meaning when compressing with multiple threads, it will * block until the flush is complete or the output buffer is full. * @return : 0 if internal buffers are entirely flushed, * >0 if some data still present within internal buffer (the value is minimal estimation of remaining size), * or an error code, which can be tested using ZSTD_isError(). * * Calling ZSTD_compressStream2() with ZSTD_e_end instructs to finish a frame. * It will perform a flush and write frame epilogue. * The epilogue is required for decoders to consider a frame completed. * flush operation is the same, and follows same rules as calling ZSTD_compressStream2() with ZSTD_e_flush. * You must continue calling ZSTD_compressStream2() with ZSTD_e_end until it returns 0, at which point you are free to * start a new frame. * note: ZSTD_e_end will flush as much output as possible, meaning when compressing with multiple threads, it will * block until the flush is complete or the output buffer is full. * @return : 0 if frame fully completed and fully flushed, * >0 if some data still present within internal buffer (the value is minimal estimation of remaining size), * or an error code, which can be tested using ZSTD_isError(). * * *******************************************************************/ typedef ZSTD_CCtx ZSTD_CStream; /**< CCtx and CStream are now effectively same object (>= v1.3.0) */ /* Continue to distinguish them for compatibility with older versions <= v1.2.0 */ /*===== ZSTD_CStream management functions =====*/ ZSTDLIB_API ZSTD_CStream* ZSTD_createCStream(void); ZSTDLIB_API size_t ZSTD_freeCStream(ZSTD_CStream* zcs); /* accept NULL pointer */ /*===== Streaming compression functions =====*/ typedef enum { ZSTD_e_continue=0, /* collect more data, encoder decides when to output compressed result, for optimal compression ratio */ ZSTD_e_flush=1, /* flush any data provided so far, * it creates (at least) one new block, that can be decoded immediately on reception; * frame will continue: any future data can still reference previously compressed data, improving compression. * note : multithreaded compression will block to flush as much output as possible. */ ZSTD_e_end=2 /* flush any remaining data _and_ close current frame. * note that frame is only closed after compressed data is fully flushed (return value == 0). * After that point, any additional data starts a new frame. * note : each frame is independent (does not reference any content from previous frame). : note : multithreaded compression will block to flush as much output as possible. */ } ZSTD_EndDirective; /*! ZSTD_compressStream2() : Requires v1.4.0+ * Behaves about the same as ZSTD_compressStream, with additional control on end directive. * - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*() * - Compression parameters cannot be changed once compression is started (save a list of exceptions in multi-threading mode) * - output->pos must be <= dstCapacity, input->pos must be <= srcSize * - output->pos and input->pos will be updated. They are guaranteed to remain below their respective limit. * - endOp must be a valid directive * - When nbWorkers==0 (default), function is blocking : it completes its job before returning to caller. * - When nbWorkers>=1, function is non-blocking : it copies a portion of input, distributes jobs to internal worker threads, flush to output whatever is available, * and then immediately returns, just indicating that there is some data remaining to be flushed. * The function nonetheless guarantees forward progress : it will return only after it reads or write at least 1+ byte. * - Exception : if the first call requests a ZSTD_e_end directive and provides enough dstCapacity, the function delegates to ZSTD_compress2() which is always blocking. * - @return provides a minimum amount of data remaining to be flushed from internal buffers * or an error code, which can be tested using ZSTD_isError(). * if @return != 0, flush is not fully completed, there is still some data left within internal buffers. * This is useful for ZSTD_e_flush, since in this case more flushes are necessary to empty all buffers. * For ZSTD_e_end, @return == 0 when internal buffers are fully flushed and frame is completed. * - after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0), * only ZSTD_e_end or ZSTD_e_flush operations are allowed. * Before starting a new compression job, or changing compression parameters, * it is required to fully flush internal buffers. */ ZSTDLIB_API size_t ZSTD_compressStream2( ZSTD_CCtx* cctx, ZSTD_outBuffer* output, ZSTD_inBuffer* input, ZSTD_EndDirective endOp); /* These buffer sizes are softly recommended. * They are not required : ZSTD_compressStream*() happily accepts any buffer size, for both input and output. * Respecting the recommended size just makes it a bit easier for ZSTD_compressStream*(), * reducing the amount of memory shuffling and buffering, resulting in minor performance savings. * * However, note that these recommendations are from the perspective of a C caller program. * If the streaming interface is invoked from some other language, * especially managed ones such as Java or Go, through a foreign function interface such as jni or cgo, * a major performance rule is to reduce crossing such interface to an absolute minimum. * It's not rare that performance ends being spent more into the interface, rather than compression itself. * In which cases, prefer using large buffers, as large as practical, * for both input and output, to reduce the nb of roundtrips. */ ZSTDLIB_API size_t ZSTD_CStreamInSize(void); /**< recommended size for input buffer */ ZSTDLIB_API size_t ZSTD_CStreamOutSize(void); /**< recommended size for output buffer. Guarantee to successfully flush at least one complete compressed block. */ /* ***************************************************************************** * This following is a legacy streaming API, available since v1.0+ . * It can be replaced by ZSTD_CCtx_reset() and ZSTD_compressStream2(). * It is redundant, but remains fully supported. ******************************************************************************/ /*! * Equivalent to: * * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); * ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any) * ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel); * * Note that ZSTD_initCStream() clears any previously set dictionary. Use the new API * to compress with a dictionary. */ ZSTDLIB_API size_t ZSTD_initCStream(ZSTD_CStream* zcs, int compressionLevel); /*! * Alternative for ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue). * NOTE: The return value is different. ZSTD_compressStream() returns a hint for * the next read size (if non-zero and not an error). ZSTD_compressStream2() * returns the minimum nb of bytes left to flush (if non-zero and not an error). */ ZSTDLIB_API size_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input); /*! Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_flush). */ ZSTDLIB_API size_t ZSTD_flushStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output); /*! Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_end). */ ZSTDLIB_API size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output); /*-*************************************************************************** * Streaming decompression - HowTo * * A ZSTD_DStream object is required to track streaming operations. * Use ZSTD_createDStream() and ZSTD_freeDStream() to create/release resources. * ZSTD_DStream objects can be re-used multiple times. * * Use ZSTD_initDStream() to start a new decompression operation. * @return : recommended first input size * Alternatively, use advanced API to set specific properties. * * Use ZSTD_decompressStream() repetitively to consume your input. * The function will update both `pos` fields. * If `input.pos < input.size`, some input has not been consumed. * It's up to the caller to present again remaining data. * The function tries to flush all data decoded immediately, respecting output buffer size. * If `output.pos < output.size`, decoder has flushed everything it could. * But if `output.pos == output.size`, there might be some data left within internal buffers., * In which case, call ZSTD_decompressStream() again to flush whatever remains in the buffer. * Note : with no additional input provided, amount of data flushed is necessarily <= ZSTD_BLOCKSIZE_MAX. * @return : 0 when a frame is completely decoded and fully flushed, * or an error code, which can be tested using ZSTD_isError(), * or any other value > 0, which means there is still some decoding or flushing to do to complete current frame : * the return value is a suggested next input size (just a hint for better latency) * that will never request more than the remaining frame size. * *******************************************************************************/ typedef ZSTD_DCtx ZSTD_DStream; /**< DCtx and DStream are now effectively same object (>= v1.3.0) */ /* For compatibility with versions <= v1.2.0, prefer differentiating them. */ /*===== ZSTD_DStream management functions =====*/ ZSTDLIB_API ZSTD_DStream* ZSTD_createDStream(void); ZSTDLIB_API size_t ZSTD_freeDStream(ZSTD_DStream* zds); /* accept NULL pointer */ /*===== Streaming decompression functions =====*/ /*! ZSTD_initDStream() : * Initialize/reset DStream state for new decompression operation. * Call before new decompression operation using same DStream. * * Note : This function is redundant with the advanced API and equivalent to: * ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); * ZSTD_DCtx_refDDict(zds, NULL); */ ZSTDLIB_API size_t ZSTD_initDStream(ZSTD_DStream* zds); /*! ZSTD_decompressStream() : * Streaming decompression function. * Call repetitively to consume full input updating it as necessary. * Function will update both input and output `pos` fields exposing current state via these fields: * - `input.pos < input.size`, some input remaining and caller should provide remaining input * on the next call. * - `output.pos < output.size`, decoder finished and flushed all remaining buffers. * - `output.pos == output.size`, potentially uncflushed data present in the internal buffers, * call ZSTD_decompressStream() again to flush remaining data to output. * Note : with no additional input, amount of data flushed <= ZSTD_BLOCKSIZE_MAX. * * @return : 0 when a frame is completely decoded and fully flushed, * or an error code, which can be tested using ZSTD_isError(), * or any other value > 0, which means there is some decoding or flushing to do to complete current frame. */ ZSTDLIB_API size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inBuffer* input); ZSTDLIB_API size_t ZSTD_DStreamInSize(void); /*!< recommended size for input buffer */ ZSTDLIB_API size_t ZSTD_DStreamOutSize(void); /*!< recommended size for output buffer. Guarantee to successfully flush at least one complete block in all circumstances. */ /************************** * Simple dictionary API ***************************/ /*! ZSTD_compress_usingDict() : * Compression at an explicit compression level using a Dictionary. * A dictionary can be any arbitrary data segment (also called a prefix), * or a buffer with specified information (see zdict.h). * Note : This function loads the dictionary, resulting in significant startup delay. * It's intended for a dictionary used only once. * Note 2 : When `dict == NULL || dictSize < 8` no dictionary is used. */ ZSTDLIB_API size_t ZSTD_compress_usingDict(ZSTD_CCtx* ctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const void* dict,size_t dictSize, int compressionLevel); /*! ZSTD_decompress_usingDict() : * Decompression using a known Dictionary. * Dictionary must be identical to the one used during compression. * Note : This function loads the dictionary, resulting in significant startup delay. * It's intended for a dictionary used only once. * Note : When `dict == NULL || dictSize < 8` no dictionary is used. */ ZSTDLIB_API size_t ZSTD_decompress_usingDict(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const void* dict,size_t dictSize); /*********************************** * Bulk processing dictionary API **********************************/ typedef struct ZSTD_CDict_s ZSTD_CDict; /*! ZSTD_createCDict() : * When compressing multiple messages or blocks using the same dictionary, * it's recommended to digest the dictionary only once, since it's a costly operation. * ZSTD_createCDict() will create a state from digesting a dictionary. * The resulting state can be used for future compression operations with very limited startup cost. * ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only. * @dictBuffer can be released after ZSTD_CDict creation, because its content is copied within CDict. * Note 1 : Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate @dictBuffer content. * Note 2 : A ZSTD_CDict can be created from an empty @dictBuffer, * in which case the only thing that it transports is the @compressionLevel. * This can be useful in a pipeline featuring ZSTD_compress_usingCDict() exclusively, * expecting a ZSTD_CDict parameter with any data, including those without a known dictionary. */ ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict(const void* dictBuffer, size_t dictSize, int compressionLevel); /*! ZSTD_freeCDict() : * Function frees memory allocated by ZSTD_createCDict(). * If a NULL pointer is passed, no operation is performed. */ ZSTDLIB_API size_t ZSTD_freeCDict(ZSTD_CDict* CDict); /*! ZSTD_compress_usingCDict() : * Compression using a digested Dictionary. * Recommended when same dictionary is used multiple times. * Note : compression level is _decided at dictionary creation time_, * and frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no) */ ZSTDLIB_API size_t ZSTD_compress_usingCDict(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const ZSTD_CDict* cdict); typedef struct ZSTD_DDict_s ZSTD_DDict; /*! ZSTD_createDDict() : * Create a digested dictionary, ready to start decompression operation without startup delay. * dictBuffer can be released after DDict creation, as its content is copied inside DDict. */ ZSTDLIB_API ZSTD_DDict* ZSTD_createDDict(const void* dictBuffer, size_t dictSize); /*! ZSTD_freeDDict() : * Function frees memory allocated with ZSTD_createDDict() * If a NULL pointer is passed, no operation is performed. */ ZSTDLIB_API size_t ZSTD_freeDDict(ZSTD_DDict* ddict); /*! ZSTD_decompress_usingDDict() : * Decompression using a digested Dictionary. * Recommended when same dictionary is used multiple times. */ ZSTDLIB_API size_t ZSTD_decompress_usingDDict(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const ZSTD_DDict* ddict); /******************************** * Dictionary helper functions *******************************/ /*! ZSTD_getDictID_fromDict() : Requires v1.4.0+ * Provides the dictID stored within dictionary. * if @return == 0, the dictionary is not conformant with Zstandard specification. * It can still be loaded, but as a content-only dictionary. */ ZSTDLIB_API unsigned ZSTD_getDictID_fromDict(const void* dict, size_t dictSize); /*! ZSTD_getDictID_fromCDict() : Requires v1.5.0+ * Provides the dictID of the dictionary loaded into `cdict`. * If @return == 0, the dictionary is not conformant to Zstandard specification, or empty. * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */ ZSTDLIB_API unsigned ZSTD_getDictID_fromCDict(const ZSTD_CDict* cdict); /*! ZSTD_getDictID_fromDDict() : Requires v1.4.0+ * Provides the dictID of the dictionary loaded into `ddict`. * If @return == 0, the dictionary is not conformant to Zstandard specification, or empty. * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */ ZSTDLIB_API unsigned ZSTD_getDictID_fromDDict(const ZSTD_DDict* ddict); /*! ZSTD_getDictID_fromFrame() : Requires v1.4.0+ * Provides the dictID required to decompressed the frame stored within `src`. * If @return == 0, the dictID could not be decoded. * This could for one of the following reasons : * - The frame does not require a dictionary to be decoded (most common case). * - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a hidden piece of information. * Note : this use case also happens when using a non-conformant dictionary. * - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`). * - This is not a Zstandard frame. * When identifying the exact failure cause, it's possible to use ZSTD_getFrameHeader(), which will provide a more precise error code. */ ZSTDLIB_API unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize); /******************************************************************************* * Advanced dictionary and prefix API (Requires v1.4.0+) * * This API allows dictionaries to be used with ZSTD_compress2(), * ZSTD_compressStream2(), and ZSTD_decompressDCtx(). * Dictionaries are sticky, they remain valid when same context is re-used, * they only reset when the context is reset * with ZSTD_reset_parameters or ZSTD_reset_session_and_parameters. * In contrast, Prefixes are single-use. ******************************************************************************/ /*! ZSTD_CCtx_loadDictionary() : Requires v1.4.0+ * Create an internal CDict from `dict` buffer. * Decompression will have to use same dictionary. * @result : 0, or an error code (which can be tested with ZSTD_isError()). * Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary, * meaning "return to no-dictionary mode". * Note 1 : Dictionary is sticky, it will be used for all future compressed frames, * until parameters are reset, a new dictionary is loaded, or the dictionary * is explicitly invalidated by loading a NULL dictionary. * Note 2 : Loading a dictionary involves building tables. * It's also a CPU consuming operation, with non-negligible impact on latency. * Tables are dependent on compression parameters, and for this reason, * compression parameters can no longer be changed after loading a dictionary. * Note 3 :`dict` content will be copied internally. * Use experimental ZSTD_CCtx_loadDictionary_byReference() to reference content instead. * In such a case, dictionary buffer must outlive its users. * Note 4 : Use ZSTD_CCtx_loadDictionary_advanced() * to precisely select how dictionary content must be interpreted. * Note 5 : This method does not benefit from LDM (long distance mode). * If you want to employ LDM on some large dictionary content, * prefer employing ZSTD_CCtx_refPrefix() described below. */ ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize); /*! ZSTD_CCtx_refCDict() : Requires v1.4.0+ * Reference a prepared dictionary, to be used for all future compressed frames. * Note that compression parameters are enforced from within CDict, * and supersede any compression parameter previously set within CCtx. * The parameters ignored are labelled as "superseded-by-cdict" in the ZSTD_cParameter enum docs. * The ignored parameters will be used again if the CCtx is returned to no-dictionary mode. * The dictionary will remain valid for future compressed frames using same CCtx. * @result : 0, or an error code (which can be tested with ZSTD_isError()). * Special : Referencing a NULL CDict means "return to no-dictionary mode". * Note 1 : Currently, only one dictionary can be managed. * Referencing a new dictionary effectively "discards" any previous one. * Note 2 : CDict is just referenced, its lifetime must outlive its usage within CCtx. */ ZSTDLIB_API size_t ZSTD_CCtx_refCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); /*! ZSTD_CCtx_refPrefix() : Requires v1.4.0+ * Reference a prefix (single-usage dictionary) for next compressed frame. * A prefix is **only used once**. Tables are discarded at end of frame (ZSTD_e_end). * Decompression will need same prefix to properly regenerate data. * Compressing with a prefix is similar in outcome as performing a diff and compressing it, * but performs much faster, especially during decompression (compression speed is tunable with compression level). * This method is compatible with LDM (long distance mode). * @result : 0, or an error code (which can be tested with ZSTD_isError()). * Special: Adding any prefix (including NULL) invalidates any previous prefix or dictionary * Note 1 : Prefix buffer is referenced. It **must** outlive compression. * Its content must remain unmodified during compression. * Note 2 : If the intention is to diff some large src data blob with some prior version of itself, * ensure that the window size is large enough to contain the entire source. * See ZSTD_c_windowLog. * Note 3 : Referencing a prefix involves building tables, which are dependent on compression parameters. * It's a CPU consuming operation, with non-negligible impact on latency. * If there is a need to use the same prefix multiple times, consider loadDictionary instead. * Note 4 : By default, the prefix is interpreted as raw content (ZSTD_dct_rawContent). * Use experimental ZSTD_CCtx_refPrefix_advanced() to alter dictionary interpretation. */ ZSTDLIB_API size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize); /*! ZSTD_DCtx_loadDictionary() : Requires v1.4.0+ * Create an internal DDict from dict buffer, to be used to decompress all future frames. * The dictionary remains valid for all future frames, until explicitly invalidated, or * a new dictionary is loaded. * @result : 0, or an error code (which can be tested with ZSTD_isError()). * Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary, * meaning "return to no-dictionary mode". * Note 1 : Loading a dictionary involves building tables, * which has a non-negligible impact on CPU usage and latency. * It's recommended to "load once, use many times", to amortize the cost * Note 2 :`dict` content will be copied internally, so `dict` can be released after loading. * Use ZSTD_DCtx_loadDictionary_byReference() to reference dictionary content instead. * Note 3 : Use ZSTD_DCtx_loadDictionary_advanced() to take control of * how dictionary content is loaded and interpreted. */ ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize); /*! ZSTD_DCtx_refDDict() : Requires v1.4.0+ * Reference a prepared dictionary, to be used to decompress next frames. * The dictionary remains active for decompression of future frames using same DCtx. * * If called with ZSTD_d_refMultipleDDicts enabled, repeated calls of this function * will store the DDict references in a table, and the DDict used for decompression * will be determined at decompression time, as per the dict ID in the frame. * The memory for the table is allocated on the first call to refDDict, and can be * freed with ZSTD_freeDCtx(). * * If called with ZSTD_d_refMultipleDDicts disabled (the default), only one dictionary * will be managed, and referencing a dictionary effectively "discards" any previous one. * * @result : 0, or an error code (which can be tested with ZSTD_isError()). * Special: referencing a NULL DDict means "return to no-dictionary mode". * Note 2 : DDict is just referenced, its lifetime must outlive its usage from DCtx. */ ZSTDLIB_API size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict); /*! ZSTD_DCtx_refPrefix() : Requires v1.4.0+ * Reference a prefix (single-usage dictionary) to decompress next frame. * This is the reverse operation of ZSTD_CCtx_refPrefix(), * and must use the same prefix as the one used during compression. * Prefix is **only used once**. Reference is discarded at end of frame. * End of frame is reached when ZSTD_decompressStream() returns 0. * @result : 0, or an error code (which can be tested with ZSTD_isError()). * Note 1 : Adding any prefix (including NULL) invalidates any previously set prefix or dictionary * Note 2 : Prefix buffer is referenced. It **must** outlive decompression. * Prefix buffer must remain unmodified up to the end of frame, * reached when ZSTD_decompressStream() returns 0. * Note 3 : By default, the prefix is treated as raw content (ZSTD_dct_rawContent). * Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode (Experimental section) * Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost. * A full dictionary is more costly, as it requires building tables. */ ZSTDLIB_API size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize); /* === Memory management === */ /*! ZSTD_sizeof_*() : Requires v1.4.0+ * These functions give the _current_ memory usage of selected object. * Note that object memory usage can evolve (increase or decrease) over time. */ ZSTDLIB_API size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx); ZSTDLIB_API size_t ZSTD_sizeof_DCtx(const ZSTD_DCtx* dctx); ZSTDLIB_API size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs); ZSTDLIB_API size_t ZSTD_sizeof_DStream(const ZSTD_DStream* zds); ZSTDLIB_API size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict); ZSTDLIB_API size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict); #endif /* ZSTD_H_235446 */ /* ************************************************************************************** * ADVANCED AND EXPERIMENTAL FUNCTIONS **************************************************************************************** * The definitions in the following section are considered experimental. * They are provided for advanced scenarios. * They should never be used with a dynamic library, as prototypes may change in the future. * Use them only in association with static linking. * ***************************************************************************************/ #if defined(ZSTD_STATIC_LINKING_ONLY) && !defined(ZSTD_H_ZSTD_STATIC_LINKING_ONLY) #define ZSTD_H_ZSTD_STATIC_LINKING_ONLY /* This can be overridden externally to hide static symbols. */ #ifndef ZSTDLIB_STATIC_API # if defined(ZSTD_DLL_EXPORT) && (ZSTD_DLL_EXPORT==1) # define ZSTDLIB_STATIC_API __declspec(dllexport) ZSTDLIB_VISIBLE # elif defined(ZSTD_DLL_IMPORT) && (ZSTD_DLL_IMPORT==1) # define ZSTDLIB_STATIC_API __declspec(dllimport) ZSTDLIB_VISIBLE # else # define ZSTDLIB_STATIC_API ZSTDLIB_VISIBLE # endif #endif /**************************************************************************************** * experimental API (static linking only) **************************************************************************************** * The following symbols and constants * are not planned to join "stable API" status in the near future. * They can still change in future versions. * Some of them are planned to remain in the static_only section indefinitely. * Some of them might be removed in the future (especially when redundant with existing stable functions) * ***************************************************************************************/ #define ZSTD_FRAMEHEADERSIZE_PREFIX(format) ((format) == ZSTD_f_zstd1 ? 5 : 1) /* minimum input size required to query frame header size */ #define ZSTD_FRAMEHEADERSIZE_MIN(format) ((format) == ZSTD_f_zstd1 ? 6 : 2) #define ZSTD_FRAMEHEADERSIZE_MAX 18 /* can be useful for static allocation */ #define ZSTD_SKIPPABLEHEADERSIZE 8 /* compression parameter bounds */ #define ZSTD_WINDOWLOG_MAX_32 30 #define ZSTD_WINDOWLOG_MAX_64 31 #define ZSTD_WINDOWLOG_MAX ((int)(sizeof(size_t) == 4 ? ZSTD_WINDOWLOG_MAX_32 : ZSTD_WINDOWLOG_MAX_64)) #define ZSTD_WINDOWLOG_MIN 10 #define ZSTD_HASHLOG_MAX ((ZSTD_WINDOWLOG_MAX < 30) ? ZSTD_WINDOWLOG_MAX : 30) #define ZSTD_HASHLOG_MIN 6 #define ZSTD_CHAINLOG_MAX_32 29 #define ZSTD_CHAINLOG_MAX_64 30 #define ZSTD_CHAINLOG_MAX ((int)(sizeof(size_t) == 4 ? ZSTD_CHAINLOG_MAX_32 : ZSTD_CHAINLOG_MAX_64)) #define ZSTD_CHAINLOG_MIN ZSTD_HASHLOG_MIN #define ZSTD_SEARCHLOG_MAX (ZSTD_WINDOWLOG_MAX-1) #define ZSTD_SEARCHLOG_MIN 1 #define ZSTD_MINMATCH_MAX 7 /* only for ZSTD_fast, other strategies are limited to 6 */ #define ZSTD_MINMATCH_MIN 3 /* only for ZSTD_btopt+, faster strategies are limited to 4 */ #define ZSTD_TARGETLENGTH_MAX ZSTD_BLOCKSIZE_MAX #define ZSTD_TARGETLENGTH_MIN 0 /* note : comparing this constant to an unsigned results in a tautological test */ #define ZSTD_STRATEGY_MIN ZSTD_fast #define ZSTD_STRATEGY_MAX ZSTD_btultra2 #define ZSTD_BLOCKSIZE_MAX_MIN (1 << 10) /* The minimum valid max blocksize. Maximum blocksizes smaller than this make compressBound() inaccurate. */ #define ZSTD_OVERLAPLOG_MIN 0 #define ZSTD_OVERLAPLOG_MAX 9 #define ZSTD_WINDOWLOG_LIMIT_DEFAULT 27 /* by default, the streaming decoder will refuse any frame * requiring larger than (1< 0: * If litLength != 0: * rep == 1 --> offset == repeat_offset_1 * rep == 2 --> offset == repeat_offset_2 * rep == 3 --> offset == repeat_offset_3 * If litLength == 0: * rep == 1 --> offset == repeat_offset_2 * rep == 2 --> offset == repeat_offset_3 * rep == 3 --> offset == repeat_offset_1 - 1 * * Note: This field is optional. ZSTD_generateSequences() will calculate the value of * 'rep', but repeat offsets do not necessarily need to be calculated from an external * sequence provider's perspective. For example, ZSTD_compressSequences() does not * use this 'rep' field at all (as of now). */ } ZSTD_Sequence; typedef struct { unsigned windowLog; /**< largest match distance : larger == more compression, more memory needed during decompression */ unsigned chainLog; /**< fully searched segment : larger == more compression, slower, more memory (useless for fast) */ unsigned hashLog; /**< dispatch table : larger == faster, more memory */ unsigned searchLog; /**< nb of searches : larger == more compression, slower */ unsigned minMatch; /**< match length searched : larger == faster decompression, sometimes less compression */ unsigned targetLength; /**< acceptable match size for optimal parser (only) : larger == more compression, slower */ ZSTD_strategy strategy; /**< see ZSTD_strategy definition above */ } ZSTD_compressionParameters; typedef struct { int contentSizeFlag; /**< 1: content size will be in frame header (when known) */ int checksumFlag; /**< 1: generate a 32-bits checksum using XXH64 algorithm at end of frame, for error detection */ int noDictIDFlag; /**< 1: no dictID will be saved into frame header (dictID is only useful for dictionary compression) */ } ZSTD_frameParameters; typedef struct { ZSTD_compressionParameters cParams; ZSTD_frameParameters fParams; } ZSTD_parameters; typedef enum { ZSTD_dct_auto = 0, /* dictionary is "full" when starting with ZSTD_MAGIC_DICTIONARY, otherwise it is "rawContent" */ ZSTD_dct_rawContent = 1, /* ensures dictionary is always loaded as rawContent, even if it starts with ZSTD_MAGIC_DICTIONARY */ ZSTD_dct_fullDict = 2 /* refuses to load a dictionary if it does not respect Zstandard's specification, starting with ZSTD_MAGIC_DICTIONARY */ } ZSTD_dictContentType_e; typedef enum { ZSTD_dlm_byCopy = 0, /**< Copy dictionary content internally */ ZSTD_dlm_byRef = 1 /**< Reference dictionary content -- the dictionary buffer must outlive its users. */ } ZSTD_dictLoadMethod_e; typedef enum { ZSTD_f_zstd1 = 0, /* zstd frame format, specified in zstd_compression_format.md (default) */ ZSTD_f_zstd1_magicless = 1 /* Variant of zstd frame format, without initial 4-bytes magic number. * Useful to save 4 bytes per generated frame. * Decoder cannot recognise automatically this format, requiring this instruction. */ } ZSTD_format_e; typedef enum { /* Note: this enum controls ZSTD_d_forceIgnoreChecksum */ ZSTD_d_validateChecksum = 0, ZSTD_d_ignoreChecksum = 1 } ZSTD_forceIgnoreChecksum_e; typedef enum { /* Note: this enum controls ZSTD_d_refMultipleDDicts */ ZSTD_rmd_refSingleDDict = 0, ZSTD_rmd_refMultipleDDicts = 1 } ZSTD_refMultipleDDicts_e; typedef enum { /* Note: this enum and the behavior it controls are effectively internal * implementation details of the compressor. They are expected to continue * to evolve and should be considered only in the context of extremely * advanced performance tuning. * * Zstd currently supports the use of a CDict in three ways: * * - The contents of the CDict can be copied into the working context. This * means that the compression can search both the dictionary and input * while operating on a single set of internal tables. This makes * the compression faster per-byte of input. However, the initial copy of * the CDict's tables incurs a fixed cost at the beginning of the * compression. For small compressions (< 8 KB), that copy can dominate * the cost of the compression. * * - The CDict's tables can be used in-place. In this model, compression is * slower per input byte, because the compressor has to search two sets of * tables. However, this model incurs no start-up cost (as long as the * working context's tables can be reused). For small inputs, this can be * faster than copying the CDict's tables. * * - The CDict's tables are not used at all, and instead we use the working * context alone to reload the dictionary and use params based on the source * size. See ZSTD_compress_insertDictionary() and ZSTD_compress_usingDict(). * This method is effective when the dictionary sizes are very small relative * to the input size, and the input size is fairly large to begin with. * * Zstd has a simple internal heuristic that selects which strategy to use * at the beginning of a compression. However, if experimentation shows that * Zstd is making poor choices, it is possible to override that choice with * this enum. */ ZSTD_dictDefaultAttach = 0, /* Use the default heuristic. */ ZSTD_dictForceAttach = 1, /* Never copy the dictionary. */ ZSTD_dictForceCopy = 2, /* Always copy the dictionary. */ ZSTD_dictForceLoad = 3 /* Always reload the dictionary */ } ZSTD_dictAttachPref_e; typedef enum { ZSTD_lcm_auto = 0, /**< Automatically determine the compression mode based on the compression level. * Negative compression levels will be uncompressed, and positive compression * levels will be compressed. */ ZSTD_lcm_huffman = 1, /**< Always attempt Huffman compression. Uncompressed literals will still be * emitted if Huffman compression is not profitable. */ ZSTD_lcm_uncompressed = 2 /**< Always emit uncompressed literals. */ } ZSTD_literalCompressionMode_e; typedef enum { /* Note: This enum controls features which are conditionally beneficial. Zstd typically will make a final * decision on whether or not to enable the feature (ZSTD_ps_auto), but setting the switch to ZSTD_ps_enable * or ZSTD_ps_disable allow for a force enable/disable the feature. */ ZSTD_ps_auto = 0, /* Let the library automatically determine whether the feature shall be enabled */ ZSTD_ps_enable = 1, /* Force-enable the feature */ ZSTD_ps_disable = 2 /* Do not use the feature */ } ZSTD_paramSwitch_e; /*************************************** * Frame header and size functions ***************************************/ /*! ZSTD_findDecompressedSize() : * `src` should point to the start of a series of ZSTD encoded and/or skippable frames * `srcSize` must be the _exact_ size of this series * (i.e. there should be a frame boundary at `src + srcSize`) * @return : - decompressed size of all data in all successive frames * - if the decompressed size cannot be determined: ZSTD_CONTENTSIZE_UNKNOWN * - if an error occurred: ZSTD_CONTENTSIZE_ERROR * * note 1 : decompressed size is an optional field, that may not be present, especially in streaming mode. * When `return==ZSTD_CONTENTSIZE_UNKNOWN`, data to decompress could be any size. * In which case, it's necessary to use streaming mode to decompress data. * note 2 : decompressed size is always present when compression is done with ZSTD_compress() * note 3 : decompressed size can be very large (64-bits value), * potentially larger than what local system can handle as a single memory segment. * In which case, it's necessary to use streaming mode to decompress data. * note 4 : If source is untrusted, decompressed size could be wrong or intentionally modified. * Always ensure result fits within application's authorized limits. * Each application can set its own limits. * note 5 : ZSTD_findDecompressedSize handles multiple frames, and so it must traverse the input to * read each contained frame header. This is fast as most of the data is skipped, * however it does mean that all frame data must be present and valid. */ ZSTDLIB_STATIC_API unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize); /*! ZSTD_decompressBound() : * `src` should point to the start of a series of ZSTD encoded and/or skippable frames * `srcSize` must be the _exact_ size of this series * (i.e. there should be a frame boundary at `src + srcSize`) * @return : - upper-bound for the decompressed size of all data in all successive frames * - if an error occurred: ZSTD_CONTENTSIZE_ERROR * * note 1 : an error can occur if `src` contains an invalid or incorrectly formatted frame. * note 2 : the upper-bound is exact when the decompressed size field is available in every ZSTD encoded frame of `src`. * in this case, `ZSTD_findDecompressedSize` and `ZSTD_decompressBound` return the same value. * note 3 : when the decompressed size field isn't available, the upper-bound for that frame is calculated by: * upper-bound = # blocks * min(128 KB, Window_Size) */ ZSTDLIB_STATIC_API unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize); /*! ZSTD_frameHeaderSize() : * srcSize must be >= ZSTD_FRAMEHEADERSIZE_PREFIX. * @return : size of the Frame Header, * or an error code (if srcSize is too small) */ ZSTDLIB_STATIC_API size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize); typedef enum { ZSTD_frame, ZSTD_skippableFrame } ZSTD_frameType_e; typedef struct { unsigned long long frameContentSize; /* if == ZSTD_CONTENTSIZE_UNKNOWN, it means this field is not available. 0 means "empty" */ unsigned long long windowSize; /* can be very large, up to <= frameContentSize */ unsigned blockSizeMax; ZSTD_frameType_e frameType; /* if == ZSTD_skippableFrame, frameContentSize is the size of skippable content */ unsigned headerSize; unsigned dictID; unsigned checksumFlag; unsigned _reserved1; unsigned _reserved2; } ZSTD_frameHeader; /*! ZSTD_getFrameHeader() : * decode Frame Header, or requires larger `srcSize`. * @return : 0, `zfhPtr` is correctly filled, * >0, `srcSize` is too small, value is wanted `srcSize` amount, * or an error code, which can be tested using ZSTD_isError() */ ZSTDLIB_STATIC_API size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize); /**< doesn't consume input */ /*! ZSTD_getFrameHeader_advanced() : * same as ZSTD_getFrameHeader(), * with added capability to select a format (like ZSTD_f_zstd1_magicless) */ ZSTDLIB_STATIC_API size_t ZSTD_getFrameHeader_advanced(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize, ZSTD_format_e format); /*! ZSTD_decompressionMargin() : * Zstd supports in-place decompression, where the input and output buffers overlap. * In this case, the output buffer must be at least (Margin + Output_Size) bytes large, * and the input buffer must be at the end of the output buffer. * * _______________________ Output Buffer ________________________ * | | * | ____ Input Buffer ____| * | | | * v v v * |---------------------------------------|-----------|----------| * ^ ^ ^ * |___________________ Output_Size ___________________|_ Margin _| * * NOTE: See also ZSTD_DECOMPRESSION_MARGIN(). * NOTE: This applies only to single-pass decompression through ZSTD_decompress() or * ZSTD_decompressDCtx(). * NOTE: This function supports multi-frame input. * * @param src The compressed frame(s) * @param srcSize The size of the compressed frame(s) * @returns The decompression margin or an error that can be checked with ZSTD_isError(). */ ZSTDLIB_STATIC_API size_t ZSTD_decompressionMargin(const void* src, size_t srcSize); /*! ZSTD_DECOMPRESS_MARGIN() : * Similar to ZSTD_decompressionMargin(), but instead of computing the margin from * the compressed frame, compute it from the original size and the blockSizeLog. * See ZSTD_decompressionMargin() for details. * * WARNING: This macro does not support multi-frame input, the input must be a single * zstd frame. If you need that support use the function, or implement it yourself. * * @param originalSize The original uncompressed size of the data. * @param blockSize The block size == MIN(windowSize, ZSTD_BLOCKSIZE_MAX). * Unless you explicitly set the windowLog smaller than * ZSTD_BLOCKSIZELOG_MAX you can just use ZSTD_BLOCKSIZE_MAX. */ #define ZSTD_DECOMPRESSION_MARGIN(originalSize, blockSize) ((size_t)( \ ZSTD_FRAMEHEADERSIZE_MAX /* Frame header */ + \ 4 /* checksum */ + \ ((originalSize) == 0 ? 0 : 3 * (((originalSize) + (blockSize) - 1) / blockSize)) /* 3 bytes per block */ + \ (blockSize) /* One block of margin */ \ )) typedef enum { ZSTD_sf_noBlockDelimiters = 0, /* Representation of ZSTD_Sequence has no block delimiters, sequences only */ ZSTD_sf_explicitBlockDelimiters = 1 /* Representation of ZSTD_Sequence contains explicit block delimiters */ } ZSTD_sequenceFormat_e; /*! ZSTD_sequenceBound() : * `srcSize` : size of the input buffer * @return : upper-bound for the number of sequences that can be generated * from a buffer of srcSize bytes * * note : returns number of sequences - to get bytes, multiply by sizeof(ZSTD_Sequence). */ ZSTDLIB_STATIC_API size_t ZSTD_sequenceBound(size_t srcSize); /*! ZSTD_generateSequences() : * Generate sequences using ZSTD_compress2(), given a source buffer. * * Each block will end with a dummy sequence * with offset == 0, matchLength == 0, and litLength == length of last literals. * litLength may be == 0, and if so, then the sequence of (of: 0 ml: 0 ll: 0) * simply acts as a block delimiter. * * @zc can be used to insert custom compression params. * This function invokes ZSTD_compress2(). * * The output of this function can be fed into ZSTD_compressSequences() with CCtx * setting of ZSTD_c_blockDelimiters as ZSTD_sf_explicitBlockDelimiters * @return : number of sequences generated */ ZSTDLIB_STATIC_API size_t ZSTD_generateSequences( ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs, size_t outSeqsSize, const void* src, size_t srcSize); /*! ZSTD_mergeBlockDelimiters() : * Given an array of ZSTD_Sequence, remove all sequences that represent block delimiters/last literals * by merging them into the literals of the next sequence. * * As such, the final generated result has no explicit representation of block boundaries, * and the final last literals segment is not represented in the sequences. * * The output of this function can be fed into ZSTD_compressSequences() with CCtx * setting of ZSTD_c_blockDelimiters as ZSTD_sf_noBlockDelimiters * @return : number of sequences left after merging */ ZSTDLIB_STATIC_API size_t ZSTD_mergeBlockDelimiters(ZSTD_Sequence* sequences, size_t seqsSize); /*! ZSTD_compressSequences() : * Compress an array of ZSTD_Sequence, associated with @src buffer, into dst. * @src contains the entire input (not just the literals). * If @srcSize > sum(sequence.length), the remaining bytes are considered all literals * If a dictionary is included, then the cctx should reference the dict. (see: ZSTD_CCtx_refCDict(), ZSTD_CCtx_loadDictionary(), etc.) * The entire source is compressed into a single frame. * * The compression behavior changes based on cctx params. In particular: * If ZSTD_c_blockDelimiters == ZSTD_sf_noBlockDelimiters, the array of ZSTD_Sequence is expected to contain * no block delimiters (defined in ZSTD_Sequence). Block boundaries are roughly determined based on * the block size derived from the cctx, and sequences may be split. This is the default setting. * * If ZSTD_c_blockDelimiters == ZSTD_sf_explicitBlockDelimiters, the array of ZSTD_Sequence is expected to contain * block delimiters (defined in ZSTD_Sequence). Behavior is undefined if no block delimiters are provided. * * If ZSTD_c_validateSequences == 0, this function will blindly accept the sequences provided. Invalid sequences cause undefined * behavior. If ZSTD_c_validateSequences == 1, then if sequence is invalid (see doc/zstd_compression_format.md for * specifics regarding offset/matchlength requirements) then the function will bail out and return an error. * * In addition to the two adjustable experimental params, there are other important cctx params. * - ZSTD_c_minMatch MUST be set as less than or equal to the smallest match generated by the match finder. It has a minimum value of ZSTD_MINMATCH_MIN. * - ZSTD_c_compressionLevel accordingly adjusts the strength of the entropy coder, as it would in typical compression. * - ZSTD_c_windowLog affects offset validation: this function will return an error at higher debug levels if a provided offset * is larger than what the spec allows for a given window log and dictionary (if present). See: doc/zstd_compression_format.md * * Note: Repcodes are, as of now, always re-calculated within this function, so ZSTD_Sequence::rep is unused. * Note 2: Once we integrate ability to ingest repcodes, the explicit block delims mode must respect those repcodes exactly, * and cannot emit an RLE block that disagrees with the repcode history * @return : final compressed size, or a ZSTD error code. */ ZSTDLIB_STATIC_API size_t ZSTD_compressSequences( ZSTD_CCtx* cctx, void* dst, size_t dstSize, const ZSTD_Sequence* inSeqs, size_t inSeqsSize, const void* src, size_t srcSize); /*! ZSTD_writeSkippableFrame() : * Generates a zstd skippable frame containing data given by src, and writes it to dst buffer. * * Skippable frames begin with a 4-byte magic number. There are 16 possible choices of magic number, * ranging from ZSTD_MAGIC_SKIPPABLE_START to ZSTD_MAGIC_SKIPPABLE_START+15. * As such, the parameter magicVariant controls the exact skippable frame magic number variant used, so * the magic number used will be ZSTD_MAGIC_SKIPPABLE_START + magicVariant. * * Returns an error if destination buffer is not large enough, if the source size is not representable * with a 4-byte unsigned int, or if the parameter magicVariant is greater than 15 (and therefore invalid). * * @return : number of bytes written or a ZSTD error. */ ZSTDLIB_STATIC_API size_t ZSTD_writeSkippableFrame(void* dst, size_t dstCapacity, const void* src, size_t srcSize, unsigned magicVariant); /*! ZSTD_readSkippableFrame() : * Retrieves a zstd skippable frame containing data given by src, and writes it to dst buffer. * * The parameter magicVariant will receive the magicVariant that was supplied when the frame was written, * i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START. This can be NULL if the caller is not interested * in the magicVariant. * * Returns an error if destination buffer is not large enough, or if the frame is not skippable. * * @return : number of bytes written or a ZSTD error. */ ZSTDLIB_API size_t ZSTD_readSkippableFrame(void* dst, size_t dstCapacity, unsigned* magicVariant, const void* src, size_t srcSize); /*! ZSTD_isSkippableFrame() : * Tells if the content of `buffer` starts with a valid Frame Identifier for a skippable frame. */ ZSTDLIB_API unsigned ZSTD_isSkippableFrame(const void* buffer, size_t size); /*************************************** * Memory management ***************************************/ /*! ZSTD_estimate*() : * These functions make it possible to estimate memory usage * of a future {D,C}Ctx, before its creation. * * ZSTD_estimateCCtxSize() will provide a memory budget large enough * for any compression level up to selected one. * Note : Unlike ZSTD_estimateCStreamSize*(), this estimate * does not include space for a window buffer. * Therefore, the estimation is only guaranteed for single-shot compressions, not streaming. * The estimate will assume the input may be arbitrarily large, * which is the worst case. * * When srcSize can be bound by a known and rather "small" value, * this fact can be used to provide a tighter estimation * because the CCtx compression context will need less memory. * This tighter estimation can be provided by more advanced functions * ZSTD_estimateCCtxSize_usingCParams(), which can be used in tandem with ZSTD_getCParams(), * and ZSTD_estimateCCtxSize_usingCCtxParams(), which can be used in tandem with ZSTD_CCtxParams_setParameter(). * Both can be used to estimate memory using custom compression parameters and arbitrary srcSize limits. * * Note : only single-threaded compression is supported. * ZSTD_estimateCCtxSize_usingCCtxParams() will return an error code if ZSTD_c_nbWorkers is >= 1. * * Note 2 : ZSTD_estimateCCtxSize* functions are not compatible with the Block-Level Sequence Producer API at this time. * Size estimates assume that no external sequence producer is registered. */ ZSTDLIB_STATIC_API size_t ZSTD_estimateCCtxSize(int compressionLevel); ZSTDLIB_STATIC_API size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams); ZSTDLIB_STATIC_API size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params); ZSTDLIB_STATIC_API size_t ZSTD_estimateDCtxSize(void); /*! ZSTD_estimateCStreamSize() : * ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one. * It will also consider src size to be arbitrarily "large", which is worst case. * If srcSize is known to always be small, ZSTD_estimateCStreamSize_usingCParams() can provide a tighter estimation. * ZSTD_estimateCStreamSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel. * ZSTD_estimateCStreamSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_c_nbWorkers is >= 1. * Note : CStream size estimation is only correct for single-threaded compression. * ZSTD_DStream memory budget depends on window Size. * This information can be passed manually, using ZSTD_estimateDStreamSize, * or deducted from a valid frame Header, using ZSTD_estimateDStreamSize_fromFrame(); * Note : if streaming is init with function ZSTD_init?Stream_usingDict(), * an internal ?Dict will be created, which additional size is not estimated here. * In this case, get total size by adding ZSTD_estimate?DictSize * Note 2 : only single-threaded compression is supported. * ZSTD_estimateCStreamSize_usingCCtxParams() will return an error code if ZSTD_c_nbWorkers is >= 1. * Note 3 : ZSTD_estimateCStreamSize* functions are not compatible with the Block-Level Sequence Producer API at this time. * Size estimates assume that no external sequence producer is registered. */ ZSTDLIB_STATIC_API size_t ZSTD_estimateCStreamSize(int compressionLevel); ZSTDLIB_STATIC_API size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams); ZSTDLIB_STATIC_API size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params); ZSTDLIB_STATIC_API size_t ZSTD_estimateDStreamSize(size_t windowSize); ZSTDLIB_STATIC_API size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize); /*! ZSTD_estimate?DictSize() : * ZSTD_estimateCDictSize() will bet that src size is relatively "small", and content is copied, like ZSTD_createCDict(). * ZSTD_estimateCDictSize_advanced() makes it possible to control compression parameters precisely, like ZSTD_createCDict_advanced(). * Note : dictionaries created by reference (`ZSTD_dlm_byRef`) are logically smaller. */ ZSTDLIB_STATIC_API size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel); ZSTDLIB_STATIC_API size_t ZSTD_estimateCDictSize_advanced(size_t dictSize, ZSTD_compressionParameters cParams, ZSTD_dictLoadMethod_e dictLoadMethod); ZSTDLIB_STATIC_API size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod); /*! ZSTD_initStatic*() : * Initialize an object using a pre-allocated fixed-size buffer. * workspace: The memory area to emplace the object into. * Provided pointer *must be 8-bytes aligned*. * Buffer must outlive object. * workspaceSize: Use ZSTD_estimate*Size() to determine * how large workspace must be to support target scenario. * @return : pointer to object (same address as workspace, just different type), * or NULL if error (size too small, incorrect alignment, etc.) * Note : zstd will never resize nor malloc() when using a static buffer. * If the object requires more memory than available, * zstd will just error out (typically ZSTD_error_memory_allocation). * Note 2 : there is no corresponding "free" function. * Since workspace is allocated externally, it must be freed externally too. * Note 3 : cParams : use ZSTD_getCParams() to convert a compression level * into its associated cParams. * Limitation 1 : currently not compatible with internal dictionary creation, triggered by * ZSTD_CCtx_loadDictionary(), ZSTD_initCStream_usingDict() or ZSTD_initDStream_usingDict(). * Limitation 2 : static cctx currently not compatible with multi-threading. * Limitation 3 : static dctx is incompatible with legacy support. */ ZSTDLIB_STATIC_API ZSTD_CCtx* ZSTD_initStaticCCtx(void* workspace, size_t workspaceSize); ZSTDLIB_STATIC_API ZSTD_CStream* ZSTD_initStaticCStream(void* workspace, size_t workspaceSize); /**< same as ZSTD_initStaticCCtx() */ ZSTDLIB_STATIC_API ZSTD_DCtx* ZSTD_initStaticDCtx(void* workspace, size_t workspaceSize); ZSTDLIB_STATIC_API ZSTD_DStream* ZSTD_initStaticDStream(void* workspace, size_t workspaceSize); /**< same as ZSTD_initStaticDCtx() */ ZSTDLIB_STATIC_API const ZSTD_CDict* ZSTD_initStaticCDict( void* workspace, size_t workspaceSize, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType, ZSTD_compressionParameters cParams); ZSTDLIB_STATIC_API const ZSTD_DDict* ZSTD_initStaticDDict( void* workspace, size_t workspaceSize, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType); /*! Custom memory allocation : * These prototypes make it possible to pass your own allocation/free functions. * ZSTD_customMem is provided at creation time, using ZSTD_create*_advanced() variants listed below. * All allocation/free operations will be completed using these custom variants instead of regular ones. */ typedef void* (*ZSTD_allocFunction) (void* opaque, size_t size); typedef void (*ZSTD_freeFunction) (void* opaque, void* address); typedef struct { ZSTD_allocFunction customAlloc; ZSTD_freeFunction customFree; void* opaque; } ZSTD_customMem; static #ifdef __GNUC__ __attribute__((__unused__)) #endif ZSTD_customMem const ZSTD_defaultCMem = { NULL, NULL, NULL }; /**< this constant defers to stdlib's functions */ ZSTDLIB_STATIC_API ZSTD_CCtx* ZSTD_createCCtx_advanced(ZSTD_customMem customMem); ZSTDLIB_STATIC_API ZSTD_CStream* ZSTD_createCStream_advanced(ZSTD_customMem customMem); ZSTDLIB_STATIC_API ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem); ZSTDLIB_STATIC_API ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem); ZSTDLIB_STATIC_API ZSTD_CDict* ZSTD_createCDict_advanced(const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType, ZSTD_compressionParameters cParams, ZSTD_customMem customMem); /*! Thread pool : * These prototypes make it possible to share a thread pool among multiple compression contexts. * This can limit resources for applications with multiple threads where each one uses * a threaded compression mode (via ZSTD_c_nbWorkers parameter). * ZSTD_createThreadPool creates a new thread pool with a given number of threads. * Note that the lifetime of such pool must exist while being used. * ZSTD_CCtx_refThreadPool assigns a thread pool to a context (use NULL argument value * to use an internal thread pool). * ZSTD_freeThreadPool frees a thread pool, accepts NULL pointer. */ typedef struct POOL_ctx_s ZSTD_threadPool; ZSTDLIB_STATIC_API ZSTD_threadPool* ZSTD_createThreadPool(size_t numThreads); ZSTDLIB_STATIC_API void ZSTD_freeThreadPool (ZSTD_threadPool* pool); /* accept NULL pointer */ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_refThreadPool(ZSTD_CCtx* cctx, ZSTD_threadPool* pool); /* * This API is temporary and is expected to change or disappear in the future! */ ZSTDLIB_STATIC_API ZSTD_CDict* ZSTD_createCDict_advanced2( const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType, const ZSTD_CCtx_params* cctxParams, ZSTD_customMem customMem); ZSTDLIB_STATIC_API ZSTD_DDict* ZSTD_createDDict_advanced( const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType, ZSTD_customMem customMem); /*************************************** * Advanced compression functions ***************************************/ /*! ZSTD_createCDict_byReference() : * Create a digested dictionary for compression * Dictionary content is just referenced, not duplicated. * As a consequence, `dictBuffer` **must** outlive CDict, * and its content must remain unmodified throughout the lifetime of CDict. * note: equivalent to ZSTD_createCDict_advanced(), with dictLoadMethod==ZSTD_dlm_byRef */ ZSTDLIB_STATIC_API ZSTD_CDict* ZSTD_createCDict_byReference(const void* dictBuffer, size_t dictSize, int compressionLevel); /*! ZSTD_getCParams() : * @return ZSTD_compressionParameters structure for a selected compression level and estimated srcSize. * `estimatedSrcSize` value is optional, select 0 if not known */ ZSTDLIB_STATIC_API ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize); /*! ZSTD_getParams() : * same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-component `ZSTD_compressionParameters`. * All fields of `ZSTD_frameParameters` are set to default : contentSize=1, checksum=0, noDictID=0 */ ZSTDLIB_STATIC_API ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize); /*! ZSTD_checkCParams() : * Ensure param values remain within authorized range. * @return 0 on success, or an error code (can be checked with ZSTD_isError()) */ ZSTDLIB_STATIC_API size_t ZSTD_checkCParams(ZSTD_compressionParameters params); /*! ZSTD_adjustCParams() : * optimize params for a given `srcSize` and `dictSize`. * `srcSize` can be unknown, in which case use ZSTD_CONTENTSIZE_UNKNOWN. * `dictSize` must be `0` when there is no dictionary. * cPar can be invalid : all parameters will be clamped within valid range in the @return struct. * This function never fails (wide contract) */ ZSTDLIB_STATIC_API ZSTD_compressionParameters ZSTD_adjustCParams(ZSTD_compressionParameters cPar, unsigned long long srcSize, size_t dictSize); /*! ZSTD_CCtx_setCParams() : * Set all parameters provided within @p cparams into the working @p cctx. * Note : if modifying parameters during compression (MT mode only), * note that changes to the .windowLog parameter will be ignored. * @return 0 on success, or an error code (can be checked with ZSTD_isError()). * On failure, no parameters are updated. */ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_setCParams(ZSTD_CCtx* cctx, ZSTD_compressionParameters cparams); /*! ZSTD_CCtx_setFParams() : * Set all parameters provided within @p fparams into the working @p cctx. * @return 0 on success, or an error code (can be checked with ZSTD_isError()). */ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_setFParams(ZSTD_CCtx* cctx, ZSTD_frameParameters fparams); /*! ZSTD_CCtx_setParams() : * Set all parameters provided within @p params into the working @p cctx. * @return 0 on success, or an error code (can be checked with ZSTD_isError()). */ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_setParams(ZSTD_CCtx* cctx, ZSTD_parameters params); /*! ZSTD_compress_advanced() : * Note : this function is now DEPRECATED. * It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_setParameter() and other parameter setters. * This prototype will generate compilation warnings. */ ZSTD_DEPRECATED("use ZSTD_compress2") ZSTDLIB_STATIC_API size_t ZSTD_compress_advanced(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const void* dict,size_t dictSize, ZSTD_parameters params); /*! ZSTD_compress_usingCDict_advanced() : * Note : this function is now DEPRECATED. * It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_loadDictionary() and other parameter setters. * This prototype will generate compilation warnings. */ ZSTD_DEPRECATED("use ZSTD_compress2 with ZSTD_CCtx_loadDictionary") ZSTDLIB_STATIC_API size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const ZSTD_CDict* cdict, ZSTD_frameParameters fParams); /*! ZSTD_CCtx_loadDictionary_byReference() : * Same as ZSTD_CCtx_loadDictionary(), but dictionary content is referenced, instead of being copied into CCtx. * It saves some memory, but also requires that `dict` outlives its usage within `cctx` */ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_loadDictionary_byReference(ZSTD_CCtx* cctx, const void* dict, size_t dictSize); /*! ZSTD_CCtx_loadDictionary_advanced() : * Same as ZSTD_CCtx_loadDictionary(), but gives finer control over * how to load the dictionary (by copy ? by reference ?) * and how to interpret it (automatic ? force raw mode ? full mode only ?) */ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_loadDictionary_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType); /*! ZSTD_CCtx_refPrefix_advanced() : * Same as ZSTD_CCtx_refPrefix(), but gives finer control over * how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) */ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType); /* === experimental parameters === */ /* these parameters can be used with ZSTD_setParameter() * they are not guaranteed to remain supported in the future */ /* Enables rsyncable mode, * which makes compressed files more rsync friendly * by adding periodic synchronization points to the compressed data. * The target average block size is ZSTD_c_jobSize / 2. * It's possible to modify the job size to increase or decrease * the granularity of the synchronization point. * Once the jobSize is smaller than the window size, * it will result in compression ratio degradation. * NOTE 1: rsyncable mode only works when multithreading is enabled. * NOTE 2: rsyncable performs poorly in combination with long range mode, * since it will decrease the effectiveness of synchronization points, * though mileage may vary. * NOTE 3: Rsyncable mode limits maximum compression speed to ~400 MB/s. * If the selected compression level is already running significantly slower, * the overall speed won't be significantly impacted. */ #define ZSTD_c_rsyncable ZSTD_c_experimentalParam1 /* Select a compression format. * The value must be of type ZSTD_format_e. * See ZSTD_format_e enum definition for details */ #define ZSTD_c_format ZSTD_c_experimentalParam2 /* Force back-reference distances to remain < windowSize, * even when referencing into Dictionary content (default:0) */ #define ZSTD_c_forceMaxWindow ZSTD_c_experimentalParam3 /* Controls whether the contents of a CDict * are used in place, or copied into the working context. * Accepts values from the ZSTD_dictAttachPref_e enum. * See the comments on that enum for an explanation of the feature. */ #define ZSTD_c_forceAttachDict ZSTD_c_experimentalParam4 /* Controlled with ZSTD_paramSwitch_e enum. * Default is ZSTD_ps_auto. * Set to ZSTD_ps_disable to never compress literals. * Set to ZSTD_ps_enable to always compress literals. (Note: uncompressed literals * may still be emitted if huffman is not beneficial to use.) * * By default, in ZSTD_ps_auto, the library will decide at runtime whether to use * literals compression based on the compression parameters - specifically, * negative compression levels do not use literal compression. */ #define ZSTD_c_literalCompressionMode ZSTD_c_experimentalParam5 /* Tries to fit compressed block size to be around targetCBlockSize. * No target when targetCBlockSize == 0. * There is no guarantee on compressed block size (default:0) */ #define ZSTD_c_targetCBlockSize ZSTD_c_experimentalParam6 /* User's best guess of source size. * Hint is not valid when srcSizeHint == 0. * There is no guarantee that hint is close to actual source size, * but compression ratio may regress significantly if guess considerably underestimates */ #define ZSTD_c_srcSizeHint ZSTD_c_experimentalParam7 /* Controls whether the new and experimental "dedicated dictionary search * structure" can be used. This feature is still rough around the edges, be * prepared for surprising behavior! * * How to use it: * * When using a CDict, whether to use this feature or not is controlled at * CDict creation, and it must be set in a CCtxParams set passed into that * construction (via ZSTD_createCDict_advanced2()). A compression will then * use the feature or not based on how the CDict was constructed; the value of * this param, set in the CCtx, will have no effect. * * However, when a dictionary buffer is passed into a CCtx, such as via * ZSTD_CCtx_loadDictionary(), this param can be set on the CCtx to control * whether the CDict that is created internally can use the feature or not. * * What it does: * * Normally, the internal data structures of the CDict are analogous to what * would be stored in a CCtx after compressing the contents of a dictionary. * To an approximation, a compression using a dictionary can then use those * data structures to simply continue what is effectively a streaming * compression where the simulated compression of the dictionary left off. * Which is to say, the search structures in the CDict are normally the same * format as in the CCtx. * * It is possible to do better, since the CDict is not like a CCtx: the search * structures are written once during CDict creation, and then are only read * after that, while the search structures in the CCtx are both read and * written as the compression goes along. This means we can choose a search * structure for the dictionary that is read-optimized. * * This feature enables the use of that different structure. * * Note that some of the members of the ZSTD_compressionParameters struct have * different semantics and constraints in the dedicated search structure. It is * highly recommended that you simply set a compression level in the CCtxParams * you pass into the CDict creation call, and avoid messing with the cParams * directly. * * Effects: * * This will only have any effect when the selected ZSTD_strategy * implementation supports this feature. Currently, that's limited to * ZSTD_greedy, ZSTD_lazy, and ZSTD_lazy2. * * Note that this means that the CDict tables can no longer be copied into the * CCtx, so the dict attachment mode ZSTD_dictForceCopy will no longer be * usable. The dictionary can only be attached or reloaded. * * In general, you should expect compression to be faster--sometimes very much * so--and CDict creation to be slightly slower. Eventually, we will probably * make this mode the default. */ #define ZSTD_c_enableDedicatedDictSearch ZSTD_c_experimentalParam8 /* ZSTD_c_stableInBuffer * Experimental parameter. * Default is 0 == disabled. Set to 1 to enable. * * Tells the compressor that input data presented with ZSTD_inBuffer * will ALWAYS be the same between calls. * Technically, the @src pointer must never be changed, * and the @pos field can only be updated by zstd. * However, it's possible to increase the @size field, * allowing scenarios where more data can be appended after compressions starts. * These conditions are checked by the compressor, * and compression will fail if they are not respected. * Also, data in the ZSTD_inBuffer within the range [src, src + pos) * MUST not be modified during compression or it will result in data corruption. * * When this flag is enabled zstd won't allocate an input window buffer, * because the user guarantees it can reference the ZSTD_inBuffer until * the frame is complete. But, it will still allocate an output buffer * large enough to fit a block (see ZSTD_c_stableOutBuffer). This will also * avoid the memcpy() from the input buffer to the input window buffer. * * NOTE: So long as the ZSTD_inBuffer always points to valid memory, using * this flag is ALWAYS memory safe, and will never access out-of-bounds * memory. However, compression WILL fail if conditions are not respected. * * WARNING: The data in the ZSTD_inBuffer in the range [src, src + pos) MUST * not be modified during compression or it will result in data corruption. * This is because zstd needs to reference data in the ZSTD_inBuffer to find * matches. Normally zstd maintains its own window buffer for this purpose, * but passing this flag tells zstd to rely on user provided buffer instead. */ #define ZSTD_c_stableInBuffer ZSTD_c_experimentalParam9 /* ZSTD_c_stableOutBuffer * Experimental parameter. * Default is 0 == disabled. Set to 1 to enable. * * Tells he compressor that the ZSTD_outBuffer will not be resized between * calls. Specifically: (out.size - out.pos) will never grow. This gives the * compressor the freedom to say: If the compressed data doesn't fit in the * output buffer then return ZSTD_error_dstSizeTooSmall. This allows us to * always decompress directly into the output buffer, instead of decompressing * into an internal buffer and copying to the output buffer. * * When this flag is enabled zstd won't allocate an output buffer, because * it can write directly to the ZSTD_outBuffer. It will still allocate the * input window buffer (see ZSTD_c_stableInBuffer). * * Zstd will check that (out.size - out.pos) never grows and return an error * if it does. While not strictly necessary, this should prevent surprises. */ #define ZSTD_c_stableOutBuffer ZSTD_c_experimentalParam10 /* ZSTD_c_blockDelimiters * Default is 0 == ZSTD_sf_noBlockDelimiters. * * For use with sequence compression API: ZSTD_compressSequences(). * * Designates whether or not the given array of ZSTD_Sequence contains block delimiters * and last literals, which are defined as sequences with offset == 0 and matchLength == 0. * See the definition of ZSTD_Sequence for more specifics. */ #define ZSTD_c_blockDelimiters ZSTD_c_experimentalParam11 /* ZSTD_c_validateSequences * Default is 0 == disabled. Set to 1 to enable sequence validation. * * For use with sequence compression API: ZSTD_compressSequences(). * Designates whether or not we validate sequences provided to ZSTD_compressSequences() * during function execution. * * Without validation, providing a sequence that does not conform to the zstd spec will cause * undefined behavior, and may produce a corrupted block. * * With validation enabled, if sequence is invalid (see doc/zstd_compression_format.md for * specifics regarding offset/matchlength requirements) then the function will bail out and * return an error. * */ #define ZSTD_c_validateSequences ZSTD_c_experimentalParam12 /* ZSTD_c_useBlockSplitter * Controlled with ZSTD_paramSwitch_e enum. * Default is ZSTD_ps_auto. * Set to ZSTD_ps_disable to never use block splitter. * Set to ZSTD_ps_enable to always use block splitter. * * By default, in ZSTD_ps_auto, the library will decide at runtime whether to use * block splitting based on the compression parameters. */ #define ZSTD_c_useBlockSplitter ZSTD_c_experimentalParam13 /* ZSTD_c_useRowMatchFinder * Controlled with ZSTD_paramSwitch_e enum. * Default is ZSTD_ps_auto. * Set to ZSTD_ps_disable to never use row-based matchfinder. * Set to ZSTD_ps_enable to force usage of row-based matchfinder. * * By default, in ZSTD_ps_auto, the library will decide at runtime whether to use * the row-based matchfinder based on support for SIMD instructions and the window log. * Note that this only pertains to compression strategies: greedy, lazy, and lazy2 */ #define ZSTD_c_useRowMatchFinder ZSTD_c_experimentalParam14 /* ZSTD_c_deterministicRefPrefix * Default is 0 == disabled. Set to 1 to enable. * * Zstd produces different results for prefix compression when the prefix is * directly adjacent to the data about to be compressed vs. when it isn't. * This is because zstd detects that the two buffers are contiguous and it can * use a more efficient match finding algorithm. However, this produces different * results than when the two buffers are non-contiguous. This flag forces zstd * to always load the prefix in non-contiguous mode, even if it happens to be * adjacent to the data, to guarantee determinism. * * If you really care about determinism when using a dictionary or prefix, * like when doing delta compression, you should select this option. It comes * at a speed penalty of about ~2.5% if the dictionary and data happened to be * contiguous, and is free if they weren't contiguous. We don't expect that * intentionally making the dictionary and data contiguous will be worth the * cost to memcpy() the data. */ #define ZSTD_c_deterministicRefPrefix ZSTD_c_experimentalParam15 /* ZSTD_c_prefetchCDictTables * Controlled with ZSTD_paramSwitch_e enum. Default is ZSTD_ps_auto. * * In some situations, zstd uses CDict tables in-place rather than copying them * into the working context. (See docs on ZSTD_dictAttachPref_e above for details). * In such situations, compression speed is seriously impacted when CDict tables are * "cold" (outside CPU cache). This parameter instructs zstd to prefetch CDict tables * when they are used in-place. * * For sufficiently small inputs, the cost of the prefetch will outweigh the benefit. * For sufficiently large inputs, zstd will by default memcpy() CDict tables * into the working context, so there is no need to prefetch. This parameter is * targeted at a middle range of input sizes, where a prefetch is cheap enough to be * useful but memcpy() is too expensive. The exact range of input sizes where this * makes sense is best determined by careful experimentation. * * Note: for this parameter, ZSTD_ps_auto is currently equivalent to ZSTD_ps_disable, * but in the future zstd may conditionally enable this feature via an auto-detection * heuristic for cold CDicts. * Use ZSTD_ps_disable to opt out of prefetching under any circumstances. */ #define ZSTD_c_prefetchCDictTables ZSTD_c_experimentalParam16 /* ZSTD_c_enableSeqProducerFallback * Allowed values are 0 (disable) and 1 (enable). The default setting is 0. * * Controls whether zstd will fall back to an internal sequence producer if an * external sequence producer is registered and returns an error code. This fallback * is block-by-block: the internal sequence producer will only be called for blocks * where the external sequence producer returns an error code. Fallback parsing will * follow any other cParam settings, such as compression level, the same as in a * normal (fully-internal) compression operation. * * The user is strongly encouraged to read the full Block-Level Sequence Producer API * documentation (below) before setting this parameter. */ #define ZSTD_c_enableSeqProducerFallback ZSTD_c_experimentalParam17 /* ZSTD_c_maxBlockSize * Allowed values are between 1KB and ZSTD_BLOCKSIZE_MAX (128KB). * The default is ZSTD_BLOCKSIZE_MAX, and setting to 0 will set to the default. * * This parameter can be used to set an upper bound on the blocksize * that overrides the default ZSTD_BLOCKSIZE_MAX. It cannot be used to set upper * bounds greater than ZSTD_BLOCKSIZE_MAX or bounds lower than 1KB (will make * compressBound() inaccurate). Only currently meant to be used for testing. * */ #define ZSTD_c_maxBlockSize ZSTD_c_experimentalParam18 /* ZSTD_c_searchForExternalRepcodes * This parameter affects how zstd parses external sequences, such as sequences * provided through the compressSequences() API or from an external block-level * sequence producer. * * If set to ZSTD_ps_enable, the library will check for repeated offsets in * external sequences, even if those repcodes are not explicitly indicated in * the "rep" field. Note that this is the only way to exploit repcode matches * while using compressSequences() or an external sequence producer, since zstd * currently ignores the "rep" field of external sequences. * * If set to ZSTD_ps_disable, the library will not exploit repeated offsets in * external sequences, regardless of whether the "rep" field has been set. This * reduces sequence compression overhead by about 25% while sacrificing some * compression ratio. * * The default value is ZSTD_ps_auto, for which the library will enable/disable * based on compression level. * * Note: for now, this param only has an effect if ZSTD_c_blockDelimiters is * set to ZSTD_sf_explicitBlockDelimiters. That may change in the future. */ #define ZSTD_c_searchForExternalRepcodes ZSTD_c_experimentalParam19 /*! ZSTD_CCtx_getParameter() : * Get the requested compression parameter value, selected by enum ZSTD_cParameter, * and store it into int* value. * @return : 0, or an error code (which can be tested with ZSTD_isError()). */ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_getParameter(const ZSTD_CCtx* cctx, ZSTD_cParameter param, int* value); /*! ZSTD_CCtx_params : * Quick howto : * - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure * - ZSTD_CCtxParams_setParameter() : Push parameters one by one into * an existing ZSTD_CCtx_params structure. * This is similar to * ZSTD_CCtx_setParameter(). * - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to * an existing CCtx. * These parameters will be applied to * all subsequent frames. * - ZSTD_compressStream2() : Do compression using the CCtx. * - ZSTD_freeCCtxParams() : Free the memory, accept NULL pointer. * * This can be used with ZSTD_estimateCCtxSize_advanced_usingCCtxParams() * for static allocation of CCtx for single-threaded compression. */ ZSTDLIB_STATIC_API ZSTD_CCtx_params* ZSTD_createCCtxParams(void); ZSTDLIB_STATIC_API size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params); /* accept NULL pointer */ /*! ZSTD_CCtxParams_reset() : * Reset params to default values. */ ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_reset(ZSTD_CCtx_params* params); /*! ZSTD_CCtxParams_init() : * Initializes the compression parameters of cctxParams according to * compression level. All other parameters are reset to their default values. */ ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_init(ZSTD_CCtx_params* cctxParams, int compressionLevel); /*! ZSTD_CCtxParams_init_advanced() : * Initializes the compression and frame parameters of cctxParams according to * params. All other parameters are reset to their default values. */ ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params); /*! ZSTD_CCtxParams_setParameter() : Requires v1.4.0+ * Similar to ZSTD_CCtx_setParameter. * Set one compression parameter, selected by enum ZSTD_cParameter. * Parameters must be applied to a ZSTD_CCtx using * ZSTD_CCtx_setParametersUsingCCtxParams(). * @result : a code representing success or failure (which can be tested with * ZSTD_isError()). */ ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* params, ZSTD_cParameter param, int value); /*! ZSTD_CCtxParams_getParameter() : * Similar to ZSTD_CCtx_getParameter. * Get the requested value of one compression parameter, selected by enum ZSTD_cParameter. * @result : 0, or an error code (which can be tested with ZSTD_isError()). */ ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_getParameter(const ZSTD_CCtx_params* params, ZSTD_cParameter param, int* value); /*! ZSTD_CCtx_setParametersUsingCCtxParams() : * Apply a set of ZSTD_CCtx_params to the compression context. * This can be done even after compression is started, * if nbWorkers==0, this will have no impact until a new compression is started. * if nbWorkers>=1, new parameters will be picked up at next job, * with a few restrictions (windowLog, pledgedSrcSize, nbWorkers, jobSize, and overlapLog are not updated). */ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_setParametersUsingCCtxParams( ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params); /*! ZSTD_compressStream2_simpleArgs() : * Same as ZSTD_compressStream2(), * but using only integral types as arguments. * This variant might be helpful for binders from dynamic languages * which have troubles handling structures containing memory pointers. */ ZSTDLIB_STATIC_API size_t ZSTD_compressStream2_simpleArgs ( ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, size_t* dstPos, const void* src, size_t srcSize, size_t* srcPos, ZSTD_EndDirective endOp); /*************************************** * Advanced decompression functions ***************************************/ /*! ZSTD_isFrame() : * Tells if the content of `buffer` starts with a valid Frame Identifier. * Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0. * Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled. * Note 3 : Skippable Frame Identifiers are considered valid. */ ZSTDLIB_STATIC_API unsigned ZSTD_isFrame(const void* buffer, size_t size); /*! ZSTD_createDDict_byReference() : * Create a digested dictionary, ready to start decompression operation without startup delay. * Dictionary content is referenced, and therefore stays in dictBuffer. * It is important that dictBuffer outlives DDict, * it must remain read accessible throughout the lifetime of DDict */ ZSTDLIB_STATIC_API ZSTD_DDict* ZSTD_createDDict_byReference(const void* dictBuffer, size_t dictSize); /*! ZSTD_DCtx_loadDictionary_byReference() : * Same as ZSTD_DCtx_loadDictionary(), * but references `dict` content instead of copying it into `dctx`. * This saves memory if `dict` remains around., * However, it's imperative that `dict` remains accessible (and unmodified) while being used, so it must outlive decompression. */ ZSTDLIB_STATIC_API size_t ZSTD_DCtx_loadDictionary_byReference(ZSTD_DCtx* dctx, const void* dict, size_t dictSize); /*! ZSTD_DCtx_loadDictionary_advanced() : * Same as ZSTD_DCtx_loadDictionary(), * but gives direct control over * how to load the dictionary (by copy ? by reference ?) * and how to interpret it (automatic ? force raw mode ? full mode only ?). */ ZSTDLIB_STATIC_API size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType); /*! ZSTD_DCtx_refPrefix_advanced() : * Same as ZSTD_DCtx_refPrefix(), but gives finer control over * how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) */ ZSTDLIB_STATIC_API size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType); /*! ZSTD_DCtx_setMaxWindowSize() : * Refuses allocating internal buffers for frames requiring a window size larger than provided limit. * This protects a decoder context from reserving too much memory for itself (potential attack scenario). * This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode. * By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT) * @return : 0, or an error code (which can be tested using ZSTD_isError()). */ ZSTDLIB_STATIC_API size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize); /*! ZSTD_DCtx_getParameter() : * Get the requested decompression parameter value, selected by enum ZSTD_dParameter, * and store it into int* value. * @return : 0, or an error code (which can be tested with ZSTD_isError()). */ ZSTDLIB_STATIC_API size_t ZSTD_DCtx_getParameter(ZSTD_DCtx* dctx, ZSTD_dParameter param, int* value); /* ZSTD_d_format * experimental parameter, * allowing selection between ZSTD_format_e input compression formats */ #define ZSTD_d_format ZSTD_d_experimentalParam1 /* ZSTD_d_stableOutBuffer * Experimental parameter. * Default is 0 == disabled. Set to 1 to enable. * * Tells the decompressor that the ZSTD_outBuffer will ALWAYS be the same * between calls, except for the modifications that zstd makes to pos (the * caller must not modify pos). This is checked by the decompressor, and * decompression will fail if it ever changes. Therefore the ZSTD_outBuffer * MUST be large enough to fit the entire decompressed frame. This will be * checked when the frame content size is known. The data in the ZSTD_outBuffer * in the range [dst, dst + pos) MUST not be modified during decompression * or you will get data corruption. * * When this flag is enabled zstd won't allocate an output buffer, because * it can write directly to the ZSTD_outBuffer, but it will still allocate * an input buffer large enough to fit any compressed block. This will also * avoid the memcpy() from the internal output buffer to the ZSTD_outBuffer. * If you need to avoid the input buffer allocation use the buffer-less * streaming API. * * NOTE: So long as the ZSTD_outBuffer always points to valid memory, using * this flag is ALWAYS memory safe, and will never access out-of-bounds * memory. However, decompression WILL fail if you violate the preconditions. * * WARNING: The data in the ZSTD_outBuffer in the range [dst, dst + pos) MUST * not be modified during decompression or you will get data corruption. This * is because zstd needs to reference data in the ZSTD_outBuffer to regenerate * matches. Normally zstd maintains its own buffer for this purpose, but passing * this flag tells zstd to use the user provided buffer. */ #define ZSTD_d_stableOutBuffer ZSTD_d_experimentalParam2 /* ZSTD_d_forceIgnoreChecksum * Experimental parameter. * Default is 0 == disabled. Set to 1 to enable * * Tells the decompressor to skip checksum validation during decompression, regardless * of whether checksumming was specified during compression. This offers some * slight performance benefits, and may be useful for debugging. * Param has values of type ZSTD_forceIgnoreChecksum_e */ #define ZSTD_d_forceIgnoreChecksum ZSTD_d_experimentalParam3 /* ZSTD_d_refMultipleDDicts * Experimental parameter. * Default is 0 == disabled. Set to 1 to enable * * If enabled and dctx is allocated on the heap, then additional memory will be allocated * to store references to multiple ZSTD_DDict. That is, multiple calls of ZSTD_refDDict() * using a given ZSTD_DCtx, rather than overwriting the previous DDict reference, will instead * store all references. At decompression time, the appropriate dictID is selected * from the set of DDicts based on the dictID in the frame. * * Usage is simply calling ZSTD_refDDict() on multiple dict buffers. * * Param has values of byte ZSTD_refMultipleDDicts_e * * WARNING: Enabling this parameter and calling ZSTD_DCtx_refDDict(), will trigger memory * allocation for the hash table. ZSTD_freeDCtx() also frees this memory. * Memory is allocated as per ZSTD_DCtx::customMem. * * Although this function allocates memory for the table, the user is still responsible for * memory management of the underlying ZSTD_DDict* themselves. */ #define ZSTD_d_refMultipleDDicts ZSTD_d_experimentalParam4 /* ZSTD_d_disableHuffmanAssembly * Set to 1 to disable the Huffman assembly implementation. * The default value is 0, which allows zstd to use the Huffman assembly * implementation if available. * * This parameter can be used to disable Huffman assembly at runtime. * If you want to disable it at compile time you can define the macro * ZSTD_DISABLE_ASM. */ #define ZSTD_d_disableHuffmanAssembly ZSTD_d_experimentalParam5 /*! ZSTD_DCtx_setFormat() : * This function is REDUNDANT. Prefer ZSTD_DCtx_setParameter(). * Instruct the decoder context about what kind of data to decode next. * This instruction is mandatory to decode data without a fully-formed header, * such ZSTD_f_zstd1_magicless for example. * @return : 0, or an error code (which can be tested using ZSTD_isError()). */ ZSTD_DEPRECATED("use ZSTD_DCtx_setParameter() instead") ZSTDLIB_STATIC_API size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format); /*! ZSTD_decompressStream_simpleArgs() : * Same as ZSTD_decompressStream(), * but using only integral types as arguments. * This can be helpful for binders from dynamic languages * which have troubles handling structures containing memory pointers. */ ZSTDLIB_STATIC_API size_t ZSTD_decompressStream_simpleArgs ( ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, size_t* dstPos, const void* src, size_t srcSize, size_t* srcPos); /******************************************************************** * Advanced streaming functions * Warning : most of these functions are now redundant with the Advanced API. * Once Advanced API reaches "stable" status, * redundant functions will be deprecated, and then at some point removed. ********************************************************************/ /*===== Advanced Streaming compression functions =====*/ /*! ZSTD_initCStream_srcSize() : * This function is DEPRECATED, and equivalent to: * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); * ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any) * ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel); * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); * * pledgedSrcSize must be correct. If it is not known at init time, use * ZSTD_CONTENTSIZE_UNKNOWN. Note that, for compatibility with older programs, * "0" also disables frame content size field. It may be enabled in the future. * This prototype will generate compilation warnings. */ ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions") ZSTDLIB_STATIC_API size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs, int compressionLevel, unsigned long long pledgedSrcSize); /*! ZSTD_initCStream_usingDict() : * This function is DEPRECATED, and is equivalent to: * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); * ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel); * ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); * * Creates of an internal CDict (incompatible with static CCtx), except if * dict == NULL or dictSize < 8, in which case no dict is used. * Note: dict is loaded with ZSTD_dct_auto (treated as a full zstd dictionary if * it begins with ZSTD_MAGIC_DICTIONARY, else as raw content) and ZSTD_dlm_byCopy. * This prototype will generate compilation warnings. */ ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions") ZSTDLIB_STATIC_API size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel); /*! ZSTD_initCStream_advanced() : * This function is DEPRECATED, and is equivalent to: * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); * ZSTD_CCtx_setParams(zcs, params); * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); * ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); * * dict is loaded with ZSTD_dct_auto and ZSTD_dlm_byCopy. * pledgedSrcSize must be correct. * If srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN. * This prototype will generate compilation warnings. */ ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions") ZSTDLIB_STATIC_API size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); /*! ZSTD_initCStream_usingCDict() : * This function is DEPRECATED, and equivalent to: * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); * ZSTD_CCtx_refCDict(zcs, cdict); * * note : cdict will just be referenced, and must outlive compression session * This prototype will generate compilation warnings. */ ZSTD_DEPRECATED("use ZSTD_CCtx_reset and ZSTD_CCtx_refCDict, see zstd.h for detailed instructions") ZSTDLIB_STATIC_API size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict); /*! ZSTD_initCStream_usingCDict_advanced() : * This function is DEPRECATED, and is equivalent to: * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); * ZSTD_CCtx_setFParams(zcs, fParams); * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); * ZSTD_CCtx_refCDict(zcs, cdict); * * same as ZSTD_initCStream_usingCDict(), with control over frame parameters. * pledgedSrcSize must be correct. If srcSize is not known at init time, use * value ZSTD_CONTENTSIZE_UNKNOWN. * This prototype will generate compilation warnings. */ ZSTD_DEPRECATED("use ZSTD_CCtx_reset and ZSTD_CCtx_refCDict, see zstd.h for detailed instructions") ZSTDLIB_STATIC_API size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs, const ZSTD_CDict* cdict, ZSTD_frameParameters fParams, unsigned long long pledgedSrcSize); /*! ZSTD_resetCStream() : * This function is DEPRECATED, and is equivalent to: * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); * Note: ZSTD_resetCStream() interprets pledgedSrcSize == 0 as ZSTD_CONTENTSIZE_UNKNOWN, but * ZSTD_CCtx_setPledgedSrcSize() does not do the same, so ZSTD_CONTENTSIZE_UNKNOWN must be * explicitly specified. * * start a new frame, using same parameters from previous frame. * This is typically useful to skip dictionary loading stage, since it will re-use it in-place. * Note that zcs must be init at least once before using ZSTD_resetCStream(). * If pledgedSrcSize is not known at reset time, use macro ZSTD_CONTENTSIZE_UNKNOWN. * If pledgedSrcSize > 0, its value must be correct, as it will be written in header, and controlled at the end. * For the time being, pledgedSrcSize==0 is interpreted as "srcSize unknown" for compatibility with older programs, * but it will change to mean "empty" in future version, so use macro ZSTD_CONTENTSIZE_UNKNOWN instead. * @return : 0, or an error code (which can be tested using ZSTD_isError()) * This prototype will generate compilation warnings. */ ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions") ZSTDLIB_STATIC_API size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize); typedef struct { unsigned long long ingested; /* nb input bytes read and buffered */ unsigned long long consumed; /* nb input bytes actually compressed */ unsigned long long produced; /* nb of compressed bytes generated and buffered */ unsigned long long flushed; /* nb of compressed bytes flushed : not provided; can be tracked from caller side */ unsigned currentJobID; /* MT only : latest started job nb */ unsigned nbActiveWorkers; /* MT only : nb of workers actively compressing at probe time */ } ZSTD_frameProgression; /* ZSTD_getFrameProgression() : * tells how much data has been ingested (read from input) * consumed (input actually compressed) and produced (output) for current frame. * Note : (ingested - consumed) is amount of input data buffered internally, not yet compressed. * Aggregates progression inside active worker threads. */ ZSTDLIB_STATIC_API ZSTD_frameProgression ZSTD_getFrameProgression(const ZSTD_CCtx* cctx); /*! ZSTD_toFlushNow() : * Tell how many bytes are ready to be flushed immediately. * Useful for multithreading scenarios (nbWorkers >= 1). * Probe the oldest active job, defined as oldest job not yet entirely flushed, * and check its output buffer. * @return : amount of data stored in oldest job and ready to be flushed immediately. * if @return == 0, it means either : * + there is no active job (could be checked with ZSTD_frameProgression()), or * + oldest job is still actively compressing data, * but everything it has produced has also been flushed so far, * therefore flush speed is limited by production speed of oldest job * irrespective of the speed of concurrent (and newer) jobs. */ ZSTDLIB_STATIC_API size_t ZSTD_toFlushNow(ZSTD_CCtx* cctx); /*===== Advanced Streaming decompression functions =====*/ /*! * This function is deprecated, and is equivalent to: * * ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); * ZSTD_DCtx_loadDictionary(zds, dict, dictSize); * * note: no dictionary will be used if dict == NULL or dictSize < 8 */ ZSTD_DEPRECATED("use ZSTD_DCtx_reset + ZSTD_DCtx_loadDictionary, see zstd.h for detailed instructions") ZSTDLIB_STATIC_API size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize); /*! * This function is deprecated, and is equivalent to: * * ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); * ZSTD_DCtx_refDDict(zds, ddict); * * note : ddict is referenced, it must outlive decompression session */ ZSTD_DEPRECATED("use ZSTD_DCtx_reset + ZSTD_DCtx_refDDict, see zstd.h for detailed instructions") ZSTDLIB_STATIC_API size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict); /*! * This function is deprecated, and is equivalent to: * * ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); * * re-use decompression parameters from previous init; saves dictionary loading */ ZSTD_DEPRECATED("use ZSTD_DCtx_reset, see zstd.h for detailed instructions") ZSTDLIB_STATIC_API size_t ZSTD_resetDStream(ZSTD_DStream* zds); /* ********************* BLOCK-LEVEL SEQUENCE PRODUCER API ********************* * * *** OVERVIEW *** * The Block-Level Sequence Producer API allows users to provide their own custom * sequence producer which libzstd invokes to process each block. The produced list * of sequences (literals and matches) is then post-processed by libzstd to produce * valid compressed blocks. * * This block-level offload API is a more granular complement of the existing * frame-level offload API compressSequences() (introduced in v1.5.1). It offers * an easier migration story for applications already integrated with libzstd: the * user application continues to invoke the same compression functions * ZSTD_compress2() or ZSTD_compressStream2() as usual, and transparently benefits * from the specific advantages of the external sequence producer. For example, * the sequence producer could be tuned to take advantage of known characteristics * of the input, to offer better speed / ratio, or could leverage hardware * acceleration not available within libzstd itself. * * See contrib/externalSequenceProducer for an example program employing the * Block-Level Sequence Producer API. * * *** USAGE *** * The user is responsible for implementing a function of type * ZSTD_sequenceProducer_F. For each block, zstd will pass the following * arguments to the user-provided function: * * - sequenceProducerState: a pointer to a user-managed state for the sequence * producer. * * - outSeqs, outSeqsCapacity: an output buffer for the sequence producer. * outSeqsCapacity is guaranteed >= ZSTD_sequenceBound(srcSize). The memory * backing outSeqs is managed by the CCtx. * * - src, srcSize: an input buffer for the sequence producer to parse. * srcSize is guaranteed to be <= ZSTD_BLOCKSIZE_MAX. * * - dict, dictSize: a history buffer, which may be empty, which the sequence * producer may reference as it parses the src buffer. Currently, zstd will * always pass dictSize == 0 into external sequence producers, but this will * change in the future. * * - compressionLevel: a signed integer representing the zstd compression level * set by the user for the current operation. The sequence producer may choose * to use this information to change its compression strategy and speed/ratio * tradeoff. Note: the compression level does not reflect zstd parameters set * through the advanced API. * * - windowSize: a size_t representing the maximum allowed offset for external * sequences. Note that sequence offsets are sometimes allowed to exceed the * windowSize if a dictionary is present, see doc/zstd_compression_format.md * for details. * * The user-provided function shall return a size_t representing the number of * sequences written to outSeqs. This return value will be treated as an error * code if it is greater than outSeqsCapacity. The return value must be non-zero * if srcSize is non-zero. The ZSTD_SEQUENCE_PRODUCER_ERROR macro is provided * for convenience, but any value greater than outSeqsCapacity will be treated as * an error code. * * If the user-provided function does not return an error code, the sequences * written to outSeqs must be a valid parse of the src buffer. Data corruption may * occur if the parse is not valid. A parse is defined to be valid if the * following conditions hold: * - The sum of matchLengths and literalLengths must equal srcSize. * - All sequences in the parse, except for the final sequence, must have * matchLength >= ZSTD_MINMATCH_MIN. The final sequence must have * matchLength >= ZSTD_MINMATCH_MIN or matchLength == 0. * - All offsets must respect the windowSize parameter as specified in * doc/zstd_compression_format.md. * - If the final sequence has matchLength == 0, it must also have offset == 0. * * zstd will only validate these conditions (and fail compression if they do not * hold) if the ZSTD_c_validateSequences cParam is enabled. Note that sequence * validation has a performance cost. * * If the user-provided function returns an error, zstd will either fall back * to an internal sequence producer or fail the compression operation. The user can * choose between the two behaviors by setting the ZSTD_c_enableSeqProducerFallback * cParam. Fallback compression will follow any other cParam settings, such as * compression level, the same as in a normal compression operation. * * The user shall instruct zstd to use a particular ZSTD_sequenceProducer_F * function by calling * ZSTD_registerSequenceProducer(cctx, * sequenceProducerState, * sequenceProducer) * This setting will persist until the next parameter reset of the CCtx. * * The sequenceProducerState must be initialized by the user before calling * ZSTD_registerSequenceProducer(). The user is responsible for destroying the * sequenceProducerState. * * *** LIMITATIONS *** * This API is compatible with all zstd compression APIs which respect advanced parameters. * However, there are three limitations: * * First, the ZSTD_c_enableLongDistanceMatching cParam is not currently supported. * COMPRESSION WILL FAIL if it is enabled and the user tries to compress with a block-level * external sequence producer. * - Note that ZSTD_c_enableLongDistanceMatching is auto-enabled by default in some * cases (see its documentation for details). Users must explicitly set * ZSTD_c_enableLongDistanceMatching to ZSTD_ps_disable in such cases if an external * sequence producer is registered. * - As of this writing, ZSTD_c_enableLongDistanceMatching is disabled by default * whenever ZSTD_c_windowLog < 128MB, but that's subject to change. Users should * check the docs on ZSTD_c_enableLongDistanceMatching whenever the Block-Level Sequence * Producer API is used in conjunction with advanced settings (like ZSTD_c_windowLog). * * Second, history buffers are not currently supported. Concretely, zstd will always pass * dictSize == 0 to the external sequence producer (for now). This has two implications: * - Dictionaries are not currently supported. Compression will *not* fail if the user * references a dictionary, but the dictionary won't have any effect. * - Stream history is not currently supported. All advanced compression APIs, including * streaming APIs, work with external sequence producers, but each block is treated as * an independent chunk without history from previous blocks. * * Third, multi-threading within a single compression is not currently supported. In other words, * COMPRESSION WILL FAIL if ZSTD_c_nbWorkers > 0 and an external sequence producer is registered. * Multi-threading across compressions is fine: simply create one CCtx per thread. * * Long-term, we plan to overcome all three limitations. There is no technical blocker to * overcoming them. It is purely a question of engineering effort. */ #define ZSTD_SEQUENCE_PRODUCER_ERROR ((size_t)(-1)) typedef size_t ZSTD_sequenceProducer_F ( void* sequenceProducerState, ZSTD_Sequence* outSeqs, size_t outSeqsCapacity, const void* src, size_t srcSize, const void* dict, size_t dictSize, int compressionLevel, size_t windowSize ); /*! ZSTD_registerSequenceProducer() : * Instruct zstd to use a block-level external sequence producer function. * * The sequenceProducerState must be initialized by the caller, and the caller is * responsible for managing its lifetime. This parameter is sticky across * compressions. It will remain set until the user explicitly resets compression * parameters. * * Sequence producer registration is considered to be an "advanced parameter", * part of the "advanced API". This means it will only have an effect on compression * APIs which respect advanced parameters, such as compress2() and compressStream2(). * Older compression APIs such as compressCCtx(), which predate the introduction of * "advanced parameters", will ignore any external sequence producer setting. * * The sequence producer can be "cleared" by registering a NULL function pointer. This * removes all limitations described above in the "LIMITATIONS" section of the API docs. * * The user is strongly encouraged to read the full API documentation (above) before * calling this function. */ ZSTDLIB_STATIC_API void ZSTD_registerSequenceProducer( ZSTD_CCtx* cctx, void* sequenceProducerState, ZSTD_sequenceProducer_F* sequenceProducer ); /********************************************************************* * Buffer-less and synchronous inner streaming functions (DEPRECATED) * * This API is deprecated, and will be removed in a future version. * It allows streaming (de)compression with user allocated buffers. * However, it is hard to use, and not as well tested as the rest of * our API. * * Please use the normal streaming API instead: ZSTD_compressStream2, * and ZSTD_decompressStream. * If there is functionality that you need, but it doesn't provide, * please open an issue on our GitHub. ********************************************************************* */ /** Buffer-less streaming compression (synchronous mode) A ZSTD_CCtx object is required to track streaming operations. Use ZSTD_createCCtx() / ZSTD_freeCCtx() to manage resource. ZSTD_CCtx object can be re-used multiple times within successive compression operations. Start by initializing a context. Use ZSTD_compressBegin(), or ZSTD_compressBegin_usingDict() for dictionary compression. Then, consume your input using ZSTD_compressContinue(). There are some important considerations to keep in mind when using this advanced function : - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only. - Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks. - Caller must ensure there is enough space in `dst` to store compressed data under worst case scenario. Worst case evaluation is provided by ZSTD_compressBound(). ZSTD_compressContinue() doesn't guarantee recover after a failed compression. - ZSTD_compressContinue() presumes prior input ***is still accessible and unmodified*** (up to maximum distance size, see WindowLog). It remembers all previous contiguous blocks, plus one separated memory segment (which can itself consists of multiple contiguous blocks) - ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps. In which case, it will "discard" the relevant memory section from its history. Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum. It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame. Without last block mark, frames are considered unfinished (hence corrupted) by compliant decoders. `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again. */ /*===== Buffer-less streaming compression functions =====*/ ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.") ZSTDLIB_STATIC_API size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel); ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.") ZSTDLIB_STATIC_API size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel); ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.") ZSTDLIB_STATIC_API size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); /**< note: fails if cdict==NULL */ ZSTD_DEPRECATED("This function will likely be removed in a future release. It is misleading and has very limited utility.") ZSTDLIB_STATIC_API size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); /**< note: if pledgedSrcSize is not known, use ZSTD_CONTENTSIZE_UNKNOWN */ ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.") ZSTDLIB_STATIC_API size_t ZSTD_compressContinue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.") ZSTDLIB_STATIC_API size_t ZSTD_compressEnd(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); /* The ZSTD_compressBegin_advanced() and ZSTD_compressBegin_usingCDict_advanced() are now DEPRECATED and will generate a compiler warning */ ZSTD_DEPRECATED("use advanced API to access custom parameters") ZSTDLIB_STATIC_API size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); /**< pledgedSrcSize : If srcSize is not known at init time, use ZSTD_CONTENTSIZE_UNKNOWN */ ZSTD_DEPRECATED("use advanced API to access custom parameters") ZSTDLIB_STATIC_API size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize); /* compression parameters are already set within cdict. pledgedSrcSize must be correct. If srcSize is not known, use macro ZSTD_CONTENTSIZE_UNKNOWN */ /** Buffer-less streaming decompression (synchronous mode) A ZSTD_DCtx object is required to track streaming operations. Use ZSTD_createDCtx() / ZSTD_freeDCtx() to manage it. A ZSTD_DCtx object can be re-used multiple times. First typical operation is to retrieve frame parameters, using ZSTD_getFrameHeader(). Frame header is extracted from the beginning of compressed frame, so providing only the frame's beginning is enough. Data fragment must be large enough to ensure successful decoding. `ZSTD_frameHeaderSize_max` bytes is guaranteed to always be large enough. result : 0 : successful decoding, the `ZSTD_frameHeader` structure is correctly filled. >0 : `srcSize` is too small, please provide at least result bytes on next attempt. errorCode, which can be tested using ZSTD_isError(). It fills a ZSTD_frameHeader structure with important information to correctly decode the frame, such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`). Note that these values could be wrong, either because of data corruption, or because a 3rd party deliberately spoofs false information. As a consequence, check that values remain within valid application range. For example, do not allocate memory blindly, check that `windowSize` is within expectation. Each application can set its own limits, depending on local restrictions. For extended interoperability, it is recommended to support `windowSize` of at least 8 MB. ZSTD_decompressContinue() needs previous data blocks during decompression, up to `windowSize` bytes. ZSTD_decompressContinue() is very sensitive to contiguity, if 2 blocks don't follow each other, make sure that either the compressor breaks contiguity at the same place, or that previous contiguous segment is large enough to properly handle maximum back-reference distance. There are multiple ways to guarantee this condition. The most memory efficient way is to use a round buffer of sufficient size. Sufficient size is determined by invoking ZSTD_decodingBufferSize_min(), which can return an error code if required value is too large for current system (in 32-bits mode). In a round buffer methodology, ZSTD_decompressContinue() decompresses each block next to previous one, up to the moment there is not enough room left in the buffer to guarantee decoding another full block, which maximum size is provided in `ZSTD_frameHeader` structure, field `blockSizeMax`. At which point, decoding can resume from the beginning of the buffer. Note that already decoded data stored in the buffer should be flushed before being overwritten. There are alternatives possible, for example using two or more buffers of size `windowSize` each, though they consume more memory. Finally, if you control the compression process, you can also ignore all buffer size rules, as long as the encoder and decoder progress in "lock-step", aka use exactly the same buffer sizes, break contiguity at the same place, etc. Once buffers are setup, start decompression, with ZSTD_decompressBegin(). If decompression requires a dictionary, use ZSTD_decompressBegin_usingDict() or ZSTD_decompressBegin_usingDDict(). Then use ZSTD_nextSrcSizeToDecompress() and ZSTD_decompressContinue() alternatively. ZSTD_nextSrcSizeToDecompress() tells how many bytes to provide as 'srcSize' to ZSTD_decompressContinue(). ZSTD_decompressContinue() requires this _exact_ amount of bytes, or it will fail. result of ZSTD_decompressContinue() is the number of bytes regenerated within 'dst' (necessarily <= dstCapacity). It can be zero : it just means ZSTD_decompressContinue() has decoded some metadata item. It can also be an error code, which can be tested with ZSTD_isError(). A frame is fully decoded when ZSTD_nextSrcSizeToDecompress() returns zero. Context can then be reset to start a new decompression. Note : it's possible to know if next input to present is a header or a block, using ZSTD_nextInputType(). This information is not required to properly decode a frame. == Special case : skippable frames == Skippable frames allow integration of user-defined data into a flow of concatenated frames. Skippable frames will be ignored (skipped) by decompressor. The format of skippable frames is as follows : a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits c) Frame Content - any content (User Data) of length equal to Frame Size For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame. For skippable frames ZSTD_decompressContinue() always returns 0 : it only skips the content. */ /*===== Buffer-less streaming decompression functions =====*/ ZSTDLIB_STATIC_API size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long long frameContentSize); /**< when frame content size is not known, pass in frameContentSize == ZSTD_CONTENTSIZE_UNKNOWN */ ZSTDLIB_STATIC_API size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx); ZSTDLIB_STATIC_API size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx* dctx, const void* dict, size_t dictSize); ZSTDLIB_STATIC_API size_t ZSTD_decompressBegin_usingDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict); ZSTDLIB_STATIC_API size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx* dctx); ZSTDLIB_STATIC_API size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); /* misc */ ZSTD_DEPRECATED("This function will likely be removed in the next minor release. It is misleading and has very limited utility.") ZSTDLIB_STATIC_API void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx); typedef enum { ZSTDnit_frameHeader, ZSTDnit_blockHeader, ZSTDnit_block, ZSTDnit_lastBlock, ZSTDnit_checksum, ZSTDnit_skippableFrame } ZSTD_nextInputType_e; ZSTDLIB_STATIC_API ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx); /* ========================================= */ /** Block level API (DEPRECATED) */ /* ========================================= */ /*! This API is deprecated in favor of the regular compression API. You can get the frame header down to 2 bytes by setting: - ZSTD_c_format = ZSTD_f_zstd1_magicless - ZSTD_c_contentSizeFlag = 0 - ZSTD_c_checksumFlag = 0 - ZSTD_c_dictIDFlag = 0 This API is not as well tested as our normal API, so we recommend not using it. We will be removing it in a future version. If the normal API doesn't provide the functionality you need, please open a GitHub issue. Block functions produce and decode raw zstd blocks, without frame metadata. Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 100 bytes). But users will have to take in charge needed metadata to regenerate data, such as compressed and content sizes. A few rules to respect : - Compressing and decompressing require a context structure + Use ZSTD_createCCtx() and ZSTD_createDCtx() - It is necessary to init context before starting + compression : any ZSTD_compressBegin*() variant, including with dictionary + decompression : any ZSTD_decompressBegin*() variant, including with dictionary - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB + If input is larger than a block size, it's necessary to split input data into multiple blocks + For inputs larger than a single block, consider using regular ZSTD_compress() instead. Frame metadata is not that costly, and quickly becomes negligible as source size grows larger than a block. - When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero) ! ===> In which case, nothing is produced into `dst` ! + User __must__ test for such outcome and deal directly with uncompressed data + A block cannot be declared incompressible if ZSTD_compressBlock() return value was != 0. Doing so would mess up with statistics history, leading to potential data corruption. + ZSTD_decompressBlock() _doesn't accept uncompressed data as input_ !! + In case of multiple successive blocks, should some of them be uncompressed, decoder must be informed of their existence in order to follow proper history. Use ZSTD_insertBlock() for such a case. */ /*===== Raw zstd block functions =====*/ ZSTD_DEPRECATED("The block API is deprecated in favor of the normal compression API. See docs.") ZSTDLIB_STATIC_API size_t ZSTD_getBlockSize (const ZSTD_CCtx* cctx); ZSTD_DEPRECATED("The block API is deprecated in favor of the normal compression API. See docs.") ZSTDLIB_STATIC_API size_t ZSTD_compressBlock (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); ZSTD_DEPRECATED("The block API is deprecated in favor of the normal compression API. See docs.") ZSTDLIB_STATIC_API size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); ZSTD_DEPRECATED("The block API is deprecated in favor of the normal compression API. See docs.") ZSTDLIB_STATIC_API size_t ZSTD_insertBlock (ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize); /**< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression. */ #endif /* ZSTD_H_ZSTD_STATIC_LINKING_ONLY */ #if defined (__cplusplus) } #endif zmat-0.9.9/src/blosc2/internal-complibs/zstd/Makefile0000644000175200007730000002545114515254731022767 0ustar rlaboissrlaboiss# ################################################################ # Copyright (c) Meta Platforms, Inc. and affiliates. # All rights reserved. # # This source code is licensed under both the BSD-style license (found in the # LICENSE file in the root directory of this source tree) and the GPLv2 (found # in the COPYING file in the root directory of this source tree). # You may select, at your option, one of the above-listed licenses. # ################################################################ # Modules ZSTD_LIB_COMPRESSION ?= 1 ZSTD_LIB_DECOMPRESSION ?= 1 ZSTD_LIB_DICTBUILDER ?= 1 ZSTD_LIB_DEPRECATED ?= 0 ZSTD_LEGACY_SUPPORT ?= 0 # Input variables for libzstd.mk ifeq ($(ZSTD_LIB_COMPRESSION), 0) ZSTD_LIB_DICTBUILDER = 0 ZSTD_LIB_DEPRECATED = 0 endif ifeq ($(ZSTD_LIB_DECOMPRESSION), 0) ZSTD_LEGACY_SUPPORT = 0 ZSTD_LIB_DEPRECATED = 0 endif include libzstd.mk ZSTD_FILES := $(ZSTD_COMMON_FILES) $(ZSTD_LEGACY_FILES) ifneq ($(ZSTD_LIB_COMPRESSION), 0) ZSTD_FILES += $(ZSTD_COMPRESS_FILES) endif ifneq ($(ZSTD_LIB_DECOMPRESSION), 0) ZSTD_FILES += $(ZSTD_DECOMPRESS_FILES) endif ifneq ($(ZSTD_LIB_DEPRECATED), 0) ZSTD_FILES += $(ZSTD_DEPRECATED_FILES) endif ifneq ($(ZSTD_LIB_DICTBUILDER), 0) ZSTD_FILES += $(ZSTD_DICTBUILDER_FILES) endif ZSTD_LOCAL_SRC := $(notdir $(ZSTD_FILES)) ZSTD_LOCAL_OBJ0 := $(ZSTD_LOCAL_SRC:.c=.o) ZSTD_LOCAL_OBJ := $(ZSTD_LOCAL_OBJ0:.S=.o) VERSION := $(ZSTD_VERSION) # Note: by default, the static library is built single-threaded and dynamic library is built # multi-threaded. It is possible to force multi or single threaded builds by appending # -mt or -nomt to the build target (like lib-mt for multi-threaded, lib-nomt for single-threaded). .PHONY: default default: lib-release CPPFLAGS_DYNLIB += -DZSTD_MULTITHREAD # dynamic library build defaults to multi-threaded LDFLAGS_DYNLIB += -pthread CPPFLAGS_STATLIB += # static library build defaults to single-threaded ifeq ($(findstring GCC,$(CCVER)),GCC) decompress/zstd_decompress_block.o : CFLAGS+=-fno-tree-vectorize endif # macOS linker doesn't support -soname, and use different extension # see : https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/DynamicLibraryDesignGuidelines.html ifeq ($(UNAME), Darwin) SHARED_EXT = dylib SHARED_EXT_MAJOR = $(LIBVER_MAJOR).$(SHARED_EXT) SHARED_EXT_VER = $(LIBVER).$(SHARED_EXT) SONAME_FLAGS = -install_name $(LIBDIR)/libzstd.$(SHARED_EXT_MAJOR) -compatibility_version $(LIBVER_MAJOR) -current_version $(LIBVER) else ifeq ($(UNAME), AIX) SONAME_FLAGS = else SONAME_FLAGS = -Wl,-soname=libzstd.$(SHARED_EXT).$(LIBVER_MAJOR) endif SHARED_EXT = so SHARED_EXT_MAJOR = $(SHARED_EXT).$(LIBVER_MAJOR) SHARED_EXT_VER = $(SHARED_EXT).$(LIBVER) endif .PHONY: all all: lib .PHONY: libzstd.a # must be run every time libzstd.a: CPPFLAGS += $(CPPFLAGS_STATLIB) libzstd.a: CFLAGS += -fPIC SET_CACHE_DIRECTORY = \ +$(MAKE) --no-print-directory $@ \ BUILD_DIR=obj/$(HASH_DIR) \ CPPFLAGS="$(CPPFLAGS)" \ CFLAGS="$(CFLAGS)" \ LDFLAGS="$(LDFLAGS)" ifndef BUILD_DIR # determine BUILD_DIR from compilation flags libzstd.a: $(SET_CACHE_DIRECTORY) else # BUILD_DIR is defined ZSTD_STATLIB_DIR := $(BUILD_DIR)/static ZSTD_STATLIB := $(ZSTD_STATLIB_DIR)/libzstd.a ZSTD_STATLIB_OBJ := $(addprefix $(ZSTD_STATLIB_DIR)/,$(ZSTD_LOCAL_OBJ)) $(ZSTD_STATLIB): ARFLAGS = rcs $(ZSTD_STATLIB): | $(ZSTD_STATLIB_DIR) $(ZSTD_STATLIB): $(ZSTD_STATLIB_OBJ) # Check for multithread flag at target execution time $(if $(filter -DZSTD_MULTITHREAD,$(CPPFLAGS)),\ @echo compiling multi-threaded static library $(LIBVER),\ @echo compiling single-threaded static library $(LIBVER)) $(AR) $(ARFLAGS) $@ $^ libzstd.a: $(ZSTD_STATLIB) cp -f $< $@ endif ifneq (,$(filter Windows%,$(TARGET_SYSTEM))) LIBZSTD = dll/libzstd.dll $(LIBZSTD): $(ZSTD_FILES) @echo compiling dynamic library $(LIBVER) $(CC) $(FLAGS) -DZSTD_DLL_EXPORT=1 -Wl,--out-implib,dll/libzstd.dll.a -shared $^ -o $@ else # not Windows LIBZSTD = libzstd.$(SHARED_EXT_VER) .PHONY: $(LIBZSTD) # must be run every time $(LIBZSTD): CPPFLAGS += $(CPPFLAGS_DYNLIB) $(LIBZSTD): CFLAGS += -fPIC -fvisibility=hidden $(LIBZSTD): LDFLAGS += -shared $(LDFLAGS_DYNLIB) ifndef BUILD_DIR # determine BUILD_DIR from compilation flags $(LIBZSTD): $(SET_CACHE_DIRECTORY) else # BUILD_DIR is defined ZSTD_DYNLIB_DIR := $(BUILD_DIR)/dynamic ZSTD_DYNLIB := $(ZSTD_DYNLIB_DIR)/$(LIBZSTD) ZSTD_DYNLIB_OBJ := $(addprefix $(ZSTD_DYNLIB_DIR)/,$(ZSTD_LOCAL_OBJ)) $(ZSTD_DYNLIB): | $(ZSTD_DYNLIB_DIR) $(ZSTD_DYNLIB): $(ZSTD_DYNLIB_OBJ) # Check for multithread flag at target execution time $(if $(filter -DZSTD_MULTITHREAD,$(CPPFLAGS)),\ @echo compiling multi-threaded dynamic library $(LIBVER),\ @echo compiling single-threaded dynamic library $(LIBVER)) $(CC) $(FLAGS) $^ $(LDFLAGS) $(SONAME_FLAGS) -o $@ @echo creating versioned links ln -sf $@ libzstd.$(SHARED_EXT_MAJOR) ln -sf $@ libzstd.$(SHARED_EXT) $(LIBZSTD): $(ZSTD_DYNLIB) cp -f $< $@ endif # ifndef BUILD_DIR endif # if windows .PHONY: libzstd libzstd : $(LIBZSTD) .PHONY: lib lib : libzstd.a libzstd # note : do not define lib-mt or lib-release as .PHONY # make does not consider implicit pattern rule for .PHONY target %-mt : CPPFLAGS_DYNLIB := -DZSTD_MULTITHREAD %-mt : CPPFLAGS_STATLIB := -DZSTD_MULTITHREAD %-mt : LDFLAGS_DYNLIB := -pthread %-mt : % @echo multi-threaded build completed %-nomt : CPPFLAGS_DYNLIB := %-nomt : LDFLAGS_DYNLIB := %-nomt : CPPFLAGS_STATLIB := %-nomt : % @echo single-threaded build completed %-release : DEBUGFLAGS := %-release : % @echo release build completed # Generate .h dependencies automatically DEPFLAGS = -MT $@ -MMD -MP -MF $(ZSTD_DYNLIB_DIR)/%.o : %.c $(ZSTD_DYNLIB_DIR)/%.d | $(ZSTD_DYNLIB_DIR) @echo CC $@ $(COMPILE.c) $(DEPFLAGS) $(ZSTD_DYNLIB_DIR)/$*.d $(OUTPUT_OPTION) $< $(ZSTD_STATLIB_DIR)/%.o : %.c $(ZSTD_STATLIB_DIR)/%.d | $(ZSTD_STATLIB_DIR) @echo CC $@ $(COMPILE.c) $(DEPFLAGS) $(ZSTD_STATLIB_DIR)/$*.d $(OUTPUT_OPTION) $< $(ZSTD_DYNLIB_DIR)/%.o : %.S | $(ZSTD_DYNLIB_DIR) @echo AS $@ $(COMPILE.S) $(OUTPUT_OPTION) $< $(ZSTD_STATLIB_DIR)/%.o : %.S | $(ZSTD_STATLIB_DIR) @echo AS $@ $(COMPILE.S) $(OUTPUT_OPTION) $< MKDIR ?= mkdir $(BUILD_DIR) $(ZSTD_DYNLIB_DIR) $(ZSTD_STATLIB_DIR): $(MKDIR) -p $@ DEPFILES := $(ZSTD_DYNLIB_OBJ:.o=.d) $(ZSTD_STATLIB_OBJ:.o=.d) $(DEPFILES): include $(wildcard $(DEPFILES)) # Special case : building library in single-thread mode _and_ without zstdmt_compress.c ZSTDMT_FILES = compress/zstdmt_compress.c ZSTD_NOMT_FILES = $(filter-out $(ZSTDMT_FILES),$(ZSTD_FILES)) libzstd-nomt: CFLAGS += -fPIC -fvisibility=hidden libzstd-nomt: LDFLAGS += -shared libzstd-nomt: $(ZSTD_NOMT_FILES) @echo compiling single-thread dynamic library $(LIBVER) @echo files : $(ZSTD_NOMT_FILES) $(CC) $(FLAGS) $^ $(LDFLAGS) $(SONAME_FLAGS) -o $@ .PHONY: clean clean: $(RM) -r *.dSYM # macOS-specific $(RM) core *.o *.a *.gcda *.$(SHARED_EXT) *.$(SHARED_EXT).* libzstd.pc $(RM) dll/libzstd.dll dll/libzstd.lib libzstd-nomt* $(RM) -r obj/* @echo Cleaning library completed #----------------------------------------------------------------------------- # make install is validated only for below listed environments #----------------------------------------------------------------------------- ifneq (,$(filter $(UNAME),Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD NetBSD DragonFly SunOS Haiku AIX)) lib: libzstd.pc HAS_EXPLICIT_EXEC_PREFIX := $(if $(or $(EXEC_PREFIX),$(exec_prefix)),1,) DESTDIR ?= # directory variables : GNU conventions prefer lowercase # see https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html # support both lower and uppercase (BSD), use uppercase in script prefix ?= /usr/local PREFIX ?= $(prefix) exec_prefix ?= $(PREFIX) EXEC_PREFIX ?= $(exec_prefix) libdir ?= $(EXEC_PREFIX)/lib LIBDIR ?= $(libdir) includedir ?= $(PREFIX)/include INCLUDEDIR ?= $(includedir) PCINCDIR := $(patsubst $(PREFIX)%,%,$(INCLUDEDIR)) PCLIBDIR := $(patsubst $(EXEC_PREFIX)%,%,$(LIBDIR)) # If we successfully stripped off a prefix, we'll add a reference to the # relevant pc variable. PCINCPREFIX := $(if $(findstring $(INCLUDEDIR),$(PCINCDIR)),,$${prefix}) PCLIBPREFIX := $(if $(findstring $(LIBDIR),$(PCLIBDIR)),,$${exec_prefix}) # If no explicit EXEC_PREFIX was set by the caller, write it out as a reference # to PREFIX, rather than as a resolved value. PCEXEC_PREFIX := $(if $(HAS_EXPLICIT_EXEC_PREFIX),$(EXEC_PREFIX),$${prefix}) ifneq (,$(filter $(UNAME),FreeBSD NetBSD DragonFly)) PKGCONFIGDIR ?= $(PREFIX)/libdata/pkgconfig else PKGCONFIGDIR ?= $(LIBDIR)/pkgconfig endif ifneq (,$(filter $(UNAME),SunOS)) INSTALL ?= ginstall else INSTALL ?= install endif INSTALL_PROGRAM ?= $(INSTALL) INSTALL_DATA ?= $(INSTALL) -m 644 libzstd.pc: libzstd.pc.in @echo creating pkgconfig @sed \ -e 's|@PREFIX@|$(PREFIX)|' \ -e 's|@EXEC_PREFIX@|$(PCEXEC_PREFIX)|' \ -e 's|@INCLUDEDIR@|$(PCINCPREFIX)$(PCINCDIR)|' \ -e 's|@LIBDIR@|$(PCLIBPREFIX)$(PCLIBDIR)|' \ -e 's|@VERSION@|$(VERSION)|' \ -e 's|@LIBS_PRIVATE@|$(LDFLAGS_DYNLIB)|' \ $< >$@ .PHONY: install install: install-pc install-static install-shared install-includes @echo zstd static and shared library installed .PHONY: install-pc install-pc: libzstd.pc [ -e $(DESTDIR)$(PKGCONFIGDIR) ] || $(INSTALL) -d -m 755 $(DESTDIR)$(PKGCONFIGDIR)/ $(INSTALL_DATA) libzstd.pc $(DESTDIR)$(PKGCONFIGDIR)/ .PHONY: install-static install-static: # only generate libzstd.a if it's not already present [ -e libzstd.a ] || $(MAKE) libzstd.a-release [ -e $(DESTDIR)$(LIBDIR) ] || $(INSTALL) -d -m 755 $(DESTDIR)$(LIBDIR)/ @echo Installing static library $(INSTALL_DATA) libzstd.a $(DESTDIR)$(LIBDIR) .PHONY: install-shared install-shared: # only generate libzstd.so if it's not already present [ -e $(LIBZSTD) ] || $(MAKE) libzstd-release [ -e $(DESTDIR)$(LIBDIR) ] || $(INSTALL) -d -m 755 $(DESTDIR)$(LIBDIR)/ @echo Installing shared library $(INSTALL_PROGRAM) $(LIBZSTD) $(DESTDIR)$(LIBDIR) ln -sf $(LIBZSTD) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT_MAJOR) ln -sf $(LIBZSTD) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT) .PHONY: install-includes install-includes: [ -e $(DESTDIR)$(INCLUDEDIR) ] || $(INSTALL) -d -m 755 $(DESTDIR)$(INCLUDEDIR)/ @echo Installing includes $(INSTALL_DATA) zstd.h $(DESTDIR)$(INCLUDEDIR) $(INSTALL_DATA) zstd_errors.h $(DESTDIR)$(INCLUDEDIR) $(INSTALL_DATA) zdict.h $(DESTDIR)$(INCLUDEDIR) .PHONY: uninstall uninstall: $(RM) $(DESTDIR)$(LIBDIR)/libzstd.a $(RM) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT) $(RM) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT_MAJOR) $(RM) $(DESTDIR)$(LIBDIR)/$(LIBZSTD) $(RM) $(DESTDIR)$(PKGCONFIGDIR)/libzstd.pc $(RM) $(DESTDIR)$(INCLUDEDIR)/zstd.h $(RM) $(DESTDIR)$(INCLUDEDIR)/zstd_errors.h $(RM) $(DESTDIR)$(INCLUDEDIR)/zdict.h @echo zstd libraries successfully uninstalled endif zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/0000755000175200007730000000000014515254731023153 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_cwksp.h0000644000175200007730000006634014515254731025530 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_CWKSP_H #define ZSTD_CWKSP_H /*-************************************* * Dependencies ***************************************/ #include "../common/allocations.h" /* ZSTD_customMalloc, ZSTD_customFree */ #include "../common/zstd_internal.h" #include "../common/portability_macros.h" #if defined (__cplusplus) extern "C" { #endif /*-************************************* * Constants ***************************************/ /* Since the workspace is effectively its own little malloc implementation / * arena, when we run under ASAN, we should similarly insert redzones between * each internal element of the workspace, so ASAN will catch overruns that * reach outside an object but that stay inside the workspace. * * This defines the size of that redzone. */ #ifndef ZSTD_CWKSP_ASAN_REDZONE_SIZE #define ZSTD_CWKSP_ASAN_REDZONE_SIZE 128 #endif /* Set our tables and aligneds to align by 64 bytes */ #define ZSTD_CWKSP_ALIGNMENT_BYTES 64 /*-************************************* * Structures ***************************************/ typedef enum { ZSTD_cwksp_alloc_objects, ZSTD_cwksp_alloc_aligned_init_once, ZSTD_cwksp_alloc_aligned, ZSTD_cwksp_alloc_buffers } ZSTD_cwksp_alloc_phase_e; /** * Used to describe whether the workspace is statically allocated (and will not * necessarily ever be freed), or if it's dynamically allocated and we can * expect a well-formed caller to free this. */ typedef enum { ZSTD_cwksp_dynamic_alloc, ZSTD_cwksp_static_alloc } ZSTD_cwksp_static_alloc_e; /** * Zstd fits all its internal datastructures into a single continuous buffer, * so that it only needs to perform a single OS allocation (or so that a buffer * can be provided to it and it can perform no allocations at all). This buffer * is called the workspace. * * Several optimizations complicate that process of allocating memory ranges * from this workspace for each internal datastructure: * * - These different internal datastructures have different setup requirements: * * - The static objects need to be cleared once and can then be trivially * reused for each compression. * * - Various buffers don't need to be initialized at all--they are always * written into before they're read. * * - The matchstate tables have a unique requirement that they don't need * their memory to be totally cleared, but they do need the memory to have * some bound, i.e., a guarantee that all values in the memory they've been * allocated is less than some maximum value (which is the starting value * for the indices that they will then use for compression). When this * guarantee is provided to them, they can use the memory without any setup * work. When it can't, they have to clear the area. * * - These buffers also have different alignment requirements. * * - We would like to reuse the objects in the workspace for multiple * compressions without having to perform any expensive reallocation or * reinitialization work. * * - We would like to be able to efficiently reuse the workspace across * multiple compressions **even when the compression parameters change** and * we need to resize some of the objects (where possible). * * To attempt to manage this buffer, given these constraints, the ZSTD_cwksp * abstraction was created. It works as follows: * * Workspace Layout: * * [ ... workspace ... ] * [objects][tables ->] free space [<- buffers][<- aligned][<- init once] * * The various objects that live in the workspace are divided into the * following categories, and are allocated separately: * * - Static objects: this is optionally the enclosing ZSTD_CCtx or ZSTD_CDict, * so that literally everything fits in a single buffer. Note: if present, * this must be the first object in the workspace, since ZSTD_customFree{CCtx, * CDict}() rely on a pointer comparison to see whether one or two frees are * required. * * - Fixed size objects: these are fixed-size, fixed-count objects that are * nonetheless "dynamically" allocated in the workspace so that we can * control how they're initialized separately from the broader ZSTD_CCtx. * Examples: * - Entropy Workspace * - 2 x ZSTD_compressedBlockState_t * - CDict dictionary contents * * - Tables: these are any of several different datastructures (hash tables, * chain tables, binary trees) that all respect a common format: they are * uint32_t arrays, all of whose values are between 0 and (nextSrc - base). * Their sizes depend on the cparams. These tables are 64-byte aligned. * * - Init once: these buffers require to be initialized at least once before * use. They should be used when we want to skip memory initialization * while not triggering memory checkers (like Valgrind) when reading from * from this memory without writing to it first. * These buffers should be used carefully as they might contain data * from previous compressions. * Buffers are aligned to 64 bytes. * * - Aligned: these buffers don't require any initialization before they're * used. The user of the buffer should make sure they write into a buffer * location before reading from it. * Buffers are aligned to 64 bytes. * * - Buffers: these buffers are used for various purposes that don't require * any alignment or initialization before they're used. This means they can * be moved around at no cost for a new compression. * * Allocating Memory: * * The various types of objects must be allocated in order, so they can be * correctly packed into the workspace buffer. That order is: * * 1. Objects * 2. Init once / Tables * 3. Aligned / Tables * 4. Buffers / Tables * * Attempts to reserve objects of different types out of order will fail. */ typedef struct { void* workspace; void* workspaceEnd; void* objectEnd; void* tableEnd; void* tableValidEnd; void* allocStart; void* initOnceStart; BYTE allocFailed; int workspaceOversizedDuration; ZSTD_cwksp_alloc_phase_e phase; ZSTD_cwksp_static_alloc_e isStatic; } ZSTD_cwksp; /*-************************************* * Functions ***************************************/ MEM_STATIC size_t ZSTD_cwksp_available_space(ZSTD_cwksp* ws); MEM_STATIC void* ZSTD_cwksp_initialAllocStart(ZSTD_cwksp* ws); MEM_STATIC void ZSTD_cwksp_assert_internal_consistency(ZSTD_cwksp* ws) { (void)ws; assert(ws->workspace <= ws->objectEnd); assert(ws->objectEnd <= ws->tableEnd); assert(ws->objectEnd <= ws->tableValidEnd); assert(ws->tableEnd <= ws->allocStart); assert(ws->tableValidEnd <= ws->allocStart); assert(ws->allocStart <= ws->workspaceEnd); assert(ws->initOnceStart <= ZSTD_cwksp_initialAllocStart(ws)); assert(ws->workspace <= ws->initOnceStart); #if ZSTD_MEMORY_SANITIZER { intptr_t const offset = __msan_test_shadow(ws->initOnceStart, (U8*)ZSTD_cwksp_initialAllocStart(ws) - (U8*)ws->initOnceStart); #if defined(ZSTD_MSAN_PRINT) if(offset!=-1) { __msan_print_shadow((U8*)ws->initOnceStart + offset - 8, 32); } #endif assert(offset==-1); }; #endif } /** * Align must be a power of 2. */ MEM_STATIC size_t ZSTD_cwksp_align(size_t size, size_t const align) { size_t const mask = align - 1; assert((align & mask) == 0); return (size + mask) & ~mask; } /** * Use this to determine how much space in the workspace we will consume to * allocate this object. (Normally it should be exactly the size of the object, * but under special conditions, like ASAN, where we pad each object, it might * be larger.) * * Since tables aren't currently redzoned, you don't need to call through this * to figure out how much space you need for the matchState tables. Everything * else is though. * * Do not use for sizing aligned buffers. Instead, use ZSTD_cwksp_aligned_alloc_size(). */ MEM_STATIC size_t ZSTD_cwksp_alloc_size(size_t size) { if (size == 0) return 0; #if ZSTD_ADDRESS_SANITIZER && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE) return size + 2 * ZSTD_CWKSP_ASAN_REDZONE_SIZE; #else return size; #endif } /** * Returns an adjusted alloc size that is the nearest larger multiple of 64 bytes. * Used to determine the number of bytes required for a given "aligned". */ MEM_STATIC size_t ZSTD_cwksp_aligned_alloc_size(size_t size) { return ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(size, ZSTD_CWKSP_ALIGNMENT_BYTES)); } /** * Returns the amount of additional space the cwksp must allocate * for internal purposes (currently only alignment). */ MEM_STATIC size_t ZSTD_cwksp_slack_space_required(void) { /* For alignment, the wksp will always allocate an additional 2*ZSTD_CWKSP_ALIGNMENT_BYTES * bytes to align the beginning of tables section and end of buffers; */ size_t const slackSpace = ZSTD_CWKSP_ALIGNMENT_BYTES * 2; return slackSpace; } /** * Return the number of additional bytes required to align a pointer to the given number of bytes. * alignBytes must be a power of two. */ MEM_STATIC size_t ZSTD_cwksp_bytes_to_align_ptr(void* ptr, const size_t alignBytes) { size_t const alignBytesMask = alignBytes - 1; size_t const bytes = (alignBytes - ((size_t)ptr & (alignBytesMask))) & alignBytesMask; assert((alignBytes & alignBytesMask) == 0); assert(bytes < alignBytes); return bytes; } /** * Returns the initial value for allocStart which is used to determine the position from * which we can allocate from the end of the workspace. */ MEM_STATIC void* ZSTD_cwksp_initialAllocStart(ZSTD_cwksp* ws) { return (void*)((size_t)ws->workspaceEnd & ~(ZSTD_CWKSP_ALIGNMENT_BYTES-1)); } /** * Internal function. Do not use directly. * Reserves the given number of bytes within the aligned/buffer segment of the wksp, * which counts from the end of the wksp (as opposed to the object/table segment). * * Returns a pointer to the beginning of that space. */ MEM_STATIC void* ZSTD_cwksp_reserve_internal_buffer_space(ZSTD_cwksp* ws, size_t const bytes) { void* const alloc = (BYTE*)ws->allocStart - bytes; void* const bottom = ws->tableEnd; DEBUGLOG(5, "cwksp: reserving %p %zd bytes, %zd bytes remaining", alloc, bytes, ZSTD_cwksp_available_space(ws) - bytes); ZSTD_cwksp_assert_internal_consistency(ws); assert(alloc >= bottom); if (alloc < bottom) { DEBUGLOG(4, "cwksp: alloc failed!"); ws->allocFailed = 1; return NULL; } /* the area is reserved from the end of wksp. * If it overlaps with tableValidEnd, it voids guarantees on values' range */ if (alloc < ws->tableValidEnd) { ws->tableValidEnd = alloc; } ws->allocStart = alloc; return alloc; } /** * Moves the cwksp to the next phase, and does any necessary allocations. * cwksp initialization must necessarily go through each phase in order. * Returns a 0 on success, or zstd error */ MEM_STATIC size_t ZSTD_cwksp_internal_advance_phase(ZSTD_cwksp* ws, ZSTD_cwksp_alloc_phase_e phase) { assert(phase >= ws->phase); if (phase > ws->phase) { /* Going from allocating objects to allocating initOnce / tables */ if (ws->phase < ZSTD_cwksp_alloc_aligned_init_once && phase >= ZSTD_cwksp_alloc_aligned_init_once) { ws->tableValidEnd = ws->objectEnd; ws->initOnceStart = ZSTD_cwksp_initialAllocStart(ws); { /* Align the start of the tables to 64 bytes. Use [0, 63] bytes */ void *const alloc = ws->objectEnd; size_t const bytesToAlign = ZSTD_cwksp_bytes_to_align_ptr(alloc, ZSTD_CWKSP_ALIGNMENT_BYTES); void *const objectEnd = (BYTE *) alloc + bytesToAlign; DEBUGLOG(5, "reserving table alignment addtl space: %zu", bytesToAlign); RETURN_ERROR_IF(objectEnd > ws->workspaceEnd, memory_allocation, "table phase - alignment initial allocation failed!"); ws->objectEnd = objectEnd; ws->tableEnd = objectEnd; /* table area starts being empty */ if (ws->tableValidEnd < ws->tableEnd) { ws->tableValidEnd = ws->tableEnd; } } } ws->phase = phase; ZSTD_cwksp_assert_internal_consistency(ws); } return 0; } /** * Returns whether this object/buffer/etc was allocated in this workspace. */ MEM_STATIC int ZSTD_cwksp_owns_buffer(const ZSTD_cwksp* ws, const void* ptr) { return (ptr != NULL) && (ws->workspace <= ptr) && (ptr < ws->workspaceEnd); } /** * Internal function. Do not use directly. */ MEM_STATIC void* ZSTD_cwksp_reserve_internal(ZSTD_cwksp* ws, size_t bytes, ZSTD_cwksp_alloc_phase_e phase) { void* alloc; if (ZSTD_isError(ZSTD_cwksp_internal_advance_phase(ws, phase)) || bytes == 0) { return NULL; } #if ZSTD_ADDRESS_SANITIZER && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE) /* over-reserve space */ bytes += 2 * ZSTD_CWKSP_ASAN_REDZONE_SIZE; #endif alloc = ZSTD_cwksp_reserve_internal_buffer_space(ws, bytes); #if ZSTD_ADDRESS_SANITIZER && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE) /* Move alloc so there's ZSTD_CWKSP_ASAN_REDZONE_SIZE unused space on * either size. */ if (alloc) { alloc = (BYTE *)alloc + ZSTD_CWKSP_ASAN_REDZONE_SIZE; if (ws->isStatic == ZSTD_cwksp_dynamic_alloc) { /* We need to keep the redzone poisoned while unpoisoning the bytes that * are actually allocated. */ __asan_unpoison_memory_region(alloc, bytes - 2 * ZSTD_CWKSP_ASAN_REDZONE_SIZE); } } #endif return alloc; } /** * Reserves and returns unaligned memory. */ MEM_STATIC BYTE* ZSTD_cwksp_reserve_buffer(ZSTD_cwksp* ws, size_t bytes) { return (BYTE*)ZSTD_cwksp_reserve_internal(ws, bytes, ZSTD_cwksp_alloc_buffers); } /** * Reserves and returns memory sized on and aligned on ZSTD_CWKSP_ALIGNMENT_BYTES (64 bytes). * This memory has been initialized at least once in the past. * This doesn't mean it has been initialized this time, and it might contain data from previous * operations. * The main usage is for algorithms that might need read access into uninitialized memory. * The algorithm must maintain safety under these conditions and must make sure it doesn't * leak any of the past data (directly or in side channels). */ MEM_STATIC void* ZSTD_cwksp_reserve_aligned_init_once(ZSTD_cwksp* ws, size_t bytes) { size_t const alignedBytes = ZSTD_cwksp_align(bytes, ZSTD_CWKSP_ALIGNMENT_BYTES); void* ptr = ZSTD_cwksp_reserve_internal(ws, alignedBytes, ZSTD_cwksp_alloc_aligned_init_once); assert(((size_t)ptr & (ZSTD_CWKSP_ALIGNMENT_BYTES-1))== 0); if(ptr && ptr < ws->initOnceStart) { /* We assume the memory following the current allocation is either: * 1. Not usable as initOnce memory (end of workspace) * 2. Another initOnce buffer that has been allocated before (and so was previously memset) * 3. An ASAN redzone, in which case we don't want to write on it * For these reasons it should be fine to not explicitly zero every byte up to ws->initOnceStart. * Note that we assume here that MSAN and ASAN cannot run in the same time. */ ZSTD_memset(ptr, 0, MIN((size_t)((U8*)ws->initOnceStart - (U8*)ptr), alignedBytes)); ws->initOnceStart = ptr; } #if ZSTD_MEMORY_SANITIZER assert(__msan_test_shadow(ptr, bytes) == -1); #endif return ptr; } /** * Reserves and returns memory sized on and aligned on ZSTD_CWKSP_ALIGNMENT_BYTES (64 bytes). */ MEM_STATIC void* ZSTD_cwksp_reserve_aligned(ZSTD_cwksp* ws, size_t bytes) { void* ptr = ZSTD_cwksp_reserve_internal(ws, ZSTD_cwksp_align(bytes, ZSTD_CWKSP_ALIGNMENT_BYTES), ZSTD_cwksp_alloc_aligned); assert(((size_t)ptr & (ZSTD_CWKSP_ALIGNMENT_BYTES-1))== 0); return ptr; } /** * Aligned on 64 bytes. These buffers have the special property that * their values remain constrained, allowing us to re-use them without * memset()-ing them. */ MEM_STATIC void* ZSTD_cwksp_reserve_table(ZSTD_cwksp* ws, size_t bytes) { const ZSTD_cwksp_alloc_phase_e phase = ZSTD_cwksp_alloc_aligned_init_once; void* alloc; void* end; void* top; /* We can only start allocating tables after we are done reserving space for objects at the * start of the workspace */ if(ws->phase < phase) { if (ZSTD_isError(ZSTD_cwksp_internal_advance_phase(ws, phase))) { return NULL; } } alloc = ws->tableEnd; end = (BYTE *)alloc + bytes; top = ws->allocStart; DEBUGLOG(5, "cwksp: reserving %p table %zd bytes, %zd bytes remaining", alloc, bytes, ZSTD_cwksp_available_space(ws) - bytes); assert((bytes & (sizeof(U32)-1)) == 0); ZSTD_cwksp_assert_internal_consistency(ws); assert(end <= top); if (end > top) { DEBUGLOG(4, "cwksp: table alloc failed!"); ws->allocFailed = 1; return NULL; } ws->tableEnd = end; #if ZSTD_ADDRESS_SANITIZER && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE) if (ws->isStatic == ZSTD_cwksp_dynamic_alloc) { __asan_unpoison_memory_region(alloc, bytes); } #endif assert((bytes & (ZSTD_CWKSP_ALIGNMENT_BYTES-1)) == 0); assert(((size_t)alloc & (ZSTD_CWKSP_ALIGNMENT_BYTES-1))== 0); return alloc; } /** * Aligned on sizeof(void*). * Note : should happen only once, at workspace first initialization */ MEM_STATIC void* ZSTD_cwksp_reserve_object(ZSTD_cwksp* ws, size_t bytes) { size_t const roundedBytes = ZSTD_cwksp_align(bytes, sizeof(void*)); void* alloc = ws->objectEnd; void* end = (BYTE*)alloc + roundedBytes; #if ZSTD_ADDRESS_SANITIZER && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE) /* over-reserve space */ end = (BYTE *)end + 2 * ZSTD_CWKSP_ASAN_REDZONE_SIZE; #endif DEBUGLOG(4, "cwksp: reserving %p object %zd bytes (rounded to %zd), %zd bytes remaining", alloc, bytes, roundedBytes, ZSTD_cwksp_available_space(ws) - roundedBytes); assert((size_t)alloc % ZSTD_ALIGNOF(void*) == 0); assert(bytes % ZSTD_ALIGNOF(void*) == 0); ZSTD_cwksp_assert_internal_consistency(ws); /* we must be in the first phase, no advance is possible */ if (ws->phase != ZSTD_cwksp_alloc_objects || end > ws->workspaceEnd) { DEBUGLOG(3, "cwksp: object alloc failed!"); ws->allocFailed = 1; return NULL; } ws->objectEnd = end; ws->tableEnd = end; ws->tableValidEnd = end; #if ZSTD_ADDRESS_SANITIZER && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE) /* Move alloc so there's ZSTD_CWKSP_ASAN_REDZONE_SIZE unused space on * either size. */ alloc = (BYTE*)alloc + ZSTD_CWKSP_ASAN_REDZONE_SIZE; if (ws->isStatic == ZSTD_cwksp_dynamic_alloc) { __asan_unpoison_memory_region(alloc, bytes); } #endif return alloc; } MEM_STATIC void ZSTD_cwksp_mark_tables_dirty(ZSTD_cwksp* ws) { DEBUGLOG(4, "cwksp: ZSTD_cwksp_mark_tables_dirty"); #if ZSTD_MEMORY_SANITIZER && !defined (ZSTD_MSAN_DONT_POISON_WORKSPACE) /* To validate that the table re-use logic is sound, and that we don't * access table space that we haven't cleaned, we re-"poison" the table * space every time we mark it dirty. * Since tableValidEnd space and initOnce space may overlap we don't poison * the initOnce portion as it break its promise. This means that this poisoning * check isn't always applied fully. */ { size_t size = (BYTE*)ws->tableValidEnd - (BYTE*)ws->objectEnd; assert(__msan_test_shadow(ws->objectEnd, size) == -1); if((BYTE*)ws->tableValidEnd < (BYTE*)ws->initOnceStart) { __msan_poison(ws->objectEnd, size); } else { assert(ws->initOnceStart >= ws->objectEnd); __msan_poison(ws->objectEnd, (BYTE*)ws->initOnceStart - (BYTE*)ws->objectEnd); } } #endif assert(ws->tableValidEnd >= ws->objectEnd); assert(ws->tableValidEnd <= ws->allocStart); ws->tableValidEnd = ws->objectEnd; ZSTD_cwksp_assert_internal_consistency(ws); } MEM_STATIC void ZSTD_cwksp_mark_tables_clean(ZSTD_cwksp* ws) { DEBUGLOG(4, "cwksp: ZSTD_cwksp_mark_tables_clean"); assert(ws->tableValidEnd >= ws->objectEnd); assert(ws->tableValidEnd <= ws->allocStart); if (ws->tableValidEnd < ws->tableEnd) { ws->tableValidEnd = ws->tableEnd; } ZSTD_cwksp_assert_internal_consistency(ws); } /** * Zero the part of the allocated tables not already marked clean. */ MEM_STATIC void ZSTD_cwksp_clean_tables(ZSTD_cwksp* ws) { DEBUGLOG(4, "cwksp: ZSTD_cwksp_clean_tables"); assert(ws->tableValidEnd >= ws->objectEnd); assert(ws->tableValidEnd <= ws->allocStart); if (ws->tableValidEnd < ws->tableEnd) { ZSTD_memset(ws->tableValidEnd, 0, (size_t)((BYTE*)ws->tableEnd - (BYTE*)ws->tableValidEnd)); } ZSTD_cwksp_mark_tables_clean(ws); } /** * Invalidates table allocations. * All other allocations remain valid. */ MEM_STATIC void ZSTD_cwksp_clear_tables(ZSTD_cwksp* ws) { DEBUGLOG(4, "cwksp: clearing tables!"); #if ZSTD_ADDRESS_SANITIZER && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE) /* We don't do this when the workspace is statically allocated, because * when that is the case, we have no capability to hook into the end of the * workspace's lifecycle to unpoison the memory. */ if (ws->isStatic == ZSTD_cwksp_dynamic_alloc) { size_t size = (BYTE*)ws->tableValidEnd - (BYTE*)ws->objectEnd; __asan_poison_memory_region(ws->objectEnd, size); } #endif ws->tableEnd = ws->objectEnd; ZSTD_cwksp_assert_internal_consistency(ws); } /** * Invalidates all buffer, aligned, and table allocations. * Object allocations remain valid. */ MEM_STATIC void ZSTD_cwksp_clear(ZSTD_cwksp* ws) { DEBUGLOG(4, "cwksp: clearing!"); #if ZSTD_MEMORY_SANITIZER && !defined (ZSTD_MSAN_DONT_POISON_WORKSPACE) /* To validate that the context re-use logic is sound, and that we don't * access stuff that this compression hasn't initialized, we re-"poison" * the workspace except for the areas in which we expect memory re-use * without initialization (objects, valid tables area and init once * memory). */ { if((BYTE*)ws->tableValidEnd < (BYTE*)ws->initOnceStart) { size_t size = (BYTE*)ws->initOnceStart - (BYTE*)ws->tableValidEnd; __msan_poison(ws->tableValidEnd, size); } } #endif #if ZSTD_ADDRESS_SANITIZER && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE) /* We don't do this when the workspace is statically allocated, because * when that is the case, we have no capability to hook into the end of the * workspace's lifecycle to unpoison the memory. */ if (ws->isStatic == ZSTD_cwksp_dynamic_alloc) { size_t size = (BYTE*)ws->workspaceEnd - (BYTE*)ws->objectEnd; __asan_poison_memory_region(ws->objectEnd, size); } #endif ws->tableEnd = ws->objectEnd; ws->allocStart = ZSTD_cwksp_initialAllocStart(ws); ws->allocFailed = 0; if (ws->phase > ZSTD_cwksp_alloc_aligned_init_once) { ws->phase = ZSTD_cwksp_alloc_aligned_init_once; } ZSTD_cwksp_assert_internal_consistency(ws); } /** * The provided workspace takes ownership of the buffer [start, start+size). * Any existing values in the workspace are ignored (the previously managed * buffer, if present, must be separately freed). */ MEM_STATIC void ZSTD_cwksp_init(ZSTD_cwksp* ws, void* start, size_t size, ZSTD_cwksp_static_alloc_e isStatic) { DEBUGLOG(4, "cwksp: init'ing workspace with %zd bytes", size); assert(((size_t)start & (sizeof(void*)-1)) == 0); /* ensure correct alignment */ ws->workspace = start; ws->workspaceEnd = (BYTE*)start + size; ws->objectEnd = ws->workspace; ws->tableValidEnd = ws->objectEnd; ws->initOnceStart = ZSTD_cwksp_initialAllocStart(ws); ws->phase = ZSTD_cwksp_alloc_objects; ws->isStatic = isStatic; ZSTD_cwksp_clear(ws); ws->workspaceOversizedDuration = 0; ZSTD_cwksp_assert_internal_consistency(ws); } MEM_STATIC size_t ZSTD_cwksp_create(ZSTD_cwksp* ws, size_t size, ZSTD_customMem customMem) { void* workspace = ZSTD_customMalloc(size, customMem); DEBUGLOG(4, "cwksp: creating new workspace with %zd bytes", size); RETURN_ERROR_IF(workspace == NULL, memory_allocation, "NULL pointer!"); ZSTD_cwksp_init(ws, workspace, size, ZSTD_cwksp_dynamic_alloc); return 0; } MEM_STATIC void ZSTD_cwksp_free(ZSTD_cwksp* ws, ZSTD_customMem customMem) { void *ptr = ws->workspace; DEBUGLOG(4, "cwksp: freeing workspace"); ZSTD_memset(ws, 0, sizeof(ZSTD_cwksp)); ZSTD_customFree(ptr, customMem); } /** * Moves the management of a workspace from one cwksp to another. The src cwksp * is left in an invalid state (src must be re-init()'ed before it's used again). */ MEM_STATIC void ZSTD_cwksp_move(ZSTD_cwksp* dst, ZSTD_cwksp* src) { *dst = *src; ZSTD_memset(src, 0, sizeof(ZSTD_cwksp)); } MEM_STATIC size_t ZSTD_cwksp_sizeof(const ZSTD_cwksp* ws) { return (size_t)((BYTE*)ws->workspaceEnd - (BYTE*)ws->workspace); } MEM_STATIC size_t ZSTD_cwksp_used(const ZSTD_cwksp* ws) { return (size_t)((BYTE*)ws->tableEnd - (BYTE*)ws->workspace) + (size_t)((BYTE*)ws->workspaceEnd - (BYTE*)ws->allocStart); } MEM_STATIC int ZSTD_cwksp_reserve_failed(const ZSTD_cwksp* ws) { return ws->allocFailed; } /*-************************************* * Functions Checking Free Space ***************************************/ /* ZSTD_alignmentSpaceWithinBounds() : * Returns if the estimated space needed for a wksp is within an acceptable limit of the * actual amount of space used. */ MEM_STATIC int ZSTD_cwksp_estimated_space_within_bounds(const ZSTD_cwksp *const ws, size_t const estimatedSpace) { /* We have an alignment space between objects and tables between tables and buffers, so we can have up to twice * the alignment bytes difference between estimation and actual usage */ return (estimatedSpace - ZSTD_cwksp_slack_space_required()) <= ZSTD_cwksp_used(ws) && ZSTD_cwksp_used(ws) <= estimatedSpace; } MEM_STATIC size_t ZSTD_cwksp_available_space(ZSTD_cwksp* ws) { return (size_t)((BYTE*)ws->allocStart - (BYTE*)ws->tableEnd); } MEM_STATIC int ZSTD_cwksp_check_available(ZSTD_cwksp* ws, size_t additionalNeededSpace) { return ZSTD_cwksp_available_space(ws) >= additionalNeededSpace; } MEM_STATIC int ZSTD_cwksp_check_too_large(ZSTD_cwksp* ws, size_t additionalNeededSpace) { return ZSTD_cwksp_check_available( ws, additionalNeededSpace * ZSTD_WORKSPACETOOLARGE_FACTOR); } MEM_STATIC int ZSTD_cwksp_check_wasteful(ZSTD_cwksp* ws, size_t additionalNeededSpace) { return ZSTD_cwksp_check_too_large(ws, additionalNeededSpace) && ws->workspaceOversizedDuration > ZSTD_WORKSPACETOOLARGE_MAXDURATION; } MEM_STATIC void ZSTD_cwksp_bump_oversized_duration( ZSTD_cwksp* ws, size_t additionalNeededSpace) { if (ZSTD_cwksp_check_too_large(ws, additionalNeededSpace)) { ws->workspaceOversizedDuration++; } else { ws->workspaceOversizedDuration = 0; } } #if defined (__cplusplus) } #endif #endif /* ZSTD_CWKSP_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_ldm.c0000644000175200007730000006742514515254731025155 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #include "zstd_ldm.h" #include "../common/debug.h" #include "../common/xxhash.h" #include "zstd_fast.h" /* ZSTD_fillHashTable() */ #include "zstd_double_fast.h" /* ZSTD_fillDoubleHashTable() */ #include "zstd_ldm_geartab.h" #define LDM_BUCKET_SIZE_LOG 3 #define LDM_MIN_MATCH_LENGTH 64 #define LDM_HASH_RLOG 7 typedef struct { U64 rolling; U64 stopMask; } ldmRollingHashState_t; /** ZSTD_ldm_gear_init(): * * Initializes the rolling hash state such that it will honor the * settings in params. */ static void ZSTD_ldm_gear_init(ldmRollingHashState_t* state, ldmParams_t const* params) { unsigned maxBitsInMask = MIN(params->minMatchLength, 64); unsigned hashRateLog = params->hashRateLog; state->rolling = ~(U32)0; /* The choice of the splitting criterion is subject to two conditions: * 1. it has to trigger on average every 2^(hashRateLog) bytes; * 2. ideally, it has to depend on a window of minMatchLength bytes. * * In the gear hash algorithm, bit n depends on the last n bytes; * so in order to obtain a good quality splitting criterion it is * preferable to use bits with high weight. * * To match condition 1 we use a mask with hashRateLog bits set * and, because of the previous remark, we make sure these bits * have the highest possible weight while still respecting * condition 2. */ if (hashRateLog > 0 && hashRateLog <= maxBitsInMask) { state->stopMask = (((U64)1 << hashRateLog) - 1) << (maxBitsInMask - hashRateLog); } else { /* In this degenerate case we simply honor the hash rate. */ state->stopMask = ((U64)1 << hashRateLog) - 1; } } /** ZSTD_ldm_gear_reset() * Feeds [data, data + minMatchLength) into the hash without registering any * splits. This effectively resets the hash state. This is used when skipping * over data, either at the beginning of a block, or skipping sections. */ static void ZSTD_ldm_gear_reset(ldmRollingHashState_t* state, BYTE const* data, size_t minMatchLength) { U64 hash = state->rolling; size_t n = 0; #define GEAR_ITER_ONCE() do { \ hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; \ n += 1; \ } while (0) while (n + 3 < minMatchLength) { GEAR_ITER_ONCE(); GEAR_ITER_ONCE(); GEAR_ITER_ONCE(); GEAR_ITER_ONCE(); } while (n < minMatchLength) { GEAR_ITER_ONCE(); } #undef GEAR_ITER_ONCE } /** ZSTD_ldm_gear_feed(): * * Registers in the splits array all the split points found in the first * size bytes following the data pointer. This function terminates when * either all the data has been processed or LDM_BATCH_SIZE splits are * present in the splits array. * * Precondition: The splits array must not be full. * Returns: The number of bytes processed. */ static size_t ZSTD_ldm_gear_feed(ldmRollingHashState_t* state, BYTE const* data, size_t size, size_t* splits, unsigned* numSplits) { size_t n; U64 hash, mask; hash = state->rolling; mask = state->stopMask; n = 0; #define GEAR_ITER_ONCE() do { \ hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; \ n += 1; \ if (UNLIKELY((hash & mask) == 0)) { \ splits[*numSplits] = n; \ *numSplits += 1; \ if (*numSplits == LDM_BATCH_SIZE) \ goto done; \ } \ } while (0) while (n + 3 < size) { GEAR_ITER_ONCE(); GEAR_ITER_ONCE(); GEAR_ITER_ONCE(); GEAR_ITER_ONCE(); } while (n < size) { GEAR_ITER_ONCE(); } #undef GEAR_ITER_ONCE done: state->rolling = hash; return n; } void ZSTD_ldm_adjustParameters(ldmParams_t* params, ZSTD_compressionParameters const* cParams) { params->windowLog = cParams->windowLog; ZSTD_STATIC_ASSERT(LDM_BUCKET_SIZE_LOG <= ZSTD_LDM_BUCKETSIZELOG_MAX); DEBUGLOG(4, "ZSTD_ldm_adjustParameters"); if (!params->bucketSizeLog) params->bucketSizeLog = LDM_BUCKET_SIZE_LOG; if (!params->minMatchLength) params->minMatchLength = LDM_MIN_MATCH_LENGTH; if (params->hashLog == 0) { params->hashLog = MAX(ZSTD_HASHLOG_MIN, params->windowLog - LDM_HASH_RLOG); assert(params->hashLog <= ZSTD_HASHLOG_MAX); } if (params->hashRateLog == 0) { params->hashRateLog = params->windowLog < params->hashLog ? 0 : params->windowLog - params->hashLog; } params->bucketSizeLog = MIN(params->bucketSizeLog, params->hashLog); } size_t ZSTD_ldm_getTableSize(ldmParams_t params) { size_t const ldmHSize = ((size_t)1) << params.hashLog; size_t const ldmBucketSizeLog = MIN(params.bucketSizeLog, params.hashLog); size_t const ldmBucketSize = ((size_t)1) << (params.hashLog - ldmBucketSizeLog); size_t const totalSize = ZSTD_cwksp_alloc_size(ldmBucketSize) + ZSTD_cwksp_alloc_size(ldmHSize * sizeof(ldmEntry_t)); return params.enableLdm == ZSTD_ps_enable ? totalSize : 0; } size_t ZSTD_ldm_getMaxNbSeq(ldmParams_t params, size_t maxChunkSize) { return params.enableLdm == ZSTD_ps_enable ? (maxChunkSize / params.minMatchLength) : 0; } /** ZSTD_ldm_getBucket() : * Returns a pointer to the start of the bucket associated with hash. */ static ldmEntry_t* ZSTD_ldm_getBucket( ldmState_t* ldmState, size_t hash, ldmParams_t const ldmParams) { return ldmState->hashTable + (hash << ldmParams.bucketSizeLog); } /** ZSTD_ldm_insertEntry() : * Insert the entry with corresponding hash into the hash table */ static void ZSTD_ldm_insertEntry(ldmState_t* ldmState, size_t const hash, const ldmEntry_t entry, ldmParams_t const ldmParams) { BYTE* const pOffset = ldmState->bucketOffsets + hash; unsigned const offset = *pOffset; *(ZSTD_ldm_getBucket(ldmState, hash, ldmParams) + offset) = entry; *pOffset = (BYTE)((offset + 1) & ((1u << ldmParams.bucketSizeLog) - 1)); } /** ZSTD_ldm_countBackwardsMatch() : * Returns the number of bytes that match backwards before pIn and pMatch. * * We count only bytes where pMatch >= pBase and pIn >= pAnchor. */ static size_t ZSTD_ldm_countBackwardsMatch( const BYTE* pIn, const BYTE* pAnchor, const BYTE* pMatch, const BYTE* pMatchBase) { size_t matchLength = 0; while (pIn > pAnchor && pMatch > pMatchBase && pIn[-1] == pMatch[-1]) { pIn--; pMatch--; matchLength++; } return matchLength; } /** ZSTD_ldm_countBackwardsMatch_2segments() : * Returns the number of bytes that match backwards from pMatch, * even with the backwards match spanning 2 different segments. * * On reaching `pMatchBase`, start counting from mEnd */ static size_t ZSTD_ldm_countBackwardsMatch_2segments( const BYTE* pIn, const BYTE* pAnchor, const BYTE* pMatch, const BYTE* pMatchBase, const BYTE* pExtDictStart, const BYTE* pExtDictEnd) { size_t matchLength = ZSTD_ldm_countBackwardsMatch(pIn, pAnchor, pMatch, pMatchBase); if (pMatch - matchLength != pMatchBase || pMatchBase == pExtDictStart) { /* If backwards match is entirely in the extDict or prefix, immediately return */ return matchLength; } DEBUGLOG(7, "ZSTD_ldm_countBackwardsMatch_2segments: found 2-parts backwards match (length in prefix==%zu)", matchLength); matchLength += ZSTD_ldm_countBackwardsMatch(pIn - matchLength, pAnchor, pExtDictEnd, pExtDictStart); DEBUGLOG(7, "final backwards match length = %zu", matchLength); return matchLength; } /** ZSTD_ldm_fillFastTables() : * * Fills the relevant tables for the ZSTD_fast and ZSTD_dfast strategies. * This is similar to ZSTD_loadDictionaryContent. * * The tables for the other strategies are filled within their * block compressors. */ static size_t ZSTD_ldm_fillFastTables(ZSTD_matchState_t* ms, void const* end) { const BYTE* const iend = (const BYTE*)end; switch(ms->cParams.strategy) { case ZSTD_fast: ZSTD_fillHashTable(ms, iend, ZSTD_dtlm_fast, ZSTD_tfp_forCCtx); break; case ZSTD_dfast: ZSTD_fillDoubleHashTable(ms, iend, ZSTD_dtlm_fast, ZSTD_tfp_forCCtx); break; case ZSTD_greedy: case ZSTD_lazy: case ZSTD_lazy2: case ZSTD_btlazy2: case ZSTD_btopt: case ZSTD_btultra: case ZSTD_btultra2: break; default: assert(0); /* not possible : not a valid strategy id */ } return 0; } void ZSTD_ldm_fillHashTable( ldmState_t* ldmState, const BYTE* ip, const BYTE* iend, ldmParams_t const* params) { U32 const minMatchLength = params->minMatchLength; U32 const hBits = params->hashLog - params->bucketSizeLog; BYTE const* const base = ldmState->window.base; BYTE const* const istart = ip; ldmRollingHashState_t hashState; size_t* const splits = ldmState->splitIndices; unsigned numSplits; DEBUGLOG(5, "ZSTD_ldm_fillHashTable"); ZSTD_ldm_gear_init(&hashState, params); while (ip < iend) { size_t hashed; unsigned n; numSplits = 0; hashed = ZSTD_ldm_gear_feed(&hashState, ip, iend - ip, splits, &numSplits); for (n = 0; n < numSplits; n++) { if (ip + splits[n] >= istart + minMatchLength) { BYTE const* const split = ip + splits[n] - minMatchLength; U64 const xxhash = XXH64(split, minMatchLength, 0); U32 const hash = (U32)(xxhash & (((U32)1 << hBits) - 1)); ldmEntry_t entry; entry.offset = (U32)(split - base); entry.checksum = (U32)(xxhash >> 32); ZSTD_ldm_insertEntry(ldmState, hash, entry, *params); } } ip += hashed; } } /** ZSTD_ldm_limitTableUpdate() : * * Sets cctx->nextToUpdate to a position corresponding closer to anchor * if it is far way * (after a long match, only update tables a limited amount). */ static void ZSTD_ldm_limitTableUpdate(ZSTD_matchState_t* ms, const BYTE* anchor) { U32 const curr = (U32)(anchor - ms->window.base); if (curr > ms->nextToUpdate + 1024) { ms->nextToUpdate = curr - MIN(512, curr - ms->nextToUpdate - 1024); } } static size_t ZSTD_ldm_generateSequences_internal( ldmState_t* ldmState, rawSeqStore_t* rawSeqStore, ldmParams_t const* params, void const* src, size_t srcSize) { /* LDM parameters */ int const extDict = ZSTD_window_hasExtDict(ldmState->window); U32 const minMatchLength = params->minMatchLength; U32 const entsPerBucket = 1U << params->bucketSizeLog; U32 const hBits = params->hashLog - params->bucketSizeLog; /* Prefix and extDict parameters */ U32 const dictLimit = ldmState->window.dictLimit; U32 const lowestIndex = extDict ? ldmState->window.lowLimit : dictLimit; BYTE const* const base = ldmState->window.base; BYTE const* const dictBase = extDict ? ldmState->window.dictBase : NULL; BYTE const* const dictStart = extDict ? dictBase + lowestIndex : NULL; BYTE const* const dictEnd = extDict ? dictBase + dictLimit : NULL; BYTE const* const lowPrefixPtr = base + dictLimit; /* Input bounds */ BYTE const* const istart = (BYTE const*)src; BYTE const* const iend = istart + srcSize; BYTE const* const ilimit = iend - HASH_READ_SIZE; /* Input positions */ BYTE const* anchor = istart; BYTE const* ip = istart; /* Rolling hash state */ ldmRollingHashState_t hashState; /* Arrays for staged-processing */ size_t* const splits = ldmState->splitIndices; ldmMatchCandidate_t* const candidates = ldmState->matchCandidates; unsigned numSplits; if (srcSize < minMatchLength) return iend - anchor; /* Initialize the rolling hash state with the first minMatchLength bytes */ ZSTD_ldm_gear_init(&hashState, params); ZSTD_ldm_gear_reset(&hashState, ip, minMatchLength); ip += minMatchLength; while (ip < ilimit) { size_t hashed; unsigned n; numSplits = 0; hashed = ZSTD_ldm_gear_feed(&hashState, ip, ilimit - ip, splits, &numSplits); for (n = 0; n < numSplits; n++) { BYTE const* const split = ip + splits[n] - minMatchLength; U64 const xxhash = XXH64(split, minMatchLength, 0); U32 const hash = (U32)(xxhash & (((U32)1 << hBits) - 1)); candidates[n].split = split; candidates[n].hash = hash; candidates[n].checksum = (U32)(xxhash >> 32); candidates[n].bucket = ZSTD_ldm_getBucket(ldmState, hash, *params); PREFETCH_L1(candidates[n].bucket); } for (n = 0; n < numSplits; n++) { size_t forwardMatchLength = 0, backwardMatchLength = 0, bestMatchLength = 0, mLength; U32 offset; BYTE const* const split = candidates[n].split; U32 const checksum = candidates[n].checksum; U32 const hash = candidates[n].hash; ldmEntry_t* const bucket = candidates[n].bucket; ldmEntry_t const* cur; ldmEntry_t const* bestEntry = NULL; ldmEntry_t newEntry; newEntry.offset = (U32)(split - base); newEntry.checksum = checksum; /* If a split point would generate a sequence overlapping with * the previous one, we merely register it in the hash table and * move on */ if (split < anchor) { ZSTD_ldm_insertEntry(ldmState, hash, newEntry, *params); continue; } for (cur = bucket; cur < bucket + entsPerBucket; cur++) { size_t curForwardMatchLength, curBackwardMatchLength, curTotalMatchLength; if (cur->checksum != checksum || cur->offset <= lowestIndex) { continue; } if (extDict) { BYTE const* const curMatchBase = cur->offset < dictLimit ? dictBase : base; BYTE const* const pMatch = curMatchBase + cur->offset; BYTE const* const matchEnd = cur->offset < dictLimit ? dictEnd : iend; BYTE const* const lowMatchPtr = cur->offset < dictLimit ? dictStart : lowPrefixPtr; curForwardMatchLength = ZSTD_count_2segments(split, pMatch, iend, matchEnd, lowPrefixPtr); if (curForwardMatchLength < minMatchLength) { continue; } curBackwardMatchLength = ZSTD_ldm_countBackwardsMatch_2segments( split, anchor, pMatch, lowMatchPtr, dictStart, dictEnd); } else { /* !extDict */ BYTE const* const pMatch = base + cur->offset; curForwardMatchLength = ZSTD_count(split, pMatch, iend); if (curForwardMatchLength < minMatchLength) { continue; } curBackwardMatchLength = ZSTD_ldm_countBackwardsMatch(split, anchor, pMatch, lowPrefixPtr); } curTotalMatchLength = curForwardMatchLength + curBackwardMatchLength; if (curTotalMatchLength > bestMatchLength) { bestMatchLength = curTotalMatchLength; forwardMatchLength = curForwardMatchLength; backwardMatchLength = curBackwardMatchLength; bestEntry = cur; } } /* No match found -- insert an entry into the hash table * and process the next candidate match */ if (bestEntry == NULL) { ZSTD_ldm_insertEntry(ldmState, hash, newEntry, *params); continue; } /* Match found */ offset = (U32)(split - base) - bestEntry->offset; mLength = forwardMatchLength + backwardMatchLength; { rawSeq* const seq = rawSeqStore->seq + rawSeqStore->size; /* Out of sequence storage */ if (rawSeqStore->size == rawSeqStore->capacity) return ERROR(dstSize_tooSmall); seq->litLength = (U32)(split - backwardMatchLength - anchor); seq->matchLength = (U32)mLength; seq->offset = offset; rawSeqStore->size++; } /* Insert the current entry into the hash table --- it must be * done after the previous block to avoid clobbering bestEntry */ ZSTD_ldm_insertEntry(ldmState, hash, newEntry, *params); anchor = split + forwardMatchLength; /* If we find a match that ends after the data that we've hashed * then we have a repeating, overlapping, pattern. E.g. all zeros. * If one repetition of the pattern matches our `stopMask` then all * repetitions will. We don't need to insert them all into out table, * only the first one. So skip over overlapping matches. * This is a major speed boost (20x) for compressing a single byte * repeated, when that byte ends up in the table. */ if (anchor > ip + hashed) { ZSTD_ldm_gear_reset(&hashState, anchor - minMatchLength, minMatchLength); /* Continue the outer loop at anchor (ip + hashed == anchor). */ ip = anchor - hashed; break; } } ip += hashed; } return iend - anchor; } /*! ZSTD_ldm_reduceTable() : * reduce table indexes by `reducerValue` */ static void ZSTD_ldm_reduceTable(ldmEntry_t* const table, U32 const size, U32 const reducerValue) { U32 u; for (u = 0; u < size; u++) { if (table[u].offset < reducerValue) table[u].offset = 0; else table[u].offset -= reducerValue; } } size_t ZSTD_ldm_generateSequences( ldmState_t* ldmState, rawSeqStore_t* sequences, ldmParams_t const* params, void const* src, size_t srcSize) { U32 const maxDist = 1U << params->windowLog; BYTE const* const istart = (BYTE const*)src; BYTE const* const iend = istart + srcSize; size_t const kMaxChunkSize = 1 << 20; size_t const nbChunks = (srcSize / kMaxChunkSize) + ((srcSize % kMaxChunkSize) != 0); size_t chunk; size_t leftoverSize = 0; assert(ZSTD_CHUNKSIZE_MAX >= kMaxChunkSize); /* Check that ZSTD_window_update() has been called for this chunk prior * to passing it to this function. */ assert(ldmState->window.nextSrc >= (BYTE const*)src + srcSize); /* The input could be very large (in zstdmt), so it must be broken up into * chunks to enforce the maximum distance and handle overflow correction. */ assert(sequences->pos <= sequences->size); assert(sequences->size <= sequences->capacity); for (chunk = 0; chunk < nbChunks && sequences->size < sequences->capacity; ++chunk) { BYTE const* const chunkStart = istart + chunk * kMaxChunkSize; size_t const remaining = (size_t)(iend - chunkStart); BYTE const *const chunkEnd = (remaining < kMaxChunkSize) ? iend : chunkStart + kMaxChunkSize; size_t const chunkSize = chunkEnd - chunkStart; size_t newLeftoverSize; size_t const prevSize = sequences->size; assert(chunkStart < iend); /* 1. Perform overflow correction if necessary. */ if (ZSTD_window_needOverflowCorrection(ldmState->window, 0, maxDist, ldmState->loadedDictEnd, chunkStart, chunkEnd)) { U32 const ldmHSize = 1U << params->hashLog; U32 const correction = ZSTD_window_correctOverflow( &ldmState->window, /* cycleLog */ 0, maxDist, chunkStart); ZSTD_ldm_reduceTable(ldmState->hashTable, ldmHSize, correction); /* invalidate dictionaries on overflow correction */ ldmState->loadedDictEnd = 0; } /* 2. We enforce the maximum offset allowed. * * kMaxChunkSize should be small enough that we don't lose too much of * the window through early invalidation. * TODO: * Test the chunk size. * * Try invalidation after the sequence generation and test the * offset against maxDist directly. * * NOTE: Because of dictionaries + sequence splitting we MUST make sure * that any offset used is valid at the END of the sequence, since it may * be split into two sequences. This condition holds when using * ZSTD_window_enforceMaxDist(), but if we move to checking offsets * against maxDist directly, we'll have to carefully handle that case. */ ZSTD_window_enforceMaxDist(&ldmState->window, chunkEnd, maxDist, &ldmState->loadedDictEnd, NULL); /* 3. Generate the sequences for the chunk, and get newLeftoverSize. */ newLeftoverSize = ZSTD_ldm_generateSequences_internal( ldmState, sequences, params, chunkStart, chunkSize); if (ZSTD_isError(newLeftoverSize)) return newLeftoverSize; /* 4. We add the leftover literals from previous iterations to the first * newly generated sequence, or add the `newLeftoverSize` if none are * generated. */ /* Prepend the leftover literals from the last call */ if (prevSize < sequences->size) { sequences->seq[prevSize].litLength += (U32)leftoverSize; leftoverSize = newLeftoverSize; } else { assert(newLeftoverSize == chunkSize); leftoverSize += chunkSize; } } return 0; } void ZSTD_ldm_skipSequences(rawSeqStore_t* rawSeqStore, size_t srcSize, U32 const minMatch) { while (srcSize > 0 && rawSeqStore->pos < rawSeqStore->size) { rawSeq* seq = rawSeqStore->seq + rawSeqStore->pos; if (srcSize <= seq->litLength) { /* Skip past srcSize literals */ seq->litLength -= (U32)srcSize; return; } srcSize -= seq->litLength; seq->litLength = 0; if (srcSize < seq->matchLength) { /* Skip past the first srcSize of the match */ seq->matchLength -= (U32)srcSize; if (seq->matchLength < minMatch) { /* The match is too short, omit it */ if (rawSeqStore->pos + 1 < rawSeqStore->size) { seq[1].litLength += seq[0].matchLength; } rawSeqStore->pos++; } return; } srcSize -= seq->matchLength; seq->matchLength = 0; rawSeqStore->pos++; } } /** * If the sequence length is longer than remaining then the sequence is split * between this block and the next. * * Returns the current sequence to handle, or if the rest of the block should * be literals, it returns a sequence with offset == 0. */ static rawSeq maybeSplitSequence(rawSeqStore_t* rawSeqStore, U32 const remaining, U32 const minMatch) { rawSeq sequence = rawSeqStore->seq[rawSeqStore->pos]; assert(sequence.offset > 0); /* Likely: No partial sequence */ if (remaining >= sequence.litLength + sequence.matchLength) { rawSeqStore->pos++; return sequence; } /* Cut the sequence short (offset == 0 ==> rest is literals). */ if (remaining <= sequence.litLength) { sequence.offset = 0; } else if (remaining < sequence.litLength + sequence.matchLength) { sequence.matchLength = remaining - sequence.litLength; if (sequence.matchLength < minMatch) { sequence.offset = 0; } } /* Skip past `remaining` bytes for the future sequences. */ ZSTD_ldm_skipSequences(rawSeqStore, remaining, minMatch); return sequence; } void ZSTD_ldm_skipRawSeqStoreBytes(rawSeqStore_t* rawSeqStore, size_t nbBytes) { U32 currPos = (U32)(rawSeqStore->posInSequence + nbBytes); while (currPos && rawSeqStore->pos < rawSeqStore->size) { rawSeq currSeq = rawSeqStore->seq[rawSeqStore->pos]; if (currPos >= currSeq.litLength + currSeq.matchLength) { currPos -= currSeq.litLength + currSeq.matchLength; rawSeqStore->pos++; } else { rawSeqStore->posInSequence = currPos; break; } } if (currPos == 0 || rawSeqStore->pos == rawSeqStore->size) { rawSeqStore->posInSequence = 0; } } size_t ZSTD_ldm_blockCompress(rawSeqStore_t* rawSeqStore, ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], ZSTD_paramSwitch_e useRowMatchFinder, void const* src, size_t srcSize) { const ZSTD_compressionParameters* const cParams = &ms->cParams; unsigned const minMatch = cParams->minMatch; ZSTD_blockCompressor const blockCompressor = ZSTD_selectBlockCompressor(cParams->strategy, useRowMatchFinder, ZSTD_matchState_dictMode(ms)); /* Input bounds */ BYTE const* const istart = (BYTE const*)src; BYTE const* const iend = istart + srcSize; /* Input positions */ BYTE const* ip = istart; DEBUGLOG(5, "ZSTD_ldm_blockCompress: srcSize=%zu", srcSize); /* If using opt parser, use LDMs only as candidates rather than always accepting them */ if (cParams->strategy >= ZSTD_btopt) { size_t lastLLSize; ms->ldmSeqStore = rawSeqStore; lastLLSize = blockCompressor(ms, seqStore, rep, src, srcSize); ZSTD_ldm_skipRawSeqStoreBytes(rawSeqStore, srcSize); return lastLLSize; } assert(rawSeqStore->pos <= rawSeqStore->size); assert(rawSeqStore->size <= rawSeqStore->capacity); /* Loop through each sequence and apply the block compressor to the literals */ while (rawSeqStore->pos < rawSeqStore->size && ip < iend) { /* maybeSplitSequence updates rawSeqStore->pos */ rawSeq const sequence = maybeSplitSequence(rawSeqStore, (U32)(iend - ip), minMatch); int i; /* End signal */ if (sequence.offset == 0) break; assert(ip + sequence.litLength + sequence.matchLength <= iend); /* Fill tables for block compressor */ ZSTD_ldm_limitTableUpdate(ms, ip); ZSTD_ldm_fillFastTables(ms, ip); /* Run the block compressor */ DEBUGLOG(5, "pos %u : calling block compressor on segment of size %u", (unsigned)(ip-istart), sequence.litLength); { size_t const newLitLength = blockCompressor(ms, seqStore, rep, ip, sequence.litLength); ip += sequence.litLength; /* Update the repcodes */ for (i = ZSTD_REP_NUM - 1; i > 0; i--) rep[i] = rep[i-1]; rep[0] = sequence.offset; /* Store the sequence */ ZSTD_storeSeq(seqStore, newLitLength, ip - newLitLength, iend, OFFSET_TO_OFFBASE(sequence.offset), sequence.matchLength); ip += sequence.matchLength; } } /* Fill the tables for the block compressor */ ZSTD_ldm_limitTableUpdate(ms, ip); ZSTD_ldm_fillFastTables(ms, ip); /* Compress the last literals */ return blockCompressor(ms, seqStore, rep, ip, iend - ip); } zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_compress_internal.h0000644000175200007730000017564014515254731030134 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* This header contains definitions * that shall **only** be used by modules within lib/compress. */ #ifndef ZSTD_COMPRESS_H #define ZSTD_COMPRESS_H /*-************************************* * Dependencies ***************************************/ #include "../common/zstd_internal.h" #include "zstd_cwksp.h" #ifdef ZSTD_MULTITHREAD # include "zstdmt_compress.h" #endif #include "../common/bits.h" /* ZSTD_highbit32, ZSTD_NbCommonBytes */ #if defined (__cplusplus) extern "C" { #endif /*-************************************* * Constants ***************************************/ #define kSearchStrength 8 #define HASH_READ_SIZE 8 #define ZSTD_DUBT_UNSORTED_MARK 1 /* For btlazy2 strategy, index ZSTD_DUBT_UNSORTED_MARK==1 means "unsorted". It could be confused for a real successor at index "1", if sorted as larger than its predecessor. It's not a big deal though : candidate will just be sorted again. Additionally, candidate position 1 will be lost. But candidate 1 cannot hide a large tree of candidates, so it's a minimal loss. The benefit is that ZSTD_DUBT_UNSORTED_MARK cannot be mishandled after table re-use with a different strategy. This constant is required by ZSTD_compressBlock_btlazy2() and ZSTD_reduceTable_internal() */ /*-************************************* * Context memory management ***************************************/ typedef enum { ZSTDcs_created=0, ZSTDcs_init, ZSTDcs_ongoing, ZSTDcs_ending } ZSTD_compressionStage_e; typedef enum { zcss_init=0, zcss_load, zcss_flush } ZSTD_cStreamStage; typedef struct ZSTD_prefixDict_s { const void* dict; size_t dictSize; ZSTD_dictContentType_e dictContentType; } ZSTD_prefixDict; typedef struct { void* dictBuffer; void const* dict; size_t dictSize; ZSTD_dictContentType_e dictContentType; ZSTD_CDict* cdict; } ZSTD_localDict; typedef struct { HUF_CElt CTable[HUF_CTABLE_SIZE_ST(255)]; HUF_repeat repeatMode; } ZSTD_hufCTables_t; typedef struct { FSE_CTable offcodeCTable[FSE_CTABLE_SIZE_U32(OffFSELog, MaxOff)]; FSE_CTable matchlengthCTable[FSE_CTABLE_SIZE_U32(MLFSELog, MaxML)]; FSE_CTable litlengthCTable[FSE_CTABLE_SIZE_U32(LLFSELog, MaxLL)]; FSE_repeat offcode_repeatMode; FSE_repeat matchlength_repeatMode; FSE_repeat litlength_repeatMode; } ZSTD_fseCTables_t; typedef struct { ZSTD_hufCTables_t huf; ZSTD_fseCTables_t fse; } ZSTD_entropyCTables_t; /*********************************************** * Entropy buffer statistics structs and funcs * ***********************************************/ /** ZSTD_hufCTablesMetadata_t : * Stores Literals Block Type for a super-block in hType, and * huffman tree description in hufDesBuffer. * hufDesSize refers to the size of huffman tree description in bytes. * This metadata is populated in ZSTD_buildBlockEntropyStats_literals() */ typedef struct { symbolEncodingType_e hType; BYTE hufDesBuffer[ZSTD_MAX_HUF_HEADER_SIZE]; size_t hufDesSize; } ZSTD_hufCTablesMetadata_t; /** ZSTD_fseCTablesMetadata_t : * Stores symbol compression modes for a super-block in {ll, ol, ml}Type, and * fse tables in fseTablesBuffer. * fseTablesSize refers to the size of fse tables in bytes. * This metadata is populated in ZSTD_buildBlockEntropyStats_sequences() */ typedef struct { symbolEncodingType_e llType; symbolEncodingType_e ofType; symbolEncodingType_e mlType; BYTE fseTablesBuffer[ZSTD_MAX_FSE_HEADERS_SIZE]; size_t fseTablesSize; size_t lastCountSize; /* This is to account for bug in 1.3.4. More detail in ZSTD_entropyCompressSeqStore_internal() */ } ZSTD_fseCTablesMetadata_t; typedef struct { ZSTD_hufCTablesMetadata_t hufMetadata; ZSTD_fseCTablesMetadata_t fseMetadata; } ZSTD_entropyCTablesMetadata_t; /** ZSTD_buildBlockEntropyStats() : * Builds entropy for the block. * @return : 0 on success or error code */ size_t ZSTD_buildBlockEntropyStats( const seqStore_t* seqStorePtr, const ZSTD_entropyCTables_t* prevEntropy, ZSTD_entropyCTables_t* nextEntropy, const ZSTD_CCtx_params* cctxParams, ZSTD_entropyCTablesMetadata_t* entropyMetadata, void* workspace, size_t wkspSize); /********************************* * Compression internals structs * *********************************/ typedef struct { U32 off; /* Offset sumtype code for the match, using ZSTD_storeSeq() format */ U32 len; /* Raw length of match */ } ZSTD_match_t; typedef struct { U32 offset; /* Offset of sequence */ U32 litLength; /* Length of literals prior to match */ U32 matchLength; /* Raw length of match */ } rawSeq; typedef struct { rawSeq* seq; /* The start of the sequences */ size_t pos; /* The index in seq where reading stopped. pos <= size. */ size_t posInSequence; /* The position within the sequence at seq[pos] where reading stopped. posInSequence <= seq[pos].litLength + seq[pos].matchLength */ size_t size; /* The number of sequences. <= capacity. */ size_t capacity; /* The capacity starting from `seq` pointer */ } rawSeqStore_t; typedef struct { U32 idx; /* Index in array of ZSTD_Sequence */ U32 posInSequence; /* Position within sequence at idx */ size_t posInSrc; /* Number of bytes given by sequences provided so far */ } ZSTD_sequencePosition; UNUSED_ATTR static const rawSeqStore_t kNullRawSeqStore = {NULL, 0, 0, 0, 0}; typedef struct { int price; U32 off; U32 mlen; U32 litlen; U32 rep[ZSTD_REP_NUM]; } ZSTD_optimal_t; typedef enum { zop_dynamic=0, zop_predef } ZSTD_OptPrice_e; typedef struct { /* All tables are allocated inside cctx->workspace by ZSTD_resetCCtx_internal() */ unsigned* litFreq; /* table of literals statistics, of size 256 */ unsigned* litLengthFreq; /* table of litLength statistics, of size (MaxLL+1) */ unsigned* matchLengthFreq; /* table of matchLength statistics, of size (MaxML+1) */ unsigned* offCodeFreq; /* table of offCode statistics, of size (MaxOff+1) */ ZSTD_match_t* matchTable; /* list of found matches, of size ZSTD_OPT_NUM+1 */ ZSTD_optimal_t* priceTable; /* All positions tracked by optimal parser, of size ZSTD_OPT_NUM+1 */ U32 litSum; /* nb of literals */ U32 litLengthSum; /* nb of litLength codes */ U32 matchLengthSum; /* nb of matchLength codes */ U32 offCodeSum; /* nb of offset codes */ U32 litSumBasePrice; /* to compare to log2(litfreq) */ U32 litLengthSumBasePrice; /* to compare to log2(llfreq) */ U32 matchLengthSumBasePrice;/* to compare to log2(mlfreq) */ U32 offCodeSumBasePrice; /* to compare to log2(offreq) */ ZSTD_OptPrice_e priceType; /* prices can be determined dynamically, or follow a pre-defined cost structure */ const ZSTD_entropyCTables_t* symbolCosts; /* pre-calculated dictionary statistics */ ZSTD_paramSwitch_e literalCompressionMode; } optState_t; typedef struct { ZSTD_entropyCTables_t entropy; U32 rep[ZSTD_REP_NUM]; } ZSTD_compressedBlockState_t; typedef struct { BYTE const* nextSrc; /* next block here to continue on current prefix */ BYTE const* base; /* All regular indexes relative to this position */ BYTE const* dictBase; /* extDict indexes relative to this position */ U32 dictLimit; /* below that point, need extDict */ U32 lowLimit; /* below that point, no more valid data */ U32 nbOverflowCorrections; /* Number of times overflow correction has run since * ZSTD_window_init(). Useful for debugging coredumps * and for ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY. */ } ZSTD_window_t; #define ZSTD_WINDOW_START_INDEX 2 typedef struct ZSTD_matchState_t ZSTD_matchState_t; #define ZSTD_ROW_HASH_CACHE_SIZE 8 /* Size of prefetching hash cache for row-based matchfinder */ struct ZSTD_matchState_t { ZSTD_window_t window; /* State for window round buffer management */ U32 loadedDictEnd; /* index of end of dictionary, within context's referential. * When loadedDictEnd != 0, a dictionary is in use, and still valid. * This relies on a mechanism to set loadedDictEnd=0 when dictionary is no longer within distance. * Such mechanism is provided within ZSTD_window_enforceMaxDist() and ZSTD_checkDictValidity(). * When dict referential is copied into active context (i.e. not attached), * loadedDictEnd == dictSize, since referential starts from zero. */ U32 nextToUpdate; /* index from which to continue table update */ U32 hashLog3; /* dispatch table for matches of len==3 : larger == faster, more memory */ U32 rowHashLog; /* For row-based matchfinder: Hashlog based on nb of rows in the hashTable.*/ BYTE* tagTable; /* For row-based matchFinder: A row-based table containing the hashes and head index. */ U32 hashCache[ZSTD_ROW_HASH_CACHE_SIZE]; /* For row-based matchFinder: a cache of hashes to improve speed */ U64 hashSalt; /* For row-based matchFinder: salts the hash for re-use of tag table */ U32 hashSaltEntropy; /* For row-based matchFinder: collects entropy for salt generation */ U32* hashTable; U32* hashTable3; U32* chainTable; U32 forceNonContiguous; /* Non-zero if we should force non-contiguous load for the next window update. */ int dedicatedDictSearch; /* Indicates whether this matchState is using the * dedicated dictionary search structure. */ optState_t opt; /* optimal parser state */ const ZSTD_matchState_t* dictMatchState; ZSTD_compressionParameters cParams; const rawSeqStore_t* ldmSeqStore; /* Controls prefetching in some dictMatchState matchfinders. * This behavior is controlled from the cctx ms. * This parameter has no effect in the cdict ms. */ int prefetchCDictTables; /* When == 0, lazy match finders insert every position. * When != 0, lazy match finders only insert positions they search. * This allows them to skip much faster over incompressible data, * at a small cost to compression ratio. */ int lazySkipping; }; typedef struct { ZSTD_compressedBlockState_t* prevCBlock; ZSTD_compressedBlockState_t* nextCBlock; ZSTD_matchState_t matchState; } ZSTD_blockState_t; typedef struct { U32 offset; U32 checksum; } ldmEntry_t; typedef struct { BYTE const* split; U32 hash; U32 checksum; ldmEntry_t* bucket; } ldmMatchCandidate_t; #define LDM_BATCH_SIZE 64 typedef struct { ZSTD_window_t window; /* State for the window round buffer management */ ldmEntry_t* hashTable; U32 loadedDictEnd; BYTE* bucketOffsets; /* Next position in bucket to insert entry */ size_t splitIndices[LDM_BATCH_SIZE]; ldmMatchCandidate_t matchCandidates[LDM_BATCH_SIZE]; } ldmState_t; typedef struct { ZSTD_paramSwitch_e enableLdm; /* ZSTD_ps_enable to enable LDM. ZSTD_ps_auto by default */ U32 hashLog; /* Log size of hashTable */ U32 bucketSizeLog; /* Log bucket size for collision resolution, at most 8 */ U32 minMatchLength; /* Minimum match length */ U32 hashRateLog; /* Log number of entries to skip */ U32 windowLog; /* Window log for the LDM */ } ldmParams_t; typedef struct { int collectSequences; ZSTD_Sequence* seqStart; size_t seqIndex; size_t maxSequences; } SeqCollector; struct ZSTD_CCtx_params_s { ZSTD_format_e format; ZSTD_compressionParameters cParams; ZSTD_frameParameters fParams; int compressionLevel; int forceWindow; /* force back-references to respect limit of * 1< 63) ? ZSTD_highbit32(litLength) + LL_deltaCode : LL_Code[litLength]; } /* ZSTD_MLcode() : * note : mlBase = matchLength - MINMATCH; * because it's the format it's stored in seqStore->sequences */ MEM_STATIC U32 ZSTD_MLcode(U32 mlBase) { static const BYTE ML_Code[128] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 32, 33, 33, 34, 34, 35, 35, 36, 36, 36, 36, 37, 37, 37, 37, 38, 38, 38, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42 }; static const U32 ML_deltaCode = 36; return (mlBase > 127) ? ZSTD_highbit32(mlBase) + ML_deltaCode : ML_Code[mlBase]; } /* ZSTD_cParam_withinBounds: * @return 1 if value is within cParam bounds, * 0 otherwise */ MEM_STATIC int ZSTD_cParam_withinBounds(ZSTD_cParameter cParam, int value) { ZSTD_bounds const bounds = ZSTD_cParam_getBounds(cParam); if (ZSTD_isError(bounds.error)) return 0; if (value < bounds.lowerBound) return 0; if (value > bounds.upperBound) return 0; return 1; } /* ZSTD_noCompressBlock() : * Writes uncompressed block to dst buffer from given src. * Returns the size of the block */ MEM_STATIC size_t ZSTD_noCompressBlock(void* dst, size_t dstCapacity, const void* src, size_t srcSize, U32 lastBlock) { U32 const cBlockHeader24 = lastBlock + (((U32)bt_raw)<<1) + (U32)(srcSize << 3); DEBUGLOG(5, "ZSTD_noCompressBlock (srcSize=%zu, dstCapacity=%zu)", srcSize, dstCapacity); RETURN_ERROR_IF(srcSize + ZSTD_blockHeaderSize > dstCapacity, dstSize_tooSmall, "dst buf too small for uncompressed block"); MEM_writeLE24(dst, cBlockHeader24); ZSTD_memcpy((BYTE*)dst + ZSTD_blockHeaderSize, src, srcSize); return ZSTD_blockHeaderSize + srcSize; } MEM_STATIC size_t ZSTD_rleCompressBlock(void* dst, size_t dstCapacity, BYTE src, size_t srcSize, U32 lastBlock) { BYTE* const op = (BYTE*)dst; U32 const cBlockHeader = lastBlock + (((U32)bt_rle)<<1) + (U32)(srcSize << 3); RETURN_ERROR_IF(dstCapacity < 4, dstSize_tooSmall, ""); MEM_writeLE24(op, cBlockHeader); op[3] = src; return 4; } /* ZSTD_minGain() : * minimum compression required * to generate a compress block or a compressed literals section. * note : use same formula for both situations */ MEM_STATIC size_t ZSTD_minGain(size_t srcSize, ZSTD_strategy strat) { U32 const minlog = (strat>=ZSTD_btultra) ? (U32)(strat) - 1 : 6; ZSTD_STATIC_ASSERT(ZSTD_btultra == 8); assert(ZSTD_cParam_withinBounds(ZSTD_c_strategy, (int)strat)); return (srcSize >> minlog) + 2; } MEM_STATIC int ZSTD_literalsCompressionIsDisabled(const ZSTD_CCtx_params* cctxParams) { switch (cctxParams->literalCompressionMode) { case ZSTD_ps_enable: return 0; case ZSTD_ps_disable: return 1; default: assert(0 /* impossible: pre-validated */); ZSTD_FALLTHROUGH; case ZSTD_ps_auto: return (cctxParams->cParams.strategy == ZSTD_fast) && (cctxParams->cParams.targetLength > 0); } } /*! ZSTD_safecopyLiterals() : * memcpy() function that won't read beyond more than WILDCOPY_OVERLENGTH bytes past ilimit_w. * Only called when the sequence ends past ilimit_w, so it only needs to be optimized for single * large copies. */ static void ZSTD_safecopyLiterals(BYTE* op, BYTE const* ip, BYTE const* const iend, BYTE const* ilimit_w) { assert(iend > ilimit_w); if (ip <= ilimit_w) { ZSTD_wildcopy(op, ip, ilimit_w - ip, ZSTD_no_overlap); op += ilimit_w - ip; ip = ilimit_w; } while (ip < iend) *op++ = *ip++; } #define REPCODE1_TO_OFFBASE REPCODE_TO_OFFBASE(1) #define REPCODE2_TO_OFFBASE REPCODE_TO_OFFBASE(2) #define REPCODE3_TO_OFFBASE REPCODE_TO_OFFBASE(3) #define REPCODE_TO_OFFBASE(r) (assert((r)>=1), assert((r)<=ZSTD_REP_NUM), (r)) /* accepts IDs 1,2,3 */ #define OFFSET_TO_OFFBASE(o) (assert((o)>0), o + ZSTD_REP_NUM) #define OFFBASE_IS_OFFSET(o) ((o) > ZSTD_REP_NUM) #define OFFBASE_IS_REPCODE(o) ( 1 <= (o) && (o) <= ZSTD_REP_NUM) #define OFFBASE_TO_OFFSET(o) (assert(OFFBASE_IS_OFFSET(o)), (o) - ZSTD_REP_NUM) #define OFFBASE_TO_REPCODE(o) (assert(OFFBASE_IS_REPCODE(o)), (o)) /* returns ID 1,2,3 */ /*! ZSTD_storeSeq() : * Store a sequence (litlen, litPtr, offBase and matchLength) into seqStore_t. * @offBase : Users should employ macros REPCODE_TO_OFFBASE() and OFFSET_TO_OFFBASE(). * @matchLength : must be >= MINMATCH * Allowed to over-read literals up to litLimit. */ HINT_INLINE UNUSED_ATTR void ZSTD_storeSeq(seqStore_t* seqStorePtr, size_t litLength, const BYTE* literals, const BYTE* litLimit, U32 offBase, size_t matchLength) { BYTE const* const litLimit_w = litLimit - WILDCOPY_OVERLENGTH; BYTE const* const litEnd = literals + litLength; #if defined(DEBUGLEVEL) && (DEBUGLEVEL >= 6) static const BYTE* g_start = NULL; if (g_start==NULL) g_start = (const BYTE*)literals; /* note : index only works for compression within a single segment */ { U32 const pos = (U32)((const BYTE*)literals - g_start); DEBUGLOG(6, "Cpos%7u :%3u literals, match%4u bytes at offBase%7u", pos, (U32)litLength, (U32)matchLength, (U32)offBase); } #endif assert((size_t)(seqStorePtr->sequences - seqStorePtr->sequencesStart) < seqStorePtr->maxNbSeq); /* copy Literals */ assert(seqStorePtr->maxNbLit <= 128 KB); assert(seqStorePtr->lit + litLength <= seqStorePtr->litStart + seqStorePtr->maxNbLit); assert(literals + litLength <= litLimit); if (litEnd <= litLimit_w) { /* Common case we can use wildcopy. * First copy 16 bytes, because literals are likely short. */ ZSTD_STATIC_ASSERT(WILDCOPY_OVERLENGTH >= 16); ZSTD_copy16(seqStorePtr->lit, literals); if (litLength > 16) { ZSTD_wildcopy(seqStorePtr->lit+16, literals+16, (ptrdiff_t)litLength-16, ZSTD_no_overlap); } } else { ZSTD_safecopyLiterals(seqStorePtr->lit, literals, litEnd, litLimit_w); } seqStorePtr->lit += litLength; /* literal Length */ if (litLength>0xFFFF) { assert(seqStorePtr->longLengthType == ZSTD_llt_none); /* there can only be a single long length */ seqStorePtr->longLengthType = ZSTD_llt_literalLength; seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart); } seqStorePtr->sequences[0].litLength = (U16)litLength; /* match offset */ seqStorePtr->sequences[0].offBase = offBase; /* match Length */ assert(matchLength >= MINMATCH); { size_t const mlBase = matchLength - MINMATCH; if (mlBase>0xFFFF) { assert(seqStorePtr->longLengthType == ZSTD_llt_none); /* there can only be a single long length */ seqStorePtr->longLengthType = ZSTD_llt_matchLength; seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart); } seqStorePtr->sequences[0].mlBase = (U16)mlBase; } seqStorePtr->sequences++; } /* ZSTD_updateRep() : * updates in-place @rep (array of repeat offsets) * @offBase : sum-type, using numeric representation of ZSTD_storeSeq() */ MEM_STATIC void ZSTD_updateRep(U32 rep[ZSTD_REP_NUM], U32 const offBase, U32 const ll0) { if (OFFBASE_IS_OFFSET(offBase)) { /* full offset */ rep[2] = rep[1]; rep[1] = rep[0]; rep[0] = OFFBASE_TO_OFFSET(offBase); } else { /* repcode */ U32 const repCode = OFFBASE_TO_REPCODE(offBase) - 1 + ll0; if (repCode > 0) { /* note : if repCode==0, no change */ U32 const currentOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode]; rep[2] = (repCode >= 2) ? rep[1] : rep[2]; rep[1] = rep[0]; rep[0] = currentOffset; } else { /* repCode == 0 */ /* nothing to do */ } } } typedef struct repcodes_s { U32 rep[3]; } repcodes_t; MEM_STATIC repcodes_t ZSTD_newRep(U32 const rep[ZSTD_REP_NUM], U32 const offBase, U32 const ll0) { repcodes_t newReps; ZSTD_memcpy(&newReps, rep, sizeof(newReps)); ZSTD_updateRep(newReps.rep, offBase, ll0); return newReps; } /*-************************************* * Match length counter ***************************************/ MEM_STATIC size_t ZSTD_count(const BYTE* pIn, const BYTE* pMatch, const BYTE* const pInLimit) { const BYTE* const pStart = pIn; const BYTE* const pInLoopLimit = pInLimit - (sizeof(size_t)-1); if (pIn < pInLoopLimit) { { size_t const diff = MEM_readST(pMatch) ^ MEM_readST(pIn); if (diff) return ZSTD_NbCommonBytes(diff); } pIn+=sizeof(size_t); pMatch+=sizeof(size_t); while (pIn < pInLoopLimit) { size_t const diff = MEM_readST(pMatch) ^ MEM_readST(pIn); if (!diff) { pIn+=sizeof(size_t); pMatch+=sizeof(size_t); continue; } pIn += ZSTD_NbCommonBytes(diff); return (size_t)(pIn - pStart); } } if (MEM_64bits() && (pIn<(pInLimit-3)) && (MEM_read32(pMatch) == MEM_read32(pIn))) { pIn+=4; pMatch+=4; } if ((pIn<(pInLimit-1)) && (MEM_read16(pMatch) == MEM_read16(pIn))) { pIn+=2; pMatch+=2; } if ((pIn> (32-h) ; } MEM_STATIC size_t ZSTD_hash3Ptr(const void* ptr, U32 h) { return ZSTD_hash3(MEM_readLE32(ptr), h, 0); } /* only in zstd_opt.h */ MEM_STATIC size_t ZSTD_hash3PtrS(const void* ptr, U32 h, U32 s) { return ZSTD_hash3(MEM_readLE32(ptr), h, s); } static const U32 prime4bytes = 2654435761U; static U32 ZSTD_hash4(U32 u, U32 h, U32 s) { assert(h <= 32); return ((u * prime4bytes) ^ s) >> (32-h) ; } static size_t ZSTD_hash4Ptr(const void* ptr, U32 h) { return ZSTD_hash4(MEM_readLE32(ptr), h, 0); } static size_t ZSTD_hash4PtrS(const void* ptr, U32 h, U32 s) { return ZSTD_hash4(MEM_readLE32(ptr), h, s); } static const U64 prime5bytes = 889523592379ULL; static size_t ZSTD_hash5(U64 u, U32 h, U64 s) { assert(h <= 64); return (size_t)((((u << (64-40)) * prime5bytes) ^ s) >> (64-h)) ; } static size_t ZSTD_hash5Ptr(const void* p, U32 h) { return ZSTD_hash5(MEM_readLE64(p), h, 0); } static size_t ZSTD_hash5PtrS(const void* p, U32 h, U64 s) { return ZSTD_hash5(MEM_readLE64(p), h, s); } static const U64 prime6bytes = 227718039650203ULL; static size_t ZSTD_hash6(U64 u, U32 h, U64 s) { assert(h <= 64); return (size_t)((((u << (64-48)) * prime6bytes) ^ s) >> (64-h)) ; } static size_t ZSTD_hash6Ptr(const void* p, U32 h) { return ZSTD_hash6(MEM_readLE64(p), h, 0); } static size_t ZSTD_hash6PtrS(const void* p, U32 h, U64 s) { return ZSTD_hash6(MEM_readLE64(p), h, s); } static const U64 prime7bytes = 58295818150454627ULL; static size_t ZSTD_hash7(U64 u, U32 h, U64 s) { assert(h <= 64); return (size_t)((((u << (64-56)) * prime7bytes) ^ s) >> (64-h)) ; } static size_t ZSTD_hash7Ptr(const void* p, U32 h) { return ZSTD_hash7(MEM_readLE64(p), h, 0); } static size_t ZSTD_hash7PtrS(const void* p, U32 h, U64 s) { return ZSTD_hash7(MEM_readLE64(p), h, s); } static const U64 prime8bytes = 0xCF1BBCDCB7A56463ULL; static size_t ZSTD_hash8(U64 u, U32 h, U64 s) { assert(h <= 64); return (size_t)((((u) * prime8bytes) ^ s) >> (64-h)) ; } static size_t ZSTD_hash8Ptr(const void* p, U32 h) { return ZSTD_hash8(MEM_readLE64(p), h, 0); } static size_t ZSTD_hash8PtrS(const void* p, U32 h, U64 s) { return ZSTD_hash8(MEM_readLE64(p), h, s); } MEM_STATIC FORCE_INLINE_ATTR size_t ZSTD_hashPtr(const void* p, U32 hBits, U32 mls) { /* Although some of these hashes do support hBits up to 64, some do not. * To be on the safe side, always avoid hBits > 32. */ assert(hBits <= 32); switch(mls) { default: case 4: return ZSTD_hash4Ptr(p, hBits); case 5: return ZSTD_hash5Ptr(p, hBits); case 6: return ZSTD_hash6Ptr(p, hBits); case 7: return ZSTD_hash7Ptr(p, hBits); case 8: return ZSTD_hash8Ptr(p, hBits); } } MEM_STATIC FORCE_INLINE_ATTR size_t ZSTD_hashPtrSalted(const void* p, U32 hBits, U32 mls, const U64 hashSalt) { /* Although some of these hashes do support hBits up to 64, some do not. * To be on the safe side, always avoid hBits > 32. */ assert(hBits <= 32); switch(mls) { default: case 4: return ZSTD_hash4PtrS(p, hBits, (U32)hashSalt); case 5: return ZSTD_hash5PtrS(p, hBits, hashSalt); case 6: return ZSTD_hash6PtrS(p, hBits, hashSalt); case 7: return ZSTD_hash7PtrS(p, hBits, hashSalt); case 8: return ZSTD_hash8PtrS(p, hBits, hashSalt); } } /** ZSTD_ipow() : * Return base^exponent. */ static U64 ZSTD_ipow(U64 base, U64 exponent) { U64 power = 1; while (exponent) { if (exponent & 1) power *= base; exponent >>= 1; base *= base; } return power; } #define ZSTD_ROLL_HASH_CHAR_OFFSET 10 /** ZSTD_rollingHash_append() : * Add the buffer to the hash value. */ static U64 ZSTD_rollingHash_append(U64 hash, void const* buf, size_t size) { BYTE const* istart = (BYTE const*)buf; size_t pos; for (pos = 0; pos < size; ++pos) { hash *= prime8bytes; hash += istart[pos] + ZSTD_ROLL_HASH_CHAR_OFFSET; } return hash; } /** ZSTD_rollingHash_compute() : * Compute the rolling hash value of the buffer. */ MEM_STATIC U64 ZSTD_rollingHash_compute(void const* buf, size_t size) { return ZSTD_rollingHash_append(0, buf, size); } /** ZSTD_rollingHash_primePower() : * Compute the primePower to be passed to ZSTD_rollingHash_rotate() for a hash * over a window of length bytes. */ MEM_STATIC U64 ZSTD_rollingHash_primePower(U32 length) { return ZSTD_ipow(prime8bytes, length - 1); } /** ZSTD_rollingHash_rotate() : * Rotate the rolling hash by one byte. */ MEM_STATIC U64 ZSTD_rollingHash_rotate(U64 hash, BYTE toRemove, BYTE toAdd, U64 primePower) { hash -= (toRemove + ZSTD_ROLL_HASH_CHAR_OFFSET) * primePower; hash *= prime8bytes; hash += toAdd + ZSTD_ROLL_HASH_CHAR_OFFSET; return hash; } /*-************************************* * Round buffer management ***************************************/ #if (ZSTD_WINDOWLOG_MAX_64 > 31) # error "ZSTD_WINDOWLOG_MAX is too large : would overflow ZSTD_CURRENT_MAX" #endif /* Max current allowed */ #define ZSTD_CURRENT_MAX ((3U << 29) + (1U << ZSTD_WINDOWLOG_MAX)) /* Maximum chunk size before overflow correction needs to be called again */ #define ZSTD_CHUNKSIZE_MAX \ ( ((U32)-1) /* Maximum ending current index */ \ - ZSTD_CURRENT_MAX) /* Maximum beginning lowLimit */ /** * ZSTD_window_clear(): * Clears the window containing the history by simply setting it to empty. */ MEM_STATIC void ZSTD_window_clear(ZSTD_window_t* window) { size_t const endT = (size_t)(window->nextSrc - window->base); U32 const end = (U32)endT; window->lowLimit = end; window->dictLimit = end; } MEM_STATIC U32 ZSTD_window_isEmpty(ZSTD_window_t const window) { return window.dictLimit == ZSTD_WINDOW_START_INDEX && window.lowLimit == ZSTD_WINDOW_START_INDEX && (window.nextSrc - window.base) == ZSTD_WINDOW_START_INDEX; } /** * ZSTD_window_hasExtDict(): * Returns non-zero if the window has a non-empty extDict. */ MEM_STATIC U32 ZSTD_window_hasExtDict(ZSTD_window_t const window) { return window.lowLimit < window.dictLimit; } /** * ZSTD_matchState_dictMode(): * Inspects the provided matchState and figures out what dictMode should be * passed to the compressor. */ MEM_STATIC ZSTD_dictMode_e ZSTD_matchState_dictMode(const ZSTD_matchState_t *ms) { return ZSTD_window_hasExtDict(ms->window) ? ZSTD_extDict : ms->dictMatchState != NULL ? (ms->dictMatchState->dedicatedDictSearch ? ZSTD_dedicatedDictSearch : ZSTD_dictMatchState) : ZSTD_noDict; } /* Defining this macro to non-zero tells zstd to run the overflow correction * code much more frequently. This is very inefficient, and should only be * used for tests and fuzzers. */ #ifndef ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY # ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION # define ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY 1 # else # define ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY 0 # endif #endif /** * ZSTD_window_canOverflowCorrect(): * Returns non-zero if the indices are large enough for overflow correction * to work correctly without impacting compression ratio. */ MEM_STATIC U32 ZSTD_window_canOverflowCorrect(ZSTD_window_t const window, U32 cycleLog, U32 maxDist, U32 loadedDictEnd, void const* src) { U32 const cycleSize = 1u << cycleLog; U32 const curr = (U32)((BYTE const*)src - window.base); U32 const minIndexToOverflowCorrect = cycleSize + MAX(maxDist, cycleSize) + ZSTD_WINDOW_START_INDEX; /* Adjust the min index to backoff the overflow correction frequency, * so we don't waste too much CPU in overflow correction. If this * computation overflows we don't really care, we just need to make * sure it is at least minIndexToOverflowCorrect. */ U32 const adjustment = window.nbOverflowCorrections + 1; U32 const adjustedIndex = MAX(minIndexToOverflowCorrect * adjustment, minIndexToOverflowCorrect); U32 const indexLargeEnough = curr > adjustedIndex; /* Only overflow correct early if the dictionary is invalidated already, * so we don't hurt compression ratio. */ U32 const dictionaryInvalidated = curr > maxDist + loadedDictEnd; return indexLargeEnough && dictionaryInvalidated; } /** * ZSTD_window_needOverflowCorrection(): * Returns non-zero if the indices are getting too large and need overflow * protection. */ MEM_STATIC U32 ZSTD_window_needOverflowCorrection(ZSTD_window_t const window, U32 cycleLog, U32 maxDist, U32 loadedDictEnd, void const* src, void const* srcEnd) { U32 const curr = (U32)((BYTE const*)srcEnd - window.base); if (ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY) { if (ZSTD_window_canOverflowCorrect(window, cycleLog, maxDist, loadedDictEnd, src)) { return 1; } } return curr > ZSTD_CURRENT_MAX; } /** * ZSTD_window_correctOverflow(): * Reduces the indices to protect from index overflow. * Returns the correction made to the indices, which must be applied to every * stored index. * * The least significant cycleLog bits of the indices must remain the same, * which may be 0. Every index up to maxDist in the past must be valid. */ MEM_STATIC U32 ZSTD_window_correctOverflow(ZSTD_window_t* window, U32 cycleLog, U32 maxDist, void const* src) { /* preemptive overflow correction: * 1. correction is large enough: * lowLimit > (3<<29) ==> current > 3<<29 + 1< (3<<29 + 1< (3<<29) - (1< (3<<29) - (1<<30) (NOTE: chainLog <= 30) * > 1<<29 * * 2. (ip+ZSTD_CHUNKSIZE_MAX - cctx->base) doesn't overflow: * After correction, current is less than (1<base < 1<<32. * 3. (cctx->lowLimit + 1< 3<<29 + 1<base); U32 const currentCycle = curr & cycleMask; /* Ensure newCurrent - maxDist >= ZSTD_WINDOW_START_INDEX. */ U32 const currentCycleCorrection = currentCycle < ZSTD_WINDOW_START_INDEX ? MAX(cycleSize, ZSTD_WINDOW_START_INDEX) : 0; U32 const newCurrent = currentCycle + currentCycleCorrection + MAX(maxDist, cycleSize); U32 const correction = curr - newCurrent; /* maxDist must be a power of two so that: * (newCurrent & cycleMask) == (curr & cycleMask) * This is required to not corrupt the chains / binary tree. */ assert((maxDist & (maxDist - 1)) == 0); assert((curr & cycleMask) == (newCurrent & cycleMask)); assert(curr > newCurrent); if (!ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY) { /* Loose bound, should be around 1<<29 (see above) */ assert(correction > 1<<28); } window->base += correction; window->dictBase += correction; if (window->lowLimit < correction + ZSTD_WINDOW_START_INDEX) { window->lowLimit = ZSTD_WINDOW_START_INDEX; } else { window->lowLimit -= correction; } if (window->dictLimit < correction + ZSTD_WINDOW_START_INDEX) { window->dictLimit = ZSTD_WINDOW_START_INDEX; } else { window->dictLimit -= correction; } /* Ensure we can still reference the full window. */ assert(newCurrent >= maxDist); assert(newCurrent - maxDist >= ZSTD_WINDOW_START_INDEX); /* Ensure that lowLimit and dictLimit didn't underflow. */ assert(window->lowLimit <= newCurrent); assert(window->dictLimit <= newCurrent); ++window->nbOverflowCorrections; DEBUGLOG(4, "Correction of 0x%x bytes to lowLimit=0x%x", correction, window->lowLimit); return correction; } /** * ZSTD_window_enforceMaxDist(): * Updates lowLimit so that: * (srcEnd - base) - lowLimit == maxDist + loadedDictEnd * * It ensures index is valid as long as index >= lowLimit. * This must be called before a block compression call. * * loadedDictEnd is only defined if a dictionary is in use for current compression. * As the name implies, loadedDictEnd represents the index at end of dictionary. * The value lies within context's referential, it can be directly compared to blockEndIdx. * * If loadedDictEndPtr is NULL, no dictionary is in use, and we use loadedDictEnd == 0. * If loadedDictEndPtr is not NULL, we set it to zero after updating lowLimit. * This is because dictionaries are allowed to be referenced fully * as long as the last byte of the dictionary is in the window. * Once input has progressed beyond window size, dictionary cannot be referenced anymore. * * In normal dict mode, the dictionary lies between lowLimit and dictLimit. * In dictMatchState mode, lowLimit and dictLimit are the same, * and the dictionary is below them. * forceWindow and dictMatchState are therefore incompatible. */ MEM_STATIC void ZSTD_window_enforceMaxDist(ZSTD_window_t* window, const void* blockEnd, U32 maxDist, U32* loadedDictEndPtr, const ZSTD_matchState_t** dictMatchStatePtr) { U32 const blockEndIdx = (U32)((BYTE const*)blockEnd - window->base); U32 const loadedDictEnd = (loadedDictEndPtr != NULL) ? *loadedDictEndPtr : 0; DEBUGLOG(5, "ZSTD_window_enforceMaxDist: blockEndIdx=%u, maxDist=%u, loadedDictEnd=%u", (unsigned)blockEndIdx, (unsigned)maxDist, (unsigned)loadedDictEnd); /* - When there is no dictionary : loadedDictEnd == 0. In which case, the test (blockEndIdx > maxDist) is merely to avoid overflowing next operation `newLowLimit = blockEndIdx - maxDist`. - When there is a standard dictionary : Index referential is copied from the dictionary, which means it starts from 0. In which case, loadedDictEnd == dictSize, and it makes sense to compare `blockEndIdx > maxDist + dictSize` since `blockEndIdx` also starts from zero. - When there is an attached dictionary : loadedDictEnd is expressed within the referential of the context, so it can be directly compared against blockEndIdx. */ if (blockEndIdx > maxDist + loadedDictEnd) { U32 const newLowLimit = blockEndIdx - maxDist; if (window->lowLimit < newLowLimit) window->lowLimit = newLowLimit; if (window->dictLimit < window->lowLimit) { DEBUGLOG(5, "Update dictLimit to match lowLimit, from %u to %u", (unsigned)window->dictLimit, (unsigned)window->lowLimit); window->dictLimit = window->lowLimit; } /* On reaching window size, dictionaries are invalidated */ if (loadedDictEndPtr) *loadedDictEndPtr = 0; if (dictMatchStatePtr) *dictMatchStatePtr = NULL; } } /* Similar to ZSTD_window_enforceMaxDist(), * but only invalidates dictionary * when input progresses beyond window size. * assumption : loadedDictEndPtr and dictMatchStatePtr are valid (non NULL) * loadedDictEnd uses same referential as window->base * maxDist is the window size */ MEM_STATIC void ZSTD_checkDictValidity(const ZSTD_window_t* window, const void* blockEnd, U32 maxDist, U32* loadedDictEndPtr, const ZSTD_matchState_t** dictMatchStatePtr) { assert(loadedDictEndPtr != NULL); assert(dictMatchStatePtr != NULL); { U32 const blockEndIdx = (U32)((BYTE const*)blockEnd - window->base); U32 const loadedDictEnd = *loadedDictEndPtr; DEBUGLOG(5, "ZSTD_checkDictValidity: blockEndIdx=%u, maxDist=%u, loadedDictEnd=%u", (unsigned)blockEndIdx, (unsigned)maxDist, (unsigned)loadedDictEnd); assert(blockEndIdx >= loadedDictEnd); if (blockEndIdx > loadedDictEnd + maxDist || loadedDictEnd != window->dictLimit) { /* On reaching window size, dictionaries are invalidated. * For simplification, if window size is reached anywhere within next block, * the dictionary is invalidated for the full block. * * We also have to invalidate the dictionary if ZSTD_window_update() has detected * non-contiguous segments, which means that loadedDictEnd != window->dictLimit. * loadedDictEnd may be 0, if forceWindow is true, but in that case we never use * dictMatchState, so setting it to NULL is not a problem. */ DEBUGLOG(6, "invalidating dictionary for current block (distance > windowSize)"); *loadedDictEndPtr = 0; *dictMatchStatePtr = NULL; } else { if (*loadedDictEndPtr != 0) { DEBUGLOG(6, "dictionary considered valid for current block"); } } } } MEM_STATIC void ZSTD_window_init(ZSTD_window_t* window) { ZSTD_memset(window, 0, sizeof(*window)); window->base = (BYTE const*)" "; window->dictBase = (BYTE const*)" "; ZSTD_STATIC_ASSERT(ZSTD_DUBT_UNSORTED_MARK < ZSTD_WINDOW_START_INDEX); /* Start above ZSTD_DUBT_UNSORTED_MARK */ window->dictLimit = ZSTD_WINDOW_START_INDEX; /* start from >0, so that 1st position is valid */ window->lowLimit = ZSTD_WINDOW_START_INDEX; /* it ensures first and later CCtx usages compress the same */ window->nextSrc = window->base + ZSTD_WINDOW_START_INDEX; /* see issue #1241 */ window->nbOverflowCorrections = 0; } /** * ZSTD_window_update(): * Updates the window by appending [src, src + srcSize) to the window. * If it is not contiguous, the current prefix becomes the extDict, and we * forget about the extDict. Handles overlap of the prefix and extDict. * Returns non-zero if the segment is contiguous. */ MEM_STATIC U32 ZSTD_window_update(ZSTD_window_t* window, void const* src, size_t srcSize, int forceNonContiguous) { BYTE const* const ip = (BYTE const*)src; U32 contiguous = 1; DEBUGLOG(5, "ZSTD_window_update"); if (srcSize == 0) return contiguous; assert(window->base != NULL); assert(window->dictBase != NULL); /* Check if blocks follow each other */ if (src != window->nextSrc || forceNonContiguous) { /* not contiguous */ size_t const distanceFromBase = (size_t)(window->nextSrc - window->base); DEBUGLOG(5, "Non contiguous blocks, new segment starts at %u", window->dictLimit); window->lowLimit = window->dictLimit; assert(distanceFromBase == (size_t)(U32)distanceFromBase); /* should never overflow */ window->dictLimit = (U32)distanceFromBase; window->dictBase = window->base; window->base = ip - distanceFromBase; /* ms->nextToUpdate = window->dictLimit; */ if (window->dictLimit - window->lowLimit < HASH_READ_SIZE) window->lowLimit = window->dictLimit; /* too small extDict */ contiguous = 0; } window->nextSrc = ip + srcSize; /* if input and dictionary overlap : reduce dictionary (area presumed modified by input) */ if ( (ip+srcSize > window->dictBase + window->lowLimit) & (ip < window->dictBase + window->dictLimit)) { ptrdiff_t const highInputIdx = (ip + srcSize) - window->dictBase; U32 const lowLimitMax = (highInputIdx > (ptrdiff_t)window->dictLimit) ? window->dictLimit : (U32)highInputIdx; window->lowLimit = lowLimitMax; DEBUGLOG(5, "Overlapping extDict and input : new lowLimit = %u", window->lowLimit); } return contiguous; } /** * Returns the lowest allowed match index. It may either be in the ext-dict or the prefix. */ MEM_STATIC U32 ZSTD_getLowestMatchIndex(const ZSTD_matchState_t* ms, U32 curr, unsigned windowLog) { U32 const maxDistance = 1U << windowLog; U32 const lowestValid = ms->window.lowLimit; U32 const withinWindow = (curr - lowestValid > maxDistance) ? curr - maxDistance : lowestValid; U32 const isDictionary = (ms->loadedDictEnd != 0); /* When using a dictionary the entire dictionary is valid if a single byte of the dictionary * is within the window. We invalidate the dictionary (and set loadedDictEnd to 0) when it isn't * valid for the entire block. So this check is sufficient to find the lowest valid match index. */ U32 const matchLowest = isDictionary ? lowestValid : withinWindow; return matchLowest; } /** * Returns the lowest allowed match index in the prefix. */ MEM_STATIC U32 ZSTD_getLowestPrefixIndex(const ZSTD_matchState_t* ms, U32 curr, unsigned windowLog) { U32 const maxDistance = 1U << windowLog; U32 const lowestValid = ms->window.dictLimit; U32 const withinWindow = (curr - lowestValid > maxDistance) ? curr - maxDistance : lowestValid; U32 const isDictionary = (ms->loadedDictEnd != 0); /* When computing the lowest prefix index we need to take the dictionary into account to handle * the edge case where the dictionary and the source are contiguous in memory. */ U32 const matchLowest = isDictionary ? lowestValid : withinWindow; return matchLowest; } /* debug functions */ #if (DEBUGLEVEL>=2) MEM_STATIC double ZSTD_fWeight(U32 rawStat) { U32 const fp_accuracy = 8; U32 const fp_multiplier = (1 << fp_accuracy); U32 const newStat = rawStat + 1; U32 const hb = ZSTD_highbit32(newStat); U32 const BWeight = hb * fp_multiplier; U32 const FWeight = (newStat << fp_accuracy) >> hb; U32 const weight = BWeight + FWeight; assert(hb + fp_accuracy < 31); return (double)weight / fp_multiplier; } /* display a table content, * listing each element, its frequency, and its predicted bit cost */ MEM_STATIC void ZSTD_debugTable(const U32* table, U32 max) { unsigned u, sum; for (u=0, sum=0; u<=max; u++) sum += table[u]; DEBUGLOG(2, "total nb elts: %u", sum); for (u=0; u<=max; u++) { DEBUGLOG(2, "%2u: %5u (%.2f)", u, table[u], ZSTD_fWeight(sum) - ZSTD_fWeight(table[u]) ); } } #endif /* Short Cache */ /* Normally, zstd matchfinders follow this flow: * 1. Compute hash at ip * 2. Load index from hashTable[hash] * 3. Check if *ip == *(base + index) * In dictionary compression, loading *(base + index) is often an L2 or even L3 miss. * * Short cache is an optimization which allows us to avoid step 3 most of the time * when the data doesn't actually match. With short cache, the flow becomes: * 1. Compute (hash, currentTag) at ip. currentTag is an 8-bit independent hash at ip. * 2. Load (index, matchTag) from hashTable[hash]. See ZSTD_writeTaggedIndex to understand how this works. * 3. Only if currentTag == matchTag, check *ip == *(base + index). Otherwise, continue. * * Currently, short cache is only implemented in CDict hashtables. Thus, its use is limited to * dictMatchState matchfinders. */ #define ZSTD_SHORT_CACHE_TAG_BITS 8 #define ZSTD_SHORT_CACHE_TAG_MASK ((1u << ZSTD_SHORT_CACHE_TAG_BITS) - 1) /* Helper function for ZSTD_fillHashTable and ZSTD_fillDoubleHashTable. * Unpacks hashAndTag into (hash, tag), then packs (index, tag) into hashTable[hash]. */ MEM_STATIC void ZSTD_writeTaggedIndex(U32* const hashTable, size_t hashAndTag, U32 index) { size_t const hash = hashAndTag >> ZSTD_SHORT_CACHE_TAG_BITS; U32 const tag = (U32)(hashAndTag & ZSTD_SHORT_CACHE_TAG_MASK); assert(index >> (32 - ZSTD_SHORT_CACHE_TAG_BITS) == 0); hashTable[hash] = (index << ZSTD_SHORT_CACHE_TAG_BITS) | tag; } /* Helper function for short cache matchfinders. * Unpacks tag1 and tag2 from lower bits of packedTag1 and packedTag2, then checks if the tags match. */ MEM_STATIC int ZSTD_comparePackedTags(size_t packedTag1, size_t packedTag2) { U32 const tag1 = packedTag1 & ZSTD_SHORT_CACHE_TAG_MASK; U32 const tag2 = packedTag2 & ZSTD_SHORT_CACHE_TAG_MASK; return tag1 == tag2; } #if defined (__cplusplus) } #endif /* =============================================================== * Shared internal declarations * These prototypes may be called from sources not in lib/compress * =============================================================== */ /* ZSTD_loadCEntropy() : * dict : must point at beginning of a valid zstd dictionary. * return : size of dictionary header (size of magic number + dict ID + entropy tables) * assumptions : magic number supposed already checked * and dictSize >= 8 */ size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace, const void* const dict, size_t dictSize); void ZSTD_reset_compressedBlockState(ZSTD_compressedBlockState_t* bs); /* ============================================================== * Private declarations * These prototypes shall only be called from within lib/compress * ============================================================== */ /* ZSTD_getCParamsFromCCtxParams() : * cParams are built depending on compressionLevel, src size hints, * LDM and manually set compression parameters. * Note: srcSizeHint == 0 means 0! */ ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams( const ZSTD_CCtx_params* CCtxParams, U64 srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode); /*! ZSTD_initCStream_internal() : * Private use only. Init streaming operation. * expects params to be valid. * must receive dict, or cdict, or none, but not both. * @return : 0, or an error code */ size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs, const void* dict, size_t dictSize, const ZSTD_CDict* cdict, const ZSTD_CCtx_params* params, unsigned long long pledgedSrcSize); void ZSTD_resetSeqStore(seqStore_t* ssPtr); /*! ZSTD_getCParamsFromCDict() : * as the name implies */ ZSTD_compressionParameters ZSTD_getCParamsFromCDict(const ZSTD_CDict* cdict); /* ZSTD_compressBegin_advanced_internal() : * Private use only. To be called from zstdmt_compress.c. */ size_t ZSTD_compressBegin_advanced_internal(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_dictContentType_e dictContentType, ZSTD_dictTableLoadMethod_e dtlm, const ZSTD_CDict* cdict, const ZSTD_CCtx_params* params, unsigned long long pledgedSrcSize); /* ZSTD_compress_advanced_internal() : * Private use only. To be called from zstdmt_compress.c. */ size_t ZSTD_compress_advanced_internal(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const void* dict,size_t dictSize, const ZSTD_CCtx_params* params); /* ZSTD_writeLastEmptyBlock() : * output an empty Block with end-of-frame mark to complete a frame * @return : size of data written into `dst` (== ZSTD_blockHeaderSize (defined in zstd_internal.h)) * or an error code if `dstCapacity` is too small ( 1 */ U32 ZSTD_cycleLog(U32 hashLog, ZSTD_strategy strat); /** ZSTD_CCtx_trace() : * Trace the end of a compression call. */ void ZSTD_CCtx_trace(ZSTD_CCtx* cctx, size_t extraCSize); /* Returns 0 on success, and a ZSTD_error otherwise. This function scans through an array of * ZSTD_Sequence, storing the sequences it finds, until it reaches a block delimiter. * Note that the block delimiter must include the last literals of the block. */ size_t ZSTD_copySequencesToSeqStoreExplicitBlockDelim(ZSTD_CCtx* cctx, ZSTD_sequencePosition* seqPos, const ZSTD_Sequence* const inSeqs, size_t inSeqsSize, const void* src, size_t blockSize, ZSTD_paramSwitch_e externalRepSearch); /* Returns the number of bytes to move the current read position back by. * Only non-zero if we ended up splitting a sequence. * Otherwise, it may return a ZSTD error if something went wrong. * * This function will attempt to scan through blockSize bytes * represented by the sequences in @inSeqs, * storing any (partial) sequences. * * Occasionally, we may want to change the actual number of bytes we consumed from inSeqs to * avoid splitting a match, or to avoid splitting a match such that it would produce a match * smaller than MINMATCH. In this case, we return the number of bytes that we didn't read from this block. */ size_t ZSTD_copySequencesToSeqStoreNoBlockDelim(ZSTD_CCtx* cctx, ZSTD_sequencePosition* seqPos, const ZSTD_Sequence* const inSeqs, size_t inSeqsSize, const void* src, size_t blockSize, ZSTD_paramSwitch_e externalRepSearch); /* =============================================================== * Deprecated definitions that are still used internally to avoid * deprecation warnings. These functions are exactly equivalent to * their public variants, but avoid the deprecation warnings. * =============================================================== */ size_t ZSTD_compressBegin_usingCDict_deprecated(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); size_t ZSTD_compressContinue_public(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); size_t ZSTD_compressEnd_public(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); size_t ZSTD_compressBlock_deprecated(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); #endif /* ZSTD_COMPRESS_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_compress_superblock.h0000644000175200007730000000222014515254731030450 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_COMPRESS_ADVANCED_H #define ZSTD_COMPRESS_ADVANCED_H /*-************************************* * Dependencies ***************************************/ #include "../zstd.h" /* ZSTD_CCtx */ /*-************************************* * Target Compressed Block Size ***************************************/ /* ZSTD_compressSuperBlock() : * Used to compress a super block when targetCBlockSize is being used. * The given block will be compressed into multiple sub blocks that are around targetCBlockSize. */ size_t ZSTD_compressSuperBlock(ZSTD_CCtx* zc, void* dst, size_t dstCapacity, void const* src, size_t srcSize, unsigned lastBlock); #endif /* ZSTD_COMPRESS_ADVANCED_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstdmt_compress.h0000644000175200007730000001106714515254731026571 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTDMT_COMPRESS_H #define ZSTDMT_COMPRESS_H #if defined (__cplusplus) extern "C" { #endif /* Note : This is an internal API. * These APIs used to be exposed with ZSTDLIB_API, * because it used to be the only way to invoke MT compression. * Now, you must use ZSTD_compress2 and ZSTD_compressStream2() instead. * * This API requires ZSTD_MULTITHREAD to be defined during compilation, * otherwise ZSTDMT_createCCtx*() will fail. */ /* === Dependencies === */ #include "../common/zstd_deps.h" /* size_t */ #define ZSTD_STATIC_LINKING_ONLY /* ZSTD_parameters */ #include "../zstd.h" /* ZSTD_inBuffer, ZSTD_outBuffer, ZSTDLIB_API */ /* === Constants === */ #ifndef ZSTDMT_NBWORKERS_MAX /* a different value can be selected at compile time */ # define ZSTDMT_NBWORKERS_MAX ((sizeof(void*)==4) /*32-bit*/ ? 64 : 256) #endif #ifndef ZSTDMT_JOBSIZE_MIN /* a different value can be selected at compile time */ # define ZSTDMT_JOBSIZE_MIN (512 KB) #endif #define ZSTDMT_JOBLOG_MAX (MEM_32bits() ? 29 : 30) #define ZSTDMT_JOBSIZE_MAX (MEM_32bits() ? (512 MB) : (1024 MB)) /* ======================================================== * === Private interface, for use by ZSTD_compress.c === * === Not exposed in libzstd. Never invoke directly === * ======================================================== */ /* === Memory management === */ typedef struct ZSTDMT_CCtx_s ZSTDMT_CCtx; /* Requires ZSTD_MULTITHREAD to be defined during compilation, otherwise it will return NULL. */ ZSTDMT_CCtx* ZSTDMT_createCCtx_advanced(unsigned nbWorkers, ZSTD_customMem cMem, ZSTD_threadPool *pool); size_t ZSTDMT_freeCCtx(ZSTDMT_CCtx* mtctx); size_t ZSTDMT_sizeof_CCtx(ZSTDMT_CCtx* mtctx); /* === Streaming functions === */ size_t ZSTDMT_nextInputSizeHint(const ZSTDMT_CCtx* mtctx); /*! ZSTDMT_initCStream_internal() : * Private use only. Init streaming operation. * expects params to be valid. * must receive dict, or cdict, or none, but not both. * mtctx can be freshly constructed or reused from a prior compression. * If mtctx is reused, memory allocations from the prior compression may not be freed, * even if they are not needed for the current compression. * @return : 0, or an error code */ size_t ZSTDMT_initCStream_internal(ZSTDMT_CCtx* mtctx, const void* dict, size_t dictSize, ZSTD_dictContentType_e dictContentType, const ZSTD_CDict* cdict, ZSTD_CCtx_params params, unsigned long long pledgedSrcSize); /*! ZSTDMT_compressStream_generic() : * Combines ZSTDMT_compressStream() with optional ZSTDMT_flushStream() or ZSTDMT_endStream() * depending on flush directive. * @return : minimum amount of data still to be flushed * 0 if fully flushed * or an error code * note : needs to be init using any ZSTD_initCStream*() variant */ size_t ZSTDMT_compressStream_generic(ZSTDMT_CCtx* mtctx, ZSTD_outBuffer* output, ZSTD_inBuffer* input, ZSTD_EndDirective endOp); /*! ZSTDMT_toFlushNow() * Tell how many bytes are ready to be flushed immediately. * Probe the oldest active job (not yet entirely flushed) and check its output buffer. * If return 0, it means there is no active job, * or, it means oldest job is still active, but everything produced has been flushed so far, * therefore flushing is limited by speed of oldest job. */ size_t ZSTDMT_toFlushNow(ZSTDMT_CCtx* mtctx); /*! ZSTDMT_updateCParams_whileCompressing() : * Updates only a selected set of compression parameters, to remain compatible with current frame. * New parameters will be applied to next compression job. */ void ZSTDMT_updateCParams_whileCompressing(ZSTDMT_CCtx* mtctx, const ZSTD_CCtx_params* cctxParams); /*! ZSTDMT_getFrameProgression(): * tells how much data has been consumed (input) and produced (output) for current frame. * able to count progression inside worker threads. */ ZSTD_frameProgression ZSTDMT_getFrameProgression(ZSTDMT_CCtx* mtctx); #if defined (__cplusplus) } #endif #endif /* ZSTDMT_COMPRESS_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/hist.h0000644000175200007730000000657014515254731024303 0ustar rlaboissrlaboiss/* ****************************************************************** * hist : Histogram functions * part of Finite State Entropy project * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy * - Public forum : https://groups.google.com/forum/#!forum/lz4c * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ /* --- dependencies --- */ #include "../common/zstd_deps.h" /* size_t */ /* --- simple histogram functions --- */ /*! HIST_count(): * Provides the precise count of each byte within a table 'count'. * 'count' is a table of unsigned int, of minimum size (*maxSymbolValuePtr+1). * Updates *maxSymbolValuePtr with actual largest symbol value detected. * @return : count of the most frequent symbol (which isn't identified). * or an error code, which can be tested using HIST_isError(). * note : if return == srcSize, there is only one symbol. */ size_t HIST_count(unsigned* count, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize); unsigned HIST_isError(size_t code); /**< tells if a return value is an error code */ /* --- advanced histogram functions --- */ #define HIST_WKSP_SIZE_U32 1024 #define HIST_WKSP_SIZE (HIST_WKSP_SIZE_U32 * sizeof(unsigned)) /** HIST_count_wksp() : * Same as HIST_count(), but using an externally provided scratch buffer. * Benefit is this function will use very little stack space. * `workSpace` is a writable buffer which must be 4-bytes aligned, * `workSpaceSize` must be >= HIST_WKSP_SIZE */ size_t HIST_count_wksp(unsigned* count, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, void* workSpace, size_t workSpaceSize); /** HIST_countFast() : * same as HIST_count(), but blindly trusts that all byte values within src are <= *maxSymbolValuePtr. * This function is unsafe, and will segfault if any value within `src` is `> *maxSymbolValuePtr` */ size_t HIST_countFast(unsigned* count, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize); /** HIST_countFast_wksp() : * Same as HIST_countFast(), but using an externally provided scratch buffer. * `workSpace` is a writable buffer which must be 4-bytes aligned, * `workSpaceSize` must be >= HIST_WKSP_SIZE */ size_t HIST_countFast_wksp(unsigned* count, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, void* workSpace, size_t workSpaceSize); /*! HIST_count_simple() : * Same as HIST_countFast(), this function is unsafe, * and will segfault if any value within `src` is `> *maxSymbolValuePtr`. * It is also a bit slower for large inputs. * However, it does not need any additional memory (not even on stack). * @return : count of the most frequent symbol. * Note this function doesn't produce any error (i.e. it must succeed). */ unsigned HIST_count_simple(unsigned* count, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize); zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/clevels.h0000644000175200007730000001531414515254731024765 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_CLEVELS_H #define ZSTD_CLEVELS_H #define ZSTD_STATIC_LINKING_ONLY /* ZSTD_compressionParameters */ #include "../zstd.h" /*-===== Pre-defined compression levels =====-*/ #define ZSTD_MAX_CLEVEL 22 #ifdef __GNUC__ __attribute__((__unused__)) #endif static const ZSTD_compressionParameters ZSTD_defaultCParameters[4][ZSTD_MAX_CLEVEL+1] = { { /* "default" - for any srcSize > 256 KB */ /* W, C, H, S, L, TL, strat */ { 19, 12, 13, 1, 6, 1, ZSTD_fast }, /* base for negative levels */ { 19, 13, 14, 1, 7, 0, ZSTD_fast }, /* level 1 */ { 20, 15, 16, 1, 6, 0, ZSTD_fast }, /* level 2 */ { 21, 16, 17, 1, 5, 0, ZSTD_dfast }, /* level 3 */ { 21, 18, 18, 1, 5, 0, ZSTD_dfast }, /* level 4 */ { 21, 18, 19, 3, 5, 2, ZSTD_greedy }, /* level 5 */ { 21, 18, 19, 3, 5, 4, ZSTD_lazy }, /* level 6 */ { 21, 19, 20, 4, 5, 8, ZSTD_lazy }, /* level 7 */ { 21, 19, 20, 4, 5, 16, ZSTD_lazy2 }, /* level 8 */ { 22, 20, 21, 4, 5, 16, ZSTD_lazy2 }, /* level 9 */ { 22, 21, 22, 5, 5, 16, ZSTD_lazy2 }, /* level 10 */ { 22, 21, 22, 6, 5, 16, ZSTD_lazy2 }, /* level 11 */ { 22, 22, 23, 6, 5, 32, ZSTD_lazy2 }, /* level 12 */ { 22, 22, 22, 4, 5, 32, ZSTD_btlazy2 }, /* level 13 */ { 22, 22, 23, 5, 5, 32, ZSTD_btlazy2 }, /* level 14 */ { 22, 23, 23, 6, 5, 32, ZSTD_btlazy2 }, /* level 15 */ { 22, 22, 22, 5, 5, 48, ZSTD_btopt }, /* level 16 */ { 23, 23, 22, 5, 4, 64, ZSTD_btopt }, /* level 17 */ { 23, 23, 22, 6, 3, 64, ZSTD_btultra }, /* level 18 */ { 23, 24, 22, 7, 3,256, ZSTD_btultra2}, /* level 19 */ { 25, 25, 23, 7, 3,256, ZSTD_btultra2}, /* level 20 */ { 26, 26, 24, 7, 3,512, ZSTD_btultra2}, /* level 21 */ { 27, 27, 25, 9, 3,999, ZSTD_btultra2}, /* level 22 */ }, { /* for srcSize <= 256 KB */ /* W, C, H, S, L, T, strat */ { 18, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */ { 18, 13, 14, 1, 6, 0, ZSTD_fast }, /* level 1 */ { 18, 14, 14, 1, 5, 0, ZSTD_dfast }, /* level 2 */ { 18, 16, 16, 1, 4, 0, ZSTD_dfast }, /* level 3 */ { 18, 16, 17, 3, 5, 2, ZSTD_greedy }, /* level 4.*/ { 18, 17, 18, 5, 5, 2, ZSTD_greedy }, /* level 5.*/ { 18, 18, 19, 3, 5, 4, ZSTD_lazy }, /* level 6.*/ { 18, 18, 19, 4, 4, 4, ZSTD_lazy }, /* level 7 */ { 18, 18, 19, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */ { 18, 18, 19, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */ { 18, 18, 19, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */ { 18, 18, 19, 5, 4, 12, ZSTD_btlazy2 }, /* level 11.*/ { 18, 19, 19, 7, 4, 12, ZSTD_btlazy2 }, /* level 12.*/ { 18, 18, 19, 4, 4, 16, ZSTD_btopt }, /* level 13 */ { 18, 18, 19, 4, 3, 32, ZSTD_btopt }, /* level 14.*/ { 18, 18, 19, 6, 3,128, ZSTD_btopt }, /* level 15.*/ { 18, 19, 19, 6, 3,128, ZSTD_btultra }, /* level 16.*/ { 18, 19, 19, 8, 3,256, ZSTD_btultra }, /* level 17.*/ { 18, 19, 19, 6, 3,128, ZSTD_btultra2}, /* level 18.*/ { 18, 19, 19, 8, 3,256, ZSTD_btultra2}, /* level 19.*/ { 18, 19, 19, 10, 3,512, ZSTD_btultra2}, /* level 20.*/ { 18, 19, 19, 12, 3,512, ZSTD_btultra2}, /* level 21.*/ { 18, 19, 19, 13, 3,999, ZSTD_btultra2}, /* level 22.*/ }, { /* for srcSize <= 128 KB */ /* W, C, H, S, L, T, strat */ { 17, 12, 12, 1, 5, 1, ZSTD_fast }, /* base for negative levels */ { 17, 12, 13, 1, 6, 0, ZSTD_fast }, /* level 1 */ { 17, 13, 15, 1, 5, 0, ZSTD_fast }, /* level 2 */ { 17, 15, 16, 2, 5, 0, ZSTD_dfast }, /* level 3 */ { 17, 17, 17, 2, 4, 0, ZSTD_dfast }, /* level 4 */ { 17, 16, 17, 3, 4, 2, ZSTD_greedy }, /* level 5 */ { 17, 16, 17, 3, 4, 4, ZSTD_lazy }, /* level 6 */ { 17, 16, 17, 3, 4, 8, ZSTD_lazy2 }, /* level 7 */ { 17, 16, 17, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */ { 17, 16, 17, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */ { 17, 16, 17, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */ { 17, 17, 17, 5, 4, 8, ZSTD_btlazy2 }, /* level 11 */ { 17, 18, 17, 7, 4, 12, ZSTD_btlazy2 }, /* level 12 */ { 17, 18, 17, 3, 4, 12, ZSTD_btopt }, /* level 13.*/ { 17, 18, 17, 4, 3, 32, ZSTD_btopt }, /* level 14.*/ { 17, 18, 17, 6, 3,256, ZSTD_btopt }, /* level 15.*/ { 17, 18, 17, 6, 3,128, ZSTD_btultra }, /* level 16.*/ { 17, 18, 17, 8, 3,256, ZSTD_btultra }, /* level 17.*/ { 17, 18, 17, 10, 3,512, ZSTD_btultra }, /* level 18.*/ { 17, 18, 17, 5, 3,256, ZSTD_btultra2}, /* level 19.*/ { 17, 18, 17, 7, 3,512, ZSTD_btultra2}, /* level 20.*/ { 17, 18, 17, 9, 3,512, ZSTD_btultra2}, /* level 21.*/ { 17, 18, 17, 11, 3,999, ZSTD_btultra2}, /* level 22.*/ }, { /* for srcSize <= 16 KB */ /* W, C, H, S, L, T, strat */ { 14, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */ { 14, 14, 15, 1, 5, 0, ZSTD_fast }, /* level 1 */ { 14, 14, 15, 1, 4, 0, ZSTD_fast }, /* level 2 */ { 14, 14, 15, 2, 4, 0, ZSTD_dfast }, /* level 3 */ { 14, 14, 14, 4, 4, 2, ZSTD_greedy }, /* level 4 */ { 14, 14, 14, 3, 4, 4, ZSTD_lazy }, /* level 5.*/ { 14, 14, 14, 4, 4, 8, ZSTD_lazy2 }, /* level 6 */ { 14, 14, 14, 6, 4, 8, ZSTD_lazy2 }, /* level 7 */ { 14, 14, 14, 8, 4, 8, ZSTD_lazy2 }, /* level 8.*/ { 14, 15, 14, 5, 4, 8, ZSTD_btlazy2 }, /* level 9.*/ { 14, 15, 14, 9, 4, 8, ZSTD_btlazy2 }, /* level 10.*/ { 14, 15, 14, 3, 4, 12, ZSTD_btopt }, /* level 11.*/ { 14, 15, 14, 4, 3, 24, ZSTD_btopt }, /* level 12.*/ { 14, 15, 14, 5, 3, 32, ZSTD_btultra }, /* level 13.*/ { 14, 15, 15, 6, 3, 64, ZSTD_btultra }, /* level 14.*/ { 14, 15, 15, 7, 3,256, ZSTD_btultra }, /* level 15.*/ { 14, 15, 15, 5, 3, 48, ZSTD_btultra2}, /* level 16.*/ { 14, 15, 15, 6, 3,128, ZSTD_btultra2}, /* level 17.*/ { 14, 15, 15, 7, 3,256, ZSTD_btultra2}, /* level 18.*/ { 14, 15, 15, 8, 3,256, ZSTD_btultra2}, /* level 19.*/ { 14, 15, 15, 8, 3,512, ZSTD_btultra2}, /* level 20.*/ { 14, 15, 15, 9, 3,512, ZSTD_btultra2}, /* level 21.*/ { 14, 15, 15, 10, 3,999, ZSTD_btultra2}, /* level 22.*/ }, }; #endif /* ZSTD_CLEVELS_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/huf_compress.c0000644000175200007730000016055614515254731026031 0ustar rlaboissrlaboiss/* ****************************************************************** * Huffman encoder, part of New Generation Entropy library * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy * - Public forum : https://groups.google.com/forum/#!forum/lz4c * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ /* ************************************************************** * Compiler specifics ****************************************************************/ #ifdef _MSC_VER /* Visual Studio */ # pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ #endif /* ************************************************************** * Includes ****************************************************************/ #include "../common/zstd_deps.h" /* ZSTD_memcpy, ZSTD_memset */ #include "../common/compiler.h" #include "../common/bitstream.h" #include "hist.h" #define FSE_STATIC_LINKING_ONLY /* FSE_optimalTableLog_internal */ #include "../common/fse.h" /* header compression */ #include "../common/huf.h" #include "../common/error_private.h" #include "../common/bits.h" /* ZSTD_highbit32 */ /* ************************************************************** * Error Management ****************************************************************/ #define HUF_isError ERR_isError #define HUF_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c) /* use only *after* variable declarations */ /* ************************************************************** * Required declarations ****************************************************************/ typedef struct nodeElt_s { U32 count; U16 parent; BYTE byte; BYTE nbBits; } nodeElt; /* ************************************************************** * Debug Traces ****************************************************************/ #if DEBUGLEVEL >= 2 static size_t showU32(const U32* arr, size_t size) { size_t u; for (u=0; u= add) { assert(add < align); assert(((size_t)aligned & mask) == 0); *workspaceSizePtr -= add; return aligned; } else { *workspaceSizePtr = 0; return NULL; } } /* HUF_compressWeights() : * Same as FSE_compress(), but dedicated to huff0's weights compression. * The use case needs much less stack memory. * Note : all elements within weightTable are supposed to be <= HUF_TABLELOG_MAX. */ #define MAX_FSE_TABLELOG_FOR_HUFF_HEADER 6 typedef struct { FSE_CTable CTable[FSE_CTABLE_SIZE_U32(MAX_FSE_TABLELOG_FOR_HUFF_HEADER, HUF_TABLELOG_MAX)]; U32 scratchBuffer[FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(HUF_TABLELOG_MAX, MAX_FSE_TABLELOG_FOR_HUFF_HEADER)]; unsigned count[HUF_TABLELOG_MAX+1]; S16 norm[HUF_TABLELOG_MAX+1]; } HUF_CompressWeightsWksp; static size_t HUF_compressWeights(void* dst, size_t dstSize, const void* weightTable, size_t wtSize, void* workspace, size_t workspaceSize) { BYTE* const ostart = (BYTE*) dst; BYTE* op = ostart; BYTE* const oend = ostart + dstSize; unsigned maxSymbolValue = HUF_TABLELOG_MAX; U32 tableLog = MAX_FSE_TABLELOG_FOR_HUFF_HEADER; HUF_CompressWeightsWksp* wksp = (HUF_CompressWeightsWksp*)HUF_alignUpWorkspace(workspace, &workspaceSize, ZSTD_ALIGNOF(U32)); if (workspaceSize < sizeof(HUF_CompressWeightsWksp)) return ERROR(GENERIC); /* init conditions */ if (wtSize <= 1) return 0; /* Not compressible */ /* Scan input and build symbol stats */ { unsigned const maxCount = HIST_count_simple(wksp->count, &maxSymbolValue, weightTable, wtSize); /* never fails */ if (maxCount == wtSize) return 1; /* only a single symbol in src : rle */ if (maxCount == 1) return 0; /* each symbol present maximum once => not compressible */ } tableLog = FSE_optimalTableLog(tableLog, wtSize, maxSymbolValue); CHECK_F( FSE_normalizeCount(wksp->norm, tableLog, wksp->count, wtSize, maxSymbolValue, /* useLowProbCount */ 0) ); /* Write table description header */ { CHECK_V_F(hSize, FSE_writeNCount(op, (size_t)(oend-op), wksp->norm, maxSymbolValue, tableLog) ); op += hSize; } /* Compress */ CHECK_F( FSE_buildCTable_wksp(wksp->CTable, wksp->norm, maxSymbolValue, tableLog, wksp->scratchBuffer, sizeof(wksp->scratchBuffer)) ); { CHECK_V_F(cSize, FSE_compress_usingCTable(op, (size_t)(oend - op), weightTable, wtSize, wksp->CTable) ); if (cSize == 0) return 0; /* not enough space for compressed data */ op += cSize; } return (size_t)(op-ostart); } static size_t HUF_getNbBits(HUF_CElt elt) { return elt & 0xFF; } static size_t HUF_getNbBitsFast(HUF_CElt elt) { return elt; } static size_t HUF_getValue(HUF_CElt elt) { return elt & ~(size_t)0xFF; } static size_t HUF_getValueFast(HUF_CElt elt) { return elt; } static void HUF_setNbBits(HUF_CElt* elt, size_t nbBits) { assert(nbBits <= HUF_TABLELOG_ABSOLUTEMAX); *elt = nbBits; } static void HUF_setValue(HUF_CElt* elt, size_t value) { size_t const nbBits = HUF_getNbBits(*elt); if (nbBits > 0) { assert((value >> nbBits) == 0); *elt |= value << (sizeof(HUF_CElt) * 8 - nbBits); } } typedef struct { HUF_CompressWeightsWksp wksp; BYTE bitsToWeight[HUF_TABLELOG_MAX + 1]; /* precomputed conversion table */ BYTE huffWeight[HUF_SYMBOLVALUE_MAX]; } HUF_WriteCTableWksp; size_t HUF_writeCTable_wksp(void* dst, size_t maxDstSize, const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog, void* workspace, size_t workspaceSize) { HUF_CElt const* const ct = CTable + 1; BYTE* op = (BYTE*)dst; U32 n; HUF_WriteCTableWksp* wksp = (HUF_WriteCTableWksp*)HUF_alignUpWorkspace(workspace, &workspaceSize, ZSTD_ALIGNOF(U32)); HUF_STATIC_ASSERT(HUF_CTABLE_WORKSPACE_SIZE >= sizeof(HUF_WriteCTableWksp)); /* check conditions */ if (workspaceSize < sizeof(HUF_WriteCTableWksp)) return ERROR(GENERIC); if (maxSymbolValue > HUF_SYMBOLVALUE_MAX) return ERROR(maxSymbolValue_tooLarge); /* convert to weight */ wksp->bitsToWeight[0] = 0; for (n=1; nbitsToWeight[n] = (BYTE)(huffLog + 1 - n); for (n=0; nhuffWeight[n] = wksp->bitsToWeight[HUF_getNbBits(ct[n])]; /* attempt weights compression by FSE */ if (maxDstSize < 1) return ERROR(dstSize_tooSmall); { CHECK_V_F(hSize, HUF_compressWeights(op+1, maxDstSize-1, wksp->huffWeight, maxSymbolValue, &wksp->wksp, sizeof(wksp->wksp)) ); if ((hSize>1) & (hSize < maxSymbolValue/2)) { /* FSE compressed */ op[0] = (BYTE)hSize; return hSize+1; } } /* write raw values as 4-bits (max : 15) */ if (maxSymbolValue > (256-128)) return ERROR(GENERIC); /* should not happen : likely means source cannot be compressed */ if (((maxSymbolValue+1)/2) + 1 > maxDstSize) return ERROR(dstSize_tooSmall); /* not enough space within dst buffer */ op[0] = (BYTE)(128 /*special case*/ + (maxSymbolValue-1)); wksp->huffWeight[maxSymbolValue] = 0; /* to be sure it doesn't cause msan issue in final combination */ for (n=0; nhuffWeight[n] << 4) + wksp->huffWeight[n+1]); return ((maxSymbolValue+1)/2) + 1; } size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, unsigned* hasZeroWeights) { BYTE huffWeight[HUF_SYMBOLVALUE_MAX + 1]; /* init not required, even though some static analyzer may complain */ U32 rankVal[HUF_TABLELOG_ABSOLUTEMAX + 1]; /* large enough for values from 0 to 16 */ U32 tableLog = 0; U32 nbSymbols = 0; HUF_CElt* const ct = CTable + 1; /* get symbol weights */ CHECK_V_F(readSize, HUF_readStats(huffWeight, HUF_SYMBOLVALUE_MAX+1, rankVal, &nbSymbols, &tableLog, src, srcSize)); *hasZeroWeights = (rankVal[0] > 0); /* check result */ if (tableLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge); if (nbSymbols > *maxSymbolValuePtr+1) return ERROR(maxSymbolValue_tooSmall); CTable[0] = tableLog; /* Prepare base value per rank */ { U32 n, nextRankStart = 0; for (n=1; n<=tableLog; n++) { U32 curr = nextRankStart; nextRankStart += (rankVal[n] << (n-1)); rankVal[n] = curr; } } /* fill nbBits */ { U32 n; for (n=0; nn=tableLog+1 */ U16 valPerRank[HUF_TABLELOG_MAX+2] = {0}; { U32 n; for (n=0; n0; n--) { /* start at n=tablelog <-> w=1 */ valPerRank[n] = min; /* get starting value within each rank */ min += nbPerRank[n]; min >>= 1; } } /* assign value within rank, symbol order */ { U32 n; for (n=0; n @targetNbBits * to employ @targetNbBits instead. Then it adjusts the tree * so that it remains a valid canonical Huffman tree. * * @pre The sum of the ranks of each symbol == 2^largestBits, * where largestBits == huffNode[lastNonNull].nbBits. * @post The sum of the ranks of each symbol == 2^largestBits, * where largestBits is the return value (expected <= targetNbBits). * * @param huffNode The Huffman tree modified in place to enforce targetNbBits. * It's presumed sorted, from most frequent to rarest symbol. * @param lastNonNull The symbol with the lowest count in the Huffman tree. * @param targetNbBits The allowed number of bits, which the Huffman tree * may not respect. After this function the Huffman tree will * respect targetNbBits. * @return The maximum number of bits of the Huffman tree after adjustment. */ static U32 HUF_setMaxHeight(nodeElt* huffNode, U32 lastNonNull, U32 targetNbBits) { const U32 largestBits = huffNode[lastNonNull].nbBits; /* early exit : no elt > targetNbBits, so the tree is already valid. */ if (largestBits <= targetNbBits) return largestBits; DEBUGLOG(5, "HUF_setMaxHeight (targetNbBits = %u)", targetNbBits); /* there are several too large elements (at least >= 2) */ { int totalCost = 0; const U32 baseCost = 1 << (largestBits - targetNbBits); int n = (int)lastNonNull; /* Adjust any ranks > targetNbBits to targetNbBits. * Compute totalCost, which is how far the sum of the ranks is * we are over 2^largestBits after adjust the offending ranks. */ while (huffNode[n].nbBits > targetNbBits) { totalCost += baseCost - (1 << (largestBits - huffNode[n].nbBits)); huffNode[n].nbBits = (BYTE)targetNbBits; n--; } /* n stops at huffNode[n].nbBits <= targetNbBits */ assert(huffNode[n].nbBits <= targetNbBits); /* n end at index of smallest symbol using < targetNbBits */ while (huffNode[n].nbBits == targetNbBits) --n; /* renorm totalCost from 2^largestBits to 2^targetNbBits * note : totalCost is necessarily a multiple of baseCost */ assert(((U32)totalCost & (baseCost - 1)) == 0); totalCost >>= (largestBits - targetNbBits); assert(totalCost > 0); /* repay normalized cost */ { U32 const noSymbol = 0xF0F0F0F0; U32 rankLast[HUF_TABLELOG_MAX+2]; /* Get pos of last (smallest = lowest cum. count) symbol per rank */ ZSTD_memset(rankLast, 0xF0, sizeof(rankLast)); { U32 currentNbBits = targetNbBits; int pos; for (pos=n ; pos >= 0; pos--) { if (huffNode[pos].nbBits >= currentNbBits) continue; currentNbBits = huffNode[pos].nbBits; /* < targetNbBits */ rankLast[targetNbBits-currentNbBits] = (U32)pos; } } while (totalCost > 0) { /* Try to reduce the next power of 2 above totalCost because we * gain back half the rank. */ U32 nBitsToDecrease = ZSTD_highbit32((U32)totalCost) + 1; for ( ; nBitsToDecrease > 1; nBitsToDecrease--) { U32 const highPos = rankLast[nBitsToDecrease]; U32 const lowPos = rankLast[nBitsToDecrease-1]; if (highPos == noSymbol) continue; /* Decrease highPos if no symbols of lowPos or if it is * not cheaper to remove 2 lowPos than highPos. */ if (lowPos == noSymbol) break; { U32 const highTotal = huffNode[highPos].count; U32 const lowTotal = 2 * huffNode[lowPos].count; if (highTotal <= lowTotal) break; } } /* only triggered when no more rank 1 symbol left => find closest one (note : there is necessarily at least one !) */ assert(rankLast[nBitsToDecrease] != noSymbol || nBitsToDecrease == 1); /* HUF_MAX_TABLELOG test just to please gcc 5+; but it should not be necessary */ while ((nBitsToDecrease<=HUF_TABLELOG_MAX) && (rankLast[nBitsToDecrease] == noSymbol)) nBitsToDecrease++; assert(rankLast[nBitsToDecrease] != noSymbol); /* Increase the number of bits to gain back half the rank cost. */ totalCost -= 1 << (nBitsToDecrease-1); huffNode[rankLast[nBitsToDecrease]].nbBits++; /* Fix up the new rank. * If the new rank was empty, this symbol is now its smallest. * Otherwise, this symbol will be the largest in the new rank so no adjustment. */ if (rankLast[nBitsToDecrease-1] == noSymbol) rankLast[nBitsToDecrease-1] = rankLast[nBitsToDecrease]; /* Fix up the old rank. * If the symbol was at position 0, meaning it was the highest weight symbol in the tree, * it must be the only symbol in its rank, so the old rank now has no symbols. * Otherwise, since the Huffman nodes are sorted by count, the previous position is now * the smallest node in the rank. If the previous position belongs to a different rank, * then the rank is now empty. */ if (rankLast[nBitsToDecrease] == 0) /* special case, reached largest symbol */ rankLast[nBitsToDecrease] = noSymbol; else { rankLast[nBitsToDecrease]--; if (huffNode[rankLast[nBitsToDecrease]].nbBits != targetNbBits-nBitsToDecrease) rankLast[nBitsToDecrease] = noSymbol; /* this rank is now empty */ } } /* while (totalCost > 0) */ /* If we've removed too much weight, then we have to add it back. * To avoid overshooting again, we only adjust the smallest rank. * We take the largest nodes from the lowest rank 0 and move them * to rank 1. There's guaranteed to be enough rank 0 symbols because * TODO. */ while (totalCost < 0) { /* Sometimes, cost correction overshoot */ /* special case : no rank 1 symbol (using targetNbBits-1); * let's create one from largest rank 0 (using targetNbBits). */ if (rankLast[1] == noSymbol) { while (huffNode[n].nbBits == targetNbBits) n--; huffNode[n+1].nbBits--; assert(n >= 0); rankLast[1] = (U32)(n+1); totalCost++; continue; } huffNode[ rankLast[1] + 1 ].nbBits--; rankLast[1]++; totalCost ++; } } /* repay normalized cost */ } /* there are several too large elements (at least >= 2) */ return targetNbBits; } typedef struct { U16 base; U16 curr; } rankPos; typedef nodeElt huffNodeTable[2 * (HUF_SYMBOLVALUE_MAX + 1)]; /* Number of buckets available for HUF_sort() */ #define RANK_POSITION_TABLE_SIZE 192 typedef struct { huffNodeTable huffNodeTbl; rankPos rankPosition[RANK_POSITION_TABLE_SIZE]; } HUF_buildCTable_wksp_tables; /* RANK_POSITION_DISTINCT_COUNT_CUTOFF == Cutoff point in HUF_sort() buckets for which we use log2 bucketing. * Strategy is to use as many buckets as possible for representing distinct * counts while using the remainder to represent all "large" counts. * * To satisfy this requirement for 192 buckets, we can do the following: * Let buckets 0-166 represent distinct counts of [0, 166] * Let buckets 166 to 192 represent all remaining counts up to RANK_POSITION_MAX_COUNT_LOG using log2 bucketing. */ #define RANK_POSITION_MAX_COUNT_LOG 32 #define RANK_POSITION_LOG_BUCKETS_BEGIN ((RANK_POSITION_TABLE_SIZE - 1) - RANK_POSITION_MAX_COUNT_LOG - 1 /* == 158 */) #define RANK_POSITION_DISTINCT_COUNT_CUTOFF (RANK_POSITION_LOG_BUCKETS_BEGIN + ZSTD_highbit32(RANK_POSITION_LOG_BUCKETS_BEGIN) /* == 166 */) /* Return the appropriate bucket index for a given count. See definition of * RANK_POSITION_DISTINCT_COUNT_CUTOFF for explanation of bucketing strategy. */ static U32 HUF_getIndex(U32 const count) { return (count < RANK_POSITION_DISTINCT_COUNT_CUTOFF) ? count : ZSTD_highbit32(count) + RANK_POSITION_LOG_BUCKETS_BEGIN; } /* Helper swap function for HUF_quickSortPartition() */ static void HUF_swapNodes(nodeElt* a, nodeElt* b) { nodeElt tmp = *a; *a = *b; *b = tmp; } /* Returns 0 if the huffNode array is not sorted by descending count */ MEM_STATIC int HUF_isSorted(nodeElt huffNode[], U32 const maxSymbolValue1) { U32 i; for (i = 1; i < maxSymbolValue1; ++i) { if (huffNode[i].count > huffNode[i-1].count) { return 0; } } return 1; } /* Insertion sort by descending order */ HINT_INLINE void HUF_insertionSort(nodeElt huffNode[], int const low, int const high) { int i; int const size = high-low+1; huffNode += low; for (i = 1; i < size; ++i) { nodeElt const key = huffNode[i]; int j = i - 1; while (j >= 0 && huffNode[j].count < key.count) { huffNode[j + 1] = huffNode[j]; j--; } huffNode[j + 1] = key; } } /* Pivot helper function for quicksort. */ static int HUF_quickSortPartition(nodeElt arr[], int const low, int const high) { /* Simply select rightmost element as pivot. "Better" selectors like * median-of-three don't experimentally appear to have any benefit. */ U32 const pivot = arr[high].count; int i = low - 1; int j = low; for ( ; j < high; j++) { if (arr[j].count > pivot) { i++; HUF_swapNodes(&arr[i], &arr[j]); } } HUF_swapNodes(&arr[i + 1], &arr[high]); return i + 1; } /* Classic quicksort by descending with partially iterative calls * to reduce worst case callstack size. */ static void HUF_simpleQuickSort(nodeElt arr[], int low, int high) { int const kInsertionSortThreshold = 8; if (high - low < kInsertionSortThreshold) { HUF_insertionSort(arr, low, high); return; } while (low < high) { int const idx = HUF_quickSortPartition(arr, low, high); if (idx - low < high - idx) { HUF_simpleQuickSort(arr, low, idx - 1); low = idx + 1; } else { HUF_simpleQuickSort(arr, idx + 1, high); high = idx - 1; } } } /** * HUF_sort(): * Sorts the symbols [0, maxSymbolValue] by count[symbol] in decreasing order. * This is a typical bucket sorting strategy that uses either quicksort or insertion sort to sort each bucket. * * @param[out] huffNode Sorted symbols by decreasing count. Only members `.count` and `.byte` are filled. * Must have (maxSymbolValue + 1) entries. * @param[in] count Histogram of the symbols. * @param[in] maxSymbolValue Maximum symbol value. * @param rankPosition This is a scratch workspace. Must have RANK_POSITION_TABLE_SIZE entries. */ static void HUF_sort(nodeElt huffNode[], const unsigned count[], U32 const maxSymbolValue, rankPos rankPosition[]) { U32 n; U32 const maxSymbolValue1 = maxSymbolValue+1; /* Compute base and set curr to base. * For symbol s let lowerRank = HUF_getIndex(count[n]) and rank = lowerRank + 1. * See HUF_getIndex to see bucketing strategy. * We attribute each symbol to lowerRank's base value, because we want to know where * each rank begins in the output, so for rank R we want to count ranks R+1 and above. */ ZSTD_memset(rankPosition, 0, sizeof(*rankPosition) * RANK_POSITION_TABLE_SIZE); for (n = 0; n < maxSymbolValue1; ++n) { U32 lowerRank = HUF_getIndex(count[n]); assert(lowerRank < RANK_POSITION_TABLE_SIZE - 1); rankPosition[lowerRank].base++; } assert(rankPosition[RANK_POSITION_TABLE_SIZE - 1].base == 0); /* Set up the rankPosition table */ for (n = RANK_POSITION_TABLE_SIZE - 1; n > 0; --n) { rankPosition[n-1].base += rankPosition[n].base; rankPosition[n-1].curr = rankPosition[n-1].base; } /* Insert each symbol into their appropriate bucket, setting up rankPosition table. */ for (n = 0; n < maxSymbolValue1; ++n) { U32 const c = count[n]; U32 const r = HUF_getIndex(c) + 1; U32 const pos = rankPosition[r].curr++; assert(pos < maxSymbolValue1); huffNode[pos].count = c; huffNode[pos].byte = (BYTE)n; } /* Sort each bucket. */ for (n = RANK_POSITION_DISTINCT_COUNT_CUTOFF; n < RANK_POSITION_TABLE_SIZE - 1; ++n) { int const bucketSize = rankPosition[n].curr - rankPosition[n].base; U32 const bucketStartIdx = rankPosition[n].base; if (bucketSize > 1) { assert(bucketStartIdx < maxSymbolValue1); HUF_simpleQuickSort(huffNode + bucketStartIdx, 0, bucketSize-1); } } assert(HUF_isSorted(huffNode, maxSymbolValue1)); } /** HUF_buildCTable_wksp() : * Same as HUF_buildCTable(), but using externally allocated scratch buffer. * `workSpace` must be aligned on 4-bytes boundaries, and be at least as large as sizeof(HUF_buildCTable_wksp_tables). */ #define STARTNODE (HUF_SYMBOLVALUE_MAX+1) /* HUF_buildTree(): * Takes the huffNode array sorted by HUF_sort() and builds an unlimited-depth Huffman tree. * * @param huffNode The array sorted by HUF_sort(). Builds the Huffman tree in this array. * @param maxSymbolValue The maximum symbol value. * @return The smallest node in the Huffman tree (by count). */ static int HUF_buildTree(nodeElt* huffNode, U32 maxSymbolValue) { nodeElt* const huffNode0 = huffNode - 1; int nonNullRank; int lowS, lowN; int nodeNb = STARTNODE; int n, nodeRoot; DEBUGLOG(5, "HUF_buildTree (alphabet size = %u)", maxSymbolValue + 1); /* init for parents */ nonNullRank = (int)maxSymbolValue; while(huffNode[nonNullRank].count == 0) nonNullRank--; lowS = nonNullRank; nodeRoot = nodeNb + lowS - 1; lowN = nodeNb; huffNode[nodeNb].count = huffNode[lowS].count + huffNode[lowS-1].count; huffNode[lowS].parent = huffNode[lowS-1].parent = (U16)nodeNb; nodeNb++; lowS-=2; for (n=nodeNb; n<=nodeRoot; n++) huffNode[n].count = (U32)(1U<<30); huffNode0[0].count = (U32)(1U<<31); /* fake entry, strong barrier */ /* create parents */ while (nodeNb <= nodeRoot) { int const n1 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++; int const n2 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++; huffNode[nodeNb].count = huffNode[n1].count + huffNode[n2].count; huffNode[n1].parent = huffNode[n2].parent = (U16)nodeNb; nodeNb++; } /* distribute weights (unlimited tree height) */ huffNode[nodeRoot].nbBits = 0; for (n=nodeRoot-1; n>=STARTNODE; n--) huffNode[n].nbBits = huffNode[ huffNode[n].parent ].nbBits + 1; for (n=0; n<=nonNullRank; n++) huffNode[n].nbBits = huffNode[ huffNode[n].parent ].nbBits + 1; DEBUGLOG(6, "Initial distribution of bits completed (%zu sorted symbols)", showHNodeBits(huffNode, maxSymbolValue+1)); return nonNullRank; } /** * HUF_buildCTableFromTree(): * Build the CTable given the Huffman tree in huffNode. * * @param[out] CTable The output Huffman CTable. * @param huffNode The Huffman tree. * @param nonNullRank The last and smallest node in the Huffman tree. * @param maxSymbolValue The maximum symbol value. * @param maxNbBits The exact maximum number of bits used in the Huffman tree. */ static void HUF_buildCTableFromTree(HUF_CElt* CTable, nodeElt const* huffNode, int nonNullRank, U32 maxSymbolValue, U32 maxNbBits) { HUF_CElt* const ct = CTable + 1; /* fill result into ctable (val, nbBits) */ int n; U16 nbPerRank[HUF_TABLELOG_MAX+1] = {0}; U16 valPerRank[HUF_TABLELOG_MAX+1] = {0}; int const alphabetSize = (int)(maxSymbolValue + 1); for (n=0; n<=nonNullRank; n++) nbPerRank[huffNode[n].nbBits]++; /* determine starting value per rank */ { U16 min = 0; for (n=(int)maxNbBits; n>0; n--) { valPerRank[n] = min; /* get starting value within each rank */ min += nbPerRank[n]; min >>= 1; } } for (n=0; nhuffNodeTbl; nodeElt* const huffNode = huffNode0+1; int nonNullRank; HUF_STATIC_ASSERT(HUF_CTABLE_WORKSPACE_SIZE == sizeof(HUF_buildCTable_wksp_tables)); DEBUGLOG(5, "HUF_buildCTable_wksp (alphabet size = %u)", maxSymbolValue+1); /* safety checks */ if (wkspSize < sizeof(HUF_buildCTable_wksp_tables)) return ERROR(workSpace_tooSmall); if (maxNbBits == 0) maxNbBits = HUF_TABLELOG_DEFAULT; if (maxSymbolValue > HUF_SYMBOLVALUE_MAX) return ERROR(maxSymbolValue_tooLarge); ZSTD_memset(huffNode0, 0, sizeof(huffNodeTable)); /* sort, decreasing order */ HUF_sort(huffNode, count, maxSymbolValue, wksp_tables->rankPosition); DEBUGLOG(6, "sorted symbols completed (%zu symbols)", showHNodeSymbols(huffNode, maxSymbolValue+1)); /* build tree */ nonNullRank = HUF_buildTree(huffNode, maxSymbolValue); /* determine and enforce maxTableLog */ maxNbBits = HUF_setMaxHeight(huffNode, (U32)nonNullRank, maxNbBits); if (maxNbBits > HUF_TABLELOG_MAX) return ERROR(GENERIC); /* check fit into table */ HUF_buildCTableFromTree(CTable, huffNode, nonNullRank, maxSymbolValue, maxNbBits); return maxNbBits; } size_t HUF_estimateCompressedSize(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue) { HUF_CElt const* ct = CTable + 1; size_t nbBits = 0; int s; for (s = 0; s <= (int)maxSymbolValue; ++s) { nbBits += HUF_getNbBits(ct[s]) * count[s]; } return nbBits >> 3; } int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue) { HUF_CElt const* ct = CTable + 1; int bad = 0; int s; for (s = 0; s <= (int)maxSymbolValue; ++s) { bad |= (count[s] != 0) & (HUF_getNbBits(ct[s]) == 0); } return !bad; } size_t HUF_compressBound(size_t size) { return HUF_COMPRESSBOUND(size); } /** HUF_CStream_t: * Huffman uses its own BIT_CStream_t implementation. * There are three major differences from BIT_CStream_t: * 1. HUF_addBits() takes a HUF_CElt (size_t) which is * the pair (nbBits, value) in the format: * format: * - Bits [0, 4) = nbBits * - Bits [4, 64 - nbBits) = 0 * - Bits [64 - nbBits, 64) = value * 2. The bitContainer is built from the upper bits and * right shifted. E.g. to add a new value of N bits * you right shift the bitContainer by N, then or in * the new value into the N upper bits. * 3. The bitstream has two bit containers. You can add * bits to the second container and merge them into * the first container. */ #define HUF_BITS_IN_CONTAINER (sizeof(size_t) * 8) typedef struct { size_t bitContainer[2]; size_t bitPos[2]; BYTE* startPtr; BYTE* ptr; BYTE* endPtr; } HUF_CStream_t; /**! HUF_initCStream(): * Initializes the bitstream. * @returns 0 or an error code. */ static size_t HUF_initCStream(HUF_CStream_t* bitC, void* startPtr, size_t dstCapacity) { ZSTD_memset(bitC, 0, sizeof(*bitC)); bitC->startPtr = (BYTE*)startPtr; bitC->ptr = bitC->startPtr; bitC->endPtr = bitC->startPtr + dstCapacity - sizeof(bitC->bitContainer[0]); if (dstCapacity <= sizeof(bitC->bitContainer[0])) return ERROR(dstSize_tooSmall); return 0; } /*! HUF_addBits(): * Adds the symbol stored in HUF_CElt elt to the bitstream. * * @param elt The element we're adding. This is a (nbBits, value) pair. * See the HUF_CStream_t docs for the format. * @param idx Insert into the bitstream at this idx. * @param kFast This is a template parameter. If the bitstream is guaranteed * to have at least 4 unused bits after this call it may be 1, * otherwise it must be 0. HUF_addBits() is faster when fast is set. */ FORCE_INLINE_TEMPLATE void HUF_addBits(HUF_CStream_t* bitC, HUF_CElt elt, int idx, int kFast) { assert(idx <= 1); assert(HUF_getNbBits(elt) <= HUF_TABLELOG_ABSOLUTEMAX); /* This is efficient on x86-64 with BMI2 because shrx * only reads the low 6 bits of the register. The compiler * knows this and elides the mask. When fast is set, * every operation can use the same value loaded from elt. */ bitC->bitContainer[idx] >>= HUF_getNbBits(elt); bitC->bitContainer[idx] |= kFast ? HUF_getValueFast(elt) : HUF_getValue(elt); /* We only read the low 8 bits of bitC->bitPos[idx] so it * doesn't matter that the high bits have noise from the value. */ bitC->bitPos[idx] += HUF_getNbBitsFast(elt); assert((bitC->bitPos[idx] & 0xFF) <= HUF_BITS_IN_CONTAINER); /* The last 4-bits of elt are dirty if fast is set, * so we must not be overwriting bits that have already been * inserted into the bit container. */ #if DEBUGLEVEL >= 1 { size_t const nbBits = HUF_getNbBits(elt); size_t const dirtyBits = nbBits == 0 ? 0 : ZSTD_highbit32((U32)nbBits) + 1; (void)dirtyBits; /* Middle bits are 0. */ assert(((elt >> dirtyBits) << (dirtyBits + nbBits)) == 0); /* We didn't overwrite any bits in the bit container. */ assert(!kFast || (bitC->bitPos[idx] & 0xFF) <= HUF_BITS_IN_CONTAINER); (void)dirtyBits; } #endif } FORCE_INLINE_TEMPLATE void HUF_zeroIndex1(HUF_CStream_t* bitC) { bitC->bitContainer[1] = 0; bitC->bitPos[1] = 0; } /*! HUF_mergeIndex1() : * Merges the bit container @ index 1 into the bit container @ index 0 * and zeros the bit container @ index 1. */ FORCE_INLINE_TEMPLATE void HUF_mergeIndex1(HUF_CStream_t* bitC) { assert((bitC->bitPos[1] & 0xFF) < HUF_BITS_IN_CONTAINER); bitC->bitContainer[0] >>= (bitC->bitPos[1] & 0xFF); bitC->bitContainer[0] |= bitC->bitContainer[1]; bitC->bitPos[0] += bitC->bitPos[1]; assert((bitC->bitPos[0] & 0xFF) <= HUF_BITS_IN_CONTAINER); } /*! HUF_flushBits() : * Flushes the bits in the bit container @ index 0. * * @post bitPos will be < 8. * @param kFast If kFast is set then we must know a-priori that * the bit container will not overflow. */ FORCE_INLINE_TEMPLATE void HUF_flushBits(HUF_CStream_t* bitC, int kFast) { /* The upper bits of bitPos are noisy, so we must mask by 0xFF. */ size_t const nbBits = bitC->bitPos[0] & 0xFF; size_t const nbBytes = nbBits >> 3; /* The top nbBits bits of bitContainer are the ones we need. */ size_t const bitContainer = bitC->bitContainer[0] >> (HUF_BITS_IN_CONTAINER - nbBits); /* Mask bitPos to account for the bytes we consumed. */ bitC->bitPos[0] &= 7; assert(nbBits > 0); assert(nbBits <= sizeof(bitC->bitContainer[0]) * 8); assert(bitC->ptr <= bitC->endPtr); MEM_writeLEST(bitC->ptr, bitContainer); bitC->ptr += nbBytes; assert(!kFast || bitC->ptr <= bitC->endPtr); if (!kFast && bitC->ptr > bitC->endPtr) bitC->ptr = bitC->endPtr; /* bitContainer doesn't need to be modified because the leftover * bits are already the top bitPos bits. And we don't care about * noise in the lower values. */ } /*! HUF_endMark() * @returns The Huffman stream end mark: A 1-bit value = 1. */ static HUF_CElt HUF_endMark(void) { HUF_CElt endMark; HUF_setNbBits(&endMark, 1); HUF_setValue(&endMark, 1); return endMark; } /*! HUF_closeCStream() : * @return Size of CStream, in bytes, * or 0 if it could not fit into dstBuffer */ static size_t HUF_closeCStream(HUF_CStream_t* bitC) { HUF_addBits(bitC, HUF_endMark(), /* idx */ 0, /* kFast */ 0); HUF_flushBits(bitC, /* kFast */ 0); { size_t const nbBits = bitC->bitPos[0] & 0xFF; if (bitC->ptr >= bitC->endPtr) return 0; /* overflow detected */ return (size_t)(bitC->ptr - bitC->startPtr) + (nbBits > 0); } } FORCE_INLINE_TEMPLATE void HUF_encodeSymbol(HUF_CStream_t* bitCPtr, U32 symbol, const HUF_CElt* CTable, int idx, int fast) { HUF_addBits(bitCPtr, CTable[symbol], idx, fast); } FORCE_INLINE_TEMPLATE void HUF_compress1X_usingCTable_internal_body_loop(HUF_CStream_t* bitC, const BYTE* ip, size_t srcSize, const HUF_CElt* ct, int kUnroll, int kFastFlush, int kLastFast) { /* Join to kUnroll */ int n = (int)srcSize; int rem = n % kUnroll; if (rem > 0) { for (; rem > 0; --rem) { HUF_encodeSymbol(bitC, ip[--n], ct, 0, /* fast */ 0); } HUF_flushBits(bitC, kFastFlush); } assert(n % kUnroll == 0); /* Join to 2 * kUnroll */ if (n % (2 * kUnroll)) { int u; for (u = 1; u < kUnroll; ++u) { HUF_encodeSymbol(bitC, ip[n - u], ct, 0, 1); } HUF_encodeSymbol(bitC, ip[n - kUnroll], ct, 0, kLastFast); HUF_flushBits(bitC, kFastFlush); n -= kUnroll; } assert(n % (2 * kUnroll) == 0); for (; n>0; n-= 2 * kUnroll) { /* Encode kUnroll symbols into the bitstream @ index 0. */ int u; for (u = 1; u < kUnroll; ++u) { HUF_encodeSymbol(bitC, ip[n - u], ct, /* idx */ 0, /* fast */ 1); } HUF_encodeSymbol(bitC, ip[n - kUnroll], ct, /* idx */ 0, /* fast */ kLastFast); HUF_flushBits(bitC, kFastFlush); /* Encode kUnroll symbols into the bitstream @ index 1. * This allows us to start filling the bit container * without any data dependencies. */ HUF_zeroIndex1(bitC); for (u = 1; u < kUnroll; ++u) { HUF_encodeSymbol(bitC, ip[n - kUnroll - u], ct, /* idx */ 1, /* fast */ 1); } HUF_encodeSymbol(bitC, ip[n - kUnroll - kUnroll], ct, /* idx */ 1, /* fast */ kLastFast); /* Merge bitstream @ index 1 into the bitstream @ index 0 */ HUF_mergeIndex1(bitC); HUF_flushBits(bitC, kFastFlush); } assert(n == 0); } /** * Returns a tight upper bound on the output space needed by Huffman * with 8 bytes buffer to handle over-writes. If the output is at least * this large we don't need to do bounds checks during Huffman encoding. */ static size_t HUF_tightCompressBound(size_t srcSize, size_t tableLog) { return ((srcSize * tableLog) >> 3) + 8; } FORCE_INLINE_TEMPLATE size_t HUF_compress1X_usingCTable_internal_body(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable) { U32 const tableLog = (U32)CTable[0]; HUF_CElt const* ct = CTable + 1; const BYTE* ip = (const BYTE*) src; BYTE* const ostart = (BYTE*)dst; BYTE* const oend = ostart + dstSize; BYTE* op = ostart; HUF_CStream_t bitC; /* init */ if (dstSize < 8) return 0; /* not enough space to compress */ { size_t const initErr = HUF_initCStream(&bitC, op, (size_t)(oend-op)); if (HUF_isError(initErr)) return 0; } if (dstSize < HUF_tightCompressBound(srcSize, (size_t)tableLog) || tableLog > 11) HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ MEM_32bits() ? 2 : 4, /* kFast */ 0, /* kLastFast */ 0); else { if (MEM_32bits()) { switch (tableLog) { case 11: HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 2, /* kFastFlush */ 1, /* kLastFast */ 0); break; case 10: ZSTD_FALLTHROUGH; case 9: ZSTD_FALLTHROUGH; case 8: HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 2, /* kFastFlush */ 1, /* kLastFast */ 1); break; case 7: ZSTD_FALLTHROUGH; default: HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 3, /* kFastFlush */ 1, /* kLastFast */ 1); break; } } else { switch (tableLog) { case 11: HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 5, /* kFastFlush */ 1, /* kLastFast */ 0); break; case 10: HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 5, /* kFastFlush */ 1, /* kLastFast */ 1); break; case 9: HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 6, /* kFastFlush */ 1, /* kLastFast */ 0); break; case 8: HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 7, /* kFastFlush */ 1, /* kLastFast */ 0); break; case 7: HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 8, /* kFastFlush */ 1, /* kLastFast */ 0); break; case 6: ZSTD_FALLTHROUGH; default: HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 9, /* kFastFlush */ 1, /* kLastFast */ 1); break; } } } assert(bitC.ptr <= bitC.endPtr); return HUF_closeCStream(&bitC); } #if DYNAMIC_BMI2 static BMI2_TARGET_ATTRIBUTE size_t HUF_compress1X_usingCTable_internal_bmi2(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable) { return HUF_compress1X_usingCTable_internal_body(dst, dstSize, src, srcSize, CTable); } static size_t HUF_compress1X_usingCTable_internal_default(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable) { return HUF_compress1X_usingCTable_internal_body(dst, dstSize, src, srcSize, CTable); } static size_t HUF_compress1X_usingCTable_internal(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, const int flags) { if (flags & HUF_flags_bmi2) { return HUF_compress1X_usingCTable_internal_bmi2(dst, dstSize, src, srcSize, CTable); } return HUF_compress1X_usingCTable_internal_default(dst, dstSize, src, srcSize, CTable); } #else static size_t HUF_compress1X_usingCTable_internal(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, const int flags) { (void)flags; return HUF_compress1X_usingCTable_internal_body(dst, dstSize, src, srcSize, CTable); } #endif size_t HUF_compress1X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, int flags) { return HUF_compress1X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, flags); } static size_t HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, int flags) { size_t const segmentSize = (srcSize+3)/4; /* first 3 segments */ const BYTE* ip = (const BYTE*) src; const BYTE* const iend = ip + srcSize; BYTE* const ostart = (BYTE*) dst; BYTE* const oend = ostart + dstSize; BYTE* op = ostart; if (dstSize < 6 + 1 + 1 + 1 + 8) return 0; /* minimum space to compress successfully */ if (srcSize < 12) return 0; /* no saving possible : too small input */ op += 6; /* jumpTable */ assert(op <= oend); { CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, flags) ); if (cSize == 0 || cSize > 65535) return 0; MEM_writeLE16(ostart, (U16)cSize); op += cSize; } ip += segmentSize; assert(op <= oend); { CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, flags) ); if (cSize == 0 || cSize > 65535) return 0; MEM_writeLE16(ostart+2, (U16)cSize); op += cSize; } ip += segmentSize; assert(op <= oend); { CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, flags) ); if (cSize == 0 || cSize > 65535) return 0; MEM_writeLE16(ostart+4, (U16)cSize); op += cSize; } ip += segmentSize; assert(op <= oend); assert(ip <= iend); { CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, (size_t)(iend-ip), CTable, flags) ); if (cSize == 0 || cSize > 65535) return 0; op += cSize; } return (size_t)(op-ostart); } size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, int flags) { return HUF_compress4X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, flags); } typedef enum { HUF_singleStream, HUF_fourStreams } HUF_nbStreams_e; static size_t HUF_compressCTable_internal( BYTE* const ostart, BYTE* op, BYTE* const oend, const void* src, size_t srcSize, HUF_nbStreams_e nbStreams, const HUF_CElt* CTable, const int flags) { size_t const cSize = (nbStreams==HUF_singleStream) ? HUF_compress1X_usingCTable_internal(op, (size_t)(oend - op), src, srcSize, CTable, flags) : HUF_compress4X_usingCTable_internal(op, (size_t)(oend - op), src, srcSize, CTable, flags); if (HUF_isError(cSize)) { return cSize; } if (cSize==0) { return 0; } /* uncompressible */ op += cSize; /* check compressibility */ assert(op >= ostart); if ((size_t)(op-ostart) >= srcSize-1) { return 0; } return (size_t)(op-ostart); } typedef struct { unsigned count[HUF_SYMBOLVALUE_MAX + 1]; HUF_CElt CTable[HUF_CTABLE_SIZE_ST(HUF_SYMBOLVALUE_MAX)]; union { HUF_buildCTable_wksp_tables buildCTable_wksp; HUF_WriteCTableWksp writeCTable_wksp; U32 hist_wksp[HIST_WKSP_SIZE_U32]; } wksps; } HUF_compress_tables_t; #define SUSPECT_INCOMPRESSIBLE_SAMPLE_SIZE 4096 #define SUSPECT_INCOMPRESSIBLE_SAMPLE_RATIO 10 /* Must be >= 2 */ unsigned HUF_cardinality(const unsigned* count, unsigned maxSymbolValue) { unsigned cardinality = 0; unsigned i; for (i = 0; i < maxSymbolValue + 1; i++) { if (count[i] != 0) cardinality += 1; } return cardinality; } unsigned HUF_minTableLog(unsigned symbolCardinality) { U32 minBitsSymbols = ZSTD_highbit32(symbolCardinality) + 1; return minBitsSymbols; } unsigned HUF_optimalTableLog( unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, void* workSpace, size_t wkspSize, HUF_CElt* table, const unsigned* count, int flags) { assert(srcSize > 1); /* Not supported, RLE should be used instead */ assert(wkspSize >= sizeof(HUF_buildCTable_wksp_tables)); if (!(flags & HUF_flags_optimalDepth)) { /* cheap evaluation, based on FSE */ return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 1); } { BYTE* dst = (BYTE*)workSpace + sizeof(HUF_WriteCTableWksp); size_t dstSize = wkspSize - sizeof(HUF_WriteCTableWksp); size_t maxBits, hSize, newSize; const unsigned symbolCardinality = HUF_cardinality(count, maxSymbolValue); const unsigned minTableLog = HUF_minTableLog(symbolCardinality); size_t optSize = ((size_t) ~0) - 1; unsigned optLog = maxTableLog, optLogGuess; DEBUGLOG(6, "HUF_optimalTableLog: probing huf depth (srcSize=%zu)", srcSize); /* Search until size increases */ for (optLogGuess = minTableLog; optLogGuess <= maxTableLog; optLogGuess++) { DEBUGLOG(7, "checking for huffLog=%u", optLogGuess); maxBits = HUF_buildCTable_wksp(table, count, maxSymbolValue, optLogGuess, workSpace, wkspSize); if (ERR_isError(maxBits)) continue; if (maxBits < optLogGuess && optLogGuess > minTableLog) break; hSize = HUF_writeCTable_wksp(dst, dstSize, table, maxSymbolValue, (U32)maxBits, workSpace, wkspSize); if (ERR_isError(hSize)) continue; newSize = HUF_estimateCompressedSize(table, count, maxSymbolValue) + hSize; if (newSize > optSize + 1) { break; } if (newSize < optSize) { optSize = newSize; optLog = optLogGuess; } } assert(optLog <= HUF_TABLELOG_MAX); return optLog; } } /* HUF_compress_internal() : * `workSpace_align4` must be aligned on 4-bytes boundaries, * and occupies the same space as a table of HUF_WORKSPACE_SIZE_U64 unsigned */ static size_t HUF_compress_internal (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned huffLog, HUF_nbStreams_e nbStreams, void* workSpace, size_t wkspSize, HUF_CElt* oldHufTable, HUF_repeat* repeat, int flags) { HUF_compress_tables_t* const table = (HUF_compress_tables_t*)HUF_alignUpWorkspace(workSpace, &wkspSize, ZSTD_ALIGNOF(size_t)); BYTE* const ostart = (BYTE*)dst; BYTE* const oend = ostart + dstSize; BYTE* op = ostart; DEBUGLOG(5, "HUF_compress_internal (srcSize=%zu)", srcSize); HUF_STATIC_ASSERT(sizeof(*table) + HUF_WORKSPACE_MAX_ALIGNMENT <= HUF_WORKSPACE_SIZE); /* checks & inits */ if (wkspSize < sizeof(*table)) return ERROR(workSpace_tooSmall); if (!srcSize) return 0; /* Uncompressed */ if (!dstSize) return 0; /* cannot fit anything within dst budget */ if (srcSize > HUF_BLOCKSIZE_MAX) return ERROR(srcSize_wrong); /* current block size limit */ if (huffLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge); if (maxSymbolValue > HUF_SYMBOLVALUE_MAX) return ERROR(maxSymbolValue_tooLarge); if (!maxSymbolValue) maxSymbolValue = HUF_SYMBOLVALUE_MAX; if (!huffLog) huffLog = HUF_TABLELOG_DEFAULT; /* Heuristic : If old table is valid, use it for small inputs */ if ((flags & HUF_flags_preferRepeat) && repeat && *repeat == HUF_repeat_valid) { return HUF_compressCTable_internal(ostart, op, oend, src, srcSize, nbStreams, oldHufTable, flags); } /* If uncompressible data is suspected, do a smaller sampling first */ DEBUG_STATIC_ASSERT(SUSPECT_INCOMPRESSIBLE_SAMPLE_RATIO >= 2); if ((flags & HUF_flags_suspectUncompressible) && srcSize >= (SUSPECT_INCOMPRESSIBLE_SAMPLE_SIZE * SUSPECT_INCOMPRESSIBLE_SAMPLE_RATIO)) { size_t largestTotal = 0; DEBUGLOG(5, "input suspected incompressible : sampling to check"); { unsigned maxSymbolValueBegin = maxSymbolValue; CHECK_V_F(largestBegin, HIST_count_simple (table->count, &maxSymbolValueBegin, (const BYTE*)src, SUSPECT_INCOMPRESSIBLE_SAMPLE_SIZE) ); largestTotal += largestBegin; } { unsigned maxSymbolValueEnd = maxSymbolValue; CHECK_V_F(largestEnd, HIST_count_simple (table->count, &maxSymbolValueEnd, (const BYTE*)src + srcSize - SUSPECT_INCOMPRESSIBLE_SAMPLE_SIZE, SUSPECT_INCOMPRESSIBLE_SAMPLE_SIZE) ); largestTotal += largestEnd; } if (largestTotal <= ((2 * SUSPECT_INCOMPRESSIBLE_SAMPLE_SIZE) >> 7)+4) return 0; /* heuristic : probably not compressible enough */ } /* Scan input and build symbol stats */ { CHECK_V_F(largest, HIST_count_wksp (table->count, &maxSymbolValue, (const BYTE*)src, srcSize, table->wksps.hist_wksp, sizeof(table->wksps.hist_wksp)) ); if (largest == srcSize) { *ostart = ((const BYTE*)src)[0]; return 1; } /* single symbol, rle */ if (largest <= (srcSize >> 7)+4) return 0; /* heuristic : probably not compressible enough */ } DEBUGLOG(6, "histogram detail completed (%zu symbols)", showU32(table->count, maxSymbolValue+1)); /* Check validity of previous table */ if ( repeat && *repeat == HUF_repeat_check && !HUF_validateCTable(oldHufTable, table->count, maxSymbolValue)) { *repeat = HUF_repeat_none; } /* Heuristic : use existing table for small inputs */ if ((flags & HUF_flags_preferRepeat) && repeat && *repeat != HUF_repeat_none) { return HUF_compressCTable_internal(ostart, op, oend, src, srcSize, nbStreams, oldHufTable, flags); } /* Build Huffman Tree */ huffLog = HUF_optimalTableLog(huffLog, srcSize, maxSymbolValue, &table->wksps, sizeof(table->wksps), table->CTable, table->count, flags); { size_t const maxBits = HUF_buildCTable_wksp(table->CTable, table->count, maxSymbolValue, huffLog, &table->wksps.buildCTable_wksp, sizeof(table->wksps.buildCTable_wksp)); CHECK_F(maxBits); huffLog = (U32)maxBits; DEBUGLOG(6, "bit distribution completed (%zu symbols)", showCTableBits(table->CTable + 1, maxSymbolValue+1)); } /* Zero unused symbols in CTable, so we can check it for validity */ { size_t const ctableSize = HUF_CTABLE_SIZE_ST(maxSymbolValue); size_t const unusedSize = sizeof(table->CTable) - ctableSize * sizeof(HUF_CElt); ZSTD_memset(table->CTable + ctableSize, 0, unusedSize); } /* Write table description header */ { CHECK_V_F(hSize, HUF_writeCTable_wksp(op, dstSize, table->CTable, maxSymbolValue, huffLog, &table->wksps.writeCTable_wksp, sizeof(table->wksps.writeCTable_wksp)) ); /* Check if using previous huffman table is beneficial */ if (repeat && *repeat != HUF_repeat_none) { size_t const oldSize = HUF_estimateCompressedSize(oldHufTable, table->count, maxSymbolValue); size_t const newSize = HUF_estimateCompressedSize(table->CTable, table->count, maxSymbolValue); if (oldSize <= hSize + newSize || hSize + 12 >= srcSize) { return HUF_compressCTable_internal(ostart, op, oend, src, srcSize, nbStreams, oldHufTable, flags); } } /* Use the new huffman table */ if (hSize + 12ul >= srcSize) { return 0; } op += hSize; if (repeat) { *repeat = HUF_repeat_none; } if (oldHufTable) ZSTD_memcpy(oldHufTable, table->CTable, sizeof(table->CTable)); /* Save new table */ } return HUF_compressCTable_internal(ostart, op, oend, src, srcSize, nbStreams, table->CTable, flags); } size_t HUF_compress1X_repeat (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned huffLog, void* workSpace, size_t wkspSize, HUF_CElt* hufTable, HUF_repeat* repeat, int flags) { DEBUGLOG(5, "HUF_compress1X_repeat (srcSize = %zu)", srcSize); return HUF_compress_internal(dst, dstSize, src, srcSize, maxSymbolValue, huffLog, HUF_singleStream, workSpace, wkspSize, hufTable, repeat, flags); } /* HUF_compress4X_repeat(): * compress input using 4 streams. * consider skipping quickly * re-use an existing huffman compression table */ size_t HUF_compress4X_repeat (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned huffLog, void* workSpace, size_t wkspSize, HUF_CElt* hufTable, HUF_repeat* repeat, int flags) { DEBUGLOG(5, "HUF_compress4X_repeat (srcSize = %zu)", srcSize); return HUF_compress_internal(dst, dstSize, src, srcSize, maxSymbolValue, huffLog, HUF_fourStreams, workSpace, wkspSize, hufTable, repeat, flags); } zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_compress_sequences.c0000644000175200007730000004704414515254731030302 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /*-************************************* * Dependencies ***************************************/ #include "zstd_compress_sequences.h" /** * -log2(x / 256) lookup table for x in [0, 256). * If x == 0: Return 0 * Else: Return floor(-log2(x / 256) * 256) */ static unsigned const kInverseProbabilityLog256[256] = { 0, 2048, 1792, 1642, 1536, 1453, 1386, 1329, 1280, 1236, 1197, 1162, 1130, 1100, 1073, 1047, 1024, 1001, 980, 960, 941, 923, 906, 889, 874, 859, 844, 830, 817, 804, 791, 779, 768, 756, 745, 734, 724, 714, 704, 694, 685, 676, 667, 658, 650, 642, 633, 626, 618, 610, 603, 595, 588, 581, 574, 567, 561, 554, 548, 542, 535, 529, 523, 517, 512, 506, 500, 495, 489, 484, 478, 473, 468, 463, 458, 453, 448, 443, 438, 434, 429, 424, 420, 415, 411, 407, 402, 398, 394, 390, 386, 382, 377, 373, 370, 366, 362, 358, 354, 350, 347, 343, 339, 336, 332, 329, 325, 322, 318, 315, 311, 308, 305, 302, 298, 295, 292, 289, 286, 282, 279, 276, 273, 270, 267, 264, 261, 258, 256, 253, 250, 247, 244, 241, 239, 236, 233, 230, 228, 225, 222, 220, 217, 215, 212, 209, 207, 204, 202, 199, 197, 194, 192, 190, 187, 185, 182, 180, 178, 175, 173, 171, 168, 166, 164, 162, 159, 157, 155, 153, 151, 149, 146, 144, 142, 140, 138, 136, 134, 132, 130, 128, 126, 123, 121, 119, 117, 115, 114, 112, 110, 108, 106, 104, 102, 100, 98, 96, 94, 93, 91, 89, 87, 85, 83, 82, 80, 78, 76, 74, 73, 71, 69, 67, 66, 64, 62, 61, 59, 57, 55, 54, 52, 50, 49, 47, 46, 44, 42, 41, 39, 37, 36, 34, 33, 31, 30, 28, 26, 25, 23, 22, 20, 19, 17, 16, 14, 13, 11, 10, 8, 7, 5, 4, 2, 1, }; static unsigned ZSTD_getFSEMaxSymbolValue(FSE_CTable const* ctable) { void const* ptr = ctable; U16 const* u16ptr = (U16 const*)ptr; U32 const maxSymbolValue = MEM_read16(u16ptr + 1); return maxSymbolValue; } /** * Returns true if we should use ncount=-1 else we should * use ncount=1 for low probability symbols instead. */ static unsigned ZSTD_useLowProbCount(size_t const nbSeq) { /* Heuristic: This should cover most blocks <= 16K and * start to fade out after 16K to about 32K depending on * compressibility. */ return nbSeq >= 2048; } /** * Returns the cost in bytes of encoding the normalized count header. * Returns an error if any of the helper functions return an error. */ static size_t ZSTD_NCountCost(unsigned const* count, unsigned const max, size_t const nbSeq, unsigned const FSELog) { BYTE wksp[FSE_NCOUNTBOUND]; S16 norm[MaxSeq + 1]; const U32 tableLog = FSE_optimalTableLog(FSELog, nbSeq, max); FORWARD_IF_ERROR(FSE_normalizeCount(norm, tableLog, count, nbSeq, max, ZSTD_useLowProbCount(nbSeq)), ""); return FSE_writeNCount(wksp, sizeof(wksp), norm, max, tableLog); } /** * Returns the cost in bits of encoding the distribution described by count * using the entropy bound. */ static size_t ZSTD_entropyCost(unsigned const* count, unsigned const max, size_t const total) { unsigned cost = 0; unsigned s; assert(total > 0); for (s = 0; s <= max; ++s) { unsigned norm = (unsigned)((256 * count[s]) / total); if (count[s] != 0 && norm == 0) norm = 1; assert(count[s] < total); cost += count[s] * kInverseProbabilityLog256[norm]; } return cost >> 8; } /** * Returns the cost in bits of encoding the distribution in count using ctable. * Returns an error if ctable cannot represent all the symbols in count. */ size_t ZSTD_fseBitCost( FSE_CTable const* ctable, unsigned const* count, unsigned const max) { unsigned const kAccuracyLog = 8; size_t cost = 0; unsigned s; FSE_CState_t cstate; FSE_initCState(&cstate, ctable); if (ZSTD_getFSEMaxSymbolValue(ctable) < max) { DEBUGLOG(5, "Repeat FSE_CTable has maxSymbolValue %u < %u", ZSTD_getFSEMaxSymbolValue(ctable), max); return ERROR(GENERIC); } for (s = 0; s <= max; ++s) { unsigned const tableLog = cstate.stateLog; unsigned const badCost = (tableLog + 1) << kAccuracyLog; unsigned const bitCost = FSE_bitCost(cstate.symbolTT, tableLog, s, kAccuracyLog); if (count[s] == 0) continue; if (bitCost >= badCost) { DEBUGLOG(5, "Repeat FSE_CTable has Prob[%u] == 0", s); return ERROR(GENERIC); } cost += (size_t)count[s] * bitCost; } return cost >> kAccuracyLog; } /** * Returns the cost in bits of encoding the distribution in count using the * table described by norm. The max symbol support by norm is assumed >= max. * norm must be valid for every symbol with non-zero probability in count. */ size_t ZSTD_crossEntropyCost(short const* norm, unsigned accuracyLog, unsigned const* count, unsigned const max) { unsigned const shift = 8 - accuracyLog; size_t cost = 0; unsigned s; assert(accuracyLog <= 8); for (s = 0; s <= max; ++s) { unsigned const normAcc = (norm[s] != -1) ? (unsigned)norm[s] : 1; unsigned const norm256 = normAcc << shift; assert(norm256 > 0); assert(norm256 < 256); cost += count[s] * kInverseProbabilityLog256[norm256]; } return cost >> 8; } symbolEncodingType_e ZSTD_selectEncodingType( FSE_repeat* repeatMode, unsigned const* count, unsigned const max, size_t const mostFrequent, size_t nbSeq, unsigned const FSELog, FSE_CTable const* prevCTable, short const* defaultNorm, U32 defaultNormLog, ZSTD_defaultPolicy_e const isDefaultAllowed, ZSTD_strategy const strategy) { ZSTD_STATIC_ASSERT(ZSTD_defaultDisallowed == 0 && ZSTD_defaultAllowed != 0); if (mostFrequent == nbSeq) { *repeatMode = FSE_repeat_none; if (isDefaultAllowed && nbSeq <= 2) { /* Prefer set_basic over set_rle when there are 2 or fewer symbols, * since RLE uses 1 byte, but set_basic uses 5-6 bits per symbol. * If basic encoding isn't possible, always choose RLE. */ DEBUGLOG(5, "Selected set_basic"); return set_basic; } DEBUGLOG(5, "Selected set_rle"); return set_rle; } if (strategy < ZSTD_lazy) { if (isDefaultAllowed) { size_t const staticFse_nbSeq_max = 1000; size_t const mult = 10 - strategy; size_t const baseLog = 3; size_t const dynamicFse_nbSeq_min = (((size_t)1 << defaultNormLog) * mult) >> baseLog; /* 28-36 for offset, 56-72 for lengths */ assert(defaultNormLog >= 5 && defaultNormLog <= 6); /* xx_DEFAULTNORMLOG */ assert(mult <= 9 && mult >= 7); if ( (*repeatMode == FSE_repeat_valid) && (nbSeq < staticFse_nbSeq_max) ) { DEBUGLOG(5, "Selected set_repeat"); return set_repeat; } if ( (nbSeq < dynamicFse_nbSeq_min) || (mostFrequent < (nbSeq >> (defaultNormLog-1))) ) { DEBUGLOG(5, "Selected set_basic"); /* The format allows default tables to be repeated, but it isn't useful. * When using simple heuristics to select encoding type, we don't want * to confuse these tables with dictionaries. When running more careful * analysis, we don't need to waste time checking both repeating tables * and default tables. */ *repeatMode = FSE_repeat_none; return set_basic; } } } else { size_t const basicCost = isDefaultAllowed ? ZSTD_crossEntropyCost(defaultNorm, defaultNormLog, count, max) : ERROR(GENERIC); size_t const repeatCost = *repeatMode != FSE_repeat_none ? ZSTD_fseBitCost(prevCTable, count, max) : ERROR(GENERIC); size_t const NCountCost = ZSTD_NCountCost(count, max, nbSeq, FSELog); size_t const compressedCost = (NCountCost << 3) + ZSTD_entropyCost(count, max, nbSeq); if (isDefaultAllowed) { assert(!ZSTD_isError(basicCost)); assert(!(*repeatMode == FSE_repeat_valid && ZSTD_isError(repeatCost))); } assert(!ZSTD_isError(NCountCost)); assert(compressedCost < ERROR(maxCode)); DEBUGLOG(5, "Estimated bit costs: basic=%u\trepeat=%u\tcompressed=%u", (unsigned)basicCost, (unsigned)repeatCost, (unsigned)compressedCost); if (basicCost <= repeatCost && basicCost <= compressedCost) { DEBUGLOG(5, "Selected set_basic"); assert(isDefaultAllowed); *repeatMode = FSE_repeat_none; return set_basic; } if (repeatCost <= compressedCost) { DEBUGLOG(5, "Selected set_repeat"); assert(!ZSTD_isError(repeatCost)); return set_repeat; } assert(compressedCost < basicCost && compressedCost < repeatCost); } DEBUGLOG(5, "Selected set_compressed"); *repeatMode = FSE_repeat_check; return set_compressed; } typedef struct { S16 norm[MaxSeq + 1]; U32 wksp[FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(MaxSeq, MaxFSELog)]; } ZSTD_BuildCTableWksp; size_t ZSTD_buildCTable(void* dst, size_t dstCapacity, FSE_CTable* nextCTable, U32 FSELog, symbolEncodingType_e type, unsigned* count, U32 max, const BYTE* codeTable, size_t nbSeq, const S16* defaultNorm, U32 defaultNormLog, U32 defaultMax, const FSE_CTable* prevCTable, size_t prevCTableSize, void* entropyWorkspace, size_t entropyWorkspaceSize) { BYTE* op = (BYTE*)dst; const BYTE* const oend = op + dstCapacity; DEBUGLOG(6, "ZSTD_buildCTable (dstCapacity=%u)", (unsigned)dstCapacity); switch (type) { case set_rle: FORWARD_IF_ERROR(FSE_buildCTable_rle(nextCTable, (BYTE)max), ""); RETURN_ERROR_IF(dstCapacity==0, dstSize_tooSmall, "not enough space"); *op = codeTable[0]; return 1; case set_repeat: ZSTD_memcpy(nextCTable, prevCTable, prevCTableSize); return 0; case set_basic: FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, defaultNorm, defaultMax, defaultNormLog, entropyWorkspace, entropyWorkspaceSize), ""); /* note : could be pre-calculated */ return 0; case set_compressed: { ZSTD_BuildCTableWksp* wksp = (ZSTD_BuildCTableWksp*)entropyWorkspace; size_t nbSeq_1 = nbSeq; const U32 tableLog = FSE_optimalTableLog(FSELog, nbSeq, max); if (count[codeTable[nbSeq-1]] > 1) { count[codeTable[nbSeq-1]]--; nbSeq_1--; } assert(nbSeq_1 > 1); assert(entropyWorkspaceSize >= sizeof(ZSTD_BuildCTableWksp)); (void)entropyWorkspaceSize; FORWARD_IF_ERROR(FSE_normalizeCount(wksp->norm, tableLog, count, nbSeq_1, max, ZSTD_useLowProbCount(nbSeq_1)), "FSE_normalizeCount failed"); assert(oend >= op); { size_t const NCountSize = FSE_writeNCount(op, (size_t)(oend - op), wksp->norm, max, tableLog); /* overflow protected */ FORWARD_IF_ERROR(NCountSize, "FSE_writeNCount failed"); FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, wksp->norm, max, tableLog, wksp->wksp, sizeof(wksp->wksp)), "FSE_buildCTable_wksp failed"); return NCountSize; } } default: assert(0); RETURN_ERROR(GENERIC, "impossible to reach"); } } FORCE_INLINE_TEMPLATE size_t ZSTD_encodeSequences_body( void* dst, size_t dstCapacity, FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable, FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable, FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable, seqDef const* sequences, size_t nbSeq, int longOffsets) { BIT_CStream_t blockStream; FSE_CState_t stateMatchLength; FSE_CState_t stateOffsetBits; FSE_CState_t stateLitLength; RETURN_ERROR_IF( ERR_isError(BIT_initCStream(&blockStream, dst, dstCapacity)), dstSize_tooSmall, "not enough space remaining"); DEBUGLOG(6, "available space for bitstream : %i (dstCapacity=%u)", (int)(blockStream.endPtr - blockStream.startPtr), (unsigned)dstCapacity); /* first symbols */ FSE_initCState2(&stateMatchLength, CTable_MatchLength, mlCodeTable[nbSeq-1]); FSE_initCState2(&stateOffsetBits, CTable_OffsetBits, ofCodeTable[nbSeq-1]); FSE_initCState2(&stateLitLength, CTable_LitLength, llCodeTable[nbSeq-1]); BIT_addBits(&blockStream, sequences[nbSeq-1].litLength, LL_bits[llCodeTable[nbSeq-1]]); if (MEM_32bits()) BIT_flushBits(&blockStream); BIT_addBits(&blockStream, sequences[nbSeq-1].mlBase, ML_bits[mlCodeTable[nbSeq-1]]); if (MEM_32bits()) BIT_flushBits(&blockStream); if (longOffsets) { U32 const ofBits = ofCodeTable[nbSeq-1]; unsigned const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN-1); if (extraBits) { BIT_addBits(&blockStream, sequences[nbSeq-1].offBase, extraBits); BIT_flushBits(&blockStream); } BIT_addBits(&blockStream, sequences[nbSeq-1].offBase >> extraBits, ofBits - extraBits); } else { BIT_addBits(&blockStream, sequences[nbSeq-1].offBase, ofCodeTable[nbSeq-1]); } BIT_flushBits(&blockStream); { size_t n; for (n=nbSeq-2 ; n= 64-7-(LLFSELog+MLFSELog+OffFSELog))) BIT_flushBits(&blockStream); /* (7)*/ BIT_addBits(&blockStream, sequences[n].litLength, llBits); if (MEM_32bits() && ((llBits+mlBits)>24)) BIT_flushBits(&blockStream); BIT_addBits(&blockStream, sequences[n].mlBase, mlBits); if (MEM_32bits() || (ofBits+mlBits+llBits > 56)) BIT_flushBits(&blockStream); if (longOffsets) { unsigned const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN-1); if (extraBits) { BIT_addBits(&blockStream, sequences[n].offBase, extraBits); BIT_flushBits(&blockStream); /* (7)*/ } BIT_addBits(&blockStream, sequences[n].offBase >> extraBits, ofBits - extraBits); /* 31 */ } else { BIT_addBits(&blockStream, sequences[n].offBase, ofBits); /* 31 */ } BIT_flushBits(&blockStream); /* (7)*/ DEBUGLOG(7, "remaining space : %i", (int)(blockStream.endPtr - blockStream.ptr)); } } DEBUGLOG(6, "ZSTD_encodeSequences: flushing ML state with %u bits", stateMatchLength.stateLog); FSE_flushCState(&blockStream, &stateMatchLength); DEBUGLOG(6, "ZSTD_encodeSequences: flushing Off state with %u bits", stateOffsetBits.stateLog); FSE_flushCState(&blockStream, &stateOffsetBits); DEBUGLOG(6, "ZSTD_encodeSequences: flushing LL state with %u bits", stateLitLength.stateLog); FSE_flushCState(&blockStream, &stateLitLength); { size_t const streamSize = BIT_closeCStream(&blockStream); RETURN_ERROR_IF(streamSize==0, dstSize_tooSmall, "not enough space"); return streamSize; } } static size_t ZSTD_encodeSequences_default( void* dst, size_t dstCapacity, FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable, FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable, FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable, seqDef const* sequences, size_t nbSeq, int longOffsets) { return ZSTD_encodeSequences_body(dst, dstCapacity, CTable_MatchLength, mlCodeTable, CTable_OffsetBits, ofCodeTable, CTable_LitLength, llCodeTable, sequences, nbSeq, longOffsets); } #if DYNAMIC_BMI2 static BMI2_TARGET_ATTRIBUTE size_t ZSTD_encodeSequences_bmi2( void* dst, size_t dstCapacity, FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable, FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable, FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable, seqDef const* sequences, size_t nbSeq, int longOffsets) { return ZSTD_encodeSequences_body(dst, dstCapacity, CTable_MatchLength, mlCodeTable, CTable_OffsetBits, ofCodeTable, CTable_LitLength, llCodeTable, sequences, nbSeq, longOffsets); } #endif size_t ZSTD_encodeSequences( void* dst, size_t dstCapacity, FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable, FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable, FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable, seqDef const* sequences, size_t nbSeq, int longOffsets, int bmi2) { DEBUGLOG(5, "ZSTD_encodeSequences: dstCapacity = %u", (unsigned)dstCapacity); #if DYNAMIC_BMI2 if (bmi2) { return ZSTD_encodeSequences_bmi2(dst, dstCapacity, CTable_MatchLength, mlCodeTable, CTable_OffsetBits, ofCodeTable, CTable_LitLength, llCodeTable, sequences, nbSeq, longOffsets); } #endif (void)bmi2; return ZSTD_encodeSequences_default(dst, dstCapacity, CTable_MatchLength, mlCodeTable, CTable_OffsetBits, ofCodeTable, CTable_LitLength, llCodeTable, sequences, nbSeq, longOffsets); } zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/fse_compress.c0000644000175200007730000005704014515254731026015 0ustar rlaboissrlaboiss/* ****************************************************************** * FSE : Finite State Entropy encoder * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy * - Public forum : https://groups.google.com/forum/#!forum/lz4c * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ /* ************************************************************** * Includes ****************************************************************/ #include "../common/compiler.h" #include "../common/mem.h" /* U32, U16, etc. */ #include "../common/debug.h" /* assert, DEBUGLOG */ #include "hist.h" /* HIST_count_wksp */ #include "../common/bitstream.h" #define FSE_STATIC_LINKING_ONLY #include "../common/fse.h" #include "../common/error_private.h" #define ZSTD_DEPS_NEED_MALLOC #define ZSTD_DEPS_NEED_MATH64 #include "../common/zstd_deps.h" /* ZSTD_malloc, ZSTD_free, ZSTD_memcpy, ZSTD_memset */ #include "../common/bits.h" /* ZSTD_highbit32 */ /* ************************************************************** * Error Management ****************************************************************/ #define FSE_isError ERR_isError /* ************************************************************** * Templates ****************************************************************/ /* designed to be included for type-specific functions (template emulation in C) Objective is to write these functions only once, for improved maintenance */ /* safety checks */ #ifndef FSE_FUNCTION_EXTENSION # error "FSE_FUNCTION_EXTENSION must be defined" #endif #ifndef FSE_FUNCTION_TYPE # error "FSE_FUNCTION_TYPE must be defined" #endif /* Function names */ #define FSE_CAT(X,Y) X##Y #define FSE_FUNCTION_NAME(X,Y) FSE_CAT(X,Y) #define FSE_TYPE_NAME(X,Y) FSE_CAT(X,Y) /* Function templates */ /* FSE_buildCTable_wksp() : * Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`). * wkspSize should be sized to handle worst case situation, which is `1<>1 : 1) ; FSE_symbolCompressionTransform* const symbolTT = (FSE_symbolCompressionTransform*) (FSCT); U32 const step = FSE_TABLESTEP(tableSize); U32 const maxSV1 = maxSymbolValue+1; U16* cumul = (U16*)workSpace; /* size = maxSV1 */ FSE_FUNCTION_TYPE* const tableSymbol = (FSE_FUNCTION_TYPE*)(cumul + (maxSV1+1)); /* size = tableSize */ U32 highThreshold = tableSize-1; assert(((size_t)workSpace & 1) == 0); /* Must be 2 bytes-aligned */ if (FSE_BUILD_CTABLE_WORKSPACE_SIZE(maxSymbolValue, tableLog) > wkspSize) return ERROR(tableLog_tooLarge); /* CTable header */ tableU16[-2] = (U16) tableLog; tableU16[-1] = (U16) maxSymbolValue; assert(tableLog < 16); /* required for threshold strategy to work */ /* For explanations on how to distribute symbol values over the table : * https://fastcompression.blogspot.fr/2014/02/fse-distributing-symbol-values.html */ #ifdef __clang_analyzer__ ZSTD_memset(tableSymbol, 0, sizeof(*tableSymbol) * tableSize); /* useless initialization, just to keep scan-build happy */ #endif /* symbol start positions */ { U32 u; cumul[0] = 0; for (u=1; u <= maxSV1; u++) { if (normalizedCounter[u-1]==-1) { /* Low proba symbol */ cumul[u] = cumul[u-1] + 1; tableSymbol[highThreshold--] = (FSE_FUNCTION_TYPE)(u-1); } else { assert(normalizedCounter[u-1] >= 0); cumul[u] = cumul[u-1] + (U16)normalizedCounter[u-1]; assert(cumul[u] >= cumul[u-1]); /* no overflow */ } } cumul[maxSV1] = (U16)(tableSize+1); } /* Spread symbols */ if (highThreshold == tableSize - 1) { /* Case for no low prob count symbols. Lay down 8 bytes at a time * to reduce branch misses since we are operating on a small block */ BYTE* const spread = tableSymbol + tableSize; /* size = tableSize + 8 (may write beyond tableSize) */ { U64 const add = 0x0101010101010101ull; size_t pos = 0; U64 sv = 0; U32 s; for (s=0; s=0); pos += (size_t)n; } } /* Spread symbols across the table. Lack of lowprob symbols means that * we don't need variable sized inner loop, so we can unroll the loop and * reduce branch misses. */ { size_t position = 0; size_t s; size_t const unroll = 2; /* Experimentally determined optimal unroll */ assert(tableSize % unroll == 0); /* FSE_MIN_TABLELOG is 5 */ for (s = 0; s < (size_t)tableSize; s += unroll) { size_t u; for (u = 0; u < unroll; ++u) { size_t const uPosition = (position + (u * step)) & tableMask; tableSymbol[uPosition] = spread[s + u]; } position = (position + (unroll * step)) & tableMask; } assert(position == 0); /* Must have initialized all positions */ } } else { U32 position = 0; U32 symbol; for (symbol=0; symbol highThreshold) position = (position + step) & tableMask; /* Low proba area */ } } assert(position==0); /* Must have initialized all positions */ } /* Build table */ { U32 u; for (u=0; u 1); { U32 const maxBitsOut = tableLog - ZSTD_highbit32 ((U32)normalizedCounter[s]-1); U32 const minStatePlus = (U32)normalizedCounter[s] << maxBitsOut; symbolTT[s].deltaNbBits = (maxBitsOut << 16) - minStatePlus; symbolTT[s].deltaFindState = (int)(total - (unsigned)normalizedCounter[s]); total += (unsigned)normalizedCounter[s]; } } } } #if 0 /* debug : symbol costs */ DEBUGLOG(5, "\n --- table statistics : "); { U32 symbol; for (symbol=0; symbol<=maxSymbolValue; symbol++) { DEBUGLOG(5, "%3u: w=%3i, maxBits=%u, fracBits=%.2f", symbol, normalizedCounter[symbol], FSE_getMaxNbBits(symbolTT, symbol), (double)FSE_bitCost(symbolTT, tableLog, symbol, 8) / 256); } } #endif return 0; } #ifndef FSE_COMMONDEFS_ONLY /*-************************************************************** * FSE NCount encoding ****************************************************************/ size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog) { size_t const maxHeaderSize = (((maxSymbolValue+1) * tableLog + 4 /* bitCount initialized at 4 */ + 2 /* first two symbols may use one additional bit each */) / 8) + 1 /* round up to whole nb bytes */ + 2 /* additional two bytes for bitstream flush */; return maxSymbolValue ? maxHeaderSize : FSE_NCOUNTBOUND; /* maxSymbolValue==0 ? use default */ } static size_t FSE_writeNCount_generic (void* header, size_t headerBufferSize, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, unsigned writeIsSafe) { BYTE* const ostart = (BYTE*) header; BYTE* out = ostart; BYTE* const oend = ostart + headerBufferSize; int nbBits; const int tableSize = 1 << tableLog; int remaining; int threshold; U32 bitStream = 0; int bitCount = 0; unsigned symbol = 0; unsigned const alphabetSize = maxSymbolValue + 1; int previousIs0 = 0; /* Table Size */ bitStream += (tableLog-FSE_MIN_TABLELOG) << bitCount; bitCount += 4; /* Init */ remaining = tableSize+1; /* +1 for extra accuracy */ threshold = tableSize; nbBits = tableLog+1; while ((symbol < alphabetSize) && (remaining>1)) { /* stops at 1 */ if (previousIs0) { unsigned start = symbol; while ((symbol < alphabetSize) && !normalizedCounter[symbol]) symbol++; if (symbol == alphabetSize) break; /* incorrect distribution */ while (symbol >= start+24) { start+=24; bitStream += 0xFFFFU << bitCount; if ((!writeIsSafe) && (out > oend-2)) return ERROR(dstSize_tooSmall); /* Buffer overflow */ out[0] = (BYTE) bitStream; out[1] = (BYTE)(bitStream>>8); out+=2; bitStream>>=16; } while (symbol >= start+3) { start+=3; bitStream += 3 << bitCount; bitCount += 2; } bitStream += (symbol-start) << bitCount; bitCount += 2; if (bitCount>16) { if ((!writeIsSafe) && (out > oend - 2)) return ERROR(dstSize_tooSmall); /* Buffer overflow */ out[0] = (BYTE)bitStream; out[1] = (BYTE)(bitStream>>8); out += 2; bitStream >>= 16; bitCount -= 16; } } { int count = normalizedCounter[symbol++]; int const max = (2*threshold-1) - remaining; remaining -= count < 0 ? -count : count; count++; /* +1 for extra accuracy */ if (count>=threshold) count += max; /* [0..max[ [max..threshold[ (...) [threshold+max 2*threshold[ */ bitStream += count << bitCount; bitCount += nbBits; bitCount -= (count>=1; } } if (bitCount>16) { if ((!writeIsSafe) && (out > oend - 2)) return ERROR(dstSize_tooSmall); /* Buffer overflow */ out[0] = (BYTE)bitStream; out[1] = (BYTE)(bitStream>>8); out += 2; bitStream >>= 16; bitCount -= 16; } } if (remaining != 1) return ERROR(GENERIC); /* incorrect normalized distribution */ assert(symbol <= alphabetSize); /* flush remaining bitStream */ if ((!writeIsSafe) && (out > oend - 2)) return ERROR(dstSize_tooSmall); /* Buffer overflow */ out[0] = (BYTE)bitStream; out[1] = (BYTE)(bitStream>>8); out+= (bitCount+7) /8; return (out-ostart); } size_t FSE_writeNCount (void* buffer, size_t bufferSize, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog) { if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge); /* Unsupported */ if (tableLog < FSE_MIN_TABLELOG) return ERROR(GENERIC); /* Unsupported */ if (bufferSize < FSE_NCountWriteBound(maxSymbolValue, tableLog)) return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 0); return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 1 /* write in buffer is safe */); } /*-************************************************************** * FSE Compression Code ****************************************************************/ /* provides the minimum logSize to safely represent a distribution */ static unsigned FSE_minTableLog(size_t srcSize, unsigned maxSymbolValue) { U32 minBitsSrc = ZSTD_highbit32((U32)(srcSize)) + 1; U32 minBitsSymbols = ZSTD_highbit32(maxSymbolValue) + 2; U32 minBits = minBitsSrc < minBitsSymbols ? minBitsSrc : minBitsSymbols; assert(srcSize > 1); /* Not supported, RLE should be used instead */ return minBits; } unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus) { U32 maxBitsSrc = ZSTD_highbit32((U32)(srcSize - 1)) - minus; U32 tableLog = maxTableLog; U32 minBits = FSE_minTableLog(srcSize, maxSymbolValue); assert(srcSize > 1); /* Not supported, RLE should be used instead */ if (tableLog==0) tableLog = FSE_DEFAULT_TABLELOG; if (maxBitsSrc < tableLog) tableLog = maxBitsSrc; /* Accuracy can be reduced */ if (minBits > tableLog) tableLog = minBits; /* Need a minimum to safely represent all symbol values */ if (tableLog < FSE_MIN_TABLELOG) tableLog = FSE_MIN_TABLELOG; if (tableLog > FSE_MAX_TABLELOG) tableLog = FSE_MAX_TABLELOG; return tableLog; } unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue) { return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 2); } /* Secondary normalization method. To be used when primary method fails. */ static size_t FSE_normalizeM2(short* norm, U32 tableLog, const unsigned* count, size_t total, U32 maxSymbolValue, short lowProbCount) { short const NOT_YET_ASSIGNED = -2; U32 s; U32 distributed = 0; U32 ToDistribute; /* Init */ U32 const lowThreshold = (U32)(total >> tableLog); U32 lowOne = (U32)((total * 3) >> (tableLog + 1)); for (s=0; s<=maxSymbolValue; s++) { if (count[s] == 0) { norm[s]=0; continue; } if (count[s] <= lowThreshold) { norm[s] = lowProbCount; distributed++; total -= count[s]; continue; } if (count[s] <= lowOne) { norm[s] = 1; distributed++; total -= count[s]; continue; } norm[s]=NOT_YET_ASSIGNED; } ToDistribute = (1 << tableLog) - distributed; if (ToDistribute == 0) return 0; if ((total / ToDistribute) > lowOne) { /* risk of rounding to zero */ lowOne = (U32)((total * 3) / (ToDistribute * 2)); for (s=0; s<=maxSymbolValue; s++) { if ((norm[s] == NOT_YET_ASSIGNED) && (count[s] <= lowOne)) { norm[s] = 1; distributed++; total -= count[s]; continue; } } ToDistribute = (1 << tableLog) - distributed; } if (distributed == maxSymbolValue+1) { /* all values are pretty poor; probably incompressible data (should have already been detected); find max, then give all remaining points to max */ U32 maxV = 0, maxC = 0; for (s=0; s<=maxSymbolValue; s++) if (count[s] > maxC) { maxV=s; maxC=count[s]; } norm[maxV] += (short)ToDistribute; return 0; } if (total == 0) { /* all of the symbols were low enough for the lowOne or lowThreshold */ for (s=0; ToDistribute > 0; s = (s+1)%(maxSymbolValue+1)) if (norm[s] > 0) { ToDistribute--; norm[s]++; } return 0; } { U64 const vStepLog = 62 - tableLog; U64 const mid = (1ULL << (vStepLog-1)) - 1; U64 const rStep = ZSTD_div64((((U64)1<> vStepLog); U32 const sEnd = (U32)(end >> vStepLog); U32 const weight = sEnd - sStart; if (weight < 1) return ERROR(GENERIC); norm[s] = (short)weight; tmpTotal = end; } } } return 0; } size_t FSE_normalizeCount (short* normalizedCounter, unsigned tableLog, const unsigned* count, size_t total, unsigned maxSymbolValue, unsigned useLowProbCount) { /* Sanity checks */ if (tableLog==0) tableLog = FSE_DEFAULT_TABLELOG; if (tableLog < FSE_MIN_TABLELOG) return ERROR(GENERIC); /* Unsupported size */ if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge); /* Unsupported size */ if (tableLog < FSE_minTableLog(total, maxSymbolValue)) return ERROR(GENERIC); /* Too small tableLog, compression potentially impossible */ { static U32 const rtbTable[] = { 0, 473195, 504333, 520860, 550000, 700000, 750000, 830000 }; short const lowProbCount = useLowProbCount ? -1 : 1; U64 const scale = 62 - tableLog; U64 const step = ZSTD_div64((U64)1<<62, (U32)total); /* <== here, one division ! */ U64 const vStep = 1ULL<<(scale-20); int stillToDistribute = 1<> tableLog); for (s=0; s<=maxSymbolValue; s++) { if (count[s] == total) return 0; /* rle special case */ if (count[s] == 0) { normalizedCounter[s]=0; continue; } if (count[s] <= lowThreshold) { normalizedCounter[s] = lowProbCount; stillToDistribute--; } else { short proba = (short)((count[s]*step) >> scale); if (proba<8) { U64 restToBeat = vStep * rtbTable[proba]; proba += (count[s]*step) - ((U64)proba< restToBeat; } if (proba > largestP) { largestP=proba; largest=s; } normalizedCounter[s] = proba; stillToDistribute -= proba; } } if (-stillToDistribute >= (normalizedCounter[largest] >> 1)) { /* corner case, need another normalization method */ size_t const errorCode = FSE_normalizeM2(normalizedCounter, tableLog, count, total, maxSymbolValue, lowProbCount); if (FSE_isError(errorCode)) return errorCode; } else normalizedCounter[largest] += (short)stillToDistribute; } #if 0 { /* Print Table (debug) */ U32 s; U32 nTotal = 0; for (s=0; s<=maxSymbolValue; s++) RAWLOG(2, "%3i: %4i \n", s, normalizedCounter[s]); for (s=0; s<=maxSymbolValue; s++) nTotal += abs(normalizedCounter[s]); if (nTotal != (1U< FSE_MAX_TABLELOG*4+7 ) && (srcSize & 2)) { /* test bit 2 */ FSE_encodeSymbol(&bitC, &CState2, *--ip); FSE_encodeSymbol(&bitC, &CState1, *--ip); FSE_FLUSHBITS(&bitC); } /* 2 or 4 encoding per loop */ while ( ip>istart ) { FSE_encodeSymbol(&bitC, &CState2, *--ip); if (sizeof(bitC.bitContainer)*8 < FSE_MAX_TABLELOG*2+7 ) /* this test must be static */ FSE_FLUSHBITS(&bitC); FSE_encodeSymbol(&bitC, &CState1, *--ip); if (sizeof(bitC.bitContainer)*8 > FSE_MAX_TABLELOG*4+7 ) { /* this test must be static */ FSE_encodeSymbol(&bitC, &CState2, *--ip); FSE_encodeSymbol(&bitC, &CState1, *--ip); } FSE_FLUSHBITS(&bitC); } FSE_flushCState(&bitC, &CState2); FSE_flushCState(&bitC, &CState1); return BIT_closeCStream(&bitC); } size_t FSE_compress_usingCTable (void* dst, size_t dstSize, const void* src, size_t srcSize, const FSE_CTable* ct) { unsigned const fast = (dstSize >= FSE_BLOCKBOUND(srcSize)); if (fast) return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 1); else return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 0); } size_t FSE_compressBound(size_t size) { return FSE_COMPRESSBOUND(size); } #endif /* FSE_COMMONDEFS_ONLY */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_opt.h0000644000175200007730000000373314515254731025200 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_OPT_H #define ZSTD_OPT_H #if defined (__cplusplus) extern "C" { #endif #include "zstd_compress_internal.h" /* used in ZSTD_loadDictionaryContent() */ void ZSTD_updateTree(ZSTD_matchState_t* ms, const BYTE* ip, const BYTE* iend); size_t ZSTD_compressBlock_btopt( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_btultra( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_btultra2( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_btopt_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_btultra_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_btopt_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_btultra_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); /* note : no btultra2 variant for extDict nor dictMatchState, * because btultra2 is not meant to work with dictionaries * and is only specific for the first block (no prefix) */ #if defined (__cplusplus) } #endif #endif /* ZSTD_OPT_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_ldm_geartab.h0000644000175200007730000001367514515254731026645 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_LDM_GEARTAB_H #define ZSTD_LDM_GEARTAB_H #include "../common/compiler.h" /* UNUSED_ATTR */ #include "../common/mem.h" /* U64 */ static UNUSED_ATTR const U64 ZSTD_ldm_gearTab[256] = { 0xf5b8f72c5f77775c, 0x84935f266b7ac412, 0xb647ada9ca730ccc, 0xb065bb4b114fb1de, 0x34584e7e8c3a9fd0, 0x4e97e17c6ae26b05, 0x3a03d743bc99a604, 0xcecd042422c4044f, 0x76de76c58524259e, 0x9c8528f65badeaca, 0x86563706e2097529, 0x2902475fa375d889, 0xafb32a9739a5ebe6, 0xce2714da3883e639, 0x21eaf821722e69e, 0x37b628620b628, 0x49a8d455d88caf5, 0x8556d711e6958140, 0x4f7ae74fc605c1f, 0x829f0c3468bd3a20, 0x4ffdc885c625179e, 0x8473de048a3daf1b, 0x51008822b05646b2, 0x69d75d12b2d1cc5f, 0x8c9d4a19159154bc, 0xc3cc10f4abbd4003, 0xd06ddc1cecb97391, 0xbe48e6e7ed80302e, 0x3481db31cee03547, 0xacc3f67cdaa1d210, 0x65cb771d8c7f96cc, 0x8eb27177055723dd, 0xc789950d44cd94be, 0x934feadc3700b12b, 0x5e485f11edbdf182, 0x1e2e2a46fd64767a, 0x2969ca71d82efa7c, 0x9d46e9935ebbba2e, 0xe056b67e05e6822b, 0x94d73f55739d03a0, 0xcd7010bdb69b5a03, 0x455ef9fcd79b82f4, 0x869cb54a8749c161, 0x38d1a4fa6185d225, 0xb475166f94bbe9bb, 0xa4143548720959f1, 0x7aed4780ba6b26ba, 0xd0ce264439e02312, 0x84366d746078d508, 0xa8ce973c72ed17be, 0x21c323a29a430b01, 0x9962d617e3af80ee, 0xab0ce91d9c8cf75b, 0x530e8ee6d19a4dbc, 0x2ef68c0cf53f5d72, 0xc03a681640a85506, 0x496e4e9f9c310967, 0x78580472b59b14a0, 0x273824c23b388577, 0x66bf923ad45cb553, 0x47ae1a5a2492ba86, 0x35e304569e229659, 0x4765182a46870b6f, 0x6cbab625e9099412, 0xddac9a2e598522c1, 0x7172086e666624f2, 0xdf5003ca503b7837, 0x88c0c1db78563d09, 0x58d51865acfc289d, 0x177671aec65224f1, 0xfb79d8a241e967d7, 0x2be1e101cad9a49a, 0x6625682f6e29186b, 0x399553457ac06e50, 0x35dffb4c23abb74, 0x429db2591f54aade, 0xc52802a8037d1009, 0x6acb27381f0b25f3, 0xf45e2551ee4f823b, 0x8b0ea2d99580c2f7, 0x3bed519cbcb4e1e1, 0xff452823dbb010a, 0x9d42ed614f3dd267, 0x5b9313c06257c57b, 0xa114b8008b5e1442, 0xc1fe311c11c13d4b, 0x66e8763ea34c5568, 0x8b982af1c262f05d, 0xee8876faaa75fbb7, 0x8a62a4d0d172bb2a, 0xc13d94a3b7449a97, 0x6dbbba9dc15d037c, 0xc786101f1d92e0f1, 0xd78681a907a0b79b, 0xf61aaf2962c9abb9, 0x2cfd16fcd3cb7ad9, 0x868c5b6744624d21, 0x25e650899c74ddd7, 0xba042af4a7c37463, 0x4eb1a539465a3eca, 0xbe09dbf03b05d5ca, 0x774e5a362b5472ba, 0x47a1221229d183cd, 0x504b0ca18ef5a2df, 0xdffbdfbde2456eb9, 0x46cd2b2fbee34634, 0xf2aef8fe819d98c3, 0x357f5276d4599d61, 0x24a5483879c453e3, 0x88026889192b4b9, 0x28da96671782dbec, 0x4ef37c40588e9aaa, 0x8837b90651bc9fb3, 0xc164f741d3f0e5d6, 0xbc135a0a704b70ba, 0x69cd868f7622ada, 0xbc37ba89e0b9c0ab, 0x47c14a01323552f6, 0x4f00794bacee98bb, 0x7107de7d637a69d5, 0x88af793bb6f2255e, 0xf3c6466b8799b598, 0xc288c616aa7f3b59, 0x81ca63cf42fca3fd, 0x88d85ace36a2674b, 0xd056bd3792389e7, 0xe55c396c4e9dd32d, 0xbefb504571e6c0a6, 0x96ab32115e91e8cc, 0xbf8acb18de8f38d1, 0x66dae58801672606, 0x833b6017872317fb, 0xb87c16f2d1c92864, 0xdb766a74e58b669c, 0x89659f85c61417be, 0xc8daad856011ea0c, 0x76a4b565b6fe7eae, 0xa469d085f6237312, 0xaaf0365683a3e96c, 0x4dbb746f8424f7b8, 0x638755af4e4acc1, 0x3d7807f5bde64486, 0x17be6d8f5bbb7639, 0x903f0cd44dc35dc, 0x67b672eafdf1196c, 0xa676ff93ed4c82f1, 0x521d1004c5053d9d, 0x37ba9ad09ccc9202, 0x84e54d297aacfb51, 0xa0b4b776a143445, 0x820d471e20b348e, 0x1874383cb83d46dc, 0x97edeec7a1efe11c, 0xb330e50b1bdc42aa, 0x1dd91955ce70e032, 0xa514cdb88f2939d5, 0x2791233fd90db9d3, 0x7b670a4cc50f7a9b, 0x77c07d2a05c6dfa5, 0xe3778b6646d0a6fa, 0xb39c8eda47b56749, 0x933ed448addbef28, 0xaf846af6ab7d0bf4, 0xe5af208eb666e49, 0x5e6622f73534cd6a, 0x297daeca42ef5b6e, 0x862daef3d35539a6, 0xe68722498f8e1ea9, 0x981c53093dc0d572, 0xfa09b0bfbf86fbf5, 0x30b1e96166219f15, 0x70e7d466bdc4fb83, 0x5a66736e35f2a8e9, 0xcddb59d2b7c1baef, 0xd6c7d247d26d8996, 0xea4e39eac8de1ba3, 0x539c8bb19fa3aff2, 0x9f90e4c5fd508d8, 0xa34e5956fbaf3385, 0x2e2f8e151d3ef375, 0x173691e9b83faec1, 0xb85a8d56bf016379, 0x8382381267408ae3, 0xb90f901bbdc0096d, 0x7c6ad32933bcec65, 0x76bb5e2f2c8ad595, 0x390f851a6cf46d28, 0xc3e6064da1c2da72, 0xc52a0c101cfa5389, 0xd78eaf84a3fbc530, 0x3781b9e2288b997e, 0x73c2f6dea83d05c4, 0x4228e364c5b5ed7, 0x9d7a3edf0da43911, 0x8edcfeda24686756, 0x5e7667a7b7a9b3a1, 0x4c4f389fa143791d, 0xb08bc1023da7cddc, 0x7ab4be3ae529b1cc, 0x754e6132dbe74ff9, 0x71635442a839df45, 0x2f6fb1643fbe52de, 0x961e0a42cf7a8177, 0xf3b45d83d89ef2ea, 0xee3de4cf4a6e3e9b, 0xcd6848542c3295e7, 0xe4cee1664c78662f, 0x9947548b474c68c4, 0x25d73777a5ed8b0b, 0xc915b1d636b7fc, 0x21c2ba75d9b0d2da, 0x5f6b5dcf608a64a1, 0xdcf333255ff9570c, 0x633b922418ced4ee, 0xc136dde0b004b34a, 0x58cc83b05d4b2f5a, 0x5eb424dda28e42d2, 0x62df47369739cd98, 0xb4e0b42485e4ce17, 0x16e1f0c1f9a8d1e7, 0x8ec3916707560ebf, 0x62ba6e2df2cc9db3, 0xcbf9f4ff77d83a16, 0x78d9d7d07d2bbcc4, 0xef554ce1e02c41f4, 0x8d7581127eccf94d, 0xa9b53336cb3c8a05, 0x38c42c0bf45c4f91, 0x640893cdf4488863, 0x80ec34bc575ea568, 0x39f324f5b48eaa40, 0xe9d9ed1f8eff527f, 0x9224fc058cc5a214, 0xbaba00b04cfe7741, 0x309a9f120fcf52af, 0xa558f3ec65626212, 0x424bec8b7adabe2f, 0x41622513a6aea433, 0xb88da2d5324ca798, 0xd287733b245528a4, 0x9a44697e6d68aec3, 0x7b1093be2f49bb28, 0x50bbec632e3d8aad, 0x6cd90723e1ea8283, 0x897b9e7431b02bf3, 0x219efdcb338a7047, 0x3b0311f0a27c0656, 0xdb17bf91c0db96e7, 0x8cd4fd6b4e85a5b2, 0xfab071054ba6409d, 0x40d6fe831fa9dfd9, 0xaf358debad7d791e, 0xeb8d0e25a65e3e58, 0xbbcbd3df14e08580, 0xcf751f27ecdab2b, 0x2b4da14f2613d8f4 }; #endif /* ZSTD_LDM_GEARTAB_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstdmt_compress.c0000644000175200007730000023666614515254731026602 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* ====== Compiler specifics ====== */ #if defined(_MSC_VER) # pragma warning(disable : 4204) /* disable: C4204: non-constant aggregate initializer */ #endif /* ====== Constants ====== */ #define ZSTDMT_OVERLAPLOG_DEFAULT 0 /* ====== Dependencies ====== */ #include "../common/allocations.h" /* ZSTD_customMalloc, ZSTD_customCalloc, ZSTD_customFree */ #include "../common/zstd_deps.h" /* ZSTD_memcpy, ZSTD_memset, INT_MAX, UINT_MAX */ #include "../common/mem.h" /* MEM_STATIC */ #include "../common/pool.h" /* threadpool */ #include "../common/threading.h" /* mutex */ #include "zstd_compress_internal.h" /* MIN, ERROR, ZSTD_*, ZSTD_highbit32 */ #include "zstd_ldm.h" #include "zstdmt_compress.h" /* Guards code to support resizing the SeqPool. * We will want to resize the SeqPool to save memory in the future. * Until then, comment the code out since it is unused. */ #define ZSTD_RESIZE_SEQPOOL 0 /* ====== Debug ====== */ #if defined(DEBUGLEVEL) && (DEBUGLEVEL>=2) \ && !defined(_MSC_VER) \ && !defined(__MINGW32__) # include # include # include # define DEBUG_PRINTHEX(l,p,n) { \ unsigned debug_u; \ for (debug_u=0; debug_u<(n); debug_u++) \ RAWLOG(l, "%02X ", ((const unsigned char*)(p))[debug_u]); \ RAWLOG(l, " \n"); \ } static unsigned long long GetCurrentClockTimeMicroseconds(void) { static clock_t _ticksPerSecond = 0; if (_ticksPerSecond <= 0) _ticksPerSecond = sysconf(_SC_CLK_TCK); { struct tms junk; clock_t newTicks = (clock_t) times(&junk); return ((((unsigned long long)newTicks)*(1000000))/_ticksPerSecond); } } #define MUTEX_WAIT_TIME_DLEVEL 6 #define ZSTD_PTHREAD_MUTEX_LOCK(mutex) { \ if (DEBUGLEVEL >= MUTEX_WAIT_TIME_DLEVEL) { \ unsigned long long const beforeTime = GetCurrentClockTimeMicroseconds(); \ ZSTD_pthread_mutex_lock(mutex); \ { unsigned long long const afterTime = GetCurrentClockTimeMicroseconds(); \ unsigned long long const elapsedTime = (afterTime-beforeTime); \ if (elapsedTime > 1000) { /* or whatever threshold you like; I'm using 1 millisecond here */ \ DEBUGLOG(MUTEX_WAIT_TIME_DLEVEL, "Thread took %llu microseconds to acquire mutex %s \n", \ elapsedTime, #mutex); \ } } \ } else { \ ZSTD_pthread_mutex_lock(mutex); \ } \ } #else # define ZSTD_PTHREAD_MUTEX_LOCK(m) ZSTD_pthread_mutex_lock(m) # define DEBUG_PRINTHEX(l,p,n) {} #endif /* ===== Buffer Pool ===== */ /* a single Buffer Pool can be invoked from multiple threads in parallel */ typedef struct buffer_s { void* start; size_t capacity; } buffer_t; static const buffer_t g_nullBuffer = { NULL, 0 }; typedef struct ZSTDMT_bufferPool_s { ZSTD_pthread_mutex_t poolMutex; size_t bufferSize; unsigned totalBuffers; unsigned nbBuffers; ZSTD_customMem cMem; buffer_t bTable[1]; /* variable size */ } ZSTDMT_bufferPool; static ZSTDMT_bufferPool* ZSTDMT_createBufferPool(unsigned maxNbBuffers, ZSTD_customMem cMem) { ZSTDMT_bufferPool* const bufPool = (ZSTDMT_bufferPool*)ZSTD_customCalloc( sizeof(ZSTDMT_bufferPool) + (maxNbBuffers-1) * sizeof(buffer_t), cMem); if (bufPool==NULL) return NULL; if (ZSTD_pthread_mutex_init(&bufPool->poolMutex, NULL)) { ZSTD_customFree(bufPool, cMem); return NULL; } bufPool->bufferSize = 64 KB; bufPool->totalBuffers = maxNbBuffers; bufPool->nbBuffers = 0; bufPool->cMem = cMem; return bufPool; } static void ZSTDMT_freeBufferPool(ZSTDMT_bufferPool* bufPool) { unsigned u; DEBUGLOG(3, "ZSTDMT_freeBufferPool (address:%08X)", (U32)(size_t)bufPool); if (!bufPool) return; /* compatibility with free on NULL */ for (u=0; utotalBuffers; u++) { DEBUGLOG(4, "free buffer %2u (address:%08X)", u, (U32)(size_t)bufPool->bTable[u].start); ZSTD_customFree(bufPool->bTable[u].start, bufPool->cMem); } ZSTD_pthread_mutex_destroy(&bufPool->poolMutex); ZSTD_customFree(bufPool, bufPool->cMem); } /* only works at initialization, not during compression */ static size_t ZSTDMT_sizeof_bufferPool(ZSTDMT_bufferPool* bufPool) { size_t const poolSize = sizeof(*bufPool) + (bufPool->totalBuffers - 1) * sizeof(buffer_t); unsigned u; size_t totalBufferSize = 0; ZSTD_pthread_mutex_lock(&bufPool->poolMutex); for (u=0; utotalBuffers; u++) totalBufferSize += bufPool->bTable[u].capacity; ZSTD_pthread_mutex_unlock(&bufPool->poolMutex); return poolSize + totalBufferSize; } /* ZSTDMT_setBufferSize() : * all future buffers provided by this buffer pool will have _at least_ this size * note : it's better for all buffers to have same size, * as they become freely interchangeable, reducing malloc/free usages and memory fragmentation */ static void ZSTDMT_setBufferSize(ZSTDMT_bufferPool* const bufPool, size_t const bSize) { ZSTD_pthread_mutex_lock(&bufPool->poolMutex); DEBUGLOG(4, "ZSTDMT_setBufferSize: bSize = %u", (U32)bSize); bufPool->bufferSize = bSize; ZSTD_pthread_mutex_unlock(&bufPool->poolMutex); } static ZSTDMT_bufferPool* ZSTDMT_expandBufferPool(ZSTDMT_bufferPool* srcBufPool, unsigned maxNbBuffers) { if (srcBufPool==NULL) return NULL; if (srcBufPool->totalBuffers >= maxNbBuffers) /* good enough */ return srcBufPool; /* need a larger buffer pool */ { ZSTD_customMem const cMem = srcBufPool->cMem; size_t const bSize = srcBufPool->bufferSize; /* forward parameters */ ZSTDMT_bufferPool* newBufPool; ZSTDMT_freeBufferPool(srcBufPool); newBufPool = ZSTDMT_createBufferPool(maxNbBuffers, cMem); if (newBufPool==NULL) return newBufPool; ZSTDMT_setBufferSize(newBufPool, bSize); return newBufPool; } } /** ZSTDMT_getBuffer() : * assumption : bufPool must be valid * @return : a buffer, with start pointer and size * note: allocation may fail, in this case, start==NULL and size==0 */ static buffer_t ZSTDMT_getBuffer(ZSTDMT_bufferPool* bufPool) { size_t const bSize = bufPool->bufferSize; DEBUGLOG(5, "ZSTDMT_getBuffer: bSize = %u", (U32)bufPool->bufferSize); ZSTD_pthread_mutex_lock(&bufPool->poolMutex); if (bufPool->nbBuffers) { /* try to use an existing buffer */ buffer_t const buf = bufPool->bTable[--(bufPool->nbBuffers)]; size_t const availBufferSize = buf.capacity; bufPool->bTable[bufPool->nbBuffers] = g_nullBuffer; if ((availBufferSize >= bSize) & ((availBufferSize>>3) <= bSize)) { /* large enough, but not too much */ DEBUGLOG(5, "ZSTDMT_getBuffer: provide buffer %u of size %u", bufPool->nbBuffers, (U32)buf.capacity); ZSTD_pthread_mutex_unlock(&bufPool->poolMutex); return buf; } /* size conditions not respected : scratch this buffer, create new one */ DEBUGLOG(5, "ZSTDMT_getBuffer: existing buffer does not meet size conditions => freeing"); ZSTD_customFree(buf.start, bufPool->cMem); } ZSTD_pthread_mutex_unlock(&bufPool->poolMutex); /* create new buffer */ DEBUGLOG(5, "ZSTDMT_getBuffer: create a new buffer"); { buffer_t buffer; void* const start = ZSTD_customMalloc(bSize, bufPool->cMem); buffer.start = start; /* note : start can be NULL if malloc fails ! */ buffer.capacity = (start==NULL) ? 0 : bSize; if (start==NULL) { DEBUGLOG(5, "ZSTDMT_getBuffer: buffer allocation failure !!"); } else { DEBUGLOG(5, "ZSTDMT_getBuffer: created buffer of size %u", (U32)bSize); } return buffer; } } #if ZSTD_RESIZE_SEQPOOL /** ZSTDMT_resizeBuffer() : * assumption : bufPool must be valid * @return : a buffer that is at least the buffer pool buffer size. * If a reallocation happens, the data in the input buffer is copied. */ static buffer_t ZSTDMT_resizeBuffer(ZSTDMT_bufferPool* bufPool, buffer_t buffer) { size_t const bSize = bufPool->bufferSize; if (buffer.capacity < bSize) { void* const start = ZSTD_customMalloc(bSize, bufPool->cMem); buffer_t newBuffer; newBuffer.start = start; newBuffer.capacity = start == NULL ? 0 : bSize; if (start != NULL) { assert(newBuffer.capacity >= buffer.capacity); ZSTD_memcpy(newBuffer.start, buffer.start, buffer.capacity); DEBUGLOG(5, "ZSTDMT_resizeBuffer: created buffer of size %u", (U32)bSize); return newBuffer; } DEBUGLOG(5, "ZSTDMT_resizeBuffer: buffer allocation failure !!"); } return buffer; } #endif /* store buffer for later re-use, up to pool capacity */ static void ZSTDMT_releaseBuffer(ZSTDMT_bufferPool* bufPool, buffer_t buf) { DEBUGLOG(5, "ZSTDMT_releaseBuffer"); if (buf.start == NULL) return; /* compatible with release on NULL */ ZSTD_pthread_mutex_lock(&bufPool->poolMutex); if (bufPool->nbBuffers < bufPool->totalBuffers) { bufPool->bTable[bufPool->nbBuffers++] = buf; /* stored for later use */ DEBUGLOG(5, "ZSTDMT_releaseBuffer: stored buffer of size %u in slot %u", (U32)buf.capacity, (U32)(bufPool->nbBuffers-1)); ZSTD_pthread_mutex_unlock(&bufPool->poolMutex); return; } ZSTD_pthread_mutex_unlock(&bufPool->poolMutex); /* Reached bufferPool capacity (should not happen) */ DEBUGLOG(5, "ZSTDMT_releaseBuffer: pool capacity reached => freeing "); ZSTD_customFree(buf.start, bufPool->cMem); } /* We need 2 output buffers per worker since each dstBuff must be flushed after it is released. * The 3 additional buffers are as follows: * 1 buffer for input loading * 1 buffer for "next input" when submitting current one * 1 buffer stuck in queue */ #define BUF_POOL_MAX_NB_BUFFERS(nbWorkers) (2*(nbWorkers) + 3) /* After a worker releases its rawSeqStore, it is immediately ready for reuse. * So we only need one seq buffer per worker. */ #define SEQ_POOL_MAX_NB_BUFFERS(nbWorkers) (nbWorkers) /* ===== Seq Pool Wrapper ====== */ typedef ZSTDMT_bufferPool ZSTDMT_seqPool; static size_t ZSTDMT_sizeof_seqPool(ZSTDMT_seqPool* seqPool) { return ZSTDMT_sizeof_bufferPool(seqPool); } static rawSeqStore_t bufferToSeq(buffer_t buffer) { rawSeqStore_t seq = kNullRawSeqStore; seq.seq = (rawSeq*)buffer.start; seq.capacity = buffer.capacity / sizeof(rawSeq); return seq; } static buffer_t seqToBuffer(rawSeqStore_t seq) { buffer_t buffer; buffer.start = seq.seq; buffer.capacity = seq.capacity * sizeof(rawSeq); return buffer; } static rawSeqStore_t ZSTDMT_getSeq(ZSTDMT_seqPool* seqPool) { if (seqPool->bufferSize == 0) { return kNullRawSeqStore; } return bufferToSeq(ZSTDMT_getBuffer(seqPool)); } #if ZSTD_RESIZE_SEQPOOL static rawSeqStore_t ZSTDMT_resizeSeq(ZSTDMT_seqPool* seqPool, rawSeqStore_t seq) { return bufferToSeq(ZSTDMT_resizeBuffer(seqPool, seqToBuffer(seq))); } #endif static void ZSTDMT_releaseSeq(ZSTDMT_seqPool* seqPool, rawSeqStore_t seq) { ZSTDMT_releaseBuffer(seqPool, seqToBuffer(seq)); } static void ZSTDMT_setNbSeq(ZSTDMT_seqPool* const seqPool, size_t const nbSeq) { ZSTDMT_setBufferSize(seqPool, nbSeq * sizeof(rawSeq)); } static ZSTDMT_seqPool* ZSTDMT_createSeqPool(unsigned nbWorkers, ZSTD_customMem cMem) { ZSTDMT_seqPool* const seqPool = ZSTDMT_createBufferPool(SEQ_POOL_MAX_NB_BUFFERS(nbWorkers), cMem); if (seqPool == NULL) return NULL; ZSTDMT_setNbSeq(seqPool, 0); return seqPool; } static void ZSTDMT_freeSeqPool(ZSTDMT_seqPool* seqPool) { ZSTDMT_freeBufferPool(seqPool); } static ZSTDMT_seqPool* ZSTDMT_expandSeqPool(ZSTDMT_seqPool* pool, U32 nbWorkers) { return ZSTDMT_expandBufferPool(pool, SEQ_POOL_MAX_NB_BUFFERS(nbWorkers)); } /* ===== CCtx Pool ===== */ /* a single CCtx Pool can be invoked from multiple threads in parallel */ typedef struct { ZSTD_pthread_mutex_t poolMutex; int totalCCtx; int availCCtx; ZSTD_customMem cMem; ZSTD_CCtx* cctx[1]; /* variable size */ } ZSTDMT_CCtxPool; /* note : all CCtx borrowed from the pool should be released back to the pool _before_ freeing the pool */ static void ZSTDMT_freeCCtxPool(ZSTDMT_CCtxPool* pool) { int cid; for (cid=0; cidtotalCCtx; cid++) ZSTD_freeCCtx(pool->cctx[cid]); /* note : compatible with free on NULL */ ZSTD_pthread_mutex_destroy(&pool->poolMutex); ZSTD_customFree(pool, pool->cMem); } /* ZSTDMT_createCCtxPool() : * implies nbWorkers >= 1 , checked by caller ZSTDMT_createCCtx() */ static ZSTDMT_CCtxPool* ZSTDMT_createCCtxPool(int nbWorkers, ZSTD_customMem cMem) { ZSTDMT_CCtxPool* const cctxPool = (ZSTDMT_CCtxPool*) ZSTD_customCalloc( sizeof(ZSTDMT_CCtxPool) + (nbWorkers-1)*sizeof(ZSTD_CCtx*), cMem); assert(nbWorkers > 0); if (!cctxPool) return NULL; if (ZSTD_pthread_mutex_init(&cctxPool->poolMutex, NULL)) { ZSTD_customFree(cctxPool, cMem); return NULL; } cctxPool->cMem = cMem; cctxPool->totalCCtx = nbWorkers; cctxPool->availCCtx = 1; /* at least one cctx for single-thread mode */ cctxPool->cctx[0] = ZSTD_createCCtx_advanced(cMem); if (!cctxPool->cctx[0]) { ZSTDMT_freeCCtxPool(cctxPool); return NULL; } DEBUGLOG(3, "cctxPool created, with %u workers", nbWorkers); return cctxPool; } static ZSTDMT_CCtxPool* ZSTDMT_expandCCtxPool(ZSTDMT_CCtxPool* srcPool, int nbWorkers) { if (srcPool==NULL) return NULL; if (nbWorkers <= srcPool->totalCCtx) return srcPool; /* good enough */ /* need a larger cctx pool */ { ZSTD_customMem const cMem = srcPool->cMem; ZSTDMT_freeCCtxPool(srcPool); return ZSTDMT_createCCtxPool(nbWorkers, cMem); } } /* only works during initialization phase, not during compression */ static size_t ZSTDMT_sizeof_CCtxPool(ZSTDMT_CCtxPool* cctxPool) { ZSTD_pthread_mutex_lock(&cctxPool->poolMutex); { unsigned const nbWorkers = cctxPool->totalCCtx; size_t const poolSize = sizeof(*cctxPool) + (nbWorkers-1) * sizeof(ZSTD_CCtx*); unsigned u; size_t totalCCtxSize = 0; for (u=0; ucctx[u]); } ZSTD_pthread_mutex_unlock(&cctxPool->poolMutex); assert(nbWorkers > 0); return poolSize + totalCCtxSize; } } static ZSTD_CCtx* ZSTDMT_getCCtx(ZSTDMT_CCtxPool* cctxPool) { DEBUGLOG(5, "ZSTDMT_getCCtx"); ZSTD_pthread_mutex_lock(&cctxPool->poolMutex); if (cctxPool->availCCtx) { cctxPool->availCCtx--; { ZSTD_CCtx* const cctx = cctxPool->cctx[cctxPool->availCCtx]; ZSTD_pthread_mutex_unlock(&cctxPool->poolMutex); return cctx; } } ZSTD_pthread_mutex_unlock(&cctxPool->poolMutex); DEBUGLOG(5, "create one more CCtx"); return ZSTD_createCCtx_advanced(cctxPool->cMem); /* note : can be NULL, when creation fails ! */ } static void ZSTDMT_releaseCCtx(ZSTDMT_CCtxPool* pool, ZSTD_CCtx* cctx) { if (cctx==NULL) return; /* compatibility with release on NULL */ ZSTD_pthread_mutex_lock(&pool->poolMutex); if (pool->availCCtx < pool->totalCCtx) pool->cctx[pool->availCCtx++] = cctx; else { /* pool overflow : should not happen, since totalCCtx==nbWorkers */ DEBUGLOG(4, "CCtx pool overflow : free cctx"); ZSTD_freeCCtx(cctx); } ZSTD_pthread_mutex_unlock(&pool->poolMutex); } /* ==== Serial State ==== */ typedef struct { void const* start; size_t size; } range_t; typedef struct { /* All variables in the struct are protected by mutex. */ ZSTD_pthread_mutex_t mutex; ZSTD_pthread_cond_t cond; ZSTD_CCtx_params params; ldmState_t ldmState; XXH64_state_t xxhState; unsigned nextJobID; /* Protects ldmWindow. * Must be acquired after the main mutex when acquiring both. */ ZSTD_pthread_mutex_t ldmWindowMutex; ZSTD_pthread_cond_t ldmWindowCond; /* Signaled when ldmWindow is updated */ ZSTD_window_t ldmWindow; /* A thread-safe copy of ldmState.window */ } serialState_t; static int ZSTDMT_serialState_reset(serialState_t* serialState, ZSTDMT_seqPool* seqPool, ZSTD_CCtx_params params, size_t jobSize, const void* dict, size_t const dictSize, ZSTD_dictContentType_e dictContentType) { /* Adjust parameters */ if (params.ldmParams.enableLdm == ZSTD_ps_enable) { DEBUGLOG(4, "LDM window size = %u KB", (1U << params.cParams.windowLog) >> 10); ZSTD_ldm_adjustParameters(¶ms.ldmParams, ¶ms.cParams); assert(params.ldmParams.hashLog >= params.ldmParams.bucketSizeLog); assert(params.ldmParams.hashRateLog < 32); } else { ZSTD_memset(¶ms.ldmParams, 0, sizeof(params.ldmParams)); } serialState->nextJobID = 0; if (params.fParams.checksumFlag) XXH64_reset(&serialState->xxhState, 0); if (params.ldmParams.enableLdm == ZSTD_ps_enable) { ZSTD_customMem cMem = params.customMem; unsigned const hashLog = params.ldmParams.hashLog; size_t const hashSize = ((size_t)1 << hashLog) * sizeof(ldmEntry_t); unsigned const bucketLog = params.ldmParams.hashLog - params.ldmParams.bucketSizeLog; unsigned const prevBucketLog = serialState->params.ldmParams.hashLog - serialState->params.ldmParams.bucketSizeLog; size_t const numBuckets = (size_t)1 << bucketLog; /* Size the seq pool tables */ ZSTDMT_setNbSeq(seqPool, ZSTD_ldm_getMaxNbSeq(params.ldmParams, jobSize)); /* Reset the window */ ZSTD_window_init(&serialState->ldmState.window); /* Resize tables and output space if necessary. */ if (serialState->ldmState.hashTable == NULL || serialState->params.ldmParams.hashLog < hashLog) { ZSTD_customFree(serialState->ldmState.hashTable, cMem); serialState->ldmState.hashTable = (ldmEntry_t*)ZSTD_customMalloc(hashSize, cMem); } if (serialState->ldmState.bucketOffsets == NULL || prevBucketLog < bucketLog) { ZSTD_customFree(serialState->ldmState.bucketOffsets, cMem); serialState->ldmState.bucketOffsets = (BYTE*)ZSTD_customMalloc(numBuckets, cMem); } if (!serialState->ldmState.hashTable || !serialState->ldmState.bucketOffsets) return 1; /* Zero the tables */ ZSTD_memset(serialState->ldmState.hashTable, 0, hashSize); ZSTD_memset(serialState->ldmState.bucketOffsets, 0, numBuckets); /* Update window state and fill hash table with dict */ serialState->ldmState.loadedDictEnd = 0; if (dictSize > 0) { if (dictContentType == ZSTD_dct_rawContent) { BYTE const* const dictEnd = (const BYTE*)dict + dictSize; ZSTD_window_update(&serialState->ldmState.window, dict, dictSize, /* forceNonContiguous */ 0); ZSTD_ldm_fillHashTable(&serialState->ldmState, (const BYTE*)dict, dictEnd, ¶ms.ldmParams); serialState->ldmState.loadedDictEnd = params.forceWindow ? 0 : (U32)(dictEnd - serialState->ldmState.window.base); } else { /* don't even load anything */ } } /* Initialize serialState's copy of ldmWindow. */ serialState->ldmWindow = serialState->ldmState.window; } serialState->params = params; serialState->params.jobSize = (U32)jobSize; return 0; } static int ZSTDMT_serialState_init(serialState_t* serialState) { int initError = 0; ZSTD_memset(serialState, 0, sizeof(*serialState)); initError |= ZSTD_pthread_mutex_init(&serialState->mutex, NULL); initError |= ZSTD_pthread_cond_init(&serialState->cond, NULL); initError |= ZSTD_pthread_mutex_init(&serialState->ldmWindowMutex, NULL); initError |= ZSTD_pthread_cond_init(&serialState->ldmWindowCond, NULL); return initError; } static void ZSTDMT_serialState_free(serialState_t* serialState) { ZSTD_customMem cMem = serialState->params.customMem; ZSTD_pthread_mutex_destroy(&serialState->mutex); ZSTD_pthread_cond_destroy(&serialState->cond); ZSTD_pthread_mutex_destroy(&serialState->ldmWindowMutex); ZSTD_pthread_cond_destroy(&serialState->ldmWindowCond); ZSTD_customFree(serialState->ldmState.hashTable, cMem); ZSTD_customFree(serialState->ldmState.bucketOffsets, cMem); } static void ZSTDMT_serialState_update(serialState_t* serialState, ZSTD_CCtx* jobCCtx, rawSeqStore_t seqStore, range_t src, unsigned jobID) { /* Wait for our turn */ ZSTD_PTHREAD_MUTEX_LOCK(&serialState->mutex); while (serialState->nextJobID < jobID) { DEBUGLOG(5, "wait for serialState->cond"); ZSTD_pthread_cond_wait(&serialState->cond, &serialState->mutex); } /* A future job may error and skip our job */ if (serialState->nextJobID == jobID) { /* It is now our turn, do any processing necessary */ if (serialState->params.ldmParams.enableLdm == ZSTD_ps_enable) { size_t error; assert(seqStore.seq != NULL && seqStore.pos == 0 && seqStore.size == 0 && seqStore.capacity > 0); assert(src.size <= serialState->params.jobSize); ZSTD_window_update(&serialState->ldmState.window, src.start, src.size, /* forceNonContiguous */ 0); error = ZSTD_ldm_generateSequences( &serialState->ldmState, &seqStore, &serialState->params.ldmParams, src.start, src.size); /* We provide a large enough buffer to never fail. */ assert(!ZSTD_isError(error)); (void)error; /* Update ldmWindow to match the ldmState.window and signal the main * thread if it is waiting for a buffer. */ ZSTD_PTHREAD_MUTEX_LOCK(&serialState->ldmWindowMutex); serialState->ldmWindow = serialState->ldmState.window; ZSTD_pthread_cond_signal(&serialState->ldmWindowCond); ZSTD_pthread_mutex_unlock(&serialState->ldmWindowMutex); } if (serialState->params.fParams.checksumFlag && src.size > 0) XXH64_update(&serialState->xxhState, src.start, src.size); } /* Now it is the next jobs turn */ serialState->nextJobID++; ZSTD_pthread_cond_broadcast(&serialState->cond); ZSTD_pthread_mutex_unlock(&serialState->mutex); if (seqStore.size > 0) { size_t const err = ZSTD_referenceExternalSequences( jobCCtx, seqStore.seq, seqStore.size); assert(serialState->params.ldmParams.enableLdm == ZSTD_ps_enable); assert(!ZSTD_isError(err)); (void)err; } } static void ZSTDMT_serialState_ensureFinished(serialState_t* serialState, unsigned jobID, size_t cSize) { ZSTD_PTHREAD_MUTEX_LOCK(&serialState->mutex); if (serialState->nextJobID <= jobID) { assert(ZSTD_isError(cSize)); (void)cSize; DEBUGLOG(5, "Skipping past job %u because of error", jobID); serialState->nextJobID = jobID + 1; ZSTD_pthread_cond_broadcast(&serialState->cond); ZSTD_PTHREAD_MUTEX_LOCK(&serialState->ldmWindowMutex); ZSTD_window_clear(&serialState->ldmWindow); ZSTD_pthread_cond_signal(&serialState->ldmWindowCond); ZSTD_pthread_mutex_unlock(&serialState->ldmWindowMutex); } ZSTD_pthread_mutex_unlock(&serialState->mutex); } /* ------------------------------------------ */ /* ===== Worker thread ===== */ /* ------------------------------------------ */ static const range_t kNullRange = { NULL, 0 }; typedef struct { size_t consumed; /* SHARED - set0 by mtctx, then modified by worker AND read by mtctx */ size_t cSize; /* SHARED - set0 by mtctx, then modified by worker AND read by mtctx, then set0 by mtctx */ ZSTD_pthread_mutex_t job_mutex; /* Thread-safe - used by mtctx and worker */ ZSTD_pthread_cond_t job_cond; /* Thread-safe - used by mtctx and worker */ ZSTDMT_CCtxPool* cctxPool; /* Thread-safe - used by mtctx and (all) workers */ ZSTDMT_bufferPool* bufPool; /* Thread-safe - used by mtctx and (all) workers */ ZSTDMT_seqPool* seqPool; /* Thread-safe - used by mtctx and (all) workers */ serialState_t* serial; /* Thread-safe - used by mtctx and (all) workers */ buffer_t dstBuff; /* set by worker (or mtctx), then read by worker & mtctx, then modified by mtctx => no barrier */ range_t prefix; /* set by mtctx, then read by worker & mtctx => no barrier */ range_t src; /* set by mtctx, then read by worker & mtctx => no barrier */ unsigned jobID; /* set by mtctx, then read by worker => no barrier */ unsigned firstJob; /* set by mtctx, then read by worker => no barrier */ unsigned lastJob; /* set by mtctx, then read by worker => no barrier */ ZSTD_CCtx_params params; /* set by mtctx, then read by worker => no barrier */ const ZSTD_CDict* cdict; /* set by mtctx, then read by worker => no barrier */ unsigned long long fullFrameSize; /* set by mtctx, then read by worker => no barrier */ size_t dstFlushed; /* used only by mtctx */ unsigned frameChecksumNeeded; /* used only by mtctx */ } ZSTDMT_jobDescription; #define JOB_ERROR(e) { \ ZSTD_PTHREAD_MUTEX_LOCK(&job->job_mutex); \ job->cSize = e; \ ZSTD_pthread_mutex_unlock(&job->job_mutex); \ goto _endJob; \ } /* ZSTDMT_compressionJob() is a POOL_function type */ static void ZSTDMT_compressionJob(void* jobDescription) { ZSTDMT_jobDescription* const job = (ZSTDMT_jobDescription*)jobDescription; ZSTD_CCtx_params jobParams = job->params; /* do not modify job->params ! copy it, modify the copy */ ZSTD_CCtx* const cctx = ZSTDMT_getCCtx(job->cctxPool); rawSeqStore_t rawSeqStore = ZSTDMT_getSeq(job->seqPool); buffer_t dstBuff = job->dstBuff; size_t lastCBlockSize = 0; /* resources */ if (cctx==NULL) JOB_ERROR(ERROR(memory_allocation)); if (dstBuff.start == NULL) { /* streaming job : doesn't provide a dstBuffer */ dstBuff = ZSTDMT_getBuffer(job->bufPool); if (dstBuff.start==NULL) JOB_ERROR(ERROR(memory_allocation)); job->dstBuff = dstBuff; /* this value can be read in ZSTDMT_flush, when it copies the whole job */ } if (jobParams.ldmParams.enableLdm == ZSTD_ps_enable && rawSeqStore.seq == NULL) JOB_ERROR(ERROR(memory_allocation)); /* Don't compute the checksum for chunks, since we compute it externally, * but write it in the header. */ if (job->jobID != 0) jobParams.fParams.checksumFlag = 0; /* Don't run LDM for the chunks, since we handle it externally */ jobParams.ldmParams.enableLdm = ZSTD_ps_disable; /* Correct nbWorkers to 0. */ jobParams.nbWorkers = 0; /* init */ if (job->cdict) { size_t const initError = ZSTD_compressBegin_advanced_internal(cctx, NULL, 0, ZSTD_dct_auto, ZSTD_dtlm_fast, job->cdict, &jobParams, job->fullFrameSize); assert(job->firstJob); /* only allowed for first job */ if (ZSTD_isError(initError)) JOB_ERROR(initError); } else { /* srcStart points at reloaded section */ U64 const pledgedSrcSize = job->firstJob ? job->fullFrameSize : job->src.size; { size_t const forceWindowError = ZSTD_CCtxParams_setParameter(&jobParams, ZSTD_c_forceMaxWindow, !job->firstJob); if (ZSTD_isError(forceWindowError)) JOB_ERROR(forceWindowError); } if (!job->firstJob) { size_t const err = ZSTD_CCtxParams_setParameter(&jobParams, ZSTD_c_deterministicRefPrefix, 0); if (ZSTD_isError(err)) JOB_ERROR(err); } { size_t const initError = ZSTD_compressBegin_advanced_internal(cctx, job->prefix.start, job->prefix.size, ZSTD_dct_rawContent, /* load dictionary in "content-only" mode (no header analysis) */ ZSTD_dtlm_fast, NULL, /*cdict*/ &jobParams, pledgedSrcSize); if (ZSTD_isError(initError)) JOB_ERROR(initError); } } /* Perform serial step as early as possible, but after CCtx initialization */ ZSTDMT_serialState_update(job->serial, cctx, rawSeqStore, job->src, job->jobID); if (!job->firstJob) { /* flush and overwrite frame header when it's not first job */ size_t const hSize = ZSTD_compressContinue_public(cctx, dstBuff.start, dstBuff.capacity, job->src.start, 0); if (ZSTD_isError(hSize)) JOB_ERROR(hSize); DEBUGLOG(5, "ZSTDMT_compressionJob: flush and overwrite %u bytes of frame header (not first job)", (U32)hSize); ZSTD_invalidateRepCodes(cctx); } /* compress */ { size_t const chunkSize = 4*ZSTD_BLOCKSIZE_MAX; int const nbChunks = (int)((job->src.size + (chunkSize-1)) / chunkSize); const BYTE* ip = (const BYTE*) job->src.start; BYTE* const ostart = (BYTE*)dstBuff.start; BYTE* op = ostart; BYTE* oend = op + dstBuff.capacity; int chunkNb; if (sizeof(size_t) > sizeof(int)) assert(job->src.size < ((size_t)INT_MAX) * chunkSize); /* check overflow */ DEBUGLOG(5, "ZSTDMT_compressionJob: compress %u bytes in %i blocks", (U32)job->src.size, nbChunks); assert(job->cSize == 0); for (chunkNb = 1; chunkNb < nbChunks; chunkNb++) { size_t const cSize = ZSTD_compressContinue_public(cctx, op, oend-op, ip, chunkSize); if (ZSTD_isError(cSize)) JOB_ERROR(cSize); ip += chunkSize; op += cSize; assert(op < oend); /* stats */ ZSTD_PTHREAD_MUTEX_LOCK(&job->job_mutex); job->cSize += cSize; job->consumed = chunkSize * chunkNb; DEBUGLOG(5, "ZSTDMT_compressionJob: compress new block : cSize==%u bytes (total: %u)", (U32)cSize, (U32)job->cSize); ZSTD_pthread_cond_signal(&job->job_cond); /* warns some more data is ready to be flushed */ ZSTD_pthread_mutex_unlock(&job->job_mutex); } /* last block */ assert(chunkSize > 0); assert((chunkSize & (chunkSize - 1)) == 0); /* chunkSize must be power of 2 for mask==(chunkSize-1) to work */ if ((nbChunks > 0) | job->lastJob /*must output a "last block" flag*/ ) { size_t const lastBlockSize1 = job->src.size & (chunkSize-1); size_t const lastBlockSize = ((lastBlockSize1==0) & (job->src.size>=chunkSize)) ? chunkSize : lastBlockSize1; size_t const cSize = (job->lastJob) ? ZSTD_compressEnd_public(cctx, op, oend-op, ip, lastBlockSize) : ZSTD_compressContinue_public(cctx, op, oend-op, ip, lastBlockSize); if (ZSTD_isError(cSize)) JOB_ERROR(cSize); lastCBlockSize = cSize; } } if (!job->firstJob) { /* Double check that we don't have an ext-dict, because then our * repcode invalidation doesn't work. */ assert(!ZSTD_window_hasExtDict(cctx->blockState.matchState.window)); } ZSTD_CCtx_trace(cctx, 0); _endJob: ZSTDMT_serialState_ensureFinished(job->serial, job->jobID, job->cSize); if (job->prefix.size > 0) DEBUGLOG(5, "Finished with prefix: %zx", (size_t)job->prefix.start); DEBUGLOG(5, "Finished with source: %zx", (size_t)job->src.start); /* release resources */ ZSTDMT_releaseSeq(job->seqPool, rawSeqStore); ZSTDMT_releaseCCtx(job->cctxPool, cctx); /* report */ ZSTD_PTHREAD_MUTEX_LOCK(&job->job_mutex); if (ZSTD_isError(job->cSize)) assert(lastCBlockSize == 0); job->cSize += lastCBlockSize; job->consumed = job->src.size; /* when job->consumed == job->src.size , compression job is presumed completed */ ZSTD_pthread_cond_signal(&job->job_cond); ZSTD_pthread_mutex_unlock(&job->job_mutex); } /* ------------------------------------------ */ /* ===== Multi-threaded compression ===== */ /* ------------------------------------------ */ typedef struct { range_t prefix; /* read-only non-owned prefix buffer */ buffer_t buffer; size_t filled; } inBuff_t; typedef struct { BYTE* buffer; /* The round input buffer. All jobs get references * to pieces of the buffer. ZSTDMT_tryGetInputRange() * handles handing out job input buffers, and makes * sure it doesn't overlap with any pieces still in use. */ size_t capacity; /* The capacity of buffer. */ size_t pos; /* The position of the current inBuff in the round * buffer. Updated past the end if the inBuff once * the inBuff is sent to the worker thread. * pos <= capacity. */ } roundBuff_t; static const roundBuff_t kNullRoundBuff = {NULL, 0, 0}; #define RSYNC_LENGTH 32 /* Don't create chunks smaller than the zstd block size. * This stops us from regressing compression ratio too much, * and ensures our output fits in ZSTD_compressBound(). * * If this is shrunk < ZSTD_BLOCKSIZELOG_MIN then * ZSTD_COMPRESSBOUND() will need to be updated. */ #define RSYNC_MIN_BLOCK_LOG ZSTD_BLOCKSIZELOG_MAX #define RSYNC_MIN_BLOCK_SIZE (1< one job is already prepared, but pool has shortage of workers. Don't create a new job. */ inBuff_t inBuff; roundBuff_t roundBuff; serialState_t serial; rsyncState_t rsync; unsigned jobIDMask; unsigned doneJobID; unsigned nextJobID; unsigned frameEnded; unsigned allJobsCompleted; unsigned long long frameContentSize; unsigned long long consumed; unsigned long long produced; ZSTD_customMem cMem; ZSTD_CDict* cdictLocal; const ZSTD_CDict* cdict; unsigned providedFactory: 1; }; static void ZSTDMT_freeJobsTable(ZSTDMT_jobDescription* jobTable, U32 nbJobs, ZSTD_customMem cMem) { U32 jobNb; if (jobTable == NULL) return; for (jobNb=0; jobNb mtctx->jobIDMask+1) { /* need more job capacity */ ZSTDMT_freeJobsTable(mtctx->jobs, mtctx->jobIDMask+1, mtctx->cMem); mtctx->jobIDMask = 0; mtctx->jobs = ZSTDMT_createJobsTable(&nbJobs, mtctx->cMem); if (mtctx->jobs==NULL) return ERROR(memory_allocation); assert((nbJobs != 0) && ((nbJobs & (nbJobs - 1)) == 0)); /* ensure nbJobs is a power of 2 */ mtctx->jobIDMask = nbJobs - 1; } return 0; } /* ZSTDMT_CCtxParam_setNbWorkers(): * Internal use only */ static size_t ZSTDMT_CCtxParam_setNbWorkers(ZSTD_CCtx_params* params, unsigned nbWorkers) { return ZSTD_CCtxParams_setParameter(params, ZSTD_c_nbWorkers, (int)nbWorkers); } MEM_STATIC ZSTDMT_CCtx* ZSTDMT_createCCtx_advanced_internal(unsigned nbWorkers, ZSTD_customMem cMem, ZSTD_threadPool* pool) { ZSTDMT_CCtx* mtctx; U32 nbJobs = nbWorkers + 2; int initError; DEBUGLOG(3, "ZSTDMT_createCCtx_advanced (nbWorkers = %u)", nbWorkers); if (nbWorkers < 1) return NULL; nbWorkers = MIN(nbWorkers , ZSTDMT_NBWORKERS_MAX); if ((cMem.customAlloc!=NULL) ^ (cMem.customFree!=NULL)) /* invalid custom allocator */ return NULL; mtctx = (ZSTDMT_CCtx*) ZSTD_customCalloc(sizeof(ZSTDMT_CCtx), cMem); if (!mtctx) return NULL; ZSTDMT_CCtxParam_setNbWorkers(&mtctx->params, nbWorkers); mtctx->cMem = cMem; mtctx->allJobsCompleted = 1; if (pool != NULL) { mtctx->factory = pool; mtctx->providedFactory = 1; } else { mtctx->factory = POOL_create_advanced(nbWorkers, 0, cMem); mtctx->providedFactory = 0; } mtctx->jobs = ZSTDMT_createJobsTable(&nbJobs, cMem); assert(nbJobs > 0); assert((nbJobs & (nbJobs - 1)) == 0); /* ensure nbJobs is a power of 2 */ mtctx->jobIDMask = nbJobs - 1; mtctx->bufPool = ZSTDMT_createBufferPool(BUF_POOL_MAX_NB_BUFFERS(nbWorkers), cMem); mtctx->cctxPool = ZSTDMT_createCCtxPool(nbWorkers, cMem); mtctx->seqPool = ZSTDMT_createSeqPool(nbWorkers, cMem); initError = ZSTDMT_serialState_init(&mtctx->serial); mtctx->roundBuff = kNullRoundBuff; if (!mtctx->factory | !mtctx->jobs | !mtctx->bufPool | !mtctx->cctxPool | !mtctx->seqPool | initError) { ZSTDMT_freeCCtx(mtctx); return NULL; } DEBUGLOG(3, "mt_cctx created, for %u threads", nbWorkers); return mtctx; } ZSTDMT_CCtx* ZSTDMT_createCCtx_advanced(unsigned nbWorkers, ZSTD_customMem cMem, ZSTD_threadPool* pool) { #ifdef ZSTD_MULTITHREAD return ZSTDMT_createCCtx_advanced_internal(nbWorkers, cMem, pool); #else (void)nbWorkers; (void)cMem; (void)pool; return NULL; #endif } /* ZSTDMT_releaseAllJobResources() : * note : ensure all workers are killed first ! */ static void ZSTDMT_releaseAllJobResources(ZSTDMT_CCtx* mtctx) { unsigned jobID; DEBUGLOG(3, "ZSTDMT_releaseAllJobResources"); for (jobID=0; jobID <= mtctx->jobIDMask; jobID++) { /* Copy the mutex/cond out */ ZSTD_pthread_mutex_t const mutex = mtctx->jobs[jobID].job_mutex; ZSTD_pthread_cond_t const cond = mtctx->jobs[jobID].job_cond; DEBUGLOG(4, "job%02u: release dst address %08X", jobID, (U32)(size_t)mtctx->jobs[jobID].dstBuff.start); ZSTDMT_releaseBuffer(mtctx->bufPool, mtctx->jobs[jobID].dstBuff); /* Clear the job description, but keep the mutex/cond */ ZSTD_memset(&mtctx->jobs[jobID], 0, sizeof(mtctx->jobs[jobID])); mtctx->jobs[jobID].job_mutex = mutex; mtctx->jobs[jobID].job_cond = cond; } mtctx->inBuff.buffer = g_nullBuffer; mtctx->inBuff.filled = 0; mtctx->allJobsCompleted = 1; } static void ZSTDMT_waitForAllJobsCompleted(ZSTDMT_CCtx* mtctx) { DEBUGLOG(4, "ZSTDMT_waitForAllJobsCompleted"); while (mtctx->doneJobID < mtctx->nextJobID) { unsigned const jobID = mtctx->doneJobID & mtctx->jobIDMask; ZSTD_PTHREAD_MUTEX_LOCK(&mtctx->jobs[jobID].job_mutex); while (mtctx->jobs[jobID].consumed < mtctx->jobs[jobID].src.size) { DEBUGLOG(4, "waiting for jobCompleted signal from job %u", mtctx->doneJobID); /* we want to block when waiting for data to flush */ ZSTD_pthread_cond_wait(&mtctx->jobs[jobID].job_cond, &mtctx->jobs[jobID].job_mutex); } ZSTD_pthread_mutex_unlock(&mtctx->jobs[jobID].job_mutex); mtctx->doneJobID++; } } size_t ZSTDMT_freeCCtx(ZSTDMT_CCtx* mtctx) { if (mtctx==NULL) return 0; /* compatible with free on NULL */ if (!mtctx->providedFactory) POOL_free(mtctx->factory); /* stop and free worker threads */ ZSTDMT_releaseAllJobResources(mtctx); /* release job resources into pools first */ ZSTDMT_freeJobsTable(mtctx->jobs, mtctx->jobIDMask+1, mtctx->cMem); ZSTDMT_freeBufferPool(mtctx->bufPool); ZSTDMT_freeCCtxPool(mtctx->cctxPool); ZSTDMT_freeSeqPool(mtctx->seqPool); ZSTDMT_serialState_free(&mtctx->serial); ZSTD_freeCDict(mtctx->cdictLocal); if (mtctx->roundBuff.buffer) ZSTD_customFree(mtctx->roundBuff.buffer, mtctx->cMem); ZSTD_customFree(mtctx, mtctx->cMem); return 0; } size_t ZSTDMT_sizeof_CCtx(ZSTDMT_CCtx* mtctx) { if (mtctx == NULL) return 0; /* supports sizeof NULL */ return sizeof(*mtctx) + POOL_sizeof(mtctx->factory) + ZSTDMT_sizeof_bufferPool(mtctx->bufPool) + (mtctx->jobIDMask+1) * sizeof(ZSTDMT_jobDescription) + ZSTDMT_sizeof_CCtxPool(mtctx->cctxPool) + ZSTDMT_sizeof_seqPool(mtctx->seqPool) + ZSTD_sizeof_CDict(mtctx->cdictLocal) + mtctx->roundBuff.capacity; } /* ZSTDMT_resize() : * @return : error code if fails, 0 on success */ static size_t ZSTDMT_resize(ZSTDMT_CCtx* mtctx, unsigned nbWorkers) { if (POOL_resize(mtctx->factory, nbWorkers)) return ERROR(memory_allocation); FORWARD_IF_ERROR( ZSTDMT_expandJobsTable(mtctx, nbWorkers) , ""); mtctx->bufPool = ZSTDMT_expandBufferPool(mtctx->bufPool, BUF_POOL_MAX_NB_BUFFERS(nbWorkers)); if (mtctx->bufPool == NULL) return ERROR(memory_allocation); mtctx->cctxPool = ZSTDMT_expandCCtxPool(mtctx->cctxPool, nbWorkers); if (mtctx->cctxPool == NULL) return ERROR(memory_allocation); mtctx->seqPool = ZSTDMT_expandSeqPool(mtctx->seqPool, nbWorkers); if (mtctx->seqPool == NULL) return ERROR(memory_allocation); ZSTDMT_CCtxParam_setNbWorkers(&mtctx->params, nbWorkers); return 0; } /*! ZSTDMT_updateCParams_whileCompressing() : * Updates a selected set of compression parameters, remaining compatible with currently active frame. * New parameters will be applied to next compression job. */ void ZSTDMT_updateCParams_whileCompressing(ZSTDMT_CCtx* mtctx, const ZSTD_CCtx_params* cctxParams) { U32 const saved_wlog = mtctx->params.cParams.windowLog; /* Do not modify windowLog while compressing */ int const compressionLevel = cctxParams->compressionLevel; DEBUGLOG(5, "ZSTDMT_updateCParams_whileCompressing (level:%i)", compressionLevel); mtctx->params.compressionLevel = compressionLevel; { ZSTD_compressionParameters cParams = ZSTD_getCParamsFromCCtxParams(cctxParams, ZSTD_CONTENTSIZE_UNKNOWN, 0, ZSTD_cpm_noAttachDict); cParams.windowLog = saved_wlog; mtctx->params.cParams = cParams; } } /* ZSTDMT_getFrameProgression(): * tells how much data has been consumed (input) and produced (output) for current frame. * able to count progression inside worker threads. * Note : mutex will be acquired during statistics collection inside workers. */ ZSTD_frameProgression ZSTDMT_getFrameProgression(ZSTDMT_CCtx* mtctx) { ZSTD_frameProgression fps; DEBUGLOG(5, "ZSTDMT_getFrameProgression"); fps.ingested = mtctx->consumed + mtctx->inBuff.filled; fps.consumed = mtctx->consumed; fps.produced = fps.flushed = mtctx->produced; fps.currentJobID = mtctx->nextJobID; fps.nbActiveWorkers = 0; { unsigned jobNb; unsigned lastJobNb = mtctx->nextJobID + mtctx->jobReady; assert(mtctx->jobReady <= 1); DEBUGLOG(6, "ZSTDMT_getFrameProgression: jobs: from %u to <%u (jobReady:%u)", mtctx->doneJobID, lastJobNb, mtctx->jobReady) for (jobNb = mtctx->doneJobID ; jobNb < lastJobNb ; jobNb++) { unsigned const wJobID = jobNb & mtctx->jobIDMask; ZSTDMT_jobDescription* jobPtr = &mtctx->jobs[wJobID]; ZSTD_pthread_mutex_lock(&jobPtr->job_mutex); { size_t const cResult = jobPtr->cSize; size_t const produced = ZSTD_isError(cResult) ? 0 : cResult; size_t const flushed = ZSTD_isError(cResult) ? 0 : jobPtr->dstFlushed; assert(flushed <= produced); fps.ingested += jobPtr->src.size; fps.consumed += jobPtr->consumed; fps.produced += produced; fps.flushed += flushed; fps.nbActiveWorkers += (jobPtr->consumed < jobPtr->src.size); } ZSTD_pthread_mutex_unlock(&mtctx->jobs[wJobID].job_mutex); } } return fps; } size_t ZSTDMT_toFlushNow(ZSTDMT_CCtx* mtctx) { size_t toFlush; unsigned const jobID = mtctx->doneJobID; assert(jobID <= mtctx->nextJobID); if (jobID == mtctx->nextJobID) return 0; /* no active job => nothing to flush */ /* look into oldest non-fully-flushed job */ { unsigned const wJobID = jobID & mtctx->jobIDMask; ZSTDMT_jobDescription* const jobPtr = &mtctx->jobs[wJobID]; ZSTD_pthread_mutex_lock(&jobPtr->job_mutex); { size_t const cResult = jobPtr->cSize; size_t const produced = ZSTD_isError(cResult) ? 0 : cResult; size_t const flushed = ZSTD_isError(cResult) ? 0 : jobPtr->dstFlushed; assert(flushed <= produced); assert(jobPtr->consumed <= jobPtr->src.size); toFlush = produced - flushed; /* if toFlush==0, nothing is available to flush. * However, jobID is expected to still be active: * if jobID was already completed and fully flushed, * ZSTDMT_flushProduced() should have already moved onto next job. * Therefore, some input has not yet been consumed. */ if (toFlush==0) { assert(jobPtr->consumed < jobPtr->src.size); } } ZSTD_pthread_mutex_unlock(&mtctx->jobs[wJobID].job_mutex); } return toFlush; } /* ------------------------------------------ */ /* ===== Multi-threaded compression ===== */ /* ------------------------------------------ */ static unsigned ZSTDMT_computeTargetJobLog(const ZSTD_CCtx_params* params) { unsigned jobLog; if (params->ldmParams.enableLdm == ZSTD_ps_enable) { /* In Long Range Mode, the windowLog is typically oversized. * In which case, it's preferable to determine the jobSize * based on cycleLog instead. */ jobLog = MAX(21, ZSTD_cycleLog(params->cParams.chainLog, params->cParams.strategy) + 3); } else { jobLog = MAX(20, params->cParams.windowLog + 2); } return MIN(jobLog, (unsigned)ZSTDMT_JOBLOG_MAX); } static int ZSTDMT_overlapLog_default(ZSTD_strategy strat) { switch(strat) { case ZSTD_btultra2: return 9; case ZSTD_btultra: case ZSTD_btopt: return 8; case ZSTD_btlazy2: case ZSTD_lazy2: return 7; case ZSTD_lazy: case ZSTD_greedy: case ZSTD_dfast: case ZSTD_fast: default:; } return 6; } static int ZSTDMT_overlapLog(int ovlog, ZSTD_strategy strat) { assert(0 <= ovlog && ovlog <= 9); if (ovlog == 0) return ZSTDMT_overlapLog_default(strat); return ovlog; } static size_t ZSTDMT_computeOverlapSize(const ZSTD_CCtx_params* params) { int const overlapRLog = 9 - ZSTDMT_overlapLog(params->overlapLog, params->cParams.strategy); int ovLog = (overlapRLog >= 8) ? 0 : (params->cParams.windowLog - overlapRLog); assert(0 <= overlapRLog && overlapRLog <= 8); if (params->ldmParams.enableLdm == ZSTD_ps_enable) { /* In Long Range Mode, the windowLog is typically oversized. * In which case, it's preferable to determine the jobSize * based on chainLog instead. * Then, ovLog becomes a fraction of the jobSize, rather than windowSize */ ovLog = MIN(params->cParams.windowLog, ZSTDMT_computeTargetJobLog(params) - 2) - overlapRLog; } assert(0 <= ovLog && ovLog <= ZSTD_WINDOWLOG_MAX); DEBUGLOG(4, "overlapLog : %i", params->overlapLog); DEBUGLOG(4, "overlap size : %i", 1 << ovLog); return (ovLog==0) ? 0 : (size_t)1 << ovLog; } /* ====================================== */ /* ======= Streaming API ======= */ /* ====================================== */ size_t ZSTDMT_initCStream_internal( ZSTDMT_CCtx* mtctx, const void* dict, size_t dictSize, ZSTD_dictContentType_e dictContentType, const ZSTD_CDict* cdict, ZSTD_CCtx_params params, unsigned long long pledgedSrcSize) { DEBUGLOG(4, "ZSTDMT_initCStream_internal (pledgedSrcSize=%u, nbWorkers=%u, cctxPool=%u)", (U32)pledgedSrcSize, params.nbWorkers, mtctx->cctxPool->totalCCtx); /* params supposed partially fully validated at this point */ assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams))); assert(!((dict) && (cdict))); /* either dict or cdict, not both */ /* init */ if (params.nbWorkers != mtctx->params.nbWorkers) FORWARD_IF_ERROR( ZSTDMT_resize(mtctx, params.nbWorkers) , ""); if (params.jobSize != 0 && params.jobSize < ZSTDMT_JOBSIZE_MIN) params.jobSize = ZSTDMT_JOBSIZE_MIN; if (params.jobSize > (size_t)ZSTDMT_JOBSIZE_MAX) params.jobSize = (size_t)ZSTDMT_JOBSIZE_MAX; DEBUGLOG(4, "ZSTDMT_initCStream_internal: %u workers", params.nbWorkers); if (mtctx->allJobsCompleted == 0) { /* previous compression not correctly finished */ ZSTDMT_waitForAllJobsCompleted(mtctx); ZSTDMT_releaseAllJobResources(mtctx); mtctx->allJobsCompleted = 1; } mtctx->params = params; mtctx->frameContentSize = pledgedSrcSize; if (dict) { ZSTD_freeCDict(mtctx->cdictLocal); mtctx->cdictLocal = ZSTD_createCDict_advanced(dict, dictSize, ZSTD_dlm_byCopy, dictContentType, /* note : a loadPrefix becomes an internal CDict */ params.cParams, mtctx->cMem); mtctx->cdict = mtctx->cdictLocal; if (mtctx->cdictLocal == NULL) return ERROR(memory_allocation); } else { ZSTD_freeCDict(mtctx->cdictLocal); mtctx->cdictLocal = NULL; mtctx->cdict = cdict; } mtctx->targetPrefixSize = ZSTDMT_computeOverlapSize(¶ms); DEBUGLOG(4, "overlapLog=%i => %u KB", params.overlapLog, (U32)(mtctx->targetPrefixSize>>10)); mtctx->targetSectionSize = params.jobSize; if (mtctx->targetSectionSize == 0) { mtctx->targetSectionSize = 1ULL << ZSTDMT_computeTargetJobLog(¶ms); } assert(mtctx->targetSectionSize <= (size_t)ZSTDMT_JOBSIZE_MAX); if (params.rsyncable) { /* Aim for the targetsectionSize as the average job size. */ U32 const jobSizeKB = (U32)(mtctx->targetSectionSize >> 10); U32 const rsyncBits = (assert(jobSizeKB >= 1), ZSTD_highbit32(jobSizeKB) + 10); /* We refuse to create jobs < RSYNC_MIN_BLOCK_SIZE bytes, so make sure our * expected job size is at least 4x larger. */ assert(rsyncBits >= RSYNC_MIN_BLOCK_LOG + 2); DEBUGLOG(4, "rsyncLog = %u", rsyncBits); mtctx->rsync.hash = 0; mtctx->rsync.hitMask = (1ULL << rsyncBits) - 1; mtctx->rsync.primePower = ZSTD_rollingHash_primePower(RSYNC_LENGTH); } if (mtctx->targetSectionSize < mtctx->targetPrefixSize) mtctx->targetSectionSize = mtctx->targetPrefixSize; /* job size must be >= overlap size */ DEBUGLOG(4, "Job Size : %u KB (note : set to %u)", (U32)(mtctx->targetSectionSize>>10), (U32)params.jobSize); DEBUGLOG(4, "inBuff Size : %u KB", (U32)(mtctx->targetSectionSize>>10)); ZSTDMT_setBufferSize(mtctx->bufPool, ZSTD_compressBound(mtctx->targetSectionSize)); { /* If ldm is enabled we need windowSize space. */ size_t const windowSize = mtctx->params.ldmParams.enableLdm == ZSTD_ps_enable ? (1U << mtctx->params.cParams.windowLog) : 0; /* Two buffers of slack, plus extra space for the overlap * This is the minimum slack that LDM works with. One extra because * flush might waste up to targetSectionSize-1 bytes. Another extra * for the overlap (if > 0), then one to fill which doesn't overlap * with the LDM window. */ size_t const nbSlackBuffers = 2 + (mtctx->targetPrefixSize > 0); size_t const slackSize = mtctx->targetSectionSize * nbSlackBuffers; /* Compute the total size, and always have enough slack */ size_t const nbWorkers = MAX(mtctx->params.nbWorkers, 1); size_t const sectionsSize = mtctx->targetSectionSize * nbWorkers; size_t const capacity = MAX(windowSize, sectionsSize) + slackSize; if (mtctx->roundBuff.capacity < capacity) { if (mtctx->roundBuff.buffer) ZSTD_customFree(mtctx->roundBuff.buffer, mtctx->cMem); mtctx->roundBuff.buffer = (BYTE*)ZSTD_customMalloc(capacity, mtctx->cMem); if (mtctx->roundBuff.buffer == NULL) { mtctx->roundBuff.capacity = 0; return ERROR(memory_allocation); } mtctx->roundBuff.capacity = capacity; } } DEBUGLOG(4, "roundBuff capacity : %u KB", (U32)(mtctx->roundBuff.capacity>>10)); mtctx->roundBuff.pos = 0; mtctx->inBuff.buffer = g_nullBuffer; mtctx->inBuff.filled = 0; mtctx->inBuff.prefix = kNullRange; mtctx->doneJobID = 0; mtctx->nextJobID = 0; mtctx->frameEnded = 0; mtctx->allJobsCompleted = 0; mtctx->consumed = 0; mtctx->produced = 0; if (ZSTDMT_serialState_reset(&mtctx->serial, mtctx->seqPool, params, mtctx->targetSectionSize, dict, dictSize, dictContentType)) return ERROR(memory_allocation); return 0; } /* ZSTDMT_writeLastEmptyBlock() * Write a single empty block with an end-of-frame to finish a frame. * Job must be created from streaming variant. * This function is always successful if expected conditions are fulfilled. */ static void ZSTDMT_writeLastEmptyBlock(ZSTDMT_jobDescription* job) { assert(job->lastJob == 1); assert(job->src.size == 0); /* last job is empty -> will be simplified into a last empty block */ assert(job->firstJob == 0); /* cannot be first job, as it also needs to create frame header */ assert(job->dstBuff.start == NULL); /* invoked from streaming variant only (otherwise, dstBuff might be user's output) */ job->dstBuff = ZSTDMT_getBuffer(job->bufPool); if (job->dstBuff.start == NULL) { job->cSize = ERROR(memory_allocation); return; } assert(job->dstBuff.capacity >= ZSTD_blockHeaderSize); /* no buffer should ever be that small */ job->src = kNullRange; job->cSize = ZSTD_writeLastEmptyBlock(job->dstBuff.start, job->dstBuff.capacity); assert(!ZSTD_isError(job->cSize)); assert(job->consumed == 0); } static size_t ZSTDMT_createCompressionJob(ZSTDMT_CCtx* mtctx, size_t srcSize, ZSTD_EndDirective endOp) { unsigned const jobID = mtctx->nextJobID & mtctx->jobIDMask; int const endFrame = (endOp == ZSTD_e_end); if (mtctx->nextJobID > mtctx->doneJobID + mtctx->jobIDMask) { DEBUGLOG(5, "ZSTDMT_createCompressionJob: will not create new job : table is full"); assert((mtctx->nextJobID & mtctx->jobIDMask) == (mtctx->doneJobID & mtctx->jobIDMask)); return 0; } if (!mtctx->jobReady) { BYTE const* src = (BYTE const*)mtctx->inBuff.buffer.start; DEBUGLOG(5, "ZSTDMT_createCompressionJob: preparing job %u to compress %u bytes with %u preload ", mtctx->nextJobID, (U32)srcSize, (U32)mtctx->inBuff.prefix.size); mtctx->jobs[jobID].src.start = src; mtctx->jobs[jobID].src.size = srcSize; assert(mtctx->inBuff.filled >= srcSize); mtctx->jobs[jobID].prefix = mtctx->inBuff.prefix; mtctx->jobs[jobID].consumed = 0; mtctx->jobs[jobID].cSize = 0; mtctx->jobs[jobID].params = mtctx->params; mtctx->jobs[jobID].cdict = mtctx->nextJobID==0 ? mtctx->cdict : NULL; mtctx->jobs[jobID].fullFrameSize = mtctx->frameContentSize; mtctx->jobs[jobID].dstBuff = g_nullBuffer; mtctx->jobs[jobID].cctxPool = mtctx->cctxPool; mtctx->jobs[jobID].bufPool = mtctx->bufPool; mtctx->jobs[jobID].seqPool = mtctx->seqPool; mtctx->jobs[jobID].serial = &mtctx->serial; mtctx->jobs[jobID].jobID = mtctx->nextJobID; mtctx->jobs[jobID].firstJob = (mtctx->nextJobID==0); mtctx->jobs[jobID].lastJob = endFrame; mtctx->jobs[jobID].frameChecksumNeeded = mtctx->params.fParams.checksumFlag && endFrame && (mtctx->nextJobID>0); mtctx->jobs[jobID].dstFlushed = 0; /* Update the round buffer pos and clear the input buffer to be reset */ mtctx->roundBuff.pos += srcSize; mtctx->inBuff.buffer = g_nullBuffer; mtctx->inBuff.filled = 0; /* Set the prefix */ if (!endFrame) { size_t const newPrefixSize = MIN(srcSize, mtctx->targetPrefixSize); mtctx->inBuff.prefix.start = src + srcSize - newPrefixSize; mtctx->inBuff.prefix.size = newPrefixSize; } else { /* endFrame==1 => no need for another input buffer */ mtctx->inBuff.prefix = kNullRange; mtctx->frameEnded = endFrame; if (mtctx->nextJobID == 0) { /* single job exception : checksum is already calculated directly within worker thread */ mtctx->params.fParams.checksumFlag = 0; } } if ( (srcSize == 0) && (mtctx->nextJobID>0)/*single job must also write frame header*/ ) { DEBUGLOG(5, "ZSTDMT_createCompressionJob: creating a last empty block to end frame"); assert(endOp == ZSTD_e_end); /* only possible case : need to end the frame with an empty last block */ ZSTDMT_writeLastEmptyBlock(mtctx->jobs + jobID); mtctx->nextJobID++; return 0; } } DEBUGLOG(5, "ZSTDMT_createCompressionJob: posting job %u : %u bytes (end:%u, jobNb == %u (mod:%u))", mtctx->nextJobID, (U32)mtctx->jobs[jobID].src.size, mtctx->jobs[jobID].lastJob, mtctx->nextJobID, jobID); if (POOL_tryAdd(mtctx->factory, ZSTDMT_compressionJob, &mtctx->jobs[jobID])) { mtctx->nextJobID++; mtctx->jobReady = 0; } else { DEBUGLOG(5, "ZSTDMT_createCompressionJob: no worker available for job %u", mtctx->nextJobID); mtctx->jobReady = 1; } return 0; } /*! ZSTDMT_flushProduced() : * flush whatever data has been produced but not yet flushed in current job. * move to next job if current one is fully flushed. * `output` : `pos` will be updated with amount of data flushed . * `blockToFlush` : if >0, the function will block and wait if there is no data available to flush . * @return : amount of data remaining within internal buffer, 0 if no more, 1 if unknown but > 0, or an error code */ static size_t ZSTDMT_flushProduced(ZSTDMT_CCtx* mtctx, ZSTD_outBuffer* output, unsigned blockToFlush, ZSTD_EndDirective end) { unsigned const wJobID = mtctx->doneJobID & mtctx->jobIDMask; DEBUGLOG(5, "ZSTDMT_flushProduced (blocking:%u , job %u <= %u)", blockToFlush, mtctx->doneJobID, mtctx->nextJobID); assert(output->size >= output->pos); ZSTD_PTHREAD_MUTEX_LOCK(&mtctx->jobs[wJobID].job_mutex); if ( blockToFlush && (mtctx->doneJobID < mtctx->nextJobID) ) { assert(mtctx->jobs[wJobID].dstFlushed <= mtctx->jobs[wJobID].cSize); while (mtctx->jobs[wJobID].dstFlushed == mtctx->jobs[wJobID].cSize) { /* nothing to flush */ if (mtctx->jobs[wJobID].consumed == mtctx->jobs[wJobID].src.size) { DEBUGLOG(5, "job %u is completely consumed (%u == %u) => don't wait for cond, there will be none", mtctx->doneJobID, (U32)mtctx->jobs[wJobID].consumed, (U32)mtctx->jobs[wJobID].src.size); break; } DEBUGLOG(5, "waiting for something to flush from job %u (currently flushed: %u bytes)", mtctx->doneJobID, (U32)mtctx->jobs[wJobID].dstFlushed); ZSTD_pthread_cond_wait(&mtctx->jobs[wJobID].job_cond, &mtctx->jobs[wJobID].job_mutex); /* block when nothing to flush but some to come */ } } /* try to flush something */ { size_t cSize = mtctx->jobs[wJobID].cSize; /* shared */ size_t const srcConsumed = mtctx->jobs[wJobID].consumed; /* shared */ size_t const srcSize = mtctx->jobs[wJobID].src.size; /* read-only, could be done after mutex lock, but no-declaration-after-statement */ ZSTD_pthread_mutex_unlock(&mtctx->jobs[wJobID].job_mutex); if (ZSTD_isError(cSize)) { DEBUGLOG(5, "ZSTDMT_flushProduced: job %u : compression error detected : %s", mtctx->doneJobID, ZSTD_getErrorName(cSize)); ZSTDMT_waitForAllJobsCompleted(mtctx); ZSTDMT_releaseAllJobResources(mtctx); return cSize; } /* add frame checksum if necessary (can only happen once) */ assert(srcConsumed <= srcSize); if ( (srcConsumed == srcSize) /* job completed -> worker no longer active */ && mtctx->jobs[wJobID].frameChecksumNeeded ) { U32 const checksum = (U32)XXH64_digest(&mtctx->serial.xxhState); DEBUGLOG(4, "ZSTDMT_flushProduced: writing checksum : %08X \n", checksum); MEM_writeLE32((char*)mtctx->jobs[wJobID].dstBuff.start + mtctx->jobs[wJobID].cSize, checksum); cSize += 4; mtctx->jobs[wJobID].cSize += 4; /* can write this shared value, as worker is no longer active */ mtctx->jobs[wJobID].frameChecksumNeeded = 0; } if (cSize > 0) { /* compression is ongoing or completed */ size_t const toFlush = MIN(cSize - mtctx->jobs[wJobID].dstFlushed, output->size - output->pos); DEBUGLOG(5, "ZSTDMT_flushProduced: Flushing %u bytes from job %u (completion:%u/%u, generated:%u)", (U32)toFlush, mtctx->doneJobID, (U32)srcConsumed, (U32)srcSize, (U32)cSize); assert(mtctx->doneJobID < mtctx->nextJobID); assert(cSize >= mtctx->jobs[wJobID].dstFlushed); assert(mtctx->jobs[wJobID].dstBuff.start != NULL); if (toFlush > 0) { ZSTD_memcpy((char*)output->dst + output->pos, (const char*)mtctx->jobs[wJobID].dstBuff.start + mtctx->jobs[wJobID].dstFlushed, toFlush); } output->pos += toFlush; mtctx->jobs[wJobID].dstFlushed += toFlush; /* can write : this value is only used by mtctx */ if ( (srcConsumed == srcSize) /* job is completed */ && (mtctx->jobs[wJobID].dstFlushed == cSize) ) { /* output buffer fully flushed => free this job position */ DEBUGLOG(5, "Job %u completed (%u bytes), moving to next one", mtctx->doneJobID, (U32)mtctx->jobs[wJobID].dstFlushed); ZSTDMT_releaseBuffer(mtctx->bufPool, mtctx->jobs[wJobID].dstBuff); DEBUGLOG(5, "dstBuffer released"); mtctx->jobs[wJobID].dstBuff = g_nullBuffer; mtctx->jobs[wJobID].cSize = 0; /* ensure this job slot is considered "not started" in future check */ mtctx->consumed += srcSize; mtctx->produced += cSize; mtctx->doneJobID++; } } /* return value : how many bytes left in buffer ; fake it to 1 when unknown but >0 */ if (cSize > mtctx->jobs[wJobID].dstFlushed) return (cSize - mtctx->jobs[wJobID].dstFlushed); if (srcSize > srcConsumed) return 1; /* current job not completely compressed */ } if (mtctx->doneJobID < mtctx->nextJobID) return 1; /* some more jobs ongoing */ if (mtctx->jobReady) return 1; /* one job is ready to push, just not yet in the list */ if (mtctx->inBuff.filled > 0) return 1; /* input is not empty, and still needs to be converted into a job */ mtctx->allJobsCompleted = mtctx->frameEnded; /* all jobs are entirely flushed => if this one is last one, frame is completed */ if (end == ZSTD_e_end) return !mtctx->frameEnded; /* for ZSTD_e_end, question becomes : is frame completed ? instead of : are internal buffers fully flushed ? */ return 0; /* internal buffers fully flushed */ } /** * Returns the range of data used by the earliest job that is not yet complete. * If the data of the first job is broken up into two segments, we cover both * sections. */ static range_t ZSTDMT_getInputDataInUse(ZSTDMT_CCtx* mtctx) { unsigned const firstJobID = mtctx->doneJobID; unsigned const lastJobID = mtctx->nextJobID; unsigned jobID; for (jobID = firstJobID; jobID < lastJobID; ++jobID) { unsigned const wJobID = jobID & mtctx->jobIDMask; size_t consumed; ZSTD_PTHREAD_MUTEX_LOCK(&mtctx->jobs[wJobID].job_mutex); consumed = mtctx->jobs[wJobID].consumed; ZSTD_pthread_mutex_unlock(&mtctx->jobs[wJobID].job_mutex); if (consumed < mtctx->jobs[wJobID].src.size) { range_t range = mtctx->jobs[wJobID].prefix; if (range.size == 0) { /* Empty prefix */ range = mtctx->jobs[wJobID].src; } /* Job source in multiple segments not supported yet */ assert(range.start <= mtctx->jobs[wJobID].src.start); return range; } } return kNullRange; } /** * Returns non-zero iff buffer and range overlap. */ static int ZSTDMT_isOverlapped(buffer_t buffer, range_t range) { BYTE const* const bufferStart = (BYTE const*)buffer.start; BYTE const* const rangeStart = (BYTE const*)range.start; if (rangeStart == NULL || bufferStart == NULL) return 0; { BYTE const* const bufferEnd = bufferStart + buffer.capacity; BYTE const* const rangeEnd = rangeStart + range.size; /* Empty ranges cannot overlap */ if (bufferStart == bufferEnd || rangeStart == rangeEnd) return 0; return bufferStart < rangeEnd && rangeStart < bufferEnd; } } static int ZSTDMT_doesOverlapWindow(buffer_t buffer, ZSTD_window_t window) { range_t extDict; range_t prefix; DEBUGLOG(5, "ZSTDMT_doesOverlapWindow"); extDict.start = window.dictBase + window.lowLimit; extDict.size = window.dictLimit - window.lowLimit; prefix.start = window.base + window.dictLimit; prefix.size = window.nextSrc - (window.base + window.dictLimit); DEBUGLOG(5, "extDict [0x%zx, 0x%zx)", (size_t)extDict.start, (size_t)extDict.start + extDict.size); DEBUGLOG(5, "prefix [0x%zx, 0x%zx)", (size_t)prefix.start, (size_t)prefix.start + prefix.size); return ZSTDMT_isOverlapped(buffer, extDict) || ZSTDMT_isOverlapped(buffer, prefix); } static void ZSTDMT_waitForLdmComplete(ZSTDMT_CCtx* mtctx, buffer_t buffer) { if (mtctx->params.ldmParams.enableLdm == ZSTD_ps_enable) { ZSTD_pthread_mutex_t* mutex = &mtctx->serial.ldmWindowMutex; DEBUGLOG(5, "ZSTDMT_waitForLdmComplete"); DEBUGLOG(5, "source [0x%zx, 0x%zx)", (size_t)buffer.start, (size_t)buffer.start + buffer.capacity); ZSTD_PTHREAD_MUTEX_LOCK(mutex); while (ZSTDMT_doesOverlapWindow(buffer, mtctx->serial.ldmWindow)) { DEBUGLOG(5, "Waiting for LDM to finish..."); ZSTD_pthread_cond_wait(&mtctx->serial.ldmWindowCond, mutex); } DEBUGLOG(6, "Done waiting for LDM to finish"); ZSTD_pthread_mutex_unlock(mutex); } } /** * Attempts to set the inBuff to the next section to fill. * If any part of the new section is still in use we give up. * Returns non-zero if the buffer is filled. */ static int ZSTDMT_tryGetInputRange(ZSTDMT_CCtx* mtctx) { range_t const inUse = ZSTDMT_getInputDataInUse(mtctx); size_t const spaceLeft = mtctx->roundBuff.capacity - mtctx->roundBuff.pos; size_t const target = mtctx->targetSectionSize; buffer_t buffer; DEBUGLOG(5, "ZSTDMT_tryGetInputRange"); assert(mtctx->inBuff.buffer.start == NULL); assert(mtctx->roundBuff.capacity >= target); if (spaceLeft < target) { /* ZSTD_invalidateRepCodes() doesn't work for extDict variants. * Simply copy the prefix to the beginning in that case. */ BYTE* const start = (BYTE*)mtctx->roundBuff.buffer; size_t const prefixSize = mtctx->inBuff.prefix.size; buffer.start = start; buffer.capacity = prefixSize; if (ZSTDMT_isOverlapped(buffer, inUse)) { DEBUGLOG(5, "Waiting for buffer..."); return 0; } ZSTDMT_waitForLdmComplete(mtctx, buffer); ZSTD_memmove(start, mtctx->inBuff.prefix.start, prefixSize); mtctx->inBuff.prefix.start = start; mtctx->roundBuff.pos = prefixSize; } buffer.start = mtctx->roundBuff.buffer + mtctx->roundBuff.pos; buffer.capacity = target; if (ZSTDMT_isOverlapped(buffer, inUse)) { DEBUGLOG(5, "Waiting for buffer..."); return 0; } assert(!ZSTDMT_isOverlapped(buffer, mtctx->inBuff.prefix)); ZSTDMT_waitForLdmComplete(mtctx, buffer); DEBUGLOG(5, "Using prefix range [%zx, %zx)", (size_t)mtctx->inBuff.prefix.start, (size_t)mtctx->inBuff.prefix.start + mtctx->inBuff.prefix.size); DEBUGLOG(5, "Using source range [%zx, %zx)", (size_t)buffer.start, (size_t)buffer.start + buffer.capacity); mtctx->inBuff.buffer = buffer; mtctx->inBuff.filled = 0; assert(mtctx->roundBuff.pos + buffer.capacity <= mtctx->roundBuff.capacity); return 1; } typedef struct { size_t toLoad; /* The number of bytes to load from the input. */ int flush; /* Boolean declaring if we must flush because we found a synchronization point. */ } syncPoint_t; /** * Searches through the input for a synchronization point. If one is found, we * will instruct the caller to flush, and return the number of bytes to load. * Otherwise, we will load as many bytes as possible and instruct the caller * to continue as normal. */ static syncPoint_t findSynchronizationPoint(ZSTDMT_CCtx const* mtctx, ZSTD_inBuffer const input) { BYTE const* const istart = (BYTE const*)input.src + input.pos; U64 const primePower = mtctx->rsync.primePower; U64 const hitMask = mtctx->rsync.hitMask; syncPoint_t syncPoint; U64 hash; BYTE const* prev; size_t pos; syncPoint.toLoad = MIN(input.size - input.pos, mtctx->targetSectionSize - mtctx->inBuff.filled); syncPoint.flush = 0; if (!mtctx->params.rsyncable) /* Rsync is disabled. */ return syncPoint; if (mtctx->inBuff.filled + input.size - input.pos < RSYNC_MIN_BLOCK_SIZE) /* We don't emit synchronization points if it would produce too small blocks. * We don't have enough input to find a synchronization point, so don't look. */ return syncPoint; if (mtctx->inBuff.filled + syncPoint.toLoad < RSYNC_LENGTH) /* Not enough to compute the hash. * We will miss any synchronization points in this RSYNC_LENGTH byte * window. However, since it depends only in the internal buffers, if the * state is already synchronized, we will remain synchronized. * Additionally, the probability that we miss a synchronization point is * low: RSYNC_LENGTH / targetSectionSize. */ return syncPoint; /* Initialize the loop variables. */ if (mtctx->inBuff.filled < RSYNC_MIN_BLOCK_SIZE) { /* We don't need to scan the first RSYNC_MIN_BLOCK_SIZE positions * because they can't possibly be a sync point. So we can start * part way through the input buffer. */ pos = RSYNC_MIN_BLOCK_SIZE - mtctx->inBuff.filled; if (pos >= RSYNC_LENGTH) { prev = istart + pos - RSYNC_LENGTH; hash = ZSTD_rollingHash_compute(prev, RSYNC_LENGTH); } else { assert(mtctx->inBuff.filled >= RSYNC_LENGTH); prev = (BYTE const*)mtctx->inBuff.buffer.start + mtctx->inBuff.filled - RSYNC_LENGTH; hash = ZSTD_rollingHash_compute(prev + pos, (RSYNC_LENGTH - pos)); hash = ZSTD_rollingHash_append(hash, istart, pos); } } else { /* We have enough bytes buffered to initialize the hash, * and have processed enough bytes to find a sync point. * Start scanning at the beginning of the input. */ assert(mtctx->inBuff.filled >= RSYNC_MIN_BLOCK_SIZE); assert(RSYNC_MIN_BLOCK_SIZE >= RSYNC_LENGTH); pos = 0; prev = (BYTE const*)mtctx->inBuff.buffer.start + mtctx->inBuff.filled - RSYNC_LENGTH; hash = ZSTD_rollingHash_compute(prev, RSYNC_LENGTH); if ((hash & hitMask) == hitMask) { /* We're already at a sync point so don't load any more until * we're able to flush this sync point. * This likely happened because the job table was full so we * couldn't add our job. */ syncPoint.toLoad = 0; syncPoint.flush = 1; return syncPoint; } } /* Starting with the hash of the previous RSYNC_LENGTH bytes, roll * through the input. If we hit a synchronization point, then cut the * job off, and tell the compressor to flush the job. Otherwise, load * all the bytes and continue as normal. * If we go too long without a synchronization point (targetSectionSize) * then a block will be emitted anyways, but this is okay, since if we * are already synchronized we will remain synchronized. */ assert(pos < RSYNC_LENGTH || ZSTD_rollingHash_compute(istart + pos - RSYNC_LENGTH, RSYNC_LENGTH) == hash); for (; pos < syncPoint.toLoad; ++pos) { BYTE const toRemove = pos < RSYNC_LENGTH ? prev[pos] : istart[pos - RSYNC_LENGTH]; /* This assert is very expensive, and Debian compiles with asserts enabled. * So disable it for now. We can get similar coverage by checking it at the * beginning & end of the loop. * assert(pos < RSYNC_LENGTH || ZSTD_rollingHash_compute(istart + pos - RSYNC_LENGTH, RSYNC_LENGTH) == hash); */ hash = ZSTD_rollingHash_rotate(hash, toRemove, istart[pos], primePower); assert(mtctx->inBuff.filled + pos >= RSYNC_MIN_BLOCK_SIZE); if ((hash & hitMask) == hitMask) { syncPoint.toLoad = pos + 1; syncPoint.flush = 1; ++pos; /* for assert */ break; } } assert(pos < RSYNC_LENGTH || ZSTD_rollingHash_compute(istart + pos - RSYNC_LENGTH, RSYNC_LENGTH) == hash); return syncPoint; } size_t ZSTDMT_nextInputSizeHint(const ZSTDMT_CCtx* mtctx) { size_t hintInSize = mtctx->targetSectionSize - mtctx->inBuff.filled; if (hintInSize==0) hintInSize = mtctx->targetSectionSize; return hintInSize; } /** ZSTDMT_compressStream_generic() : * internal use only - exposed to be invoked from zstd_compress.c * assumption : output and input are valid (pos <= size) * @return : minimum amount of data remaining to flush, 0 if none */ size_t ZSTDMT_compressStream_generic(ZSTDMT_CCtx* mtctx, ZSTD_outBuffer* output, ZSTD_inBuffer* input, ZSTD_EndDirective endOp) { unsigned forwardInputProgress = 0; DEBUGLOG(5, "ZSTDMT_compressStream_generic (endOp=%u, srcSize=%u)", (U32)endOp, (U32)(input->size - input->pos)); assert(output->pos <= output->size); assert(input->pos <= input->size); if ((mtctx->frameEnded) && (endOp==ZSTD_e_continue)) { /* current frame being ended. Only flush/end are allowed */ return ERROR(stage_wrong); } /* fill input buffer */ if ( (!mtctx->jobReady) && (input->size > input->pos) ) { /* support NULL input */ if (mtctx->inBuff.buffer.start == NULL) { assert(mtctx->inBuff.filled == 0); /* Can't fill an empty buffer */ if (!ZSTDMT_tryGetInputRange(mtctx)) { /* It is only possible for this operation to fail if there are * still compression jobs ongoing. */ DEBUGLOG(5, "ZSTDMT_tryGetInputRange failed"); assert(mtctx->doneJobID != mtctx->nextJobID); } else DEBUGLOG(5, "ZSTDMT_tryGetInputRange completed successfully : mtctx->inBuff.buffer.start = %p", mtctx->inBuff.buffer.start); } if (mtctx->inBuff.buffer.start != NULL) { syncPoint_t const syncPoint = findSynchronizationPoint(mtctx, *input); if (syncPoint.flush && endOp == ZSTD_e_continue) { endOp = ZSTD_e_flush; } assert(mtctx->inBuff.buffer.capacity >= mtctx->targetSectionSize); DEBUGLOG(5, "ZSTDMT_compressStream_generic: adding %u bytes on top of %u to buffer of size %u", (U32)syncPoint.toLoad, (U32)mtctx->inBuff.filled, (U32)mtctx->targetSectionSize); ZSTD_memcpy((char*)mtctx->inBuff.buffer.start + mtctx->inBuff.filled, (const char*)input->src + input->pos, syncPoint.toLoad); input->pos += syncPoint.toLoad; mtctx->inBuff.filled += syncPoint.toLoad; forwardInputProgress = syncPoint.toLoad>0; } } if ((input->pos < input->size) && (endOp == ZSTD_e_end)) { /* Can't end yet because the input is not fully consumed. * We are in one of these cases: * - mtctx->inBuff is NULL & empty: we couldn't get an input buffer so don't create a new job. * - We filled the input buffer: flush this job but don't end the frame. * - We hit a synchronization point: flush this job but don't end the frame. */ assert(mtctx->inBuff.filled == 0 || mtctx->inBuff.filled == mtctx->targetSectionSize || mtctx->params.rsyncable); endOp = ZSTD_e_flush; } if ( (mtctx->jobReady) || (mtctx->inBuff.filled >= mtctx->targetSectionSize) /* filled enough : let's compress */ || ((endOp != ZSTD_e_continue) && (mtctx->inBuff.filled > 0)) /* something to flush : let's go */ || ((endOp == ZSTD_e_end) && (!mtctx->frameEnded)) ) { /* must finish the frame with a zero-size block */ size_t const jobSize = mtctx->inBuff.filled; assert(mtctx->inBuff.filled <= mtctx->targetSectionSize); FORWARD_IF_ERROR( ZSTDMT_createCompressionJob(mtctx, jobSize, endOp) , ""); } /* check for potential compressed data ready to be flushed */ { size_t const remainingToFlush = ZSTDMT_flushProduced(mtctx, output, !forwardInputProgress, endOp); /* block if there was no forward input progress */ if (input->pos < input->size) return MAX(remainingToFlush, 1); /* input not consumed : do not end flush yet */ DEBUGLOG(5, "end of ZSTDMT_compressStream_generic: remainingToFlush = %u", (U32)remainingToFlush); return remainingToFlush; } } zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_double_fast.h0000644000175200007730000000250314515254731026657 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_DOUBLE_FAST_H #define ZSTD_DOUBLE_FAST_H #if defined (__cplusplus) extern "C" { #endif #include "../common/mem.h" /* U32 */ #include "zstd_compress_internal.h" /* ZSTD_CCtx, size_t */ void ZSTD_fillDoubleHashTable(ZSTD_matchState_t* ms, void const* end, ZSTD_dictTableLoadMethod_e dtlm, ZSTD_tableFillPurpose_e tfp); size_t ZSTD_compressBlock_doubleFast( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_doubleFast_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_doubleFast_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); #if defined (__cplusplus) } #endif #endif /* ZSTD_DOUBLE_FAST_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_fast.h0000644000175200007730000000235514515254731025332 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_FAST_H #define ZSTD_FAST_H #if defined (__cplusplus) extern "C" { #endif #include "../common/mem.h" /* U32 */ #include "zstd_compress_internal.h" void ZSTD_fillHashTable(ZSTD_matchState_t* ms, void const* end, ZSTD_dictTableLoadMethod_e dtlm, ZSTD_tableFillPurpose_e tfp); size_t ZSTD_compressBlock_fast( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_fast_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_fast_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); #if defined (__cplusplus) } #endif #endif /* ZSTD_FAST_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_fast.c0000644000175200007730000011012614515254731025321 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #include "zstd_compress_internal.h" /* ZSTD_hashPtr, ZSTD_count, ZSTD_storeSeq */ #include "zstd_fast.h" static void ZSTD_fillHashTableForCDict(ZSTD_matchState_t* ms, const void* const end, ZSTD_dictTableLoadMethod_e dtlm) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32* const hashTable = ms->hashTable; U32 const hBits = cParams->hashLog + ZSTD_SHORT_CACHE_TAG_BITS; U32 const mls = cParams->minMatch; const BYTE* const base = ms->window.base; const BYTE* ip = base + ms->nextToUpdate; const BYTE* const iend = ((const BYTE*)end) - HASH_READ_SIZE; const U32 fastHashFillStep = 3; /* Currently, we always use ZSTD_dtlm_full for filling CDict tables. * Feel free to remove this assert if there's a good reason! */ assert(dtlm == ZSTD_dtlm_full); /* Always insert every fastHashFillStep position into the hash table. * Insert the other positions if their hash entry is empty. */ for ( ; ip + fastHashFillStep < iend + 2; ip += fastHashFillStep) { U32 const curr = (U32)(ip - base); { size_t const hashAndTag = ZSTD_hashPtr(ip, hBits, mls); ZSTD_writeTaggedIndex(hashTable, hashAndTag, curr); } if (dtlm == ZSTD_dtlm_fast) continue; /* Only load extra positions for ZSTD_dtlm_full */ { U32 p; for (p = 1; p < fastHashFillStep; ++p) { size_t const hashAndTag = ZSTD_hashPtr(ip + p, hBits, mls); if (hashTable[hashAndTag >> ZSTD_SHORT_CACHE_TAG_BITS] == 0) { /* not yet filled */ ZSTD_writeTaggedIndex(hashTable, hashAndTag, curr + p); } } } } } static void ZSTD_fillHashTableForCCtx(ZSTD_matchState_t* ms, const void* const end, ZSTD_dictTableLoadMethod_e dtlm) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32* const hashTable = ms->hashTable; U32 const hBits = cParams->hashLog; U32 const mls = cParams->minMatch; const BYTE* const base = ms->window.base; const BYTE* ip = base + ms->nextToUpdate; const BYTE* const iend = ((const BYTE*)end) - HASH_READ_SIZE; const U32 fastHashFillStep = 3; /* Currently, we always use ZSTD_dtlm_fast for filling CCtx tables. * Feel free to remove this assert if there's a good reason! */ assert(dtlm == ZSTD_dtlm_fast); /* Always insert every fastHashFillStep position into the hash table. * Insert the other positions if their hash entry is empty. */ for ( ; ip + fastHashFillStep < iend + 2; ip += fastHashFillStep) { U32 const curr = (U32)(ip - base); size_t const hash0 = ZSTD_hashPtr(ip, hBits, mls); hashTable[hash0] = curr; if (dtlm == ZSTD_dtlm_fast) continue; /* Only load extra positions for ZSTD_dtlm_full */ { U32 p; for (p = 1; p < fastHashFillStep; ++p) { size_t const hash = ZSTD_hashPtr(ip + p, hBits, mls); if (hashTable[hash] == 0) { /* not yet filled */ hashTable[hash] = curr + p; } } } } } void ZSTD_fillHashTable(ZSTD_matchState_t* ms, const void* const end, ZSTD_dictTableLoadMethod_e dtlm, ZSTD_tableFillPurpose_e tfp) { if (tfp == ZSTD_tfp_forCDict) { ZSTD_fillHashTableForCDict(ms, end, dtlm); } else { ZSTD_fillHashTableForCCtx(ms, end, dtlm); } } /** * If you squint hard enough (and ignore repcodes), the search operation at any * given position is broken into 4 stages: * * 1. Hash (map position to hash value via input read) * 2. Lookup (map hash val to index via hashtable read) * 3. Load (map index to value at that position via input read) * 4. Compare * * Each of these steps involves a memory read at an address which is computed * from the previous step. This means these steps must be sequenced and their * latencies are cumulative. * * Rather than do 1->2->3->4 sequentially for a single position before moving * onto the next, this implementation interleaves these operations across the * next few positions: * * R = Repcode Read & Compare * H = Hash * T = Table Lookup * M = Match Read & Compare * * Pos | Time --> * ----+------------------- * N | ... M * N+1 | ... TM * N+2 | R H T M * N+3 | H TM * N+4 | R H T M * N+5 | H ... * N+6 | R ... * * This is very much analogous to the pipelining of execution in a CPU. And just * like a CPU, we have to dump the pipeline when we find a match (i.e., take a * branch). * * When this happens, we throw away our current state, and do the following prep * to re-enter the loop: * * Pos | Time --> * ----+------------------- * N | H T * N+1 | H * * This is also the work we do at the beginning to enter the loop initially. */ FORCE_INLINE_TEMPLATE size_t ZSTD_compressBlock_fast_noDict_generic( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize, U32 const mls, U32 const hasStep) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32* const hashTable = ms->hashTable; U32 const hlog = cParams->hashLog; /* support stepSize of 0 */ size_t const stepSize = hasStep ? (cParams->targetLength + !(cParams->targetLength) + 1) : 2; const BYTE* const base = ms->window.base; const BYTE* const istart = (const BYTE*)src; const U32 endIndex = (U32)((size_t)(istart - base) + srcSize); const U32 prefixStartIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog); const BYTE* const prefixStart = base + prefixStartIndex; const BYTE* const iend = istart + srcSize; const BYTE* const ilimit = iend - HASH_READ_SIZE; const BYTE* anchor = istart; const BYTE* ip0 = istart; const BYTE* ip1; const BYTE* ip2; const BYTE* ip3; U32 current0; U32 rep_offset1 = rep[0]; U32 rep_offset2 = rep[1]; U32 offsetSaved1 = 0, offsetSaved2 = 0; size_t hash0; /* hash for ip0 */ size_t hash1; /* hash for ip1 */ U32 idx; /* match idx for ip0 */ U32 mval; /* src value at match idx */ U32 offcode; const BYTE* match0; size_t mLength; /* ip0 and ip1 are always adjacent. The targetLength skipping and * uncompressibility acceleration is applied to every other position, * matching the behavior of #1562. step therefore represents the gap * between pairs of positions, from ip0 to ip2 or ip1 to ip3. */ size_t step; const BYTE* nextStep; const size_t kStepIncr = (1 << (kSearchStrength - 1)); DEBUGLOG(5, "ZSTD_compressBlock_fast_generic"); ip0 += (ip0 == prefixStart); { U32 const curr = (U32)(ip0 - base); U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, curr, cParams->windowLog); U32 const maxRep = curr - windowLow; if (rep_offset2 > maxRep) offsetSaved2 = rep_offset2, rep_offset2 = 0; if (rep_offset1 > maxRep) offsetSaved1 = rep_offset1, rep_offset1 = 0; } /* start each op */ _start: /* Requires: ip0 */ step = stepSize; nextStep = ip0 + kStepIncr; /* calculate positions, ip0 - anchor == 0, so we skip step calc */ ip1 = ip0 + 1; ip2 = ip0 + step; ip3 = ip2 + 1; if (ip3 >= ilimit) { goto _cleanup; } hash0 = ZSTD_hashPtr(ip0, hlog, mls); hash1 = ZSTD_hashPtr(ip1, hlog, mls); idx = hashTable[hash0]; do { /* load repcode match for ip[2]*/ const U32 rval = MEM_read32(ip2 - rep_offset1); /* write back hash table entry */ current0 = (U32)(ip0 - base); hashTable[hash0] = current0; /* check repcode at ip[2] */ if ((MEM_read32(ip2) == rval) & (rep_offset1 > 0)) { ip0 = ip2; match0 = ip0 - rep_offset1; mLength = ip0[-1] == match0[-1]; ip0 -= mLength; match0 -= mLength; offcode = REPCODE1_TO_OFFBASE; mLength += 4; /* First write next hash table entry; we've already calculated it. * This write is known to be safe because the ip1 is before the * repcode (ip2). */ hashTable[hash1] = (U32)(ip1 - base); goto _match; } /* load match for ip[0] */ if (idx >= prefixStartIndex) { mval = MEM_read32(base + idx); } else { mval = MEM_read32(ip0) ^ 1; /* guaranteed to not match. */ } /* check match at ip[0] */ if (MEM_read32(ip0) == mval) { /* found a match! */ /* First write next hash table entry; we've already calculated it. * This write is known to be safe because the ip1 == ip0 + 1, so * we know we will resume searching after ip1 */ hashTable[hash1] = (U32)(ip1 - base); goto _offset; } /* lookup ip[1] */ idx = hashTable[hash1]; /* hash ip[2] */ hash0 = hash1; hash1 = ZSTD_hashPtr(ip2, hlog, mls); /* advance to next positions */ ip0 = ip1; ip1 = ip2; ip2 = ip3; /* write back hash table entry */ current0 = (U32)(ip0 - base); hashTable[hash0] = current0; /* load match for ip[0] */ if (idx >= prefixStartIndex) { mval = MEM_read32(base + idx); } else { mval = MEM_read32(ip0) ^ 1; /* guaranteed to not match. */ } /* check match at ip[0] */ if (MEM_read32(ip0) == mval) { /* found a match! */ /* first write next hash table entry; we've already calculated it */ if (step <= 4) { /* We need to avoid writing an index into the hash table >= the * position at which we will pick up our searching after we've * taken this match. * * The minimum possible match has length 4, so the earliest ip0 * can be after we take this match will be the current ip0 + 4. * ip1 is ip0 + step - 1. If ip1 is >= ip0 + 4, we can't safely * write this position. */ hashTable[hash1] = (U32)(ip1 - base); } goto _offset; } /* lookup ip[1] */ idx = hashTable[hash1]; /* hash ip[2] */ hash0 = hash1; hash1 = ZSTD_hashPtr(ip2, hlog, mls); /* advance to next positions */ ip0 = ip1; ip1 = ip2; ip2 = ip0 + step; ip3 = ip1 + step; /* calculate step */ if (ip2 >= nextStep) { step++; PREFETCH_L1(ip1 + 64); PREFETCH_L1(ip1 + 128); nextStep += kStepIncr; } } while (ip3 < ilimit); _cleanup: /* Note that there are probably still a couple positions we could search. * However, it seems to be a meaningful performance hit to try to search * them. So let's not. */ /* When the repcodes are outside of the prefix, we set them to zero before the loop. * When the offsets are still zero, we need to restore them after the block to have a correct * repcode history. If only one offset was invalid, it is easy. The tricky case is when both * offsets were invalid. We need to figure out which offset to refill with. * - If both offsets are zero they are in the same order. * - If both offsets are non-zero, we won't restore the offsets from `offsetSaved[12]`. * - If only one is zero, we need to decide which offset to restore. * - If rep_offset1 is non-zero, then rep_offset2 must be offsetSaved1. * - It is impossible for rep_offset2 to be non-zero. * * So if rep_offset1 started invalid (offsetSaved1 != 0) and became valid (rep_offset1 != 0), then * set rep[0] = rep_offset1 and rep[1] = offsetSaved1. */ offsetSaved2 = ((offsetSaved1 != 0) && (rep_offset1 != 0)) ? offsetSaved1 : offsetSaved2; /* save reps for next block */ rep[0] = rep_offset1 ? rep_offset1 : offsetSaved1; rep[1] = rep_offset2 ? rep_offset2 : offsetSaved2; /* Return the last literals size */ return (size_t)(iend - anchor); _offset: /* Requires: ip0, idx */ /* Compute the offset code. */ match0 = base + idx; rep_offset2 = rep_offset1; rep_offset1 = (U32)(ip0-match0); offcode = OFFSET_TO_OFFBASE(rep_offset1); mLength = 4; /* Count the backwards match length. */ while (((ip0>anchor) & (match0>prefixStart)) && (ip0[-1] == match0[-1])) { ip0--; match0--; mLength++; } _match: /* Requires: ip0, match0, offcode */ /* Count the forward length. */ mLength += ZSTD_count(ip0 + mLength, match0 + mLength, iend); ZSTD_storeSeq(seqStore, (size_t)(ip0 - anchor), anchor, iend, offcode, mLength); ip0 += mLength; anchor = ip0; /* Fill table and check for immediate repcode. */ if (ip0 <= ilimit) { /* Fill Table */ assert(base+current0+2 > istart); /* check base overflow */ hashTable[ZSTD_hashPtr(base+current0+2, hlog, mls)] = current0+2; /* here because current+2 could be > iend-8 */ hashTable[ZSTD_hashPtr(ip0-2, hlog, mls)] = (U32)(ip0-2-base); if (rep_offset2 > 0) { /* rep_offset2==0 means rep_offset2 is invalidated */ while ( (ip0 <= ilimit) && (MEM_read32(ip0) == MEM_read32(ip0 - rep_offset2)) ) { /* store sequence */ size_t const rLength = ZSTD_count(ip0+4, ip0+4-rep_offset2, iend) + 4; { U32 const tmpOff = rep_offset2; rep_offset2 = rep_offset1; rep_offset1 = tmpOff; } /* swap rep_offset2 <=> rep_offset1 */ hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = (U32)(ip0-base); ip0 += rLength; ZSTD_storeSeq(seqStore, 0 /*litLen*/, anchor, iend, REPCODE1_TO_OFFBASE, rLength); anchor = ip0; continue; /* faster when present (confirmed on gcc-8) ... (?) */ } } } goto _start; } #define ZSTD_GEN_FAST_FN(dictMode, mls, step) \ static size_t ZSTD_compressBlock_fast_##dictMode##_##mls##_##step( \ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], \ void const* src, size_t srcSize) \ { \ return ZSTD_compressBlock_fast_##dictMode##_generic(ms, seqStore, rep, src, srcSize, mls, step); \ } ZSTD_GEN_FAST_FN(noDict, 4, 1) ZSTD_GEN_FAST_FN(noDict, 5, 1) ZSTD_GEN_FAST_FN(noDict, 6, 1) ZSTD_GEN_FAST_FN(noDict, 7, 1) ZSTD_GEN_FAST_FN(noDict, 4, 0) ZSTD_GEN_FAST_FN(noDict, 5, 0) ZSTD_GEN_FAST_FN(noDict, 6, 0) ZSTD_GEN_FAST_FN(noDict, 7, 0) size_t ZSTD_compressBlock_fast( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { U32 const mls = ms->cParams.minMatch; assert(ms->dictMatchState == NULL); if (ms->cParams.targetLength > 1) { switch(mls) { default: /* includes case 3 */ case 4 : return ZSTD_compressBlock_fast_noDict_4_1(ms, seqStore, rep, src, srcSize); case 5 : return ZSTD_compressBlock_fast_noDict_5_1(ms, seqStore, rep, src, srcSize); case 6 : return ZSTD_compressBlock_fast_noDict_6_1(ms, seqStore, rep, src, srcSize); case 7 : return ZSTD_compressBlock_fast_noDict_7_1(ms, seqStore, rep, src, srcSize); } } else { switch(mls) { default: /* includes case 3 */ case 4 : return ZSTD_compressBlock_fast_noDict_4_0(ms, seqStore, rep, src, srcSize); case 5 : return ZSTD_compressBlock_fast_noDict_5_0(ms, seqStore, rep, src, srcSize); case 6 : return ZSTD_compressBlock_fast_noDict_6_0(ms, seqStore, rep, src, srcSize); case 7 : return ZSTD_compressBlock_fast_noDict_7_0(ms, seqStore, rep, src, srcSize); } } } FORCE_INLINE_TEMPLATE size_t ZSTD_compressBlock_fast_dictMatchState_generic( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize, U32 const mls, U32 const hasStep) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32* const hashTable = ms->hashTable; U32 const hlog = cParams->hashLog; /* support stepSize of 0 */ U32 const stepSize = cParams->targetLength + !(cParams->targetLength); const BYTE* const base = ms->window.base; const BYTE* const istart = (const BYTE*)src; const BYTE* ip0 = istart; const BYTE* ip1 = ip0 + stepSize; /* we assert below that stepSize >= 1 */ const BYTE* anchor = istart; const U32 prefixStartIndex = ms->window.dictLimit; const BYTE* const prefixStart = base + prefixStartIndex; const BYTE* const iend = istart + srcSize; const BYTE* const ilimit = iend - HASH_READ_SIZE; U32 offset_1=rep[0], offset_2=rep[1]; const ZSTD_matchState_t* const dms = ms->dictMatchState; const ZSTD_compressionParameters* const dictCParams = &dms->cParams ; const U32* const dictHashTable = dms->hashTable; const U32 dictStartIndex = dms->window.dictLimit; const BYTE* const dictBase = dms->window.base; const BYTE* const dictStart = dictBase + dictStartIndex; const BYTE* const dictEnd = dms->window.nextSrc; const U32 dictIndexDelta = prefixStartIndex - (U32)(dictEnd - dictBase); const U32 dictAndPrefixLength = (U32)(istart - prefixStart + dictEnd - dictStart); const U32 dictHBits = dictCParams->hashLog + ZSTD_SHORT_CACHE_TAG_BITS; /* if a dictionary is still attached, it necessarily means that * it is within window size. So we just check it. */ const U32 maxDistance = 1U << cParams->windowLog; const U32 endIndex = (U32)((size_t)(istart - base) + srcSize); assert(endIndex - prefixStartIndex <= maxDistance); (void)maxDistance; (void)endIndex; /* these variables are not used when assert() is disabled */ (void)hasStep; /* not currently specialized on whether it's accelerated */ /* ensure there will be no underflow * when translating a dict index into a local index */ assert(prefixStartIndex >= (U32)(dictEnd - dictBase)); if (ms->prefetchCDictTables) { size_t const hashTableBytes = (((size_t)1) << dictCParams->hashLog) * sizeof(U32); PREFETCH_AREA(dictHashTable, hashTableBytes) } /* init */ DEBUGLOG(5, "ZSTD_compressBlock_fast_dictMatchState_generic"); ip0 += (dictAndPrefixLength == 0); /* dictMatchState repCode checks don't currently handle repCode == 0 * disabling. */ assert(offset_1 <= dictAndPrefixLength); assert(offset_2 <= dictAndPrefixLength); /* Outer search loop */ assert(stepSize >= 1); while (ip1 <= ilimit) { /* repcode check at (ip0 + 1) is safe because ip0 < ip1 */ size_t mLength; size_t hash0 = ZSTD_hashPtr(ip0, hlog, mls); size_t const dictHashAndTag0 = ZSTD_hashPtr(ip0, dictHBits, mls); U32 dictMatchIndexAndTag = dictHashTable[dictHashAndTag0 >> ZSTD_SHORT_CACHE_TAG_BITS]; int dictTagsMatch = ZSTD_comparePackedTags(dictMatchIndexAndTag, dictHashAndTag0); U32 matchIndex = hashTable[hash0]; U32 curr = (U32)(ip0 - base); size_t step = stepSize; const size_t kStepIncr = 1 << kSearchStrength; const BYTE* nextStep = ip0 + kStepIncr; /* Inner search loop */ while (1) { const BYTE* match = base + matchIndex; const U32 repIndex = curr + 1 - offset_1; const BYTE* repMatch = (repIndex < prefixStartIndex) ? dictBase + (repIndex - dictIndexDelta) : base + repIndex; const size_t hash1 = ZSTD_hashPtr(ip1, hlog, mls); size_t const dictHashAndTag1 = ZSTD_hashPtr(ip1, dictHBits, mls); hashTable[hash0] = curr; /* update hash table */ if (((U32) ((prefixStartIndex - 1) - repIndex) >= 3) /* intentional underflow : ensure repIndex isn't overlapping dict + prefix */ && (MEM_read32(repMatch) == MEM_read32(ip0 + 1))) { const BYTE* const repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend; mLength = ZSTD_count_2segments(ip0 + 1 + 4, repMatch + 4, iend, repMatchEnd, prefixStart) + 4; ip0++; ZSTD_storeSeq(seqStore, (size_t) (ip0 - anchor), anchor, iend, REPCODE1_TO_OFFBASE, mLength); break; } if (dictTagsMatch) { /* Found a possible dict match */ const U32 dictMatchIndex = dictMatchIndexAndTag >> ZSTD_SHORT_CACHE_TAG_BITS; const BYTE* dictMatch = dictBase + dictMatchIndex; if (dictMatchIndex > dictStartIndex && MEM_read32(dictMatch) == MEM_read32(ip0)) { /* To replicate extDict parse behavior, we only use dict matches when the normal matchIndex is invalid */ if (matchIndex <= prefixStartIndex) { U32 const offset = (U32) (curr - dictMatchIndex - dictIndexDelta); mLength = ZSTD_count_2segments(ip0 + 4, dictMatch + 4, iend, dictEnd, prefixStart) + 4; while (((ip0 > anchor) & (dictMatch > dictStart)) && (ip0[-1] == dictMatch[-1])) { ip0--; dictMatch--; mLength++; } /* catch up */ offset_2 = offset_1; offset_1 = offset; ZSTD_storeSeq(seqStore, (size_t) (ip0 - anchor), anchor, iend, OFFSET_TO_OFFBASE(offset), mLength); break; } } } if (matchIndex > prefixStartIndex && MEM_read32(match) == MEM_read32(ip0)) { /* found a regular match */ U32 const offset = (U32) (ip0 - match); mLength = ZSTD_count(ip0 + 4, match + 4, iend) + 4; while (((ip0 > anchor) & (match > prefixStart)) && (ip0[-1] == match[-1])) { ip0--; match--; mLength++; } /* catch up */ offset_2 = offset_1; offset_1 = offset; ZSTD_storeSeq(seqStore, (size_t) (ip0 - anchor), anchor, iend, OFFSET_TO_OFFBASE(offset), mLength); break; } /* Prepare for next iteration */ dictMatchIndexAndTag = dictHashTable[dictHashAndTag1 >> ZSTD_SHORT_CACHE_TAG_BITS]; dictTagsMatch = ZSTD_comparePackedTags(dictMatchIndexAndTag, dictHashAndTag1); matchIndex = hashTable[hash1]; if (ip1 >= nextStep) { step++; nextStep += kStepIncr; } ip0 = ip1; ip1 = ip1 + step; if (ip1 > ilimit) goto _cleanup; curr = (U32)(ip0 - base); hash0 = hash1; } /* end inner search loop */ /* match found */ assert(mLength); ip0 += mLength; anchor = ip0; if (ip0 <= ilimit) { /* Fill Table */ assert(base+curr+2 > istart); /* check base overflow */ hashTable[ZSTD_hashPtr(base+curr+2, hlog, mls)] = curr+2; /* here because curr+2 could be > iend-8 */ hashTable[ZSTD_hashPtr(ip0-2, hlog, mls)] = (U32)(ip0-2-base); /* check immediate repcode */ while (ip0 <= ilimit) { U32 const current2 = (U32)(ip0-base); U32 const repIndex2 = current2 - offset_2; const BYTE* repMatch2 = repIndex2 < prefixStartIndex ? dictBase - dictIndexDelta + repIndex2 : base + repIndex2; if ( ((U32)((prefixStartIndex-1) - (U32)repIndex2) >= 3 /* intentional overflow */) && (MEM_read32(repMatch2) == MEM_read32(ip0))) { const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend; size_t const repLength2 = ZSTD_count_2segments(ip0+4, repMatch2+4, iend, repEnd2, prefixStart) + 4; U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */ ZSTD_storeSeq(seqStore, 0, anchor, iend, REPCODE1_TO_OFFBASE, repLength2); hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = current2; ip0 += repLength2; anchor = ip0; continue; } break; } } /* Prepare for next iteration */ assert(ip0 == anchor); ip1 = ip0 + stepSize; } _cleanup: /* save reps for next block */ rep[0] = offset_1; rep[1] = offset_2; /* Return the last literals size */ return (size_t)(iend - anchor); } ZSTD_GEN_FAST_FN(dictMatchState, 4, 0) ZSTD_GEN_FAST_FN(dictMatchState, 5, 0) ZSTD_GEN_FAST_FN(dictMatchState, 6, 0) ZSTD_GEN_FAST_FN(dictMatchState, 7, 0) size_t ZSTD_compressBlock_fast_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { U32 const mls = ms->cParams.minMatch; assert(ms->dictMatchState != NULL); switch(mls) { default: /* includes case 3 */ case 4 : return ZSTD_compressBlock_fast_dictMatchState_4_0(ms, seqStore, rep, src, srcSize); case 5 : return ZSTD_compressBlock_fast_dictMatchState_5_0(ms, seqStore, rep, src, srcSize); case 6 : return ZSTD_compressBlock_fast_dictMatchState_6_0(ms, seqStore, rep, src, srcSize); case 7 : return ZSTD_compressBlock_fast_dictMatchState_7_0(ms, seqStore, rep, src, srcSize); } } static size_t ZSTD_compressBlock_fast_extDict_generic( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize, U32 const mls, U32 const hasStep) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32* const hashTable = ms->hashTable; U32 const hlog = cParams->hashLog; /* support stepSize of 0 */ size_t const stepSize = cParams->targetLength + !(cParams->targetLength) + 1; const BYTE* const base = ms->window.base; const BYTE* const dictBase = ms->window.dictBase; const BYTE* const istart = (const BYTE*)src; const BYTE* anchor = istart; const U32 endIndex = (U32)((size_t)(istart - base) + srcSize); const U32 lowLimit = ZSTD_getLowestMatchIndex(ms, endIndex, cParams->windowLog); const U32 dictStartIndex = lowLimit; const BYTE* const dictStart = dictBase + dictStartIndex; const U32 dictLimit = ms->window.dictLimit; const U32 prefixStartIndex = dictLimit < lowLimit ? lowLimit : dictLimit; const BYTE* const prefixStart = base + prefixStartIndex; const BYTE* const dictEnd = dictBase + prefixStartIndex; const BYTE* const iend = istart + srcSize; const BYTE* const ilimit = iend - 8; U32 offset_1=rep[0], offset_2=rep[1]; U32 offsetSaved1 = 0, offsetSaved2 = 0; const BYTE* ip0 = istart; const BYTE* ip1; const BYTE* ip2; const BYTE* ip3; U32 current0; size_t hash0; /* hash for ip0 */ size_t hash1; /* hash for ip1 */ U32 idx; /* match idx for ip0 */ const BYTE* idxBase; /* base pointer for idx */ U32 offcode; const BYTE* match0; size_t mLength; const BYTE* matchEnd = 0; /* initialize to avoid warning, assert != 0 later */ size_t step; const BYTE* nextStep; const size_t kStepIncr = (1 << (kSearchStrength - 1)); (void)hasStep; /* not currently specialized on whether it's accelerated */ DEBUGLOG(5, "ZSTD_compressBlock_fast_extDict_generic (offset_1=%u)", offset_1); /* switch to "regular" variant if extDict is invalidated due to maxDistance */ if (prefixStartIndex == dictStartIndex) return ZSTD_compressBlock_fast(ms, seqStore, rep, src, srcSize); { U32 const curr = (U32)(ip0 - base); U32 const maxRep = curr - dictStartIndex; if (offset_2 >= maxRep) offsetSaved2 = offset_2, offset_2 = 0; if (offset_1 >= maxRep) offsetSaved1 = offset_1, offset_1 = 0; } /* start each op */ _start: /* Requires: ip0 */ step = stepSize; nextStep = ip0 + kStepIncr; /* calculate positions, ip0 - anchor == 0, so we skip step calc */ ip1 = ip0 + 1; ip2 = ip0 + step; ip3 = ip2 + 1; if (ip3 >= ilimit) { goto _cleanup; } hash0 = ZSTD_hashPtr(ip0, hlog, mls); hash1 = ZSTD_hashPtr(ip1, hlog, mls); idx = hashTable[hash0]; idxBase = idx < prefixStartIndex ? dictBase : base; do { { /* load repcode match for ip[2] */ U32 const current2 = (U32)(ip2 - base); U32 const repIndex = current2 - offset_1; const BYTE* const repBase = repIndex < prefixStartIndex ? dictBase : base; U32 rval; if ( ((U32)(prefixStartIndex - repIndex) >= 4) /* intentional underflow */ & (offset_1 > 0) ) { rval = MEM_read32(repBase + repIndex); } else { rval = MEM_read32(ip2) ^ 1; /* guaranteed to not match. */ } /* write back hash table entry */ current0 = (U32)(ip0 - base); hashTable[hash0] = current0; /* check repcode at ip[2] */ if (MEM_read32(ip2) == rval) { ip0 = ip2; match0 = repBase + repIndex; matchEnd = repIndex < prefixStartIndex ? dictEnd : iend; assert((match0 != prefixStart) & (match0 != dictStart)); mLength = ip0[-1] == match0[-1]; ip0 -= mLength; match0 -= mLength; offcode = REPCODE1_TO_OFFBASE; mLength += 4; goto _match; } } { /* load match for ip[0] */ U32 const mval = idx >= dictStartIndex ? MEM_read32(idxBase + idx) : MEM_read32(ip0) ^ 1; /* guaranteed not to match */ /* check match at ip[0] */ if (MEM_read32(ip0) == mval) { /* found a match! */ goto _offset; } } /* lookup ip[1] */ idx = hashTable[hash1]; idxBase = idx < prefixStartIndex ? dictBase : base; /* hash ip[2] */ hash0 = hash1; hash1 = ZSTD_hashPtr(ip2, hlog, mls); /* advance to next positions */ ip0 = ip1; ip1 = ip2; ip2 = ip3; /* write back hash table entry */ current0 = (U32)(ip0 - base); hashTable[hash0] = current0; { /* load match for ip[0] */ U32 const mval = idx >= dictStartIndex ? MEM_read32(idxBase + idx) : MEM_read32(ip0) ^ 1; /* guaranteed not to match */ /* check match at ip[0] */ if (MEM_read32(ip0) == mval) { /* found a match! */ goto _offset; } } /* lookup ip[1] */ idx = hashTable[hash1]; idxBase = idx < prefixStartIndex ? dictBase : base; /* hash ip[2] */ hash0 = hash1; hash1 = ZSTD_hashPtr(ip2, hlog, mls); /* advance to next positions */ ip0 = ip1; ip1 = ip2; ip2 = ip0 + step; ip3 = ip1 + step; /* calculate step */ if (ip2 >= nextStep) { step++; PREFETCH_L1(ip1 + 64); PREFETCH_L1(ip1 + 128); nextStep += kStepIncr; } } while (ip3 < ilimit); _cleanup: /* Note that there are probably still a couple positions we could search. * However, it seems to be a meaningful performance hit to try to search * them. So let's not. */ /* If offset_1 started invalid (offsetSaved1 != 0) and became valid (offset_1 != 0), * rotate saved offsets. See comment in ZSTD_compressBlock_fast_noDict for more context. */ offsetSaved2 = ((offsetSaved1 != 0) && (offset_1 != 0)) ? offsetSaved1 : offsetSaved2; /* save reps for next block */ rep[0] = offset_1 ? offset_1 : offsetSaved1; rep[1] = offset_2 ? offset_2 : offsetSaved2; /* Return the last literals size */ return (size_t)(iend - anchor); _offset: /* Requires: ip0, idx, idxBase */ /* Compute the offset code. */ { U32 const offset = current0 - idx; const BYTE* const lowMatchPtr = idx < prefixStartIndex ? dictStart : prefixStart; matchEnd = idx < prefixStartIndex ? dictEnd : iend; match0 = idxBase + idx; offset_2 = offset_1; offset_1 = offset; offcode = OFFSET_TO_OFFBASE(offset); mLength = 4; /* Count the backwards match length. */ while (((ip0>anchor) & (match0>lowMatchPtr)) && (ip0[-1] == match0[-1])) { ip0--; match0--; mLength++; } } _match: /* Requires: ip0, match0, offcode, matchEnd */ /* Count the forward length. */ assert(matchEnd != 0); mLength += ZSTD_count_2segments(ip0 + mLength, match0 + mLength, iend, matchEnd, prefixStart); ZSTD_storeSeq(seqStore, (size_t)(ip0 - anchor), anchor, iend, offcode, mLength); ip0 += mLength; anchor = ip0; /* write next hash table entry */ if (ip1 < ip0) { hashTable[hash1] = (U32)(ip1 - base); } /* Fill table and check for immediate repcode. */ if (ip0 <= ilimit) { /* Fill Table */ assert(base+current0+2 > istart); /* check base overflow */ hashTable[ZSTD_hashPtr(base+current0+2, hlog, mls)] = current0+2; /* here because current+2 could be > iend-8 */ hashTable[ZSTD_hashPtr(ip0-2, hlog, mls)] = (U32)(ip0-2-base); while (ip0 <= ilimit) { U32 const repIndex2 = (U32)(ip0-base) - offset_2; const BYTE* const repMatch2 = repIndex2 < prefixStartIndex ? dictBase + repIndex2 : base + repIndex2; if ( (((U32)((prefixStartIndex-1) - repIndex2) >= 3) & (offset_2 > 0)) /* intentional underflow */ && (MEM_read32(repMatch2) == MEM_read32(ip0)) ) { const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend; size_t const repLength2 = ZSTD_count_2segments(ip0+4, repMatch2+4, iend, repEnd2, prefixStart) + 4; { U32 const tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; } /* swap offset_2 <=> offset_1 */ ZSTD_storeSeq(seqStore, 0 /*litlen*/, anchor, iend, REPCODE1_TO_OFFBASE, repLength2); hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = (U32)(ip0-base); ip0 += repLength2; anchor = ip0; continue; } break; } } goto _start; } ZSTD_GEN_FAST_FN(extDict, 4, 0) ZSTD_GEN_FAST_FN(extDict, 5, 0) ZSTD_GEN_FAST_FN(extDict, 6, 0) ZSTD_GEN_FAST_FN(extDict, 7, 0) size_t ZSTD_compressBlock_fast_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { U32 const mls = ms->cParams.minMatch; assert(ms->dictMatchState == NULL); switch(mls) { default: /* includes case 3 */ case 4 : return ZSTD_compressBlock_fast_extDict_4_0(ms, seqStore, rep, src, srcSize); case 5 : return ZSTD_compressBlock_fast_extDict_5_0(ms, seqStore, rep, src, srcSize); case 6 : return ZSTD_compressBlock_fast_extDict_6_0(ms, seqStore, rep, src, srcSize); case 7 : return ZSTD_compressBlock_fast_extDict_7_0(ms, seqStore, rep, src, srcSize); } } zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_compress_superblock.c0000644000175200007730000006752314515254731030464 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /*-************************************* * Dependencies ***************************************/ #include "zstd_compress_superblock.h" #include "../common/zstd_internal.h" /* ZSTD_getSequenceLength */ #include "hist.h" /* HIST_countFast_wksp */ #include "zstd_compress_internal.h" /* ZSTD_[huf|fse|entropy]CTablesMetadata_t */ #include "zstd_compress_sequences.h" #include "zstd_compress_literals.h" /** ZSTD_compressSubBlock_literal() : * Compresses literals section for a sub-block. * When we have to write the Huffman table we will sometimes choose a header * size larger than necessary. This is because we have to pick the header size * before we know the table size + compressed size, so we have a bound on the * table size. If we guessed incorrectly, we fall back to uncompressed literals. * * We write the header when writeEntropy=1 and set entropyWritten=1 when we succeeded * in writing the header, otherwise it is set to 0. * * hufMetadata->hType has literals block type info. * If it is set_basic, all sub-blocks literals section will be Raw_Literals_Block. * If it is set_rle, all sub-blocks literals section will be RLE_Literals_Block. * If it is set_compressed, first sub-block's literals section will be Compressed_Literals_Block * If it is set_compressed, first sub-block's literals section will be Treeless_Literals_Block * and the following sub-blocks' literals sections will be Treeless_Literals_Block. * @return : compressed size of literals section of a sub-block * Or 0 if unable to compress. * Or error code */ static size_t ZSTD_compressSubBlock_literal(const HUF_CElt* hufTable, const ZSTD_hufCTablesMetadata_t* hufMetadata, const BYTE* literals, size_t litSize, void* dst, size_t dstSize, const int bmi2, int writeEntropy, int* entropyWritten) { size_t const header = writeEntropy ? 200 : 0; size_t const lhSize = 3 + (litSize >= (1 KB - header)) + (litSize >= (16 KB - header)); BYTE* const ostart = (BYTE*)dst; BYTE* const oend = ostart + dstSize; BYTE* op = ostart + lhSize; U32 const singleStream = lhSize == 3; symbolEncodingType_e hType = writeEntropy ? hufMetadata->hType : set_repeat; size_t cLitSize = 0; DEBUGLOG(5, "ZSTD_compressSubBlock_literal (litSize=%zu, lhSize=%zu, writeEntropy=%d)", litSize, lhSize, writeEntropy); *entropyWritten = 0; if (litSize == 0 || hufMetadata->hType == set_basic) { DEBUGLOG(5, "ZSTD_compressSubBlock_literal using raw literal"); return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize); } else if (hufMetadata->hType == set_rle) { DEBUGLOG(5, "ZSTD_compressSubBlock_literal using rle literal"); return ZSTD_compressRleLiteralsBlock(dst, dstSize, literals, litSize); } assert(litSize > 0); assert(hufMetadata->hType == set_compressed || hufMetadata->hType == set_repeat); if (writeEntropy && hufMetadata->hType == set_compressed) { ZSTD_memcpy(op, hufMetadata->hufDesBuffer, hufMetadata->hufDesSize); op += hufMetadata->hufDesSize; cLitSize += hufMetadata->hufDesSize; DEBUGLOG(5, "ZSTD_compressSubBlock_literal (hSize=%zu)", hufMetadata->hufDesSize); } { int const flags = bmi2 ? HUF_flags_bmi2 : 0; const size_t cSize = singleStream ? HUF_compress1X_usingCTable(op, oend-op, literals, litSize, hufTable, flags) : HUF_compress4X_usingCTable(op, oend-op, literals, litSize, hufTable, flags); op += cSize; cLitSize += cSize; if (cSize == 0 || ERR_isError(cSize)) { DEBUGLOG(5, "Failed to write entropy tables %s", ZSTD_getErrorName(cSize)); return 0; } /* If we expand and we aren't writing a header then emit uncompressed */ if (!writeEntropy && cLitSize >= litSize) { DEBUGLOG(5, "ZSTD_compressSubBlock_literal using raw literal because uncompressible"); return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize); } /* If we are writing headers then allow expansion that doesn't change our header size. */ if (lhSize < (size_t)(3 + (cLitSize >= 1 KB) + (cLitSize >= 16 KB))) { assert(cLitSize > litSize); DEBUGLOG(5, "Literals expanded beyond allowed header size"); return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize); } DEBUGLOG(5, "ZSTD_compressSubBlock_literal (cSize=%zu)", cSize); } /* Build header */ switch(lhSize) { case 3: /* 2 - 2 - 10 - 10 */ { U32 const lhc = hType + ((!singleStream) << 2) + ((U32)litSize<<4) + ((U32)cLitSize<<14); MEM_writeLE24(ostart, lhc); break; } case 4: /* 2 - 2 - 14 - 14 */ { U32 const lhc = hType + (2 << 2) + ((U32)litSize<<4) + ((U32)cLitSize<<18); MEM_writeLE32(ostart, lhc); break; } case 5: /* 2 - 2 - 18 - 18 */ { U32 const lhc = hType + (3 << 2) + ((U32)litSize<<4) + ((U32)cLitSize<<22); MEM_writeLE32(ostart, lhc); ostart[4] = (BYTE)(cLitSize >> 10); break; } default: /* not possible : lhSize is {3,4,5} */ assert(0); } *entropyWritten = 1; DEBUGLOG(5, "Compressed literals: %u -> %u", (U32)litSize, (U32)(op-ostart)); return op-ostart; } static size_t ZSTD_seqDecompressedSize(seqStore_t const* seqStore, const seqDef* sequences, size_t nbSeq, size_t litSize, int lastSequence) { const seqDef* const sstart = sequences; const seqDef* const send = sequences + nbSeq; const seqDef* sp = sstart; size_t matchLengthSum = 0; size_t litLengthSum = 0; (void)(litLengthSum); /* suppress unused variable warning on some environments */ while (send-sp > 0) { ZSTD_sequenceLength const seqLen = ZSTD_getSequenceLength(seqStore, sp); litLengthSum += seqLen.litLength; matchLengthSum += seqLen.matchLength; sp++; } assert(litLengthSum <= litSize); if (!lastSequence) { assert(litLengthSum == litSize); } return matchLengthSum + litSize; } /** ZSTD_compressSubBlock_sequences() : * Compresses sequences section for a sub-block. * fseMetadata->llType, fseMetadata->ofType, and fseMetadata->mlType have * symbol compression modes for the super-block. * The first successfully compressed block will have these in its header. * We set entropyWritten=1 when we succeed in compressing the sequences. * The following sub-blocks will always have repeat mode. * @return : compressed size of sequences section of a sub-block * Or 0 if it is unable to compress * Or error code. */ static size_t ZSTD_compressSubBlock_sequences(const ZSTD_fseCTables_t* fseTables, const ZSTD_fseCTablesMetadata_t* fseMetadata, const seqDef* sequences, size_t nbSeq, const BYTE* llCode, const BYTE* mlCode, const BYTE* ofCode, const ZSTD_CCtx_params* cctxParams, void* dst, size_t dstCapacity, const int bmi2, int writeEntropy, int* entropyWritten) { const int longOffsets = cctxParams->cParams.windowLog > STREAM_ACCUMULATOR_MIN; BYTE* const ostart = (BYTE*)dst; BYTE* const oend = ostart + dstCapacity; BYTE* op = ostart; BYTE* seqHead; DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (nbSeq=%zu, writeEntropy=%d, longOffsets=%d)", nbSeq, writeEntropy, longOffsets); *entropyWritten = 0; /* Sequences Header */ RETURN_ERROR_IF((oend-op) < 3 /*max nbSeq Size*/ + 1 /*seqHead*/, dstSize_tooSmall, ""); if (nbSeq < 0x7F) *op++ = (BYTE)nbSeq; else if (nbSeq < LONGNBSEQ) op[0] = (BYTE)((nbSeq>>8) + 0x80), op[1] = (BYTE)nbSeq, op+=2; else op[0]=0xFF, MEM_writeLE16(op+1, (U16)(nbSeq - LONGNBSEQ)), op+=3; if (nbSeq==0) { return op - ostart; } /* seqHead : flags for FSE encoding type */ seqHead = op++; DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (seqHeadSize=%u)", (unsigned)(op-ostart)); if (writeEntropy) { const U32 LLtype = fseMetadata->llType; const U32 Offtype = fseMetadata->ofType; const U32 MLtype = fseMetadata->mlType; DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (fseTablesSize=%zu)", fseMetadata->fseTablesSize); *seqHead = (BYTE)((LLtype<<6) + (Offtype<<4) + (MLtype<<2)); ZSTD_memcpy(op, fseMetadata->fseTablesBuffer, fseMetadata->fseTablesSize); op += fseMetadata->fseTablesSize; } else { const U32 repeat = set_repeat; *seqHead = (BYTE)((repeat<<6) + (repeat<<4) + (repeat<<2)); } { size_t const bitstreamSize = ZSTD_encodeSequences( op, oend - op, fseTables->matchlengthCTable, mlCode, fseTables->offcodeCTable, ofCode, fseTables->litlengthCTable, llCode, sequences, nbSeq, longOffsets, bmi2); FORWARD_IF_ERROR(bitstreamSize, "ZSTD_encodeSequences failed"); op += bitstreamSize; /* zstd versions <= 1.3.4 mistakenly report corruption when * FSE_readNCount() receives a buffer < 4 bytes. * Fixed by https://github.com/facebook/zstd/pull/1146. * This can happen when the last set_compressed table present is 2 * bytes and the bitstream is only one byte. * In this exceedingly rare case, we will simply emit an uncompressed * block, since it isn't worth optimizing. */ #ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION if (writeEntropy && fseMetadata->lastCountSize && fseMetadata->lastCountSize + bitstreamSize < 4) { /* NCountSize >= 2 && bitstreamSize > 0 ==> lastCountSize == 3 */ assert(fseMetadata->lastCountSize + bitstreamSize == 3); DEBUGLOG(5, "Avoiding bug in zstd decoder in versions <= 1.3.4 by " "emitting an uncompressed block."); return 0; } #endif DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (bitstreamSize=%zu)", bitstreamSize); } /* zstd versions <= 1.4.0 mistakenly report error when * sequences section body size is less than 3 bytes. * Fixed by https://github.com/facebook/zstd/pull/1664. * This can happen when the previous sequences section block is compressed * with rle mode and the current block's sequences section is compressed * with repeat mode where sequences section body size can be 1 byte. */ #ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION if (op-seqHead < 4) { DEBUGLOG(5, "Avoiding bug in zstd decoder in versions <= 1.4.0 by emitting " "an uncompressed block when sequences are < 4 bytes"); return 0; } #endif *entropyWritten = 1; return op - ostart; } /** ZSTD_compressSubBlock() : * Compresses a single sub-block. * @return : compressed size of the sub-block * Or 0 if it failed to compress. */ static size_t ZSTD_compressSubBlock(const ZSTD_entropyCTables_t* entropy, const ZSTD_entropyCTablesMetadata_t* entropyMetadata, const seqDef* sequences, size_t nbSeq, const BYTE* literals, size_t litSize, const BYTE* llCode, const BYTE* mlCode, const BYTE* ofCode, const ZSTD_CCtx_params* cctxParams, void* dst, size_t dstCapacity, const int bmi2, int writeLitEntropy, int writeSeqEntropy, int* litEntropyWritten, int* seqEntropyWritten, U32 lastBlock) { BYTE* const ostart = (BYTE*)dst; BYTE* const oend = ostart + dstCapacity; BYTE* op = ostart + ZSTD_blockHeaderSize; DEBUGLOG(5, "ZSTD_compressSubBlock (litSize=%zu, nbSeq=%zu, writeLitEntropy=%d, writeSeqEntropy=%d, lastBlock=%d)", litSize, nbSeq, writeLitEntropy, writeSeqEntropy, lastBlock); { size_t cLitSize = ZSTD_compressSubBlock_literal((const HUF_CElt*)entropy->huf.CTable, &entropyMetadata->hufMetadata, literals, litSize, op, oend-op, bmi2, writeLitEntropy, litEntropyWritten); FORWARD_IF_ERROR(cLitSize, "ZSTD_compressSubBlock_literal failed"); if (cLitSize == 0) return 0; op += cLitSize; } { size_t cSeqSize = ZSTD_compressSubBlock_sequences(&entropy->fse, &entropyMetadata->fseMetadata, sequences, nbSeq, llCode, mlCode, ofCode, cctxParams, op, oend-op, bmi2, writeSeqEntropy, seqEntropyWritten); FORWARD_IF_ERROR(cSeqSize, "ZSTD_compressSubBlock_sequences failed"); if (cSeqSize == 0) return 0; op += cSeqSize; } /* Write block header */ { size_t cSize = (op-ostart)-ZSTD_blockHeaderSize; U32 const cBlockHeader24 = lastBlock + (((U32)bt_compressed)<<1) + (U32)(cSize << 3); MEM_writeLE24(ostart, cBlockHeader24); } return op-ostart; } static size_t ZSTD_estimateSubBlockSize_literal(const BYTE* literals, size_t litSize, const ZSTD_hufCTables_t* huf, const ZSTD_hufCTablesMetadata_t* hufMetadata, void* workspace, size_t wkspSize, int writeEntropy) { unsigned* const countWksp = (unsigned*)workspace; unsigned maxSymbolValue = 255; size_t literalSectionHeaderSize = 3; /* Use hard coded size of 3 bytes */ if (hufMetadata->hType == set_basic) return litSize; else if (hufMetadata->hType == set_rle) return 1; else if (hufMetadata->hType == set_compressed || hufMetadata->hType == set_repeat) { size_t const largest = HIST_count_wksp (countWksp, &maxSymbolValue, (const BYTE*)literals, litSize, workspace, wkspSize); if (ZSTD_isError(largest)) return litSize; { size_t cLitSizeEstimate = HUF_estimateCompressedSize((const HUF_CElt*)huf->CTable, countWksp, maxSymbolValue); if (writeEntropy) cLitSizeEstimate += hufMetadata->hufDesSize; return cLitSizeEstimate + literalSectionHeaderSize; } } assert(0); /* impossible */ return 0; } static size_t ZSTD_estimateSubBlockSize_symbolType(symbolEncodingType_e type, const BYTE* codeTable, unsigned maxCode, size_t nbSeq, const FSE_CTable* fseCTable, const U8* additionalBits, short const* defaultNorm, U32 defaultNormLog, U32 defaultMax, void* workspace, size_t wkspSize) { unsigned* const countWksp = (unsigned*)workspace; const BYTE* ctp = codeTable; const BYTE* const ctStart = ctp; const BYTE* const ctEnd = ctStart + nbSeq; size_t cSymbolTypeSizeEstimateInBits = 0; unsigned max = maxCode; HIST_countFast_wksp(countWksp, &max, codeTable, nbSeq, workspace, wkspSize); /* can't fail */ if (type == set_basic) { /* We selected this encoding type, so it must be valid. */ assert(max <= defaultMax); cSymbolTypeSizeEstimateInBits = max <= defaultMax ? ZSTD_crossEntropyCost(defaultNorm, defaultNormLog, countWksp, max) : ERROR(GENERIC); } else if (type == set_rle) { cSymbolTypeSizeEstimateInBits = 0; } else if (type == set_compressed || type == set_repeat) { cSymbolTypeSizeEstimateInBits = ZSTD_fseBitCost(fseCTable, countWksp, max); } if (ZSTD_isError(cSymbolTypeSizeEstimateInBits)) return nbSeq * 10; while (ctp < ctEnd) { if (additionalBits) cSymbolTypeSizeEstimateInBits += additionalBits[*ctp]; else cSymbolTypeSizeEstimateInBits += *ctp; /* for offset, offset code is also the number of additional bits */ ctp++; } return cSymbolTypeSizeEstimateInBits / 8; } static size_t ZSTD_estimateSubBlockSize_sequences(const BYTE* ofCodeTable, const BYTE* llCodeTable, const BYTE* mlCodeTable, size_t nbSeq, const ZSTD_fseCTables_t* fseTables, const ZSTD_fseCTablesMetadata_t* fseMetadata, void* workspace, size_t wkspSize, int writeEntropy) { size_t const sequencesSectionHeaderSize = 3; /* Use hard coded size of 3 bytes */ size_t cSeqSizeEstimate = 0; if (nbSeq == 0) return sequencesSectionHeaderSize; cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->ofType, ofCodeTable, MaxOff, nbSeq, fseTables->offcodeCTable, NULL, OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff, workspace, wkspSize); cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->llType, llCodeTable, MaxLL, nbSeq, fseTables->litlengthCTable, LL_bits, LL_defaultNorm, LL_defaultNormLog, MaxLL, workspace, wkspSize); cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->mlType, mlCodeTable, MaxML, nbSeq, fseTables->matchlengthCTable, ML_bits, ML_defaultNorm, ML_defaultNormLog, MaxML, workspace, wkspSize); if (writeEntropy) cSeqSizeEstimate += fseMetadata->fseTablesSize; return cSeqSizeEstimate + sequencesSectionHeaderSize; } static size_t ZSTD_estimateSubBlockSize(const BYTE* literals, size_t litSize, const BYTE* ofCodeTable, const BYTE* llCodeTable, const BYTE* mlCodeTable, size_t nbSeq, const ZSTD_entropyCTables_t* entropy, const ZSTD_entropyCTablesMetadata_t* entropyMetadata, void* workspace, size_t wkspSize, int writeLitEntropy, int writeSeqEntropy) { size_t cSizeEstimate = 0; cSizeEstimate += ZSTD_estimateSubBlockSize_literal(literals, litSize, &entropy->huf, &entropyMetadata->hufMetadata, workspace, wkspSize, writeLitEntropy); cSizeEstimate += ZSTD_estimateSubBlockSize_sequences(ofCodeTable, llCodeTable, mlCodeTable, nbSeq, &entropy->fse, &entropyMetadata->fseMetadata, workspace, wkspSize, writeSeqEntropy); return cSizeEstimate + ZSTD_blockHeaderSize; } static int ZSTD_needSequenceEntropyTables(ZSTD_fseCTablesMetadata_t const* fseMetadata) { if (fseMetadata->llType == set_compressed || fseMetadata->llType == set_rle) return 1; if (fseMetadata->mlType == set_compressed || fseMetadata->mlType == set_rle) return 1; if (fseMetadata->ofType == set_compressed || fseMetadata->ofType == set_rle) return 1; return 0; } /** ZSTD_compressSubBlock_multi() : * Breaks super-block into multiple sub-blocks and compresses them. * Entropy will be written to the first block. * The following blocks will use repeat mode to compress. * All sub-blocks are compressed blocks (no raw or rle blocks). * @return : compressed size of the super block (which is multiple ZSTD blocks) * Or 0 if it failed to compress. */ static size_t ZSTD_compressSubBlock_multi(const seqStore_t* seqStorePtr, const ZSTD_compressedBlockState_t* prevCBlock, ZSTD_compressedBlockState_t* nextCBlock, const ZSTD_entropyCTablesMetadata_t* entropyMetadata, const ZSTD_CCtx_params* cctxParams, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const int bmi2, U32 lastBlock, void* workspace, size_t wkspSize) { const seqDef* const sstart = seqStorePtr->sequencesStart; const seqDef* const send = seqStorePtr->sequences; const seqDef* sp = sstart; const BYTE* const lstart = seqStorePtr->litStart; const BYTE* const lend = seqStorePtr->lit; const BYTE* lp = lstart; BYTE const* ip = (BYTE const*)src; BYTE const* const iend = ip + srcSize; BYTE* const ostart = (BYTE*)dst; BYTE* const oend = ostart + dstCapacity; BYTE* op = ostart; const BYTE* llCodePtr = seqStorePtr->llCode; const BYTE* mlCodePtr = seqStorePtr->mlCode; const BYTE* ofCodePtr = seqStorePtr->ofCode; size_t targetCBlockSize = cctxParams->targetCBlockSize; size_t litSize, seqCount; int writeLitEntropy = entropyMetadata->hufMetadata.hType == set_compressed; int writeSeqEntropy = 1; int lastSequence = 0; DEBUGLOG(5, "ZSTD_compressSubBlock_multi (litSize=%u, nbSeq=%u)", (unsigned)(lend-lp), (unsigned)(send-sstart)); litSize = 0; seqCount = 0; do { size_t cBlockSizeEstimate = 0; if (sstart == send) { lastSequence = 1; } else { const seqDef* const sequence = sp + seqCount; lastSequence = sequence == send - 1; litSize += ZSTD_getSequenceLength(seqStorePtr, sequence).litLength; seqCount++; } if (lastSequence) { assert(lp <= lend); assert(litSize <= (size_t)(lend - lp)); litSize = (size_t)(lend - lp); } /* I think there is an optimization opportunity here. * Calling ZSTD_estimateSubBlockSize for every sequence can be wasteful * since it recalculates estimate from scratch. * For example, it would recount literal distribution and symbol codes every time. */ cBlockSizeEstimate = ZSTD_estimateSubBlockSize(lp, litSize, ofCodePtr, llCodePtr, mlCodePtr, seqCount, &nextCBlock->entropy, entropyMetadata, workspace, wkspSize, writeLitEntropy, writeSeqEntropy); if (cBlockSizeEstimate > targetCBlockSize || lastSequence) { int litEntropyWritten = 0; int seqEntropyWritten = 0; const size_t decompressedSize = ZSTD_seqDecompressedSize(seqStorePtr, sp, seqCount, litSize, lastSequence); const size_t cSize = ZSTD_compressSubBlock(&nextCBlock->entropy, entropyMetadata, sp, seqCount, lp, litSize, llCodePtr, mlCodePtr, ofCodePtr, cctxParams, op, oend-op, bmi2, writeLitEntropy, writeSeqEntropy, &litEntropyWritten, &seqEntropyWritten, lastBlock && lastSequence); FORWARD_IF_ERROR(cSize, "ZSTD_compressSubBlock failed"); if (cSize > 0 && cSize < decompressedSize) { DEBUGLOG(5, "Committed the sub-block"); assert(ip + decompressedSize <= iend); ip += decompressedSize; sp += seqCount; lp += litSize; op += cSize; llCodePtr += seqCount; mlCodePtr += seqCount; ofCodePtr += seqCount; litSize = 0; seqCount = 0; /* Entropy only needs to be written once */ if (litEntropyWritten) { writeLitEntropy = 0; } if (seqEntropyWritten) { writeSeqEntropy = 0; } } } } while (!lastSequence); if (writeLitEntropy) { DEBUGLOG(5, "ZSTD_compressSubBlock_multi has literal entropy tables unwritten"); ZSTD_memcpy(&nextCBlock->entropy.huf, &prevCBlock->entropy.huf, sizeof(prevCBlock->entropy.huf)); } if (writeSeqEntropy && ZSTD_needSequenceEntropyTables(&entropyMetadata->fseMetadata)) { /* If we haven't written our entropy tables, then we've violated our contract and * must emit an uncompressed block. */ DEBUGLOG(5, "ZSTD_compressSubBlock_multi has sequence entropy tables unwritten"); return 0; } if (ip < iend) { size_t const cSize = ZSTD_noCompressBlock(op, oend - op, ip, iend - ip, lastBlock); DEBUGLOG(5, "ZSTD_compressSubBlock_multi last sub-block uncompressed, %zu bytes", (size_t)(iend - ip)); FORWARD_IF_ERROR(cSize, "ZSTD_noCompressBlock failed"); assert(cSize != 0); op += cSize; /* We have to regenerate the repcodes because we've skipped some sequences */ if (sp < send) { seqDef const* seq; repcodes_t rep; ZSTD_memcpy(&rep, prevCBlock->rep, sizeof(rep)); for (seq = sstart; seq < sp; ++seq) { ZSTD_updateRep(rep.rep, seq->offBase, ZSTD_getSequenceLength(seqStorePtr, seq).litLength == 0); } ZSTD_memcpy(nextCBlock->rep, &rep, sizeof(rep)); } } DEBUGLOG(5, "ZSTD_compressSubBlock_multi compressed"); return op-ostart; } size_t ZSTD_compressSuperBlock(ZSTD_CCtx* zc, void* dst, size_t dstCapacity, void const* src, size_t srcSize, unsigned lastBlock) { ZSTD_entropyCTablesMetadata_t entropyMetadata; FORWARD_IF_ERROR(ZSTD_buildBlockEntropyStats(&zc->seqStore, &zc->blockState.prevCBlock->entropy, &zc->blockState.nextCBlock->entropy, &zc->appliedParams, &entropyMetadata, zc->entropyWorkspace, ENTROPY_WORKSPACE_SIZE /* statically allocated in resetCCtx */), ""); return ZSTD_compressSubBlock_multi(&zc->seqStore, zc->blockState.prevCBlock, zc->blockState.nextCBlock, &entropyMetadata, &zc->appliedParams, dst, dstCapacity, src, srcSize, zc->bmi2, lastBlock, zc->entropyWorkspace, ENTROPY_WORKSPACE_SIZE /* statically allocated in resetCCtx */); } zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_lazy.c0000644000175200007730000030710014515254731025343 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #include "zstd_compress_internal.h" #include "zstd_lazy.h" #include "../common/bits.h" /* ZSTD_countTrailingZeros64 */ #define kLazySkippingStep 8 /*-************************************* * Binary Tree search ***************************************/ static void ZSTD_updateDUBT(ZSTD_matchState_t* ms, const BYTE* ip, const BYTE* iend, U32 mls) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32* const hashTable = ms->hashTable; U32 const hashLog = cParams->hashLog; U32* const bt = ms->chainTable; U32 const btLog = cParams->chainLog - 1; U32 const btMask = (1 << btLog) - 1; const BYTE* const base = ms->window.base; U32 const target = (U32)(ip - base); U32 idx = ms->nextToUpdate; if (idx != target) DEBUGLOG(7, "ZSTD_updateDUBT, from %u to %u (dictLimit:%u)", idx, target, ms->window.dictLimit); assert(ip + 8 <= iend); /* condition for ZSTD_hashPtr */ (void)iend; assert(idx >= ms->window.dictLimit); /* condition for valid base+idx */ for ( ; idx < target ; idx++) { size_t const h = ZSTD_hashPtr(base + idx, hashLog, mls); /* assumption : ip + 8 <= iend */ U32 const matchIndex = hashTable[h]; U32* const nextCandidatePtr = bt + 2*(idx&btMask); U32* const sortMarkPtr = nextCandidatePtr + 1; DEBUGLOG(8, "ZSTD_updateDUBT: insert %u", idx); hashTable[h] = idx; /* Update Hash Table */ *nextCandidatePtr = matchIndex; /* update BT like a chain */ *sortMarkPtr = ZSTD_DUBT_UNSORTED_MARK; } ms->nextToUpdate = target; } /** ZSTD_insertDUBT1() : * sort one already inserted but unsorted position * assumption : curr >= btlow == (curr - btmask) * doesn't fail */ static void ZSTD_insertDUBT1(const ZSTD_matchState_t* ms, U32 curr, const BYTE* inputEnd, U32 nbCompares, U32 btLow, const ZSTD_dictMode_e dictMode) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32* const bt = ms->chainTable; U32 const btLog = cParams->chainLog - 1; U32 const btMask = (1 << btLog) - 1; size_t commonLengthSmaller=0, commonLengthLarger=0; const BYTE* const base = ms->window.base; const BYTE* const dictBase = ms->window.dictBase; const U32 dictLimit = ms->window.dictLimit; const BYTE* const ip = (curr>=dictLimit) ? base + curr : dictBase + curr; const BYTE* const iend = (curr>=dictLimit) ? inputEnd : dictBase + dictLimit; const BYTE* const dictEnd = dictBase + dictLimit; const BYTE* const prefixStart = base + dictLimit; const BYTE* match; U32* smallerPtr = bt + 2*(curr&btMask); U32* largerPtr = smallerPtr + 1; U32 matchIndex = *smallerPtr; /* this candidate is unsorted : next sorted candidate is reached through *smallerPtr, while *largerPtr contains previous unsorted candidate (which is already saved and can be overwritten) */ U32 dummy32; /* to be nullified at the end */ U32 const windowValid = ms->window.lowLimit; U32 const maxDistance = 1U << cParams->windowLog; U32 const windowLow = (curr - windowValid > maxDistance) ? curr - maxDistance : windowValid; DEBUGLOG(8, "ZSTD_insertDUBT1(%u) (dictLimit=%u, lowLimit=%u)", curr, dictLimit, windowLow); assert(curr >= btLow); assert(ip < iend); /* condition for ZSTD_count */ for (; nbCompares && (matchIndex > windowLow); --nbCompares) { U32* const nextPtr = bt + 2*(matchIndex & btMask); size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */ assert(matchIndex < curr); /* note : all candidates are now supposed sorted, * but it's still possible to have nextPtr[1] == ZSTD_DUBT_UNSORTED_MARK * when a real index has the same value as ZSTD_DUBT_UNSORTED_MARK */ if ( (dictMode != ZSTD_extDict) || (matchIndex+matchLength >= dictLimit) /* both in current segment*/ || (curr < dictLimit) /* both in extDict */) { const BYTE* const mBase = ( (dictMode != ZSTD_extDict) || (matchIndex+matchLength >= dictLimit)) ? base : dictBase; assert( (matchIndex+matchLength >= dictLimit) /* might be wrong if extDict is incorrectly set to 0 */ || (curr < dictLimit) ); match = mBase + matchIndex; matchLength += ZSTD_count(ip+matchLength, match+matchLength, iend); } else { match = dictBase + matchIndex; matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart); if (matchIndex+matchLength >= dictLimit) match = base + matchIndex; /* preparation for next read of match[matchLength] */ } DEBUGLOG(8, "ZSTD_insertDUBT1: comparing %u with %u : found %u common bytes ", curr, matchIndex, (U32)matchLength); if (ip+matchLength == iend) { /* equal : no way to know if inf or sup */ break; /* drop , to guarantee consistency ; miss a bit of compression, but other solutions can corrupt tree */ } if (match[matchLength] < ip[matchLength]) { /* necessarily within buffer */ /* match is smaller than current */ *smallerPtr = matchIndex; /* update smaller idx */ commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */ if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop searching */ DEBUGLOG(8, "ZSTD_insertDUBT1: %u (>btLow=%u) is smaller : next => %u", matchIndex, btLow, nextPtr[1]); smallerPtr = nextPtr+1; /* new "candidate" => larger than match, which was smaller than target */ matchIndex = nextPtr[1]; /* new matchIndex, larger than previous and closer to current */ } else { /* match is larger than current */ *largerPtr = matchIndex; commonLengthLarger = matchLength; if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop searching */ DEBUGLOG(8, "ZSTD_insertDUBT1: %u (>btLow=%u) is larger => %u", matchIndex, btLow, nextPtr[0]); largerPtr = nextPtr; matchIndex = nextPtr[0]; } } *smallerPtr = *largerPtr = 0; } static size_t ZSTD_DUBT_findBetterDictMatch ( const ZSTD_matchState_t* ms, const BYTE* const ip, const BYTE* const iend, size_t* offsetPtr, size_t bestLength, U32 nbCompares, U32 const mls, const ZSTD_dictMode_e dictMode) { const ZSTD_matchState_t * const dms = ms->dictMatchState; const ZSTD_compressionParameters* const dmsCParams = &dms->cParams; const U32 * const dictHashTable = dms->hashTable; U32 const hashLog = dmsCParams->hashLog; size_t const h = ZSTD_hashPtr(ip, hashLog, mls); U32 dictMatchIndex = dictHashTable[h]; const BYTE* const base = ms->window.base; const BYTE* const prefixStart = base + ms->window.dictLimit; U32 const curr = (U32)(ip-base); const BYTE* const dictBase = dms->window.base; const BYTE* const dictEnd = dms->window.nextSrc; U32 const dictHighLimit = (U32)(dms->window.nextSrc - dms->window.base); U32 const dictLowLimit = dms->window.lowLimit; U32 const dictIndexDelta = ms->window.lowLimit - dictHighLimit; U32* const dictBt = dms->chainTable; U32 const btLog = dmsCParams->chainLog - 1; U32 const btMask = (1 << btLog) - 1; U32 const btLow = (btMask >= dictHighLimit - dictLowLimit) ? dictLowLimit : dictHighLimit - btMask; size_t commonLengthSmaller=0, commonLengthLarger=0; (void)dictMode; assert(dictMode == ZSTD_dictMatchState); for (; nbCompares && (dictMatchIndex > dictLowLimit); --nbCompares) { U32* const nextPtr = dictBt + 2*(dictMatchIndex & btMask); size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */ const BYTE* match = dictBase + dictMatchIndex; matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart); if (dictMatchIndex+matchLength >= dictHighLimit) match = base + dictMatchIndex + dictIndexDelta; /* to prepare for next usage of match[matchLength] */ if (matchLength > bestLength) { U32 matchIndex = dictMatchIndex + dictIndexDelta; if ( (4*(int)(matchLength-bestLength)) > (int)(ZSTD_highbit32(curr-matchIndex+1) - ZSTD_highbit32((U32)offsetPtr[0]+1)) ) { DEBUGLOG(9, "ZSTD_DUBT_findBetterDictMatch(%u) : found better match length %u -> %u and offsetCode %u -> %u (dictMatchIndex %u, matchIndex %u)", curr, (U32)bestLength, (U32)matchLength, (U32)*offsetPtr, OFFSET_TO_OFFBASE(curr - matchIndex), dictMatchIndex, matchIndex); bestLength = matchLength, *offsetPtr = OFFSET_TO_OFFBASE(curr - matchIndex); } if (ip+matchLength == iend) { /* reached end of input : ip[matchLength] is not valid, no way to know if it's larger or smaller than match */ break; /* drop, to guarantee consistency (miss a little bit of compression) */ } } if (match[matchLength] < ip[matchLength]) { if (dictMatchIndex <= btLow) { break; } /* beyond tree size, stop the search */ commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */ dictMatchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */ } else { /* match is larger than current */ if (dictMatchIndex <= btLow) { break; } /* beyond tree size, stop the search */ commonLengthLarger = matchLength; dictMatchIndex = nextPtr[0]; } } if (bestLength >= MINMATCH) { U32 const mIndex = curr - (U32)OFFBASE_TO_OFFSET(*offsetPtr); (void)mIndex; DEBUGLOG(8, "ZSTD_DUBT_findBetterDictMatch(%u) : found match of length %u and offsetCode %u (pos %u)", curr, (U32)bestLength, (U32)*offsetPtr, mIndex); } return bestLength; } static size_t ZSTD_DUBT_findBestMatch(ZSTD_matchState_t* ms, const BYTE* const ip, const BYTE* const iend, size_t* offBasePtr, U32 const mls, const ZSTD_dictMode_e dictMode) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32* const hashTable = ms->hashTable; U32 const hashLog = cParams->hashLog; size_t const h = ZSTD_hashPtr(ip, hashLog, mls); U32 matchIndex = hashTable[h]; const BYTE* const base = ms->window.base; U32 const curr = (U32)(ip-base); U32 const windowLow = ZSTD_getLowestMatchIndex(ms, curr, cParams->windowLog); U32* const bt = ms->chainTable; U32 const btLog = cParams->chainLog - 1; U32 const btMask = (1 << btLog) - 1; U32 const btLow = (btMask >= curr) ? 0 : curr - btMask; U32 const unsortLimit = MAX(btLow, windowLow); U32* nextCandidate = bt + 2*(matchIndex&btMask); U32* unsortedMark = bt + 2*(matchIndex&btMask) + 1; U32 nbCompares = 1U << cParams->searchLog; U32 nbCandidates = nbCompares; U32 previousCandidate = 0; DEBUGLOG(7, "ZSTD_DUBT_findBestMatch (%u) ", curr); assert(ip <= iend-8); /* required for h calculation */ assert(dictMode != ZSTD_dedicatedDictSearch); /* reach end of unsorted candidates list */ while ( (matchIndex > unsortLimit) && (*unsortedMark == ZSTD_DUBT_UNSORTED_MARK) && (nbCandidates > 1) ) { DEBUGLOG(8, "ZSTD_DUBT_findBestMatch: candidate %u is unsorted", matchIndex); *unsortedMark = previousCandidate; /* the unsortedMark becomes a reversed chain, to move up back to original position */ previousCandidate = matchIndex; matchIndex = *nextCandidate; nextCandidate = bt + 2*(matchIndex&btMask); unsortedMark = bt + 2*(matchIndex&btMask) + 1; nbCandidates --; } /* nullify last candidate if it's still unsorted * simplification, detrimental to compression ratio, beneficial for speed */ if ( (matchIndex > unsortLimit) && (*unsortedMark==ZSTD_DUBT_UNSORTED_MARK) ) { DEBUGLOG(7, "ZSTD_DUBT_findBestMatch: nullify last unsorted candidate %u", matchIndex); *nextCandidate = *unsortedMark = 0; } /* batch sort stacked candidates */ matchIndex = previousCandidate; while (matchIndex) { /* will end on matchIndex == 0 */ U32* const nextCandidateIdxPtr = bt + 2*(matchIndex&btMask) + 1; U32 const nextCandidateIdx = *nextCandidateIdxPtr; ZSTD_insertDUBT1(ms, matchIndex, iend, nbCandidates, unsortLimit, dictMode); matchIndex = nextCandidateIdx; nbCandidates++; } /* find longest match */ { size_t commonLengthSmaller = 0, commonLengthLarger = 0; const BYTE* const dictBase = ms->window.dictBase; const U32 dictLimit = ms->window.dictLimit; const BYTE* const dictEnd = dictBase + dictLimit; const BYTE* const prefixStart = base + dictLimit; U32* smallerPtr = bt + 2*(curr&btMask); U32* largerPtr = bt + 2*(curr&btMask) + 1; U32 matchEndIdx = curr + 8 + 1; U32 dummy32; /* to be nullified at the end */ size_t bestLength = 0; matchIndex = hashTable[h]; hashTable[h] = curr; /* Update Hash Table */ for (; nbCompares && (matchIndex > windowLow); --nbCompares) { U32* const nextPtr = bt + 2*(matchIndex & btMask); size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */ const BYTE* match; if ((dictMode != ZSTD_extDict) || (matchIndex+matchLength >= dictLimit)) { match = base + matchIndex; matchLength += ZSTD_count(ip+matchLength, match+matchLength, iend); } else { match = dictBase + matchIndex; matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart); if (matchIndex+matchLength >= dictLimit) match = base + matchIndex; /* to prepare for next usage of match[matchLength] */ } if (matchLength > bestLength) { if (matchLength > matchEndIdx - matchIndex) matchEndIdx = matchIndex + (U32)matchLength; if ( (4*(int)(matchLength-bestLength)) > (int)(ZSTD_highbit32(curr - matchIndex + 1) - ZSTD_highbit32((U32)*offBasePtr)) ) bestLength = matchLength, *offBasePtr = OFFSET_TO_OFFBASE(curr - matchIndex); if (ip+matchLength == iend) { /* equal : no way to know if inf or sup */ if (dictMode == ZSTD_dictMatchState) { nbCompares = 0; /* in addition to avoiding checking any * further in this loop, make sure we * skip checking in the dictionary. */ } break; /* drop, to guarantee consistency (miss a little bit of compression) */ } } if (match[matchLength] < ip[matchLength]) { /* match is smaller than current */ *smallerPtr = matchIndex; /* update smaller idx */ commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */ if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */ smallerPtr = nextPtr+1; /* new "smaller" => larger of match */ matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */ } else { /* match is larger than current */ *largerPtr = matchIndex; commonLengthLarger = matchLength; if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */ largerPtr = nextPtr; matchIndex = nextPtr[0]; } } *smallerPtr = *largerPtr = 0; assert(nbCompares <= (1U << ZSTD_SEARCHLOG_MAX)); /* Check we haven't underflowed. */ if (dictMode == ZSTD_dictMatchState && nbCompares) { bestLength = ZSTD_DUBT_findBetterDictMatch( ms, ip, iend, offBasePtr, bestLength, nbCompares, mls, dictMode); } assert(matchEndIdx > curr+8); /* ensure nextToUpdate is increased */ ms->nextToUpdate = matchEndIdx - 8; /* skip repetitive patterns */ if (bestLength >= MINMATCH) { U32 const mIndex = curr - (U32)OFFBASE_TO_OFFSET(*offBasePtr); (void)mIndex; DEBUGLOG(8, "ZSTD_DUBT_findBestMatch(%u) : found match of length %u and offsetCode %u (pos %u)", curr, (U32)bestLength, (U32)*offBasePtr, mIndex); } return bestLength; } } /** ZSTD_BtFindBestMatch() : Tree updater, providing best match */ FORCE_INLINE_TEMPLATE size_t ZSTD_BtFindBestMatch( ZSTD_matchState_t* ms, const BYTE* const ip, const BYTE* const iLimit, size_t* offBasePtr, const U32 mls /* template */, const ZSTD_dictMode_e dictMode) { DEBUGLOG(7, "ZSTD_BtFindBestMatch"); if (ip < ms->window.base + ms->nextToUpdate) return 0; /* skipped area */ ZSTD_updateDUBT(ms, ip, iLimit, mls); return ZSTD_DUBT_findBestMatch(ms, ip, iLimit, offBasePtr, mls, dictMode); } /*********************************** * Dedicated dict search ***********************************/ void ZSTD_dedicatedDictSearch_lazy_loadDictionary(ZSTD_matchState_t* ms, const BYTE* const ip) { const BYTE* const base = ms->window.base; U32 const target = (U32)(ip - base); U32* const hashTable = ms->hashTable; U32* const chainTable = ms->chainTable; U32 const chainSize = 1 << ms->cParams.chainLog; U32 idx = ms->nextToUpdate; U32 const minChain = chainSize < target - idx ? target - chainSize : idx; U32 const bucketSize = 1 << ZSTD_LAZY_DDSS_BUCKET_LOG; U32 const cacheSize = bucketSize - 1; U32 const chainAttempts = (1 << ms->cParams.searchLog) - cacheSize; U32 const chainLimit = chainAttempts > 255 ? 255 : chainAttempts; /* We know the hashtable is oversized by a factor of `bucketSize`. * We are going to temporarily pretend `bucketSize == 1`, keeping only a * single entry. We will use the rest of the space to construct a temporary * chaintable. */ U32 const hashLog = ms->cParams.hashLog - ZSTD_LAZY_DDSS_BUCKET_LOG; U32* const tmpHashTable = hashTable; U32* const tmpChainTable = hashTable + ((size_t)1 << hashLog); U32 const tmpChainSize = (U32)((1 << ZSTD_LAZY_DDSS_BUCKET_LOG) - 1) << hashLog; U32 const tmpMinChain = tmpChainSize < target ? target - tmpChainSize : idx; U32 hashIdx; assert(ms->cParams.chainLog <= 24); assert(ms->cParams.hashLog > ms->cParams.chainLog); assert(idx != 0); assert(tmpMinChain <= minChain); /* fill conventional hash table and conventional chain table */ for ( ; idx < target; idx++) { U32 const h = (U32)ZSTD_hashPtr(base + idx, hashLog, ms->cParams.minMatch); if (idx >= tmpMinChain) { tmpChainTable[idx - tmpMinChain] = hashTable[h]; } tmpHashTable[h] = idx; } /* sort chains into ddss chain table */ { U32 chainPos = 0; for (hashIdx = 0; hashIdx < (1U << hashLog); hashIdx++) { U32 count; U32 countBeyondMinChain = 0; U32 i = tmpHashTable[hashIdx]; for (count = 0; i >= tmpMinChain && count < cacheSize; count++) { /* skip through the chain to the first position that won't be * in the hash cache bucket */ if (i < minChain) { countBeyondMinChain++; } i = tmpChainTable[i - tmpMinChain]; } if (count == cacheSize) { for (count = 0; count < chainLimit;) { if (i < minChain) { if (!i || ++countBeyondMinChain > cacheSize) { /* only allow pulling `cacheSize` number of entries * into the cache or chainTable beyond `minChain`, * to replace the entries pulled out of the * chainTable into the cache. This lets us reach * back further without increasing the total number * of entries in the chainTable, guaranteeing the * DDSS chain table will fit into the space * allocated for the regular one. */ break; } } chainTable[chainPos++] = i; count++; if (i < tmpMinChain) { break; } i = tmpChainTable[i - tmpMinChain]; } } else { count = 0; } if (count) { tmpHashTable[hashIdx] = ((chainPos - count) << 8) + count; } else { tmpHashTable[hashIdx] = 0; } } assert(chainPos <= chainSize); /* I believe this is guaranteed... */ } /* move chain pointers into the last entry of each hash bucket */ for (hashIdx = (1 << hashLog); hashIdx; ) { U32 const bucketIdx = --hashIdx << ZSTD_LAZY_DDSS_BUCKET_LOG; U32 const chainPackedPointer = tmpHashTable[hashIdx]; U32 i; for (i = 0; i < cacheSize; i++) { hashTable[bucketIdx + i] = 0; } hashTable[bucketIdx + bucketSize - 1] = chainPackedPointer; } /* fill the buckets of the hash table */ for (idx = ms->nextToUpdate; idx < target; idx++) { U32 const h = (U32)ZSTD_hashPtr(base + idx, hashLog, ms->cParams.minMatch) << ZSTD_LAZY_DDSS_BUCKET_LOG; U32 i; /* Shift hash cache down 1. */ for (i = cacheSize - 1; i; i--) hashTable[h + i] = hashTable[h + i - 1]; hashTable[h] = idx; } ms->nextToUpdate = target; } /* Returns the longest match length found in the dedicated dict search structure. * If none are longer than the argument ml, then ml will be returned. */ FORCE_INLINE_TEMPLATE size_t ZSTD_dedicatedDictSearch_lazy_search(size_t* offsetPtr, size_t ml, U32 nbAttempts, const ZSTD_matchState_t* const dms, const BYTE* const ip, const BYTE* const iLimit, const BYTE* const prefixStart, const U32 curr, const U32 dictLimit, const size_t ddsIdx) { const U32 ddsLowestIndex = dms->window.dictLimit; const BYTE* const ddsBase = dms->window.base; const BYTE* const ddsEnd = dms->window.nextSrc; const U32 ddsSize = (U32)(ddsEnd - ddsBase); const U32 ddsIndexDelta = dictLimit - ddsSize; const U32 bucketSize = (1 << ZSTD_LAZY_DDSS_BUCKET_LOG); const U32 bucketLimit = nbAttempts < bucketSize - 1 ? nbAttempts : bucketSize - 1; U32 ddsAttempt; U32 matchIndex; for (ddsAttempt = 0; ddsAttempt < bucketSize - 1; ddsAttempt++) { PREFETCH_L1(ddsBase + dms->hashTable[ddsIdx + ddsAttempt]); } { U32 const chainPackedPointer = dms->hashTable[ddsIdx + bucketSize - 1]; U32 const chainIndex = chainPackedPointer >> 8; PREFETCH_L1(&dms->chainTable[chainIndex]); } for (ddsAttempt = 0; ddsAttempt < bucketLimit; ddsAttempt++) { size_t currentMl=0; const BYTE* match; matchIndex = dms->hashTable[ddsIdx + ddsAttempt]; match = ddsBase + matchIndex; if (!matchIndex) { return ml; } /* guaranteed by table construction */ (void)ddsLowestIndex; assert(matchIndex >= ddsLowestIndex); assert(match+4 <= ddsEnd); if (MEM_read32(match) == MEM_read32(ip)) { /* assumption : matchIndex <= dictLimit-4 (by table construction) */ currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, ddsEnd, prefixStart) + 4; } /* save best solution */ if (currentMl > ml) { ml = currentMl; *offsetPtr = OFFSET_TO_OFFBASE(curr - (matchIndex + ddsIndexDelta)); if (ip+currentMl == iLimit) { /* best possible, avoids read overflow on next attempt */ return ml; } } } { U32 const chainPackedPointer = dms->hashTable[ddsIdx + bucketSize - 1]; U32 chainIndex = chainPackedPointer >> 8; U32 const chainLength = chainPackedPointer & 0xFF; U32 const chainAttempts = nbAttempts - ddsAttempt; U32 const chainLimit = chainAttempts > chainLength ? chainLength : chainAttempts; U32 chainAttempt; for (chainAttempt = 0 ; chainAttempt < chainLimit; chainAttempt++) { PREFETCH_L1(ddsBase + dms->chainTable[chainIndex + chainAttempt]); } for (chainAttempt = 0 ; chainAttempt < chainLimit; chainAttempt++, chainIndex++) { size_t currentMl=0; const BYTE* match; matchIndex = dms->chainTable[chainIndex]; match = ddsBase + matchIndex; /* guaranteed by table construction */ assert(matchIndex >= ddsLowestIndex); assert(match+4 <= ddsEnd); if (MEM_read32(match) == MEM_read32(ip)) { /* assumption : matchIndex <= dictLimit-4 (by table construction) */ currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, ddsEnd, prefixStart) + 4; } /* save best solution */ if (currentMl > ml) { ml = currentMl; *offsetPtr = OFFSET_TO_OFFBASE(curr - (matchIndex + ddsIndexDelta)); if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */ } } } return ml; } /* ********************************* * Hash Chain ***********************************/ #define NEXT_IN_CHAIN(d, mask) chainTable[(d) & (mask)] /* Update chains up to ip (excluded) Assumption : always within prefix (i.e. not within extDict) */ FORCE_INLINE_TEMPLATE U32 ZSTD_insertAndFindFirstIndex_internal( ZSTD_matchState_t* ms, const ZSTD_compressionParameters* const cParams, const BYTE* ip, U32 const mls, U32 const lazySkipping) { U32* const hashTable = ms->hashTable; const U32 hashLog = cParams->hashLog; U32* const chainTable = ms->chainTable; const U32 chainMask = (1 << cParams->chainLog) - 1; const BYTE* const base = ms->window.base; const U32 target = (U32)(ip - base); U32 idx = ms->nextToUpdate; while(idx < target) { /* catch up */ size_t const h = ZSTD_hashPtr(base+idx, hashLog, mls); NEXT_IN_CHAIN(idx, chainMask) = hashTable[h]; hashTable[h] = idx; idx++; /* Stop inserting every position when in the lazy skipping mode. */ if (lazySkipping) break; } ms->nextToUpdate = target; return hashTable[ZSTD_hashPtr(ip, hashLog, mls)]; } U32 ZSTD_insertAndFindFirstIndex(ZSTD_matchState_t* ms, const BYTE* ip) { const ZSTD_compressionParameters* const cParams = &ms->cParams; return ZSTD_insertAndFindFirstIndex_internal(ms, cParams, ip, ms->cParams.minMatch, /* lazySkipping*/ 0); } /* inlining is important to hardwire a hot branch (template emulation) */ FORCE_INLINE_TEMPLATE size_t ZSTD_HcFindBestMatch( ZSTD_matchState_t* ms, const BYTE* const ip, const BYTE* const iLimit, size_t* offsetPtr, const U32 mls, const ZSTD_dictMode_e dictMode) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32* const chainTable = ms->chainTable; const U32 chainSize = (1 << cParams->chainLog); const U32 chainMask = chainSize-1; const BYTE* const base = ms->window.base; const BYTE* const dictBase = ms->window.dictBase; const U32 dictLimit = ms->window.dictLimit; const BYTE* const prefixStart = base + dictLimit; const BYTE* const dictEnd = dictBase + dictLimit; const U32 curr = (U32)(ip-base); const U32 maxDistance = 1U << cParams->windowLog; const U32 lowestValid = ms->window.lowLimit; const U32 withinMaxDistance = (curr - lowestValid > maxDistance) ? curr - maxDistance : lowestValid; const U32 isDictionary = (ms->loadedDictEnd != 0); const U32 lowLimit = isDictionary ? lowestValid : withinMaxDistance; const U32 minChain = curr > chainSize ? curr - chainSize : 0; U32 nbAttempts = 1U << cParams->searchLog; size_t ml=4-1; const ZSTD_matchState_t* const dms = ms->dictMatchState; const U32 ddsHashLog = dictMode == ZSTD_dedicatedDictSearch ? dms->cParams.hashLog - ZSTD_LAZY_DDSS_BUCKET_LOG : 0; const size_t ddsIdx = dictMode == ZSTD_dedicatedDictSearch ? ZSTD_hashPtr(ip, ddsHashLog, mls) << ZSTD_LAZY_DDSS_BUCKET_LOG : 0; U32 matchIndex; if (dictMode == ZSTD_dedicatedDictSearch) { const U32* entry = &dms->hashTable[ddsIdx]; PREFETCH_L1(entry); } /* HC4 match finder */ matchIndex = ZSTD_insertAndFindFirstIndex_internal(ms, cParams, ip, mls, ms->lazySkipping); for ( ; (matchIndex>=lowLimit) & (nbAttempts>0) ; nbAttempts--) { size_t currentMl=0; if ((dictMode != ZSTD_extDict) || matchIndex >= dictLimit) { const BYTE* const match = base + matchIndex; assert(matchIndex >= dictLimit); /* ensures this is true if dictMode != ZSTD_extDict */ /* read 4B starting from (match + ml + 1 - sizeof(U32)) */ if (MEM_read32(match + ml - 3) == MEM_read32(ip + ml - 3)) /* potentially better */ currentMl = ZSTD_count(ip, match, iLimit); } else { const BYTE* const match = dictBase + matchIndex; assert(match+4 <= dictEnd); if (MEM_read32(match) == MEM_read32(ip)) /* assumption : matchIndex <= dictLimit-4 (by table construction) */ currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, dictEnd, prefixStart) + 4; } /* save best solution */ if (currentMl > ml) { ml = currentMl; *offsetPtr = OFFSET_TO_OFFBASE(curr - matchIndex); if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */ } if (matchIndex <= minChain) break; matchIndex = NEXT_IN_CHAIN(matchIndex, chainMask); } assert(nbAttempts <= (1U << ZSTD_SEARCHLOG_MAX)); /* Check we haven't underflowed. */ if (dictMode == ZSTD_dedicatedDictSearch) { ml = ZSTD_dedicatedDictSearch_lazy_search(offsetPtr, ml, nbAttempts, dms, ip, iLimit, prefixStart, curr, dictLimit, ddsIdx); } else if (dictMode == ZSTD_dictMatchState) { const U32* const dmsChainTable = dms->chainTable; const U32 dmsChainSize = (1 << dms->cParams.chainLog); const U32 dmsChainMask = dmsChainSize - 1; const U32 dmsLowestIndex = dms->window.dictLimit; const BYTE* const dmsBase = dms->window.base; const BYTE* const dmsEnd = dms->window.nextSrc; const U32 dmsSize = (U32)(dmsEnd - dmsBase); const U32 dmsIndexDelta = dictLimit - dmsSize; const U32 dmsMinChain = dmsSize > dmsChainSize ? dmsSize - dmsChainSize : 0; matchIndex = dms->hashTable[ZSTD_hashPtr(ip, dms->cParams.hashLog, mls)]; for ( ; (matchIndex>=dmsLowestIndex) & (nbAttempts>0) ; nbAttempts--) { size_t currentMl=0; const BYTE* const match = dmsBase + matchIndex; assert(match+4 <= dmsEnd); if (MEM_read32(match) == MEM_read32(ip)) /* assumption : matchIndex <= dictLimit-4 (by table construction) */ currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, dmsEnd, prefixStart) + 4; /* save best solution */ if (currentMl > ml) { ml = currentMl; assert(curr > matchIndex + dmsIndexDelta); *offsetPtr = OFFSET_TO_OFFBASE(curr - (matchIndex + dmsIndexDelta)); if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */ } if (matchIndex <= dmsMinChain) break; matchIndex = dmsChainTable[matchIndex & dmsChainMask]; } } return ml; } /* ********************************* * (SIMD) Row-based matchfinder ***********************************/ /* Constants for row-based hash */ #define ZSTD_ROW_HASH_TAG_MASK ((1u << ZSTD_ROW_HASH_TAG_BITS) - 1) #define ZSTD_ROW_HASH_MAX_ENTRIES 64 /* absolute maximum number of entries per row, for all configurations */ #define ZSTD_ROW_HASH_CACHE_MASK (ZSTD_ROW_HASH_CACHE_SIZE - 1) typedef U64 ZSTD_VecMask; /* Clarifies when we are interacting with a U64 representing a mask of matches */ /* ZSTD_VecMask_next(): * Starting from the LSB, returns the idx of the next non-zero bit. * Basically counting the nb of trailing zeroes. */ MEM_STATIC U32 ZSTD_VecMask_next(ZSTD_VecMask val) { return ZSTD_countTrailingZeros64(val); } /* ZSTD_row_nextIndex(): * Returns the next index to insert at within a tagTable row, and updates the "head" * value to reflect the update. Essentially cycles backwards from [1, {entries per row}) */ FORCE_INLINE_TEMPLATE U32 ZSTD_row_nextIndex(BYTE* const tagRow, U32 const rowMask) { U32 next = (*tagRow-1) & rowMask; next += (next == 0) ? rowMask : 0; /* skip first position */ *tagRow = (BYTE)next; return next; } /* ZSTD_isAligned(): * Checks that a pointer is aligned to "align" bytes which must be a power of 2. */ MEM_STATIC int ZSTD_isAligned(void const* ptr, size_t align) { assert((align & (align - 1)) == 0); return (((size_t)ptr) & (align - 1)) == 0; } /* ZSTD_row_prefetch(): * Performs prefetching for the hashTable and tagTable at a given row. */ FORCE_INLINE_TEMPLATE void ZSTD_row_prefetch(U32 const* hashTable, BYTE const* tagTable, U32 const relRow, U32 const rowLog) { PREFETCH_L1(hashTable + relRow); if (rowLog >= 5) { PREFETCH_L1(hashTable + relRow + 16); /* Note: prefetching more of the hash table does not appear to be beneficial for 128-entry rows */ } PREFETCH_L1(tagTable + relRow); if (rowLog == 6) { PREFETCH_L1(tagTable + relRow + 32); } assert(rowLog == 4 || rowLog == 5 || rowLog == 6); assert(ZSTD_isAligned(hashTable + relRow, 64)); /* prefetched hash row always 64-byte aligned */ assert(ZSTD_isAligned(tagTable + relRow, (size_t)1 << rowLog)); /* prefetched tagRow sits on correct multiple of bytes (32,64,128) */ } /* ZSTD_row_fillHashCache(): * Fill up the hash cache starting at idx, prefetching up to ZSTD_ROW_HASH_CACHE_SIZE entries, * but not beyond iLimit. */ FORCE_INLINE_TEMPLATE void ZSTD_row_fillHashCache(ZSTD_matchState_t* ms, const BYTE* base, U32 const rowLog, U32 const mls, U32 idx, const BYTE* const iLimit) { U32 const* const hashTable = ms->hashTable; BYTE const* const tagTable = ms->tagTable; U32 const hashLog = ms->rowHashLog; U32 const maxElemsToPrefetch = (base + idx) > iLimit ? 0 : (U32)(iLimit - (base + idx) + 1); U32 const lim = idx + MIN(ZSTD_ROW_HASH_CACHE_SIZE, maxElemsToPrefetch); for (; idx < lim; ++idx) { U32 const hash = (U32)ZSTD_hashPtrSalted(base + idx, hashLog + ZSTD_ROW_HASH_TAG_BITS, mls, ms->hashSalt); U32 const row = (hash >> ZSTD_ROW_HASH_TAG_BITS) << rowLog; ZSTD_row_prefetch(hashTable, tagTable, row, rowLog); ms->hashCache[idx & ZSTD_ROW_HASH_CACHE_MASK] = hash; } DEBUGLOG(6, "ZSTD_row_fillHashCache(): [%u %u %u %u %u %u %u %u]", ms->hashCache[0], ms->hashCache[1], ms->hashCache[2], ms->hashCache[3], ms->hashCache[4], ms->hashCache[5], ms->hashCache[6], ms->hashCache[7]); } /* ZSTD_row_nextCachedHash(): * Returns the hash of base + idx, and replaces the hash in the hash cache with the byte at * base + idx + ZSTD_ROW_HASH_CACHE_SIZE. Also prefetches the appropriate rows from hashTable and tagTable. */ FORCE_INLINE_TEMPLATE U32 ZSTD_row_nextCachedHash(U32* cache, U32 const* hashTable, BYTE const* tagTable, BYTE const* base, U32 idx, U32 const hashLog, U32 const rowLog, U32 const mls, U64 const hashSalt) { U32 const newHash = (U32)ZSTD_hashPtrSalted(base+idx+ZSTD_ROW_HASH_CACHE_SIZE, hashLog + ZSTD_ROW_HASH_TAG_BITS, mls, hashSalt); U32 const row = (newHash >> ZSTD_ROW_HASH_TAG_BITS) << rowLog; ZSTD_row_prefetch(hashTable, tagTable, row, rowLog); { U32 const hash = cache[idx & ZSTD_ROW_HASH_CACHE_MASK]; cache[idx & ZSTD_ROW_HASH_CACHE_MASK] = newHash; return hash; } } /* ZSTD_row_update_internalImpl(): * Updates the hash table with positions starting from updateStartIdx until updateEndIdx. */ FORCE_INLINE_TEMPLATE void ZSTD_row_update_internalImpl(ZSTD_matchState_t* ms, U32 updateStartIdx, U32 const updateEndIdx, U32 const mls, U32 const rowLog, U32 const rowMask, U32 const useCache) { U32* const hashTable = ms->hashTable; BYTE* const tagTable = ms->tagTable; U32 const hashLog = ms->rowHashLog; const BYTE* const base = ms->window.base; DEBUGLOG(6, "ZSTD_row_update_internalImpl(): updateStartIdx=%u, updateEndIdx=%u", updateStartIdx, updateEndIdx); for (; updateStartIdx < updateEndIdx; ++updateStartIdx) { U32 const hash = useCache ? ZSTD_row_nextCachedHash(ms->hashCache, hashTable, tagTable, base, updateStartIdx, hashLog, rowLog, mls, ms->hashSalt) : (U32)ZSTD_hashPtrSalted(base + updateStartIdx, hashLog + ZSTD_ROW_HASH_TAG_BITS, mls, ms->hashSalt); U32 const relRow = (hash >> ZSTD_ROW_HASH_TAG_BITS) << rowLog; U32* const row = hashTable + relRow; BYTE* tagRow = tagTable + relRow; U32 const pos = ZSTD_row_nextIndex(tagRow, rowMask); assert(hash == ZSTD_hashPtrSalted(base + updateStartIdx, hashLog + ZSTD_ROW_HASH_TAG_BITS, mls, ms->hashSalt)); tagRow[pos] = hash & ZSTD_ROW_HASH_TAG_MASK; row[pos] = updateStartIdx; } } /* ZSTD_row_update_internal(): * Inserts the byte at ip into the appropriate position in the hash table, and updates ms->nextToUpdate. * Skips sections of long matches as is necessary. */ FORCE_INLINE_TEMPLATE void ZSTD_row_update_internal(ZSTD_matchState_t* ms, const BYTE* ip, U32 const mls, U32 const rowLog, U32 const rowMask, U32 const useCache) { U32 idx = ms->nextToUpdate; const BYTE* const base = ms->window.base; const U32 target = (U32)(ip - base); const U32 kSkipThreshold = 384; const U32 kMaxMatchStartPositionsToUpdate = 96; const U32 kMaxMatchEndPositionsToUpdate = 32; if (useCache) { /* Only skip positions when using hash cache, i.e. * if we are loading a dict, don't skip anything. * If we decide to skip, then we only update a set number * of positions at the beginning and end of the match. */ if (UNLIKELY(target - idx > kSkipThreshold)) { U32 const bound = idx + kMaxMatchStartPositionsToUpdate; ZSTD_row_update_internalImpl(ms, idx, bound, mls, rowLog, rowMask, useCache); idx = target - kMaxMatchEndPositionsToUpdate; ZSTD_row_fillHashCache(ms, base, rowLog, mls, idx, ip+1); } } assert(target >= idx); ZSTD_row_update_internalImpl(ms, idx, target, mls, rowLog, rowMask, useCache); ms->nextToUpdate = target; } /* ZSTD_row_update(): * External wrapper for ZSTD_row_update_internal(). Used for filling the hashtable during dictionary * processing. */ void ZSTD_row_update(ZSTD_matchState_t* const ms, const BYTE* ip) { const U32 rowLog = BOUNDED(4, ms->cParams.searchLog, 6); const U32 rowMask = (1u << rowLog) - 1; const U32 mls = MIN(ms->cParams.minMatch, 6 /* mls caps out at 6 */); DEBUGLOG(5, "ZSTD_row_update(), rowLog=%u", rowLog); ZSTD_row_update_internal(ms, ip, mls, rowLog, rowMask, 0 /* don't use cache */); } /* Returns the mask width of bits group of which will be set to 1. Given not all * architectures have easy movemask instruction, this helps to iterate over * groups of bits easier and faster. */ FORCE_INLINE_TEMPLATE U32 ZSTD_row_matchMaskGroupWidth(const U32 rowEntries) { assert((rowEntries == 16) || (rowEntries == 32) || rowEntries == 64); assert(rowEntries <= ZSTD_ROW_HASH_MAX_ENTRIES); (void)rowEntries; #if defined(ZSTD_ARCH_ARM_NEON) /* NEON path only works for little endian */ if (!MEM_isLittleEndian()) { return 1; } if (rowEntries == 16) { return 4; } if (rowEntries == 32) { return 2; } if (rowEntries == 64) { return 1; } #endif return 1; } #if defined(ZSTD_ARCH_X86_SSE2) FORCE_INLINE_TEMPLATE ZSTD_VecMask ZSTD_row_getSSEMask(int nbChunks, const BYTE* const src, const BYTE tag, const U32 head) { const __m128i comparisonMask = _mm_set1_epi8((char)tag); int matches[4] = {0}; int i; assert(nbChunks == 1 || nbChunks == 2 || nbChunks == 4); for (i=0; i> chunkSize; do { size_t chunk = MEM_readST(&src[i]); chunk ^= splatChar; chunk = (((chunk | x80) - x01) | chunk) & x80; matches <<= chunkSize; matches |= (chunk * extractMagic) >> shiftAmount; i -= chunkSize; } while (i >= 0); } else { /* big endian: reverse bits during extraction */ const size_t msb = xFF ^ (xFF >> 1); const size_t extractMagic = (msb / 0x1FF) | msb; do { size_t chunk = MEM_readST(&src[i]); chunk ^= splatChar; chunk = (((chunk | x80) - x01) | chunk) & x80; matches <<= chunkSize; matches |= ((chunk >> 7) * extractMagic) >> shiftAmount; i -= chunkSize; } while (i >= 0); } matches = ~matches; if (rowEntries == 16) { return ZSTD_rotateRight_U16((U16)matches, headGrouped); } else if (rowEntries == 32) { return ZSTD_rotateRight_U32((U32)matches, headGrouped); } else { return ZSTD_rotateRight_U64((U64)matches, headGrouped); } } #endif } /* The high-level approach of the SIMD row based match finder is as follows: * - Figure out where to insert the new entry: * - Generate a hash from a byte along with an additional 1-byte "short hash". The additional byte is our "tag" * - The hashTable is effectively split into groups or "rows" of 16 or 32 entries of U32, and the hash determines * which row to insert into. * - Determine the correct position within the row to insert the entry into. Each row of 16 or 32 can * be considered as a circular buffer with a "head" index that resides in the tagTable. * - Also insert the "tag" into the equivalent row and position in the tagTable. * - Note: The tagTable has 17 or 33 1-byte entries per row, due to 16 or 32 tags, and 1 "head" entry. * The 17 or 33 entry rows are spaced out to occur every 32 or 64 bytes, respectively, * for alignment/performance reasons, leaving some bytes unused. * - Use SIMD to efficiently compare the tags in the tagTable to the 1-byte "short hash" and * generate a bitfield that we can cycle through to check the collisions in the hash table. * - Pick the longest match. */ FORCE_INLINE_TEMPLATE size_t ZSTD_RowFindBestMatch( ZSTD_matchState_t* ms, const BYTE* const ip, const BYTE* const iLimit, size_t* offsetPtr, const U32 mls, const ZSTD_dictMode_e dictMode, const U32 rowLog) { U32* const hashTable = ms->hashTable; BYTE* const tagTable = ms->tagTable; U32* const hashCache = ms->hashCache; const U32 hashLog = ms->rowHashLog; const ZSTD_compressionParameters* const cParams = &ms->cParams; const BYTE* const base = ms->window.base; const BYTE* const dictBase = ms->window.dictBase; const U32 dictLimit = ms->window.dictLimit; const BYTE* const prefixStart = base + dictLimit; const BYTE* const dictEnd = dictBase + dictLimit; const U32 curr = (U32)(ip-base); const U32 maxDistance = 1U << cParams->windowLog; const U32 lowestValid = ms->window.lowLimit; const U32 withinMaxDistance = (curr - lowestValid > maxDistance) ? curr - maxDistance : lowestValid; const U32 isDictionary = (ms->loadedDictEnd != 0); const U32 lowLimit = isDictionary ? lowestValid : withinMaxDistance; const U32 rowEntries = (1U << rowLog); const U32 rowMask = rowEntries - 1; const U32 cappedSearchLog = MIN(cParams->searchLog, rowLog); /* nb of searches is capped at nb entries per row */ const U32 groupWidth = ZSTD_row_matchMaskGroupWidth(rowEntries); const U64 hashSalt = ms->hashSalt; U32 nbAttempts = 1U << cappedSearchLog; size_t ml=4-1; U32 hash; /* DMS/DDS variables that may be referenced laster */ const ZSTD_matchState_t* const dms = ms->dictMatchState; /* Initialize the following variables to satisfy static analyzer */ size_t ddsIdx = 0; U32 ddsExtraAttempts = 0; /* cctx hash tables are limited in searches, but allow extra searches into DDS */ U32 dmsTag = 0; U32* dmsRow = NULL; BYTE* dmsTagRow = NULL; if (dictMode == ZSTD_dedicatedDictSearch) { const U32 ddsHashLog = dms->cParams.hashLog - ZSTD_LAZY_DDSS_BUCKET_LOG; { /* Prefetch DDS hashtable entry */ ddsIdx = ZSTD_hashPtr(ip, ddsHashLog, mls) << ZSTD_LAZY_DDSS_BUCKET_LOG; PREFETCH_L1(&dms->hashTable[ddsIdx]); } ddsExtraAttempts = cParams->searchLog > rowLog ? 1U << (cParams->searchLog - rowLog) : 0; } if (dictMode == ZSTD_dictMatchState) { /* Prefetch DMS rows */ U32* const dmsHashTable = dms->hashTable; BYTE* const dmsTagTable = dms->tagTable; U32 const dmsHash = (U32)ZSTD_hashPtr(ip, dms->rowHashLog + ZSTD_ROW_HASH_TAG_BITS, mls); U32 const dmsRelRow = (dmsHash >> ZSTD_ROW_HASH_TAG_BITS) << rowLog; dmsTag = dmsHash & ZSTD_ROW_HASH_TAG_MASK; dmsTagRow = (BYTE*)(dmsTagTable + dmsRelRow); dmsRow = dmsHashTable + dmsRelRow; ZSTD_row_prefetch(dmsHashTable, dmsTagTable, dmsRelRow, rowLog); } /* Update the hashTable and tagTable up to (but not including) ip */ if (!ms->lazySkipping) { ZSTD_row_update_internal(ms, ip, mls, rowLog, rowMask, 1 /* useCache */); hash = ZSTD_row_nextCachedHash(hashCache, hashTable, tagTable, base, curr, hashLog, rowLog, mls, hashSalt); } else { /* Stop inserting every position when in the lazy skipping mode. * The hash cache is also not kept up to date in this mode. */ hash = (U32)ZSTD_hashPtrSalted(ip, hashLog + ZSTD_ROW_HASH_TAG_BITS, mls, hashSalt); ms->nextToUpdate = curr; } ms->hashSaltEntropy += hash; /* collect salt entropy */ { /* Get the hash for ip, compute the appropriate row */ U32 const relRow = (hash >> ZSTD_ROW_HASH_TAG_BITS) << rowLog; U32 const tag = hash & ZSTD_ROW_HASH_TAG_MASK; U32* const row = hashTable + relRow; BYTE* tagRow = (BYTE*)(tagTable + relRow); U32 const headGrouped = (*tagRow & rowMask) * groupWidth; U32 matchBuffer[ZSTD_ROW_HASH_MAX_ENTRIES]; size_t numMatches = 0; size_t currMatch = 0; ZSTD_VecMask matches = ZSTD_row_getMatchMask(tagRow, (BYTE)tag, headGrouped, rowEntries); /* Cycle through the matches and prefetch */ for (; (matches > 0) && (nbAttempts > 0); matches &= (matches - 1)) { U32 const matchPos = ((headGrouped + ZSTD_VecMask_next(matches)) / groupWidth) & rowMask; U32 const matchIndex = row[matchPos]; if(matchPos == 0) continue; assert(numMatches < rowEntries); if (matchIndex < lowLimit) break; if ((dictMode != ZSTD_extDict) || matchIndex >= dictLimit) { PREFETCH_L1(base + matchIndex); } else { PREFETCH_L1(dictBase + matchIndex); } matchBuffer[numMatches++] = matchIndex; --nbAttempts; } /* Speed opt: insert current byte into hashtable too. This allows us to avoid one iteration of the loop in ZSTD_row_update_internal() at the next search. */ { U32 const pos = ZSTD_row_nextIndex(tagRow, rowMask); tagRow[pos] = (BYTE)tag; row[pos] = ms->nextToUpdate++; } /* Return the longest match */ for (; currMatch < numMatches; ++currMatch) { U32 const matchIndex = matchBuffer[currMatch]; size_t currentMl=0; assert(matchIndex < curr); assert(matchIndex >= lowLimit); if ((dictMode != ZSTD_extDict) || matchIndex >= dictLimit) { const BYTE* const match = base + matchIndex; assert(matchIndex >= dictLimit); /* ensures this is true if dictMode != ZSTD_extDict */ /* read 4B starting from (match + ml + 1 - sizeof(U32)) */ if (MEM_read32(match + ml - 3) == MEM_read32(ip + ml - 3)) /* potentially better */ currentMl = ZSTD_count(ip, match, iLimit); } else { const BYTE* const match = dictBase + matchIndex; assert(match+4 <= dictEnd); if (MEM_read32(match) == MEM_read32(ip)) /* assumption : matchIndex <= dictLimit-4 (by table construction) */ currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, dictEnd, prefixStart) + 4; } /* Save best solution */ if (currentMl > ml) { ml = currentMl; *offsetPtr = OFFSET_TO_OFFBASE(curr - matchIndex); if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */ } } } assert(nbAttempts <= (1U << ZSTD_SEARCHLOG_MAX)); /* Check we haven't underflowed. */ if (dictMode == ZSTD_dedicatedDictSearch) { ml = ZSTD_dedicatedDictSearch_lazy_search(offsetPtr, ml, nbAttempts + ddsExtraAttempts, dms, ip, iLimit, prefixStart, curr, dictLimit, ddsIdx); } else if (dictMode == ZSTD_dictMatchState) { /* TODO: Measure and potentially add prefetching to DMS */ const U32 dmsLowestIndex = dms->window.dictLimit; const BYTE* const dmsBase = dms->window.base; const BYTE* const dmsEnd = dms->window.nextSrc; const U32 dmsSize = (U32)(dmsEnd - dmsBase); const U32 dmsIndexDelta = dictLimit - dmsSize; { U32 const headGrouped = (*dmsTagRow & rowMask) * groupWidth; U32 matchBuffer[ZSTD_ROW_HASH_MAX_ENTRIES]; size_t numMatches = 0; size_t currMatch = 0; ZSTD_VecMask matches = ZSTD_row_getMatchMask(dmsTagRow, (BYTE)dmsTag, headGrouped, rowEntries); for (; (matches > 0) && (nbAttempts > 0); matches &= (matches - 1)) { U32 const matchPos = ((headGrouped + ZSTD_VecMask_next(matches)) / groupWidth) & rowMask; U32 const matchIndex = dmsRow[matchPos]; if(matchPos == 0) continue; if (matchIndex < dmsLowestIndex) break; PREFETCH_L1(dmsBase + matchIndex); matchBuffer[numMatches++] = matchIndex; --nbAttempts; } /* Return the longest match */ for (; currMatch < numMatches; ++currMatch) { U32 const matchIndex = matchBuffer[currMatch]; size_t currentMl=0; assert(matchIndex >= dmsLowestIndex); assert(matchIndex < curr); { const BYTE* const match = dmsBase + matchIndex; assert(match+4 <= dmsEnd); if (MEM_read32(match) == MEM_read32(ip)) currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, dmsEnd, prefixStart) + 4; } if (currentMl > ml) { ml = currentMl; assert(curr > matchIndex + dmsIndexDelta); *offsetPtr = OFFSET_TO_OFFBASE(curr - (matchIndex + dmsIndexDelta)); if (ip+currentMl == iLimit) break; } } } } return ml; } /** * Generate search functions templated on (dictMode, mls, rowLog). * These functions are outlined for code size & compilation time. * ZSTD_searchMax() dispatches to the correct implementation function. * * TODO: The start of the search function involves loading and calculating a * bunch of constants from the ZSTD_matchState_t. These computations could be * done in an initialization function, and saved somewhere in the match state. * Then we could pass a pointer to the saved state instead of the match state, * and avoid duplicate computations. * * TODO: Move the match re-winding into searchMax. This improves compression * ratio, and unlocks further simplifications with the next TODO. * * TODO: Try moving the repcode search into searchMax. After the re-winding * and repcode search are in searchMax, there is no more logic in the match * finder loop that requires knowledge about the dictMode. So we should be * able to avoid force inlining it, and we can join the extDict loop with * the single segment loop. It should go in searchMax instead of its own * function to avoid having multiple virtual function calls per search. */ #define ZSTD_BT_SEARCH_FN(dictMode, mls) ZSTD_BtFindBestMatch_##dictMode##_##mls #define ZSTD_HC_SEARCH_FN(dictMode, mls) ZSTD_HcFindBestMatch_##dictMode##_##mls #define ZSTD_ROW_SEARCH_FN(dictMode, mls, rowLog) ZSTD_RowFindBestMatch_##dictMode##_##mls##_##rowLog #define ZSTD_SEARCH_FN_ATTRS FORCE_NOINLINE #define GEN_ZSTD_BT_SEARCH_FN(dictMode, mls) \ ZSTD_SEARCH_FN_ATTRS size_t ZSTD_BT_SEARCH_FN(dictMode, mls)( \ ZSTD_matchState_t* ms, \ const BYTE* ip, const BYTE* const iLimit, \ size_t* offBasePtr) \ { \ assert(MAX(4, MIN(6, ms->cParams.minMatch)) == mls); \ return ZSTD_BtFindBestMatch(ms, ip, iLimit, offBasePtr, mls, ZSTD_##dictMode); \ } \ #define GEN_ZSTD_HC_SEARCH_FN(dictMode, mls) \ ZSTD_SEARCH_FN_ATTRS size_t ZSTD_HC_SEARCH_FN(dictMode, mls)( \ ZSTD_matchState_t* ms, \ const BYTE* ip, const BYTE* const iLimit, \ size_t* offsetPtr) \ { \ assert(MAX(4, MIN(6, ms->cParams.minMatch)) == mls); \ return ZSTD_HcFindBestMatch(ms, ip, iLimit, offsetPtr, mls, ZSTD_##dictMode); \ } \ #define GEN_ZSTD_ROW_SEARCH_FN(dictMode, mls, rowLog) \ ZSTD_SEARCH_FN_ATTRS size_t ZSTD_ROW_SEARCH_FN(dictMode, mls, rowLog)( \ ZSTD_matchState_t* ms, \ const BYTE* ip, const BYTE* const iLimit, \ size_t* offsetPtr) \ { \ assert(MAX(4, MIN(6, ms->cParams.minMatch)) == mls); \ assert(MAX(4, MIN(6, ms->cParams.searchLog)) == rowLog); \ return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, mls, ZSTD_##dictMode, rowLog); \ } \ #define ZSTD_FOR_EACH_ROWLOG(X, dictMode, mls) \ X(dictMode, mls, 4) \ X(dictMode, mls, 5) \ X(dictMode, mls, 6) #define ZSTD_FOR_EACH_MLS_ROWLOG(X, dictMode) \ ZSTD_FOR_EACH_ROWLOG(X, dictMode, 4) \ ZSTD_FOR_EACH_ROWLOG(X, dictMode, 5) \ ZSTD_FOR_EACH_ROWLOG(X, dictMode, 6) #define ZSTD_FOR_EACH_MLS(X, dictMode) \ X(dictMode, 4) \ X(dictMode, 5) \ X(dictMode, 6) #define ZSTD_FOR_EACH_DICT_MODE(X, ...) \ X(__VA_ARGS__, noDict) \ X(__VA_ARGS__, extDict) \ X(__VA_ARGS__, dictMatchState) \ X(__VA_ARGS__, dedicatedDictSearch) /* Generate row search fns for each combination of (dictMode, mls, rowLog) */ ZSTD_FOR_EACH_DICT_MODE(ZSTD_FOR_EACH_MLS_ROWLOG, GEN_ZSTD_ROW_SEARCH_FN) /* Generate binary Tree search fns for each combination of (dictMode, mls) */ ZSTD_FOR_EACH_DICT_MODE(ZSTD_FOR_EACH_MLS, GEN_ZSTD_BT_SEARCH_FN) /* Generate hash chain search fns for each combination of (dictMode, mls) */ ZSTD_FOR_EACH_DICT_MODE(ZSTD_FOR_EACH_MLS, GEN_ZSTD_HC_SEARCH_FN) typedef enum { search_hashChain=0, search_binaryTree=1, search_rowHash=2 } searchMethod_e; #define GEN_ZSTD_CALL_BT_SEARCH_FN(dictMode, mls) \ case mls: \ return ZSTD_BT_SEARCH_FN(dictMode, mls)(ms, ip, iend, offsetPtr); #define GEN_ZSTD_CALL_HC_SEARCH_FN(dictMode, mls) \ case mls: \ return ZSTD_HC_SEARCH_FN(dictMode, mls)(ms, ip, iend, offsetPtr); #define GEN_ZSTD_CALL_ROW_SEARCH_FN(dictMode, mls, rowLog) \ case rowLog: \ return ZSTD_ROW_SEARCH_FN(dictMode, mls, rowLog)(ms, ip, iend, offsetPtr); #define ZSTD_SWITCH_MLS(X, dictMode) \ switch (mls) { \ ZSTD_FOR_EACH_MLS(X, dictMode) \ } #define ZSTD_SWITCH_ROWLOG(dictMode, mls) \ case mls: \ switch (rowLog) { \ ZSTD_FOR_EACH_ROWLOG(GEN_ZSTD_CALL_ROW_SEARCH_FN, dictMode, mls) \ } \ ZSTD_UNREACHABLE; \ break; #define ZSTD_SWITCH_SEARCH_METHOD(dictMode) \ switch (searchMethod) { \ case search_hashChain: \ ZSTD_SWITCH_MLS(GEN_ZSTD_CALL_HC_SEARCH_FN, dictMode) \ break; \ case search_binaryTree: \ ZSTD_SWITCH_MLS(GEN_ZSTD_CALL_BT_SEARCH_FN, dictMode) \ break; \ case search_rowHash: \ ZSTD_SWITCH_MLS(ZSTD_SWITCH_ROWLOG, dictMode) \ break; \ } \ ZSTD_UNREACHABLE; /** * Searches for the longest match at @p ip. * Dispatches to the correct implementation function based on the * (searchMethod, dictMode, mls, rowLog). We use switch statements * here instead of using an indirect function call through a function * pointer because after Spectre and Meltdown mitigations, indirect * function calls can be very costly, especially in the kernel. * * NOTE: dictMode and searchMethod should be templated, so those switch * statements should be optimized out. Only the mls & rowLog switches * should be left. * * @param ms The match state. * @param ip The position to search at. * @param iend The end of the input data. * @param[out] offsetPtr Stores the match offset into this pointer. * @param mls The minimum search length, in the range [4, 6]. * @param rowLog The row log (if applicable), in the range [4, 6]. * @param searchMethod The search method to use (templated). * @param dictMode The dictMode (templated). * * @returns The length of the longest match found, or < mls if no match is found. * If a match is found its offset is stored in @p offsetPtr. */ FORCE_INLINE_TEMPLATE size_t ZSTD_searchMax( ZSTD_matchState_t* ms, const BYTE* ip, const BYTE* iend, size_t* offsetPtr, U32 const mls, U32 const rowLog, searchMethod_e const searchMethod, ZSTD_dictMode_e const dictMode) { if (dictMode == ZSTD_noDict) { ZSTD_SWITCH_SEARCH_METHOD(noDict) } else if (dictMode == ZSTD_extDict) { ZSTD_SWITCH_SEARCH_METHOD(extDict) } else if (dictMode == ZSTD_dictMatchState) { ZSTD_SWITCH_SEARCH_METHOD(dictMatchState) } else if (dictMode == ZSTD_dedicatedDictSearch) { ZSTD_SWITCH_SEARCH_METHOD(dedicatedDictSearch) } ZSTD_UNREACHABLE; return 0; } /* ******************************* * Common parser - lazy strategy *********************************/ FORCE_INLINE_TEMPLATE size_t ZSTD_compressBlock_lazy_generic( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize, const searchMethod_e searchMethod, const U32 depth, ZSTD_dictMode_e const dictMode) { const BYTE* const istart = (const BYTE*)src; const BYTE* ip = istart; const BYTE* anchor = istart; const BYTE* const iend = istart + srcSize; const BYTE* const ilimit = (searchMethod == search_rowHash) ? iend - 8 - ZSTD_ROW_HASH_CACHE_SIZE : iend - 8; const BYTE* const base = ms->window.base; const U32 prefixLowestIndex = ms->window.dictLimit; const BYTE* const prefixLowest = base + prefixLowestIndex; const U32 mls = BOUNDED(4, ms->cParams.minMatch, 6); const U32 rowLog = BOUNDED(4, ms->cParams.searchLog, 6); U32 offset_1 = rep[0], offset_2 = rep[1]; U32 offsetSaved1 = 0, offsetSaved2 = 0; const int isDMS = dictMode == ZSTD_dictMatchState; const int isDDS = dictMode == ZSTD_dedicatedDictSearch; const int isDxS = isDMS || isDDS; const ZSTD_matchState_t* const dms = ms->dictMatchState; const U32 dictLowestIndex = isDxS ? dms->window.dictLimit : 0; const BYTE* const dictBase = isDxS ? dms->window.base : NULL; const BYTE* const dictLowest = isDxS ? dictBase + dictLowestIndex : NULL; const BYTE* const dictEnd = isDxS ? dms->window.nextSrc : NULL; const U32 dictIndexDelta = isDxS ? prefixLowestIndex - (U32)(dictEnd - dictBase) : 0; const U32 dictAndPrefixLength = (U32)((ip - prefixLowest) + (dictEnd - dictLowest)); DEBUGLOG(5, "ZSTD_compressBlock_lazy_generic (dictMode=%u) (searchFunc=%u)", (U32)dictMode, (U32)searchMethod); ip += (dictAndPrefixLength == 0); if (dictMode == ZSTD_noDict) { U32 const curr = (U32)(ip - base); U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, curr, ms->cParams.windowLog); U32 const maxRep = curr - windowLow; if (offset_2 > maxRep) offsetSaved2 = offset_2, offset_2 = 0; if (offset_1 > maxRep) offsetSaved1 = offset_1, offset_1 = 0; } if (isDxS) { /* dictMatchState repCode checks don't currently handle repCode == 0 * disabling. */ assert(offset_1 <= dictAndPrefixLength); assert(offset_2 <= dictAndPrefixLength); } /* Reset the lazy skipping state */ ms->lazySkipping = 0; if (searchMethod == search_rowHash) { ZSTD_row_fillHashCache(ms, base, rowLog, mls, ms->nextToUpdate, ilimit); } /* Match Loop */ #if defined(__GNUC__) && defined(__x86_64__) /* I've measured random a 5% speed loss on levels 5 & 6 (greedy) when the * code alignment is perturbed. To fix the instability align the loop on 32-bytes. */ __asm__(".p2align 5"); #endif while (ip < ilimit) { size_t matchLength=0; size_t offBase = REPCODE1_TO_OFFBASE; const BYTE* start=ip+1; DEBUGLOG(7, "search baseline (depth 0)"); /* check repCode */ if (isDxS) { const U32 repIndex = (U32)(ip - base) + 1 - offset_1; const BYTE* repMatch = ((dictMode == ZSTD_dictMatchState || dictMode == ZSTD_dedicatedDictSearch) && repIndex < prefixLowestIndex) ? dictBase + (repIndex - dictIndexDelta) : base + repIndex; if (((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */) && (MEM_read32(repMatch) == MEM_read32(ip+1)) ) { const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend; matchLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4; if (depth==0) goto _storeSequence; } } if ( dictMode == ZSTD_noDict && ((offset_1 > 0) & (MEM_read32(ip+1-offset_1) == MEM_read32(ip+1)))) { matchLength = ZSTD_count(ip+1+4, ip+1+4-offset_1, iend) + 4; if (depth==0) goto _storeSequence; } /* first search (depth 0) */ { size_t offbaseFound = 999999999; size_t const ml2 = ZSTD_searchMax(ms, ip, iend, &offbaseFound, mls, rowLog, searchMethod, dictMode); if (ml2 > matchLength) matchLength = ml2, start = ip, offBase = offbaseFound; } if (matchLength < 4) { size_t const step = ((size_t)(ip-anchor) >> kSearchStrength) + 1; /* jump faster over incompressible sections */; ip += step; /* Enter the lazy skipping mode once we are skipping more than 8 bytes at a time. * In this mode we stop inserting every position into our tables, and only insert * positions that we search, which is one in step positions. * The exact cutoff is flexible, I've just chosen a number that is reasonably high, * so we minimize the compression ratio loss in "normal" scenarios. This mode gets * triggered once we've gone 2KB without finding any matches. */ ms->lazySkipping = step > kLazySkippingStep; continue; } /* let's try to find a better solution */ if (depth>=1) while (ip0) & (MEM_read32(ip) == MEM_read32(ip - offset_1)))) { size_t const mlRep = ZSTD_count(ip+4, ip+4-offset_1, iend) + 4; int const gain2 = (int)(mlRep * 3); int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offBase) + 1); if ((mlRep >= 4) && (gain2 > gain1)) matchLength = mlRep, offBase = REPCODE1_TO_OFFBASE, start = ip; } if (isDxS) { const U32 repIndex = (U32)(ip - base) - offset_1; const BYTE* repMatch = repIndex < prefixLowestIndex ? dictBase + (repIndex - dictIndexDelta) : base + repIndex; if (((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */) && (MEM_read32(repMatch) == MEM_read32(ip)) ) { const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend; size_t const mlRep = ZSTD_count_2segments(ip+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4; int const gain2 = (int)(mlRep * 3); int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offBase) + 1); if ((mlRep >= 4) && (gain2 > gain1)) matchLength = mlRep, offBase = REPCODE1_TO_OFFBASE, start = ip; } } { size_t ofbCandidate=999999999; size_t const ml2 = ZSTD_searchMax(ms, ip, iend, &ofbCandidate, mls, rowLog, searchMethod, dictMode); int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)ofbCandidate)); /* raw approx */ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offBase) + 4); if ((ml2 >= 4) && (gain2 > gain1)) { matchLength = ml2, offBase = ofbCandidate, start = ip; continue; /* search a better one */ } } /* let's find an even better one */ if ((depth==2) && (ip0) & (MEM_read32(ip) == MEM_read32(ip - offset_1)))) { size_t const mlRep = ZSTD_count(ip+4, ip+4-offset_1, iend) + 4; int const gain2 = (int)(mlRep * 4); int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offBase) + 1); if ((mlRep >= 4) && (gain2 > gain1)) matchLength = mlRep, offBase = REPCODE1_TO_OFFBASE, start = ip; } if (isDxS) { const U32 repIndex = (U32)(ip - base) - offset_1; const BYTE* repMatch = repIndex < prefixLowestIndex ? dictBase + (repIndex - dictIndexDelta) : base + repIndex; if (((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */) && (MEM_read32(repMatch) == MEM_read32(ip)) ) { const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend; size_t const mlRep = ZSTD_count_2segments(ip+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4; int const gain2 = (int)(mlRep * 4); int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offBase) + 1); if ((mlRep >= 4) && (gain2 > gain1)) matchLength = mlRep, offBase = REPCODE1_TO_OFFBASE, start = ip; } } { size_t ofbCandidate=999999999; size_t const ml2 = ZSTD_searchMax(ms, ip, iend, &ofbCandidate, mls, rowLog, searchMethod, dictMode); int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)ofbCandidate)); /* raw approx */ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offBase) + 7); if ((ml2 >= 4) && (gain2 > gain1)) { matchLength = ml2, offBase = ofbCandidate, start = ip; continue; } } } break; /* nothing found : store previous solution */ } /* NOTE: * Pay attention that `start[-value]` can lead to strange undefined behavior * notably if `value` is unsigned, resulting in a large positive `-value`. */ /* catch up */ if (OFFBASE_IS_OFFSET(offBase)) { if (dictMode == ZSTD_noDict) { while ( ((start > anchor) & (start - OFFBASE_TO_OFFSET(offBase) > prefixLowest)) && (start[-1] == (start-OFFBASE_TO_OFFSET(offBase))[-1]) ) /* only search for offset within prefix */ { start--; matchLength++; } } if (isDxS) { U32 const matchIndex = (U32)((size_t)(start-base) - OFFBASE_TO_OFFSET(offBase)); const BYTE* match = (matchIndex < prefixLowestIndex) ? dictBase + matchIndex - dictIndexDelta : base + matchIndex; const BYTE* const mStart = (matchIndex < prefixLowestIndex) ? dictLowest : prefixLowest; while ((start>anchor) && (match>mStart) && (start[-1] == match[-1])) { start--; match--; matchLength++; } /* catch up */ } offset_2 = offset_1; offset_1 = (U32)OFFBASE_TO_OFFSET(offBase); } /* store sequence */ _storeSequence: { size_t const litLength = (size_t)(start - anchor); ZSTD_storeSeq(seqStore, litLength, anchor, iend, (U32)offBase, matchLength); anchor = ip = start + matchLength; } if (ms->lazySkipping) { /* We've found a match, disable lazy skipping mode, and refill the hash cache. */ if (searchMethod == search_rowHash) { ZSTD_row_fillHashCache(ms, base, rowLog, mls, ms->nextToUpdate, ilimit); } ms->lazySkipping = 0; } /* check immediate repcode */ if (isDxS) { while (ip <= ilimit) { U32 const current2 = (U32)(ip-base); U32 const repIndex = current2 - offset_2; const BYTE* repMatch = repIndex < prefixLowestIndex ? dictBase - dictIndexDelta + repIndex : base + repIndex; if ( ((U32)((prefixLowestIndex-1) - (U32)repIndex) >= 3 /* intentional overflow */) && (MEM_read32(repMatch) == MEM_read32(ip)) ) { const BYTE* const repEnd2 = repIndex < prefixLowestIndex ? dictEnd : iend; matchLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd2, prefixLowest) + 4; offBase = offset_2; offset_2 = offset_1; offset_1 = (U32)offBase; /* swap offset_2 <=> offset_1 */ ZSTD_storeSeq(seqStore, 0, anchor, iend, REPCODE1_TO_OFFBASE, matchLength); ip += matchLength; anchor = ip; continue; } break; } } if (dictMode == ZSTD_noDict) { while ( ((ip <= ilimit) & (offset_2>0)) && (MEM_read32(ip) == MEM_read32(ip - offset_2)) ) { /* store sequence */ matchLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4; offBase = offset_2; offset_2 = offset_1; offset_1 = (U32)offBase; /* swap repcodes */ ZSTD_storeSeq(seqStore, 0, anchor, iend, REPCODE1_TO_OFFBASE, matchLength); ip += matchLength; anchor = ip; continue; /* faster when present ... (?) */ } } } /* If offset_1 started invalid (offsetSaved1 != 0) and became valid (offset_1 != 0), * rotate saved offsets. See comment in ZSTD_compressBlock_fast_noDict for more context. */ offsetSaved2 = ((offsetSaved1 != 0) && (offset_1 != 0)) ? offsetSaved1 : offsetSaved2; /* save reps for next block */ rep[0] = offset_1 ? offset_1 : offsetSaved1; rep[1] = offset_2 ? offset_2 : offsetSaved2; /* Return the last literals size */ return (size_t)(iend - anchor); } size_t ZSTD_compressBlock_btlazy2( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_binaryTree, 2, ZSTD_noDict); } size_t ZSTD_compressBlock_lazy2( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 2, ZSTD_noDict); } size_t ZSTD_compressBlock_lazy( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 1, ZSTD_noDict); } size_t ZSTD_compressBlock_greedy( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 0, ZSTD_noDict); } size_t ZSTD_compressBlock_btlazy2_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_binaryTree, 2, ZSTD_dictMatchState); } size_t ZSTD_compressBlock_lazy2_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 2, ZSTD_dictMatchState); } size_t ZSTD_compressBlock_lazy_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 1, ZSTD_dictMatchState); } size_t ZSTD_compressBlock_greedy_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 0, ZSTD_dictMatchState); } size_t ZSTD_compressBlock_lazy2_dedicatedDictSearch( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 2, ZSTD_dedicatedDictSearch); } size_t ZSTD_compressBlock_lazy_dedicatedDictSearch( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 1, ZSTD_dedicatedDictSearch); } size_t ZSTD_compressBlock_greedy_dedicatedDictSearch( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 0, ZSTD_dedicatedDictSearch); } /* Row-based matchfinder */ size_t ZSTD_compressBlock_lazy2_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 2, ZSTD_noDict); } size_t ZSTD_compressBlock_lazy_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 1, ZSTD_noDict); } size_t ZSTD_compressBlock_greedy_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 0, ZSTD_noDict); } size_t ZSTD_compressBlock_lazy2_dictMatchState_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 2, ZSTD_dictMatchState); } size_t ZSTD_compressBlock_lazy_dictMatchState_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 1, ZSTD_dictMatchState); } size_t ZSTD_compressBlock_greedy_dictMatchState_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 0, ZSTD_dictMatchState); } size_t ZSTD_compressBlock_lazy2_dedicatedDictSearch_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 2, ZSTD_dedicatedDictSearch); } size_t ZSTD_compressBlock_lazy_dedicatedDictSearch_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 1, ZSTD_dedicatedDictSearch); } size_t ZSTD_compressBlock_greedy_dedicatedDictSearch_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 0, ZSTD_dedicatedDictSearch); } FORCE_INLINE_TEMPLATE size_t ZSTD_compressBlock_lazy_extDict_generic( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize, const searchMethod_e searchMethod, const U32 depth) { const BYTE* const istart = (const BYTE*)src; const BYTE* ip = istart; const BYTE* anchor = istart; const BYTE* const iend = istart + srcSize; const BYTE* const ilimit = searchMethod == search_rowHash ? iend - 8 - ZSTD_ROW_HASH_CACHE_SIZE : iend - 8; const BYTE* const base = ms->window.base; const U32 dictLimit = ms->window.dictLimit; const BYTE* const prefixStart = base + dictLimit; const BYTE* const dictBase = ms->window.dictBase; const BYTE* const dictEnd = dictBase + dictLimit; const BYTE* const dictStart = dictBase + ms->window.lowLimit; const U32 windowLog = ms->cParams.windowLog; const U32 mls = BOUNDED(4, ms->cParams.minMatch, 6); const U32 rowLog = BOUNDED(4, ms->cParams.searchLog, 6); U32 offset_1 = rep[0], offset_2 = rep[1]; DEBUGLOG(5, "ZSTD_compressBlock_lazy_extDict_generic (searchFunc=%u)", (U32)searchMethod); /* Reset the lazy skipping state */ ms->lazySkipping = 0; /* init */ ip += (ip == prefixStart); if (searchMethod == search_rowHash) { ZSTD_row_fillHashCache(ms, base, rowLog, mls, ms->nextToUpdate, ilimit); } /* Match Loop */ #if defined(__GNUC__) && defined(__x86_64__) /* I've measured random a 5% speed loss on levels 5 & 6 (greedy) when the * code alignment is perturbed. To fix the instability align the loop on 32-bytes. */ __asm__(".p2align 5"); #endif while (ip < ilimit) { size_t matchLength=0; size_t offBase = REPCODE1_TO_OFFBASE; const BYTE* start=ip+1; U32 curr = (U32)(ip-base); /* check repCode */ { const U32 windowLow = ZSTD_getLowestMatchIndex(ms, curr+1, windowLog); const U32 repIndex = (U32)(curr+1 - offset_1); const BYTE* const repBase = repIndex < dictLimit ? dictBase : base; const BYTE* const repMatch = repBase + repIndex; if ( ((U32)((dictLimit-1) - repIndex) >= 3) /* intentional overflow */ & (offset_1 <= curr+1 - windowLow) ) /* note: we are searching at curr+1 */ if (MEM_read32(ip+1) == MEM_read32(repMatch)) { /* repcode detected we should take it */ const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend; matchLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repEnd, prefixStart) + 4; if (depth==0) goto _storeSequence; } } /* first search (depth 0) */ { size_t ofbCandidate = 999999999; size_t const ml2 = ZSTD_searchMax(ms, ip, iend, &ofbCandidate, mls, rowLog, searchMethod, ZSTD_extDict); if (ml2 > matchLength) matchLength = ml2, start = ip, offBase = ofbCandidate; } if (matchLength < 4) { size_t const step = ((size_t)(ip-anchor) >> kSearchStrength); ip += step + 1; /* jump faster over incompressible sections */ /* Enter the lazy skipping mode once we are skipping more than 8 bytes at a time. * In this mode we stop inserting every position into our tables, and only insert * positions that we search, which is one in step positions. * The exact cutoff is flexible, I've just chosen a number that is reasonably high, * so we minimize the compression ratio loss in "normal" scenarios. This mode gets * triggered once we've gone 2KB without finding any matches. */ ms->lazySkipping = step > kLazySkippingStep; continue; } /* let's try to find a better solution */ if (depth>=1) while (ip= 3) /* intentional overflow : do not test positions overlapping 2 memory segments */ & (offset_1 <= curr - windowLow) ) /* equivalent to `curr > repIndex >= windowLow` */ if (MEM_read32(ip) == MEM_read32(repMatch)) { /* repcode detected */ const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend; size_t const repLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4; int const gain2 = (int)(repLength * 3); int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offBase) + 1); if ((repLength >= 4) && (gain2 > gain1)) matchLength = repLength, offBase = REPCODE1_TO_OFFBASE, start = ip; } } /* search match, depth 1 */ { size_t ofbCandidate = 999999999; size_t const ml2 = ZSTD_searchMax(ms, ip, iend, &ofbCandidate, mls, rowLog, searchMethod, ZSTD_extDict); int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)ofbCandidate)); /* raw approx */ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offBase) + 4); if ((ml2 >= 4) && (gain2 > gain1)) { matchLength = ml2, offBase = ofbCandidate, start = ip; continue; /* search a better one */ } } /* let's find an even better one */ if ((depth==2) && (ip= 3) /* intentional overflow : do not test positions overlapping 2 memory segments */ & (offset_1 <= curr - windowLow) ) /* equivalent to `curr > repIndex >= windowLow` */ if (MEM_read32(ip) == MEM_read32(repMatch)) { /* repcode detected */ const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend; size_t const repLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4; int const gain2 = (int)(repLength * 4); int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offBase) + 1); if ((repLength >= 4) && (gain2 > gain1)) matchLength = repLength, offBase = REPCODE1_TO_OFFBASE, start = ip; } } /* search match, depth 2 */ { size_t ofbCandidate = 999999999; size_t const ml2 = ZSTD_searchMax(ms, ip, iend, &ofbCandidate, mls, rowLog, searchMethod, ZSTD_extDict); int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)ofbCandidate)); /* raw approx */ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offBase) + 7); if ((ml2 >= 4) && (gain2 > gain1)) { matchLength = ml2, offBase = ofbCandidate, start = ip; continue; } } } break; /* nothing found : store previous solution */ } /* catch up */ if (OFFBASE_IS_OFFSET(offBase)) { U32 const matchIndex = (U32)((size_t)(start-base) - OFFBASE_TO_OFFSET(offBase)); const BYTE* match = (matchIndex < dictLimit) ? dictBase + matchIndex : base + matchIndex; const BYTE* const mStart = (matchIndex < dictLimit) ? dictStart : prefixStart; while ((start>anchor) && (match>mStart) && (start[-1] == match[-1])) { start--; match--; matchLength++; } /* catch up */ offset_2 = offset_1; offset_1 = (U32)OFFBASE_TO_OFFSET(offBase); } /* store sequence */ _storeSequence: { size_t const litLength = (size_t)(start - anchor); ZSTD_storeSeq(seqStore, litLength, anchor, iend, (U32)offBase, matchLength); anchor = ip = start + matchLength; } if (ms->lazySkipping) { /* We've found a match, disable lazy skipping mode, and refill the hash cache. */ if (searchMethod == search_rowHash) { ZSTD_row_fillHashCache(ms, base, rowLog, mls, ms->nextToUpdate, ilimit); } ms->lazySkipping = 0; } /* check immediate repcode */ while (ip <= ilimit) { const U32 repCurrent = (U32)(ip-base); const U32 windowLow = ZSTD_getLowestMatchIndex(ms, repCurrent, windowLog); const U32 repIndex = repCurrent - offset_2; const BYTE* const repBase = repIndex < dictLimit ? dictBase : base; const BYTE* const repMatch = repBase + repIndex; if ( ((U32)((dictLimit-1) - repIndex) >= 3) /* intentional overflow : do not test positions overlapping 2 memory segments */ & (offset_2 <= repCurrent - windowLow) ) /* equivalent to `curr > repIndex >= windowLow` */ if (MEM_read32(ip) == MEM_read32(repMatch)) { /* repcode detected we should take it */ const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend; matchLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4; offBase = offset_2; offset_2 = offset_1; offset_1 = (U32)offBase; /* swap offset history */ ZSTD_storeSeq(seqStore, 0, anchor, iend, REPCODE1_TO_OFFBASE, matchLength); ip += matchLength; anchor = ip; continue; /* faster when present ... (?) */ } break; } } /* Save reps for next block */ rep[0] = offset_1; rep[1] = offset_2; /* Return the last literals size */ return (size_t)(iend - anchor); } size_t ZSTD_compressBlock_greedy_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 0); } size_t ZSTD_compressBlock_lazy_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 1); } size_t ZSTD_compressBlock_lazy2_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 2); } size_t ZSTD_compressBlock_btlazy2_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_binaryTree, 2); } size_t ZSTD_compressBlock_greedy_extDict_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 0); } size_t ZSTD_compressBlock_lazy_extDict_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 1); } size_t ZSTD_compressBlock_lazy2_extDict_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 2); } zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_double_fast.c0000644000175200007730000010325714515254731026662 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #include "zstd_compress_internal.h" #include "zstd_double_fast.h" static void ZSTD_fillDoubleHashTableForCDict(ZSTD_matchState_t* ms, void const* end, ZSTD_dictTableLoadMethod_e dtlm) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32* const hashLarge = ms->hashTable; U32 const hBitsL = cParams->hashLog + ZSTD_SHORT_CACHE_TAG_BITS; U32 const mls = cParams->minMatch; U32* const hashSmall = ms->chainTable; U32 const hBitsS = cParams->chainLog + ZSTD_SHORT_CACHE_TAG_BITS; const BYTE* const base = ms->window.base; const BYTE* ip = base + ms->nextToUpdate; const BYTE* const iend = ((const BYTE*)end) - HASH_READ_SIZE; const U32 fastHashFillStep = 3; /* Always insert every fastHashFillStep position into the hash tables. * Insert the other positions into the large hash table if their entry * is empty. */ for (; ip + fastHashFillStep - 1 <= iend; ip += fastHashFillStep) { U32 const curr = (U32)(ip - base); U32 i; for (i = 0; i < fastHashFillStep; ++i) { size_t const smHashAndTag = ZSTD_hashPtr(ip + i, hBitsS, mls); size_t const lgHashAndTag = ZSTD_hashPtr(ip + i, hBitsL, 8); if (i == 0) { ZSTD_writeTaggedIndex(hashSmall, smHashAndTag, curr + i); } if (i == 0 || hashLarge[lgHashAndTag >> ZSTD_SHORT_CACHE_TAG_BITS] == 0) { ZSTD_writeTaggedIndex(hashLarge, lgHashAndTag, curr + i); } /* Only load extra positions for ZSTD_dtlm_full */ if (dtlm == ZSTD_dtlm_fast) break; } } } static void ZSTD_fillDoubleHashTableForCCtx(ZSTD_matchState_t* ms, void const* end, ZSTD_dictTableLoadMethod_e dtlm) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32* const hashLarge = ms->hashTable; U32 const hBitsL = cParams->hashLog; U32 const mls = cParams->minMatch; U32* const hashSmall = ms->chainTable; U32 const hBitsS = cParams->chainLog; const BYTE* const base = ms->window.base; const BYTE* ip = base + ms->nextToUpdate; const BYTE* const iend = ((const BYTE*)end) - HASH_READ_SIZE; const U32 fastHashFillStep = 3; /* Always insert every fastHashFillStep position into the hash tables. * Insert the other positions into the large hash table if their entry * is empty. */ for (; ip + fastHashFillStep - 1 <= iend; ip += fastHashFillStep) { U32 const curr = (U32)(ip - base); U32 i; for (i = 0; i < fastHashFillStep; ++i) { size_t const smHash = ZSTD_hashPtr(ip + i, hBitsS, mls); size_t const lgHash = ZSTD_hashPtr(ip + i, hBitsL, 8); if (i == 0) hashSmall[smHash] = curr + i; if (i == 0 || hashLarge[lgHash] == 0) hashLarge[lgHash] = curr + i; /* Only load extra positions for ZSTD_dtlm_full */ if (dtlm == ZSTD_dtlm_fast) break; } } } void ZSTD_fillDoubleHashTable(ZSTD_matchState_t* ms, const void* const end, ZSTD_dictTableLoadMethod_e dtlm, ZSTD_tableFillPurpose_e tfp) { if (tfp == ZSTD_tfp_forCDict) { ZSTD_fillDoubleHashTableForCDict(ms, end, dtlm); } else { ZSTD_fillDoubleHashTableForCCtx(ms, end, dtlm); } } FORCE_INLINE_TEMPLATE size_t ZSTD_compressBlock_doubleFast_noDict_generic( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize, U32 const mls /* template */) { ZSTD_compressionParameters const* cParams = &ms->cParams; U32* const hashLong = ms->hashTable; const U32 hBitsL = cParams->hashLog; U32* const hashSmall = ms->chainTable; const U32 hBitsS = cParams->chainLog; const BYTE* const base = ms->window.base; const BYTE* const istart = (const BYTE*)src; const BYTE* anchor = istart; const U32 endIndex = (U32)((size_t)(istart - base) + srcSize); /* presumes that, if there is a dictionary, it must be using Attach mode */ const U32 prefixLowestIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog); const BYTE* const prefixLowest = base + prefixLowestIndex; const BYTE* const iend = istart + srcSize; const BYTE* const ilimit = iend - HASH_READ_SIZE; U32 offset_1=rep[0], offset_2=rep[1]; U32 offsetSaved1 = 0, offsetSaved2 = 0; size_t mLength; U32 offset; U32 curr; /* how many positions to search before increasing step size */ const size_t kStepIncr = 1 << kSearchStrength; /* the position at which to increment the step size if no match is found */ const BYTE* nextStep; size_t step; /* the current step size */ size_t hl0; /* the long hash at ip */ size_t hl1; /* the long hash at ip1 */ U32 idxl0; /* the long match index for ip */ U32 idxl1; /* the long match index for ip1 */ const BYTE* matchl0; /* the long match for ip */ const BYTE* matchs0; /* the short match for ip */ const BYTE* matchl1; /* the long match for ip1 */ const BYTE* ip = istart; /* the current position */ const BYTE* ip1; /* the next position */ DEBUGLOG(5, "ZSTD_compressBlock_doubleFast_noDict_generic"); /* init */ ip += ((ip - prefixLowest) == 0); { U32 const current = (U32)(ip - base); U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, current, cParams->windowLog); U32 const maxRep = current - windowLow; if (offset_2 > maxRep) offsetSaved2 = offset_2, offset_2 = 0; if (offset_1 > maxRep) offsetSaved1 = offset_1, offset_1 = 0; } /* Outer Loop: one iteration per match found and stored */ while (1) { step = 1; nextStep = ip + kStepIncr; ip1 = ip + step; if (ip1 > ilimit) { goto _cleanup; } hl0 = ZSTD_hashPtr(ip, hBitsL, 8); idxl0 = hashLong[hl0]; matchl0 = base + idxl0; /* Inner Loop: one iteration per search / position */ do { const size_t hs0 = ZSTD_hashPtr(ip, hBitsS, mls); const U32 idxs0 = hashSmall[hs0]; curr = (U32)(ip-base); matchs0 = base + idxs0; hashLong[hl0] = hashSmall[hs0] = curr; /* update hash tables */ /* check noDict repcode */ if ((offset_1 > 0) & (MEM_read32(ip+1-offset_1) == MEM_read32(ip+1))) { mLength = ZSTD_count(ip+1+4, ip+1+4-offset_1, iend) + 4; ip++; ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, REPCODE1_TO_OFFBASE, mLength); goto _match_stored; } hl1 = ZSTD_hashPtr(ip1, hBitsL, 8); if (idxl0 > prefixLowestIndex) { /* check prefix long match */ if (MEM_read64(matchl0) == MEM_read64(ip)) { mLength = ZSTD_count(ip+8, matchl0+8, iend) + 8; offset = (U32)(ip-matchl0); while (((ip>anchor) & (matchl0>prefixLowest)) && (ip[-1] == matchl0[-1])) { ip--; matchl0--; mLength++; } /* catch up */ goto _match_found; } } idxl1 = hashLong[hl1]; matchl1 = base + idxl1; if (idxs0 > prefixLowestIndex) { /* check prefix short match */ if (MEM_read32(matchs0) == MEM_read32(ip)) { goto _search_next_long; } } if (ip1 >= nextStep) { PREFETCH_L1(ip1 + 64); PREFETCH_L1(ip1 + 128); step++; nextStep += kStepIncr; } ip = ip1; ip1 += step; hl0 = hl1; idxl0 = idxl1; matchl0 = matchl1; #if defined(__aarch64__) PREFETCH_L1(ip+256); #endif } while (ip1 <= ilimit); _cleanup: /* If offset_1 started invalid (offsetSaved1 != 0) and became valid (offset_1 != 0), * rotate saved offsets. See comment in ZSTD_compressBlock_fast_noDict for more context. */ offsetSaved2 = ((offsetSaved1 != 0) && (offset_1 != 0)) ? offsetSaved1 : offsetSaved2; /* save reps for next block */ rep[0] = offset_1 ? offset_1 : offsetSaved1; rep[1] = offset_2 ? offset_2 : offsetSaved2; /* Return the last literals size */ return (size_t)(iend - anchor); _search_next_long: /* check prefix long +1 match */ if (idxl1 > prefixLowestIndex) { if (MEM_read64(matchl1) == MEM_read64(ip1)) { ip = ip1; mLength = ZSTD_count(ip+8, matchl1+8, iend) + 8; offset = (U32)(ip-matchl1); while (((ip>anchor) & (matchl1>prefixLowest)) && (ip[-1] == matchl1[-1])) { ip--; matchl1--; mLength++; } /* catch up */ goto _match_found; } } /* if no long +1 match, explore the short match we found */ mLength = ZSTD_count(ip+4, matchs0+4, iend) + 4; offset = (U32)(ip - matchs0); while (((ip>anchor) & (matchs0>prefixLowest)) && (ip[-1] == matchs0[-1])) { ip--; matchs0--; mLength++; } /* catch up */ /* fall-through */ _match_found: /* requires ip, offset, mLength */ offset_2 = offset_1; offset_1 = offset; if (step < 4) { /* It is unsafe to write this value back to the hashtable when ip1 is * greater than or equal to the new ip we will have after we're done * processing this match. Rather than perform that test directly * (ip1 >= ip + mLength), which costs speed in practice, we do a simpler * more predictable test. The minmatch even if we take a short match is * 4 bytes, so as long as step, the distance between ip and ip1 * (initially) is less than 4, we know ip1 < new ip. */ hashLong[hl1] = (U32)(ip1 - base); } ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, OFFSET_TO_OFFBASE(offset), mLength); _match_stored: /* match found */ ip += mLength; anchor = ip; if (ip <= ilimit) { /* Complementary insertion */ /* done after iLimit test, as candidates could be > iend-8 */ { U32 const indexToInsert = curr+2; hashLong[ZSTD_hashPtr(base+indexToInsert, hBitsL, 8)] = indexToInsert; hashLong[ZSTD_hashPtr(ip-2, hBitsL, 8)] = (U32)(ip-2-base); hashSmall[ZSTD_hashPtr(base+indexToInsert, hBitsS, mls)] = indexToInsert; hashSmall[ZSTD_hashPtr(ip-1, hBitsS, mls)] = (U32)(ip-1-base); } /* check immediate repcode */ while ( (ip <= ilimit) && ( (offset_2>0) & (MEM_read32(ip) == MEM_read32(ip - offset_2)) )) { /* store sequence */ size_t const rLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4; U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; /* swap offset_2 <=> offset_1 */ hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = (U32)(ip-base); hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = (U32)(ip-base); ZSTD_storeSeq(seqStore, 0, anchor, iend, REPCODE1_TO_OFFBASE, rLength); ip += rLength; anchor = ip; continue; /* faster when present ... (?) */ } } } } FORCE_INLINE_TEMPLATE size_t ZSTD_compressBlock_doubleFast_dictMatchState_generic( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize, U32 const mls /* template */) { ZSTD_compressionParameters const* cParams = &ms->cParams; U32* const hashLong = ms->hashTable; const U32 hBitsL = cParams->hashLog; U32* const hashSmall = ms->chainTable; const U32 hBitsS = cParams->chainLog; const BYTE* const base = ms->window.base; const BYTE* const istart = (const BYTE*)src; const BYTE* ip = istart; const BYTE* anchor = istart; const U32 endIndex = (U32)((size_t)(istart - base) + srcSize); /* presumes that, if there is a dictionary, it must be using Attach mode */ const U32 prefixLowestIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog); const BYTE* const prefixLowest = base + prefixLowestIndex; const BYTE* const iend = istart + srcSize; const BYTE* const ilimit = iend - HASH_READ_SIZE; U32 offset_1=rep[0], offset_2=rep[1]; const ZSTD_matchState_t* const dms = ms->dictMatchState; const ZSTD_compressionParameters* const dictCParams = &dms->cParams; const U32* const dictHashLong = dms->hashTable; const U32* const dictHashSmall = dms->chainTable; const U32 dictStartIndex = dms->window.dictLimit; const BYTE* const dictBase = dms->window.base; const BYTE* const dictStart = dictBase + dictStartIndex; const BYTE* const dictEnd = dms->window.nextSrc; const U32 dictIndexDelta = prefixLowestIndex - (U32)(dictEnd - dictBase); const U32 dictHBitsL = dictCParams->hashLog + ZSTD_SHORT_CACHE_TAG_BITS; const U32 dictHBitsS = dictCParams->chainLog + ZSTD_SHORT_CACHE_TAG_BITS; const U32 dictAndPrefixLength = (U32)((ip - prefixLowest) + (dictEnd - dictStart)); DEBUGLOG(5, "ZSTD_compressBlock_doubleFast_dictMatchState_generic"); /* if a dictionary is attached, it must be within window range */ assert(ms->window.dictLimit + (1U << cParams->windowLog) >= endIndex); if (ms->prefetchCDictTables) { size_t const hashTableBytes = (((size_t)1) << dictCParams->hashLog) * sizeof(U32); size_t const chainTableBytes = (((size_t)1) << dictCParams->chainLog) * sizeof(U32); PREFETCH_AREA(dictHashLong, hashTableBytes) PREFETCH_AREA(dictHashSmall, chainTableBytes) } /* init */ ip += (dictAndPrefixLength == 0); /* dictMatchState repCode checks don't currently handle repCode == 0 * disabling. */ assert(offset_1 <= dictAndPrefixLength); assert(offset_2 <= dictAndPrefixLength); /* Main Search Loop */ while (ip < ilimit) { /* < instead of <=, because repcode check at (ip+1) */ size_t mLength; U32 offset; size_t const h2 = ZSTD_hashPtr(ip, hBitsL, 8); size_t const h = ZSTD_hashPtr(ip, hBitsS, mls); size_t const dictHashAndTagL = ZSTD_hashPtr(ip, dictHBitsL, 8); size_t const dictHashAndTagS = ZSTD_hashPtr(ip, dictHBitsS, mls); U32 const dictMatchIndexAndTagL = dictHashLong[dictHashAndTagL >> ZSTD_SHORT_CACHE_TAG_BITS]; U32 const dictMatchIndexAndTagS = dictHashSmall[dictHashAndTagS >> ZSTD_SHORT_CACHE_TAG_BITS]; int const dictTagsMatchL = ZSTD_comparePackedTags(dictMatchIndexAndTagL, dictHashAndTagL); int const dictTagsMatchS = ZSTD_comparePackedTags(dictMatchIndexAndTagS, dictHashAndTagS); U32 const curr = (U32)(ip-base); U32 const matchIndexL = hashLong[h2]; U32 matchIndexS = hashSmall[h]; const BYTE* matchLong = base + matchIndexL; const BYTE* match = base + matchIndexS; const U32 repIndex = curr + 1 - offset_1; const BYTE* repMatch = (repIndex < prefixLowestIndex) ? dictBase + (repIndex - dictIndexDelta) : base + repIndex; hashLong[h2] = hashSmall[h] = curr; /* update hash tables */ /* check repcode */ if (((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */) && (MEM_read32(repMatch) == MEM_read32(ip+1)) ) { const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend; mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4; ip++; ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, REPCODE1_TO_OFFBASE, mLength); goto _match_stored; } if (matchIndexL > prefixLowestIndex) { /* check prefix long match */ if (MEM_read64(matchLong) == MEM_read64(ip)) { mLength = ZSTD_count(ip+8, matchLong+8, iend) + 8; offset = (U32)(ip-matchLong); while (((ip>anchor) & (matchLong>prefixLowest)) && (ip[-1] == matchLong[-1])) { ip--; matchLong--; mLength++; } /* catch up */ goto _match_found; } } else if (dictTagsMatchL) { /* check dictMatchState long match */ U32 const dictMatchIndexL = dictMatchIndexAndTagL >> ZSTD_SHORT_CACHE_TAG_BITS; const BYTE* dictMatchL = dictBase + dictMatchIndexL; assert(dictMatchL < dictEnd); if (dictMatchL > dictStart && MEM_read64(dictMatchL) == MEM_read64(ip)) { mLength = ZSTD_count_2segments(ip+8, dictMatchL+8, iend, dictEnd, prefixLowest) + 8; offset = (U32)(curr - dictMatchIndexL - dictIndexDelta); while (((ip>anchor) & (dictMatchL>dictStart)) && (ip[-1] == dictMatchL[-1])) { ip--; dictMatchL--; mLength++; } /* catch up */ goto _match_found; } } if (matchIndexS > prefixLowestIndex) { /* check prefix short match */ if (MEM_read32(match) == MEM_read32(ip)) { goto _search_next_long; } } else if (dictTagsMatchS) { /* check dictMatchState short match */ U32 const dictMatchIndexS = dictMatchIndexAndTagS >> ZSTD_SHORT_CACHE_TAG_BITS; match = dictBase + dictMatchIndexS; matchIndexS = dictMatchIndexS + dictIndexDelta; if (match > dictStart && MEM_read32(match) == MEM_read32(ip)) { goto _search_next_long; } } ip += ((ip-anchor) >> kSearchStrength) + 1; #if defined(__aarch64__) PREFETCH_L1(ip+256); #endif continue; _search_next_long: { size_t const hl3 = ZSTD_hashPtr(ip+1, hBitsL, 8); size_t const dictHashAndTagL3 = ZSTD_hashPtr(ip+1, dictHBitsL, 8); U32 const matchIndexL3 = hashLong[hl3]; U32 const dictMatchIndexAndTagL3 = dictHashLong[dictHashAndTagL3 >> ZSTD_SHORT_CACHE_TAG_BITS]; int const dictTagsMatchL3 = ZSTD_comparePackedTags(dictMatchIndexAndTagL3, dictHashAndTagL3); const BYTE* matchL3 = base + matchIndexL3; hashLong[hl3] = curr + 1; /* check prefix long +1 match */ if (matchIndexL3 > prefixLowestIndex) { if (MEM_read64(matchL3) == MEM_read64(ip+1)) { mLength = ZSTD_count(ip+9, matchL3+8, iend) + 8; ip++; offset = (U32)(ip-matchL3); while (((ip>anchor) & (matchL3>prefixLowest)) && (ip[-1] == matchL3[-1])) { ip--; matchL3--; mLength++; } /* catch up */ goto _match_found; } } else if (dictTagsMatchL3) { /* check dict long +1 match */ U32 const dictMatchIndexL3 = dictMatchIndexAndTagL3 >> ZSTD_SHORT_CACHE_TAG_BITS; const BYTE* dictMatchL3 = dictBase + dictMatchIndexL3; assert(dictMatchL3 < dictEnd); if (dictMatchL3 > dictStart && MEM_read64(dictMatchL3) == MEM_read64(ip+1)) { mLength = ZSTD_count_2segments(ip+1+8, dictMatchL3+8, iend, dictEnd, prefixLowest) + 8; ip++; offset = (U32)(curr + 1 - dictMatchIndexL3 - dictIndexDelta); while (((ip>anchor) & (dictMatchL3>dictStart)) && (ip[-1] == dictMatchL3[-1])) { ip--; dictMatchL3--; mLength++; } /* catch up */ goto _match_found; } } } /* if no long +1 match, explore the short match we found */ if (matchIndexS < prefixLowestIndex) { mLength = ZSTD_count_2segments(ip+4, match+4, iend, dictEnd, prefixLowest) + 4; offset = (U32)(curr - matchIndexS); while (((ip>anchor) & (match>dictStart)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */ } else { mLength = ZSTD_count(ip+4, match+4, iend) + 4; offset = (U32)(ip - match); while (((ip>anchor) & (match>prefixLowest)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */ } _match_found: offset_2 = offset_1; offset_1 = offset; ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, OFFSET_TO_OFFBASE(offset), mLength); _match_stored: /* match found */ ip += mLength; anchor = ip; if (ip <= ilimit) { /* Complementary insertion */ /* done after iLimit test, as candidates could be > iend-8 */ { U32 const indexToInsert = curr+2; hashLong[ZSTD_hashPtr(base+indexToInsert, hBitsL, 8)] = indexToInsert; hashLong[ZSTD_hashPtr(ip-2, hBitsL, 8)] = (U32)(ip-2-base); hashSmall[ZSTD_hashPtr(base+indexToInsert, hBitsS, mls)] = indexToInsert; hashSmall[ZSTD_hashPtr(ip-1, hBitsS, mls)] = (U32)(ip-1-base); } /* check immediate repcode */ while (ip <= ilimit) { U32 const current2 = (U32)(ip-base); U32 const repIndex2 = current2 - offset_2; const BYTE* repMatch2 = repIndex2 < prefixLowestIndex ? dictBase + repIndex2 - dictIndexDelta : base + repIndex2; if ( ((U32)((prefixLowestIndex-1) - (U32)repIndex2) >= 3 /* intentional overflow */) && (MEM_read32(repMatch2) == MEM_read32(ip)) ) { const BYTE* const repEnd2 = repIndex2 < prefixLowestIndex ? dictEnd : iend; size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixLowest) + 4; U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */ ZSTD_storeSeq(seqStore, 0, anchor, iend, REPCODE1_TO_OFFBASE, repLength2); hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2; hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2; ip += repLength2; anchor = ip; continue; } break; } } } /* while (ip < ilimit) */ /* save reps for next block */ rep[0] = offset_1; rep[1] = offset_2; /* Return the last literals size */ return (size_t)(iend - anchor); } #define ZSTD_GEN_DFAST_FN(dictMode, mls) \ static size_t ZSTD_compressBlock_doubleFast_##dictMode##_##mls( \ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], \ void const* src, size_t srcSize) \ { \ return ZSTD_compressBlock_doubleFast_##dictMode##_generic(ms, seqStore, rep, src, srcSize, mls); \ } ZSTD_GEN_DFAST_FN(noDict, 4) ZSTD_GEN_DFAST_FN(noDict, 5) ZSTD_GEN_DFAST_FN(noDict, 6) ZSTD_GEN_DFAST_FN(noDict, 7) ZSTD_GEN_DFAST_FN(dictMatchState, 4) ZSTD_GEN_DFAST_FN(dictMatchState, 5) ZSTD_GEN_DFAST_FN(dictMatchState, 6) ZSTD_GEN_DFAST_FN(dictMatchState, 7) size_t ZSTD_compressBlock_doubleFast( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { const U32 mls = ms->cParams.minMatch; switch(mls) { default: /* includes case 3 */ case 4 : return ZSTD_compressBlock_doubleFast_noDict_4(ms, seqStore, rep, src, srcSize); case 5 : return ZSTD_compressBlock_doubleFast_noDict_5(ms, seqStore, rep, src, srcSize); case 6 : return ZSTD_compressBlock_doubleFast_noDict_6(ms, seqStore, rep, src, srcSize); case 7 : return ZSTD_compressBlock_doubleFast_noDict_7(ms, seqStore, rep, src, srcSize); } } size_t ZSTD_compressBlock_doubleFast_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { const U32 mls = ms->cParams.minMatch; switch(mls) { default: /* includes case 3 */ case 4 : return ZSTD_compressBlock_doubleFast_dictMatchState_4(ms, seqStore, rep, src, srcSize); case 5 : return ZSTD_compressBlock_doubleFast_dictMatchState_5(ms, seqStore, rep, src, srcSize); case 6 : return ZSTD_compressBlock_doubleFast_dictMatchState_6(ms, seqStore, rep, src, srcSize); case 7 : return ZSTD_compressBlock_doubleFast_dictMatchState_7(ms, seqStore, rep, src, srcSize); } } static size_t ZSTD_compressBlock_doubleFast_extDict_generic( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize, U32 const mls /* template */) { ZSTD_compressionParameters const* cParams = &ms->cParams; U32* const hashLong = ms->hashTable; U32 const hBitsL = cParams->hashLog; U32* const hashSmall = ms->chainTable; U32 const hBitsS = cParams->chainLog; const BYTE* const istart = (const BYTE*)src; const BYTE* ip = istart; const BYTE* anchor = istart; const BYTE* const iend = istart + srcSize; const BYTE* const ilimit = iend - 8; const BYTE* const base = ms->window.base; const U32 endIndex = (U32)((size_t)(istart - base) + srcSize); const U32 lowLimit = ZSTD_getLowestMatchIndex(ms, endIndex, cParams->windowLog); const U32 dictStartIndex = lowLimit; const U32 dictLimit = ms->window.dictLimit; const U32 prefixStartIndex = (dictLimit > lowLimit) ? dictLimit : lowLimit; const BYTE* const prefixStart = base + prefixStartIndex; const BYTE* const dictBase = ms->window.dictBase; const BYTE* const dictStart = dictBase + dictStartIndex; const BYTE* const dictEnd = dictBase + prefixStartIndex; U32 offset_1=rep[0], offset_2=rep[1]; DEBUGLOG(5, "ZSTD_compressBlock_doubleFast_extDict_generic (srcSize=%zu)", srcSize); /* if extDict is invalidated due to maxDistance, switch to "regular" variant */ if (prefixStartIndex == dictStartIndex) return ZSTD_compressBlock_doubleFast(ms, seqStore, rep, src, srcSize); /* Search Loop */ while (ip < ilimit) { /* < instead of <=, because (ip+1) */ const size_t hSmall = ZSTD_hashPtr(ip, hBitsS, mls); const U32 matchIndex = hashSmall[hSmall]; const BYTE* const matchBase = matchIndex < prefixStartIndex ? dictBase : base; const BYTE* match = matchBase + matchIndex; const size_t hLong = ZSTD_hashPtr(ip, hBitsL, 8); const U32 matchLongIndex = hashLong[hLong]; const BYTE* const matchLongBase = matchLongIndex < prefixStartIndex ? dictBase : base; const BYTE* matchLong = matchLongBase + matchLongIndex; const U32 curr = (U32)(ip-base); const U32 repIndex = curr + 1 - offset_1; /* offset_1 expected <= curr +1 */ const BYTE* const repBase = repIndex < prefixStartIndex ? dictBase : base; const BYTE* const repMatch = repBase + repIndex; size_t mLength; hashSmall[hSmall] = hashLong[hLong] = curr; /* update hash table */ if ((((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow : ensure repIndex doesn't overlap dict + prefix */ & (offset_1 <= curr+1 - dictStartIndex)) /* note: we are searching at curr+1 */ && (MEM_read32(repMatch) == MEM_read32(ip+1)) ) { const BYTE* repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend; mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixStart) + 4; ip++; ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, REPCODE1_TO_OFFBASE, mLength); } else { if ((matchLongIndex > dictStartIndex) && (MEM_read64(matchLong) == MEM_read64(ip))) { const BYTE* const matchEnd = matchLongIndex < prefixStartIndex ? dictEnd : iend; const BYTE* const lowMatchPtr = matchLongIndex < prefixStartIndex ? dictStart : prefixStart; U32 offset; mLength = ZSTD_count_2segments(ip+8, matchLong+8, iend, matchEnd, prefixStart) + 8; offset = curr - matchLongIndex; while (((ip>anchor) & (matchLong>lowMatchPtr)) && (ip[-1] == matchLong[-1])) { ip--; matchLong--; mLength++; } /* catch up */ offset_2 = offset_1; offset_1 = offset; ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, OFFSET_TO_OFFBASE(offset), mLength); } else if ((matchIndex > dictStartIndex) && (MEM_read32(match) == MEM_read32(ip))) { size_t const h3 = ZSTD_hashPtr(ip+1, hBitsL, 8); U32 const matchIndex3 = hashLong[h3]; const BYTE* const match3Base = matchIndex3 < prefixStartIndex ? dictBase : base; const BYTE* match3 = match3Base + matchIndex3; U32 offset; hashLong[h3] = curr + 1; if ( (matchIndex3 > dictStartIndex) && (MEM_read64(match3) == MEM_read64(ip+1)) ) { const BYTE* const matchEnd = matchIndex3 < prefixStartIndex ? dictEnd : iend; const BYTE* const lowMatchPtr = matchIndex3 < prefixStartIndex ? dictStart : prefixStart; mLength = ZSTD_count_2segments(ip+9, match3+8, iend, matchEnd, prefixStart) + 8; ip++; offset = curr+1 - matchIndex3; while (((ip>anchor) & (match3>lowMatchPtr)) && (ip[-1] == match3[-1])) { ip--; match3--; mLength++; } /* catch up */ } else { const BYTE* const matchEnd = matchIndex < prefixStartIndex ? dictEnd : iend; const BYTE* const lowMatchPtr = matchIndex < prefixStartIndex ? dictStart : prefixStart; mLength = ZSTD_count_2segments(ip+4, match+4, iend, matchEnd, prefixStart) + 4; offset = curr - matchIndex; while (((ip>anchor) & (match>lowMatchPtr)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */ } offset_2 = offset_1; offset_1 = offset; ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, OFFSET_TO_OFFBASE(offset), mLength); } else { ip += ((ip-anchor) >> kSearchStrength) + 1; continue; } } /* move to next sequence start */ ip += mLength; anchor = ip; if (ip <= ilimit) { /* Complementary insertion */ /* done after iLimit test, as candidates could be > iend-8 */ { U32 const indexToInsert = curr+2; hashLong[ZSTD_hashPtr(base+indexToInsert, hBitsL, 8)] = indexToInsert; hashLong[ZSTD_hashPtr(ip-2, hBitsL, 8)] = (U32)(ip-2-base); hashSmall[ZSTD_hashPtr(base+indexToInsert, hBitsS, mls)] = indexToInsert; hashSmall[ZSTD_hashPtr(ip-1, hBitsS, mls)] = (U32)(ip-1-base); } /* check immediate repcode */ while (ip <= ilimit) { U32 const current2 = (U32)(ip-base); U32 const repIndex2 = current2 - offset_2; const BYTE* repMatch2 = repIndex2 < prefixStartIndex ? dictBase + repIndex2 : base + repIndex2; if ( (((U32)((prefixStartIndex-1) - repIndex2) >= 3) /* intentional overflow : ensure repIndex2 doesn't overlap dict + prefix */ & (offset_2 <= current2 - dictStartIndex)) && (MEM_read32(repMatch2) == MEM_read32(ip)) ) { const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend; size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4; U32 const tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */ ZSTD_storeSeq(seqStore, 0, anchor, iend, REPCODE1_TO_OFFBASE, repLength2); hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2; hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2; ip += repLength2; anchor = ip; continue; } break; } } } /* save reps for next block */ rep[0] = offset_1; rep[1] = offset_2; /* Return the last literals size */ return (size_t)(iend - anchor); } ZSTD_GEN_DFAST_FN(extDict, 4) ZSTD_GEN_DFAST_FN(extDict, 5) ZSTD_GEN_DFAST_FN(extDict, 6) ZSTD_GEN_DFAST_FN(extDict, 7) size_t ZSTD_compressBlock_doubleFast_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize) { U32 const mls = ms->cParams.minMatch; switch(mls) { default: /* includes case 3 */ case 4 : return ZSTD_compressBlock_doubleFast_extDict_4(ms, seqStore, rep, src, srcSize); case 5 : return ZSTD_compressBlock_doubleFast_extDict_5(ms, seqStore, rep, src, srcSize); case 6 : return ZSTD_compressBlock_doubleFast_extDict_6(ms, seqStore, rep, src, srcSize); case 7 : return ZSTD_compressBlock_doubleFast_extDict_7(ms, seqStore, rep, src, srcSize); } } zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_compress_literals.h0000644000175200007730000000311514515254731030122 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_COMPRESS_LITERALS_H #define ZSTD_COMPRESS_LITERALS_H #include "zstd_compress_internal.h" /* ZSTD_hufCTables_t, ZSTD_minGain() */ size_t ZSTD_noCompressLiterals (void* dst, size_t dstCapacity, const void* src, size_t srcSize); /* ZSTD_compressRleLiteralsBlock() : * Conditions : * - All bytes in @src are identical * - dstCapacity >= 4 */ size_t ZSTD_compressRleLiteralsBlock (void* dst, size_t dstCapacity, const void* src, size_t srcSize); /* ZSTD_compressLiterals(): * @entropyWorkspace: must be aligned on 4-bytes boundaries * @entropyWorkspaceSize : must be >= HUF_WORKSPACE_SIZE * @suspectUncompressible: sampling checks, to potentially skip huffman coding */ size_t ZSTD_compressLiterals (void* dst, size_t dstCapacity, const void* src, size_t srcSize, void* entropyWorkspace, size_t entropyWorkspaceSize, const ZSTD_hufCTables_t* prevHuf, ZSTD_hufCTables_t* nextHuf, ZSTD_strategy strategy, int disableLiteralCompression, int suspectUncompressible, int bmi2); #endif /* ZSTD_COMPRESS_LITERALS_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_compress_literals.c0000644000175200007730000002170314515254731030120 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /*-************************************* * Dependencies ***************************************/ #include "zstd_compress_literals.h" /* ************************************************************** * Debug Traces ****************************************************************/ #if DEBUGLEVEL >= 2 static size_t showHexa(const void* src, size_t srcSize) { const BYTE* const ip = (const BYTE*)src; size_t u; for (u=0; u31) + (srcSize>4095); DEBUGLOG(5, "ZSTD_noCompressLiterals: srcSize=%zu, dstCapacity=%zu", srcSize, dstCapacity); RETURN_ERROR_IF(srcSize + flSize > dstCapacity, dstSize_tooSmall, ""); switch(flSize) { case 1: /* 2 - 1 - 5 */ ostart[0] = (BYTE)((U32)set_basic + (srcSize<<3)); break; case 2: /* 2 - 2 - 12 */ MEM_writeLE16(ostart, (U16)((U32)set_basic + (1<<2) + (srcSize<<4))); break; case 3: /* 2 - 2 - 20 */ MEM_writeLE32(ostart, (U32)((U32)set_basic + (3<<2) + (srcSize<<4))); break; default: /* not necessary : flSize is {1,2,3} */ assert(0); } ZSTD_memcpy(ostart + flSize, src, srcSize); DEBUGLOG(5, "Raw (uncompressed) literals: %u -> %u", (U32)srcSize, (U32)(srcSize + flSize)); return srcSize + flSize; } static int allBytesIdentical(const void* src, size_t srcSize) { assert(srcSize >= 1); assert(src != NULL); { const BYTE b = ((const BYTE*)src)[0]; size_t p; for (p=1; p31) + (srcSize>4095); assert(dstCapacity >= 4); (void)dstCapacity; assert(allBytesIdentical(src, srcSize)); switch(flSize) { case 1: /* 2 - 1 - 5 */ ostart[0] = (BYTE)((U32)set_rle + (srcSize<<3)); break; case 2: /* 2 - 2 - 12 */ MEM_writeLE16(ostart, (U16)((U32)set_rle + (1<<2) + (srcSize<<4))); break; case 3: /* 2 - 2 - 20 */ MEM_writeLE32(ostart, (U32)((U32)set_rle + (3<<2) + (srcSize<<4))); break; default: /* not necessary : flSize is {1,2,3} */ assert(0); } ostart[flSize] = *(const BYTE*)src; DEBUGLOG(5, "RLE : Repeated Literal (%02X: %u times) -> %u bytes encoded", ((const BYTE*)src)[0], (U32)srcSize, (U32)flSize + 1); return flSize+1; } /* ZSTD_minLiteralsToCompress() : * returns minimal amount of literals * for literal compression to even be attempted. * Minimum is made tighter as compression strategy increases. */ static size_t ZSTD_minLiteralsToCompress(ZSTD_strategy strategy, HUF_repeat huf_repeat) { assert((int)strategy >= 0); assert((int)strategy <= 9); /* btultra2 : min 8 bytes; * then 2x larger for each successive compression strategy * max threshold 64 bytes */ { int const shift = MIN(9-(int)strategy, 3); size_t const mintc = (huf_repeat == HUF_repeat_valid) ? 6 : (size_t)8 << shift; DEBUGLOG(7, "minLiteralsToCompress = %zu", mintc); return mintc; } } size_t ZSTD_compressLiterals ( void* dst, size_t dstCapacity, const void* src, size_t srcSize, void* entropyWorkspace, size_t entropyWorkspaceSize, const ZSTD_hufCTables_t* prevHuf, ZSTD_hufCTables_t* nextHuf, ZSTD_strategy strategy, int disableLiteralCompression, int suspectUncompressible, int bmi2) { size_t const lhSize = 3 + (srcSize >= 1 KB) + (srcSize >= 16 KB); BYTE* const ostart = (BYTE*)dst; U32 singleStream = srcSize < 256; symbolEncodingType_e hType = set_compressed; size_t cLitSize; DEBUGLOG(5,"ZSTD_compressLiterals (disableLiteralCompression=%i, srcSize=%u, dstCapacity=%zu)", disableLiteralCompression, (U32)srcSize, dstCapacity); DEBUGLOG(6, "Completed literals listing (%zu bytes)", showHexa(src, srcSize)); /* Prepare nextEntropy assuming reusing the existing table */ ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf)); if (disableLiteralCompression) return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize); /* if too small, don't even attempt compression (speed opt) */ if (srcSize < ZSTD_minLiteralsToCompress(strategy, prevHuf->repeatMode)) return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize); RETURN_ERROR_IF(dstCapacity < lhSize+1, dstSize_tooSmall, "not enough space for compression"); { HUF_repeat repeat = prevHuf->repeatMode; int const flags = 0 | (bmi2 ? HUF_flags_bmi2 : 0) | (strategy < ZSTD_lazy && srcSize <= 1024 ? HUF_flags_preferRepeat : 0) | (strategy >= HUF_OPTIMAL_DEPTH_THRESHOLD ? HUF_flags_optimalDepth : 0) | (suspectUncompressible ? HUF_flags_suspectUncompressible : 0); typedef size_t (*huf_compress_f)(void*, size_t, const void*, size_t, unsigned, unsigned, void*, size_t, HUF_CElt*, HUF_repeat*, int); huf_compress_f huf_compress; if (repeat == HUF_repeat_valid && lhSize == 3) singleStream = 1; huf_compress = singleStream ? HUF_compress1X_repeat : HUF_compress4X_repeat; cLitSize = huf_compress(ostart+lhSize, dstCapacity-lhSize, src, srcSize, HUF_SYMBOLVALUE_MAX, LitHufLog, entropyWorkspace, entropyWorkspaceSize, (HUF_CElt*)nextHuf->CTable, &repeat, flags); DEBUGLOG(5, "%zu literals compressed into %zu bytes (before header)", srcSize, cLitSize); if (repeat != HUF_repeat_none) { /* reused the existing table */ DEBUGLOG(5, "reusing statistics from previous huffman block"); hType = set_repeat; } } { size_t const minGain = ZSTD_minGain(srcSize, strategy); if ((cLitSize==0) || (cLitSize >= srcSize - minGain) || ERR_isError(cLitSize)) { ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf)); return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize); } } if (cLitSize==1) { /* A return value of 1 signals that the alphabet consists of a single symbol. * However, in some rare circumstances, it could be the compressed size (a single byte). * For that outcome to have a chance to happen, it's necessary that `srcSize < 8`. * (it's also necessary to not generate statistics). * Therefore, in such a case, actively check that all bytes are identical. */ if ((srcSize >= 8) || allBytesIdentical(src, srcSize)) { ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf)); return ZSTD_compressRleLiteralsBlock(dst, dstCapacity, src, srcSize); } } if (hType == set_compressed) { /* using a newly constructed table */ nextHuf->repeatMode = HUF_repeat_check; } /* Build header */ switch(lhSize) { case 3: /* 2 - 2 - 10 - 10 */ if (!singleStream) assert(srcSize >= MIN_LITERALS_FOR_4_STREAMS); { U32 const lhc = hType + ((U32)(!singleStream) << 2) + ((U32)srcSize<<4) + ((U32)cLitSize<<14); MEM_writeLE24(ostart, lhc); break; } case 4: /* 2 - 2 - 14 - 14 */ assert(srcSize >= MIN_LITERALS_FOR_4_STREAMS); { U32 const lhc = hType + (2 << 2) + ((U32)srcSize<<4) + ((U32)cLitSize<<18); MEM_writeLE32(ostart, lhc); break; } case 5: /* 2 - 2 - 18 - 18 */ assert(srcSize >= MIN_LITERALS_FOR_4_STREAMS); { U32 const lhc = hType + (3 << 2) + ((U32)srcSize<<4) + ((U32)cLitSize<<22); MEM_writeLE32(ostart, lhc); ostart[4] = (BYTE)(cLitSize >> 10); break; } default: /* not possible : lhSize is {3,4,5} */ assert(0); } DEBUGLOG(5, "Compressed literals: %u -> %u", (U32)srcSize, (U32)(lhSize+cLitSize)); return lhSize+cLitSize; } zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_opt.c0000644000175200007730000020360314515254731025171 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #include "zstd_compress_internal.h" #include "hist.h" #include "zstd_opt.h" #define ZSTD_LITFREQ_ADD 2 /* scaling factor for litFreq, so that frequencies adapt faster to new stats */ #define ZSTD_MAX_PRICE (1<<30) #define ZSTD_PREDEF_THRESHOLD 8 /* if srcSize < ZSTD_PREDEF_THRESHOLD, symbols' cost is assumed static, directly determined by pre-defined distributions */ /*-************************************* * Price functions for optimal parser ***************************************/ #if 0 /* approximation at bit level (for tests) */ # define BITCOST_ACCURACY 0 # define BITCOST_MULTIPLIER (1 << BITCOST_ACCURACY) # define WEIGHT(stat, opt) ((void)(opt), ZSTD_bitWeight(stat)) #elif 0 /* fractional bit accuracy (for tests) */ # define BITCOST_ACCURACY 8 # define BITCOST_MULTIPLIER (1 << BITCOST_ACCURACY) # define WEIGHT(stat,opt) ((void)(opt), ZSTD_fracWeight(stat)) #else /* opt==approx, ultra==accurate */ # define BITCOST_ACCURACY 8 # define BITCOST_MULTIPLIER (1 << BITCOST_ACCURACY) # define WEIGHT(stat,opt) ((opt) ? ZSTD_fracWeight(stat) : ZSTD_bitWeight(stat)) #endif /* ZSTD_bitWeight() : * provide estimated "cost" of a stat in full bits only */ MEM_STATIC U32 ZSTD_bitWeight(U32 stat) { return (ZSTD_highbit32(stat+1) * BITCOST_MULTIPLIER); } /* ZSTD_fracWeight() : * provide fractional-bit "cost" of a stat, * using linear interpolation approximation */ MEM_STATIC U32 ZSTD_fracWeight(U32 rawStat) { U32 const stat = rawStat + 1; U32 const hb = ZSTD_highbit32(stat); U32 const BWeight = hb * BITCOST_MULTIPLIER; /* Fweight was meant for "Fractional weight" * but it's effectively a value between 1 and 2 * using fixed point arithmetic */ U32 const FWeight = (stat << BITCOST_ACCURACY) >> hb; U32 const weight = BWeight + FWeight; assert(hb + BITCOST_ACCURACY < 31); return weight; } #if (DEBUGLEVEL>=2) /* debugging function, * @return price in bytes as fractional value * for debug messages only */ MEM_STATIC double ZSTD_fCost(int price) { return (double)price / (BITCOST_MULTIPLIER*8); } #endif static int ZSTD_compressedLiterals(optState_t const* const optPtr) { return optPtr->literalCompressionMode != ZSTD_ps_disable; } static void ZSTD_setBasePrices(optState_t* optPtr, int optLevel) { if (ZSTD_compressedLiterals(optPtr)) optPtr->litSumBasePrice = WEIGHT(optPtr->litSum, optLevel); optPtr->litLengthSumBasePrice = WEIGHT(optPtr->litLengthSum, optLevel); optPtr->matchLengthSumBasePrice = WEIGHT(optPtr->matchLengthSum, optLevel); optPtr->offCodeSumBasePrice = WEIGHT(optPtr->offCodeSum, optLevel); } static U32 sum_u32(const unsigned table[], size_t nbElts) { size_t n; U32 total = 0; for (n=0; n0); unsigned const newStat = base + (table[s] >> shift); sum += newStat; table[s] = newStat; } return sum; } /* ZSTD_scaleStats() : * reduce all elt frequencies in table if sum too large * return the resulting sum of elements */ static U32 ZSTD_scaleStats(unsigned* table, U32 lastEltIndex, U32 logTarget) { U32 const prevsum = sum_u32(table, lastEltIndex+1); U32 const factor = prevsum >> logTarget; DEBUGLOG(5, "ZSTD_scaleStats (nbElts=%u, target=%u)", (unsigned)lastEltIndex+1, (unsigned)logTarget); assert(logTarget < 30); if (factor <= 1) return prevsum; return ZSTD_downscaleStats(table, lastEltIndex, ZSTD_highbit32(factor), base_1guaranteed); } /* ZSTD_rescaleFreqs() : * if first block (detected by optPtr->litLengthSum == 0) : init statistics * take hints from dictionary if there is one * and init from zero if there is none, * using src for literals stats, and baseline stats for sequence symbols * otherwise downscale existing stats, to be used as seed for next block. */ static void ZSTD_rescaleFreqs(optState_t* const optPtr, const BYTE* const src, size_t const srcSize, int const optLevel) { int const compressedLiterals = ZSTD_compressedLiterals(optPtr); DEBUGLOG(5, "ZSTD_rescaleFreqs (srcSize=%u)", (unsigned)srcSize); optPtr->priceType = zop_dynamic; if (optPtr->litLengthSum == 0) { /* no literals stats collected -> first block assumed -> init */ /* heuristic: use pre-defined stats for too small inputs */ if (srcSize <= ZSTD_PREDEF_THRESHOLD) { DEBUGLOG(5, "srcSize <= %i : use predefined stats", ZSTD_PREDEF_THRESHOLD); optPtr->priceType = zop_predef; } assert(optPtr->symbolCosts != NULL); if (optPtr->symbolCosts->huf.repeatMode == HUF_repeat_valid) { /* huffman stats covering the full value set : table presumed generated by dictionary */ optPtr->priceType = zop_dynamic; if (compressedLiterals) { /* generate literals statistics from huffman table */ unsigned lit; assert(optPtr->litFreq != NULL); optPtr->litSum = 0; for (lit=0; lit<=MaxLit; lit++) { U32 const scaleLog = 11; /* scale to 2K */ U32 const bitCost = HUF_getNbBitsFromCTable(optPtr->symbolCosts->huf.CTable, lit); assert(bitCost <= scaleLog); optPtr->litFreq[lit] = bitCost ? 1 << (scaleLog-bitCost) : 1 /*minimum to calculate cost*/; optPtr->litSum += optPtr->litFreq[lit]; } } { unsigned ll; FSE_CState_t llstate; FSE_initCState(&llstate, optPtr->symbolCosts->fse.litlengthCTable); optPtr->litLengthSum = 0; for (ll=0; ll<=MaxLL; ll++) { U32 const scaleLog = 10; /* scale to 1K */ U32 const bitCost = FSE_getMaxNbBits(llstate.symbolTT, ll); assert(bitCost < scaleLog); optPtr->litLengthFreq[ll] = bitCost ? 1 << (scaleLog-bitCost) : 1 /*minimum to calculate cost*/; optPtr->litLengthSum += optPtr->litLengthFreq[ll]; } } { unsigned ml; FSE_CState_t mlstate; FSE_initCState(&mlstate, optPtr->symbolCosts->fse.matchlengthCTable); optPtr->matchLengthSum = 0; for (ml=0; ml<=MaxML; ml++) { U32 const scaleLog = 10; U32 const bitCost = FSE_getMaxNbBits(mlstate.symbolTT, ml); assert(bitCost < scaleLog); optPtr->matchLengthFreq[ml] = bitCost ? 1 << (scaleLog-bitCost) : 1 /*minimum to calculate cost*/; optPtr->matchLengthSum += optPtr->matchLengthFreq[ml]; } } { unsigned of; FSE_CState_t ofstate; FSE_initCState(&ofstate, optPtr->symbolCosts->fse.offcodeCTable); optPtr->offCodeSum = 0; for (of=0; of<=MaxOff; of++) { U32 const scaleLog = 10; U32 const bitCost = FSE_getMaxNbBits(ofstate.symbolTT, of); assert(bitCost < scaleLog); optPtr->offCodeFreq[of] = bitCost ? 1 << (scaleLog-bitCost) : 1 /*minimum to calculate cost*/; optPtr->offCodeSum += optPtr->offCodeFreq[of]; } } } else { /* first block, no dictionary */ assert(optPtr->litFreq != NULL); if (compressedLiterals) { /* base initial cost of literals on direct frequency within src */ unsigned lit = MaxLit; HIST_count_simple(optPtr->litFreq, &lit, src, srcSize); /* use raw first block to init statistics */ optPtr->litSum = ZSTD_downscaleStats(optPtr->litFreq, MaxLit, 8, base_0possible); } { unsigned const baseLLfreqs[MaxLL+1] = { 4, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 }; ZSTD_memcpy(optPtr->litLengthFreq, baseLLfreqs, sizeof(baseLLfreqs)); optPtr->litLengthSum = sum_u32(baseLLfreqs, MaxLL+1); } { unsigned ml; for (ml=0; ml<=MaxML; ml++) optPtr->matchLengthFreq[ml] = 1; } optPtr->matchLengthSum = MaxML+1; { unsigned const baseOFCfreqs[MaxOff+1] = { 6, 2, 1, 1, 2, 3, 4, 4, 4, 3, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 }; ZSTD_memcpy(optPtr->offCodeFreq, baseOFCfreqs, sizeof(baseOFCfreqs)); optPtr->offCodeSum = sum_u32(baseOFCfreqs, MaxOff+1); } } } else { /* new block : scale down accumulated statistics */ if (compressedLiterals) optPtr->litSum = ZSTD_scaleStats(optPtr->litFreq, MaxLit, 12); optPtr->litLengthSum = ZSTD_scaleStats(optPtr->litLengthFreq, MaxLL, 11); optPtr->matchLengthSum = ZSTD_scaleStats(optPtr->matchLengthFreq, MaxML, 11); optPtr->offCodeSum = ZSTD_scaleStats(optPtr->offCodeFreq, MaxOff, 11); } ZSTD_setBasePrices(optPtr, optLevel); } /* ZSTD_rawLiteralsCost() : * price of literals (only) in specified segment (which length can be 0). * does not include price of literalLength symbol */ static U32 ZSTD_rawLiteralsCost(const BYTE* const literals, U32 const litLength, const optState_t* const optPtr, int optLevel) { if (litLength == 0) return 0; if (!ZSTD_compressedLiterals(optPtr)) return (litLength << 3) * BITCOST_MULTIPLIER; /* Uncompressed - 8 bytes per literal. */ if (optPtr->priceType == zop_predef) return (litLength*6) * BITCOST_MULTIPLIER; /* 6 bit per literal - no statistic used */ /* dynamic statistics */ { U32 price = optPtr->litSumBasePrice * litLength; U32 const litPriceMax = optPtr->litSumBasePrice - BITCOST_MULTIPLIER; U32 u; assert(optPtr->litSumBasePrice >= BITCOST_MULTIPLIER); for (u=0; u < litLength; u++) { U32 litPrice = WEIGHT(optPtr->litFreq[literals[u]], optLevel); if (UNLIKELY(litPrice > litPriceMax)) litPrice = litPriceMax; price -= litPrice; } return price; } } /* ZSTD_litLengthPrice() : * cost of literalLength symbol */ static U32 ZSTD_litLengthPrice(U32 const litLength, const optState_t* const optPtr, int optLevel) { assert(litLength <= ZSTD_BLOCKSIZE_MAX); if (optPtr->priceType == zop_predef) return WEIGHT(litLength, optLevel); /* ZSTD_LLcode() can't compute litLength price for sizes >= ZSTD_BLOCKSIZE_MAX * because it isn't representable in the zstd format. * So instead just pretend it would cost 1 bit more than ZSTD_BLOCKSIZE_MAX - 1. * In such a case, the block would be all literals. */ if (litLength == ZSTD_BLOCKSIZE_MAX) return BITCOST_MULTIPLIER + ZSTD_litLengthPrice(ZSTD_BLOCKSIZE_MAX - 1, optPtr, optLevel); /* dynamic statistics */ { U32 const llCode = ZSTD_LLcode(litLength); return (LL_bits[llCode] * BITCOST_MULTIPLIER) + optPtr->litLengthSumBasePrice - WEIGHT(optPtr->litLengthFreq[llCode], optLevel); } } /* ZSTD_getMatchPrice() : * Provides the cost of the match part (offset + matchLength) of a sequence. * Must be combined with ZSTD_fullLiteralsCost() to get the full cost of a sequence. * @offBase : sumtype, representing an offset or a repcode, and using numeric representation of ZSTD_storeSeq() * @optLevel: when <2, favors small offset for decompression speed (improved cache efficiency) */ FORCE_INLINE_TEMPLATE U32 ZSTD_getMatchPrice(U32 const offBase, U32 const matchLength, const optState_t* const optPtr, int const optLevel) { U32 price; U32 const offCode = ZSTD_highbit32(offBase); U32 const mlBase = matchLength - MINMATCH; assert(matchLength >= MINMATCH); if (optPtr->priceType == zop_predef) /* fixed scheme, does not use statistics */ return WEIGHT(mlBase, optLevel) + ((16 + offCode) * BITCOST_MULTIPLIER); /* emulated offset cost */ /* dynamic statistics */ price = (offCode * BITCOST_MULTIPLIER) + (optPtr->offCodeSumBasePrice - WEIGHT(optPtr->offCodeFreq[offCode], optLevel)); if ((optLevel<2) /*static*/ && offCode >= 20) price += (offCode-19)*2 * BITCOST_MULTIPLIER; /* handicap for long distance offsets, favor decompression speed */ /* match Length */ { U32 const mlCode = ZSTD_MLcode(mlBase); price += (ML_bits[mlCode] * BITCOST_MULTIPLIER) + (optPtr->matchLengthSumBasePrice - WEIGHT(optPtr->matchLengthFreq[mlCode], optLevel)); } price += BITCOST_MULTIPLIER / 5; /* heuristic : make matches a bit more costly to favor less sequences -> faster decompression speed */ DEBUGLOG(8, "ZSTD_getMatchPrice(ml:%u) = %u", matchLength, price); return price; } /* ZSTD_updateStats() : * assumption : literals + litLength <= iend */ static void ZSTD_updateStats(optState_t* const optPtr, U32 litLength, const BYTE* literals, U32 offBase, U32 matchLength) { /* literals */ if (ZSTD_compressedLiterals(optPtr)) { U32 u; for (u=0; u < litLength; u++) optPtr->litFreq[literals[u]] += ZSTD_LITFREQ_ADD; optPtr->litSum += litLength*ZSTD_LITFREQ_ADD; } /* literal Length */ { U32 const llCode = ZSTD_LLcode(litLength); optPtr->litLengthFreq[llCode]++; optPtr->litLengthSum++; } /* offset code : follows storeSeq() numeric representation */ { U32 const offCode = ZSTD_highbit32(offBase); assert(offCode <= MaxOff); optPtr->offCodeFreq[offCode]++; optPtr->offCodeSum++; } /* match Length */ { U32 const mlBase = matchLength - MINMATCH; U32 const mlCode = ZSTD_MLcode(mlBase); optPtr->matchLengthFreq[mlCode]++; optPtr->matchLengthSum++; } } /* ZSTD_readMINMATCH() : * function safe only for comparisons * assumption : memPtr must be at least 4 bytes before end of buffer */ MEM_STATIC U32 ZSTD_readMINMATCH(const void* memPtr, U32 length) { switch (length) { default : case 4 : return MEM_read32(memPtr); case 3 : if (MEM_isLittleEndian()) return MEM_read32(memPtr)<<8; else return MEM_read32(memPtr)>>8; } } /* Update hashTable3 up to ip (excluded) Assumption : always within prefix (i.e. not within extDict) */ static U32 ZSTD_insertAndFindFirstIndexHash3 (const ZSTD_matchState_t* ms, U32* nextToUpdate3, const BYTE* const ip) { U32* const hashTable3 = ms->hashTable3; U32 const hashLog3 = ms->hashLog3; const BYTE* const base = ms->window.base; U32 idx = *nextToUpdate3; U32 const target = (U32)(ip - base); size_t const hash3 = ZSTD_hash3Ptr(ip, hashLog3); assert(hashLog3 > 0); while(idx < target) { hashTable3[ZSTD_hash3Ptr(base+idx, hashLog3)] = idx; idx++; } *nextToUpdate3 = target; return hashTable3[hash3]; } /*-************************************* * Binary Tree search ***************************************/ /** ZSTD_insertBt1() : add one or multiple positions to tree. * @param ip assumed <= iend-8 . * @param target The target of ZSTD_updateTree_internal() - we are filling to this position * @return : nb of positions added */ static U32 ZSTD_insertBt1( const ZSTD_matchState_t* ms, const BYTE* const ip, const BYTE* const iend, U32 const target, U32 const mls, const int extDict) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32* const hashTable = ms->hashTable; U32 const hashLog = cParams->hashLog; size_t const h = ZSTD_hashPtr(ip, hashLog, mls); U32* const bt = ms->chainTable; U32 const btLog = cParams->chainLog - 1; U32 const btMask = (1 << btLog) - 1; U32 matchIndex = hashTable[h]; size_t commonLengthSmaller=0, commonLengthLarger=0; const BYTE* const base = ms->window.base; const BYTE* const dictBase = ms->window.dictBase; const U32 dictLimit = ms->window.dictLimit; const BYTE* const dictEnd = dictBase + dictLimit; const BYTE* const prefixStart = base + dictLimit; const BYTE* match; const U32 curr = (U32)(ip-base); const U32 btLow = btMask >= curr ? 0 : curr - btMask; U32* smallerPtr = bt + 2*(curr&btMask); U32* largerPtr = smallerPtr + 1; U32 dummy32; /* to be nullified at the end */ /* windowLow is based on target because * we only need positions that will be in the window at the end of the tree update. */ U32 const windowLow = ZSTD_getLowestMatchIndex(ms, target, cParams->windowLog); U32 matchEndIdx = curr+8+1; size_t bestLength = 8; U32 nbCompares = 1U << cParams->searchLog; #ifdef ZSTD_C_PREDICT U32 predictedSmall = *(bt + 2*((curr-1)&btMask) + 0); U32 predictedLarge = *(bt + 2*((curr-1)&btMask) + 1); predictedSmall += (predictedSmall>0); predictedLarge += (predictedLarge>0); #endif /* ZSTD_C_PREDICT */ DEBUGLOG(8, "ZSTD_insertBt1 (%u)", curr); assert(curr <= target); assert(ip <= iend-8); /* required for h calculation */ hashTable[h] = curr; /* Update Hash Table */ assert(windowLow > 0); for (; nbCompares && (matchIndex >= windowLow); --nbCompares) { U32* const nextPtr = bt + 2*(matchIndex & btMask); size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */ assert(matchIndex < curr); #ifdef ZSTD_C_PREDICT /* note : can create issues when hlog small <= 11 */ const U32* predictPtr = bt + 2*((matchIndex-1) & btMask); /* written this way, as bt is a roll buffer */ if (matchIndex == predictedSmall) { /* no need to check length, result known */ *smallerPtr = matchIndex; if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */ smallerPtr = nextPtr+1; /* new "smaller" => larger of match */ matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */ predictedSmall = predictPtr[1] + (predictPtr[1]>0); continue; } if (matchIndex == predictedLarge) { *largerPtr = matchIndex; if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */ largerPtr = nextPtr; matchIndex = nextPtr[0]; predictedLarge = predictPtr[0] + (predictPtr[0]>0); continue; } #endif if (!extDict || (matchIndex+matchLength >= dictLimit)) { assert(matchIndex+matchLength >= dictLimit); /* might be wrong if actually extDict */ match = base + matchIndex; matchLength += ZSTD_count(ip+matchLength, match+matchLength, iend); } else { match = dictBase + matchIndex; matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart); if (matchIndex+matchLength >= dictLimit) match = base + matchIndex; /* to prepare for next usage of match[matchLength] */ } if (matchLength > bestLength) { bestLength = matchLength; if (matchLength > matchEndIdx - matchIndex) matchEndIdx = matchIndex + (U32)matchLength; } if (ip+matchLength == iend) { /* equal : no way to know if inf or sup */ break; /* drop , to guarantee consistency ; miss a bit of compression, but other solutions can corrupt tree */ } if (match[matchLength] < ip[matchLength]) { /* necessarily within buffer */ /* match is smaller than current */ *smallerPtr = matchIndex; /* update smaller idx */ commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */ if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop searching */ smallerPtr = nextPtr+1; /* new "candidate" => larger than match, which was smaller than target */ matchIndex = nextPtr[1]; /* new matchIndex, larger than previous and closer to current */ } else { /* match is larger than current */ *largerPtr = matchIndex; commonLengthLarger = matchLength; if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop searching */ largerPtr = nextPtr; matchIndex = nextPtr[0]; } } *smallerPtr = *largerPtr = 0; { U32 positions = 0; if (bestLength > 384) positions = MIN(192, (U32)(bestLength - 384)); /* speed optimization */ assert(matchEndIdx > curr + 8); return MAX(positions, matchEndIdx - (curr + 8)); } } FORCE_INLINE_TEMPLATE void ZSTD_updateTree_internal( ZSTD_matchState_t* ms, const BYTE* const ip, const BYTE* const iend, const U32 mls, const ZSTD_dictMode_e dictMode) { const BYTE* const base = ms->window.base; U32 const target = (U32)(ip - base); U32 idx = ms->nextToUpdate; DEBUGLOG(6, "ZSTD_updateTree_internal, from %u to %u (dictMode:%u)", idx, target, dictMode); while(idx < target) { U32 const forward = ZSTD_insertBt1(ms, base+idx, iend, target, mls, dictMode == ZSTD_extDict); assert(idx < (U32)(idx + forward)); idx += forward; } assert((size_t)(ip - base) <= (size_t)(U32)(-1)); assert((size_t)(iend - base) <= (size_t)(U32)(-1)); ms->nextToUpdate = target; } void ZSTD_updateTree(ZSTD_matchState_t* ms, const BYTE* ip, const BYTE* iend) { ZSTD_updateTree_internal(ms, ip, iend, ms->cParams.minMatch, ZSTD_noDict); } FORCE_INLINE_TEMPLATE U32 ZSTD_insertBtAndGetAllMatches ( ZSTD_match_t* matches, /* store result (found matches) in this table (presumed large enough) */ ZSTD_matchState_t* ms, U32* nextToUpdate3, const BYTE* const ip, const BYTE* const iLimit, const ZSTD_dictMode_e dictMode, const U32 rep[ZSTD_REP_NUM], const U32 ll0, /* tells if associated literal length is 0 or not. This value must be 0 or 1 */ const U32 lengthToBeat, const U32 mls /* template */) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32 const sufficient_len = MIN(cParams->targetLength, ZSTD_OPT_NUM -1); const BYTE* const base = ms->window.base; U32 const curr = (U32)(ip-base); U32 const hashLog = cParams->hashLog; U32 const minMatch = (mls==3) ? 3 : 4; U32* const hashTable = ms->hashTable; size_t const h = ZSTD_hashPtr(ip, hashLog, mls); U32 matchIndex = hashTable[h]; U32* const bt = ms->chainTable; U32 const btLog = cParams->chainLog - 1; U32 const btMask= (1U << btLog) - 1; size_t commonLengthSmaller=0, commonLengthLarger=0; const BYTE* const dictBase = ms->window.dictBase; U32 const dictLimit = ms->window.dictLimit; const BYTE* const dictEnd = dictBase + dictLimit; const BYTE* const prefixStart = base + dictLimit; U32 const btLow = (btMask >= curr) ? 0 : curr - btMask; U32 const windowLow = ZSTD_getLowestMatchIndex(ms, curr, cParams->windowLog); U32 const matchLow = windowLow ? windowLow : 1; U32* smallerPtr = bt + 2*(curr&btMask); U32* largerPtr = bt + 2*(curr&btMask) + 1; U32 matchEndIdx = curr+8+1; /* farthest referenced position of any match => detects repetitive patterns */ U32 dummy32; /* to be nullified at the end */ U32 mnum = 0; U32 nbCompares = 1U << cParams->searchLog; const ZSTD_matchState_t* dms = dictMode == ZSTD_dictMatchState ? ms->dictMatchState : NULL; const ZSTD_compressionParameters* const dmsCParams = dictMode == ZSTD_dictMatchState ? &dms->cParams : NULL; const BYTE* const dmsBase = dictMode == ZSTD_dictMatchState ? dms->window.base : NULL; const BYTE* const dmsEnd = dictMode == ZSTD_dictMatchState ? dms->window.nextSrc : NULL; U32 const dmsHighLimit = dictMode == ZSTD_dictMatchState ? (U32)(dmsEnd - dmsBase) : 0; U32 const dmsLowLimit = dictMode == ZSTD_dictMatchState ? dms->window.lowLimit : 0; U32 const dmsIndexDelta = dictMode == ZSTD_dictMatchState ? windowLow - dmsHighLimit : 0; U32 const dmsHashLog = dictMode == ZSTD_dictMatchState ? dmsCParams->hashLog : hashLog; U32 const dmsBtLog = dictMode == ZSTD_dictMatchState ? dmsCParams->chainLog - 1 : btLog; U32 const dmsBtMask = dictMode == ZSTD_dictMatchState ? (1U << dmsBtLog) - 1 : 0; U32 const dmsBtLow = dictMode == ZSTD_dictMatchState && dmsBtMask < dmsHighLimit - dmsLowLimit ? dmsHighLimit - dmsBtMask : dmsLowLimit; size_t bestLength = lengthToBeat-1; DEBUGLOG(8, "ZSTD_insertBtAndGetAllMatches: current=%u", curr); /* check repCode */ assert(ll0 <= 1); /* necessarily 1 or 0 */ { U32 const lastR = ZSTD_REP_NUM + ll0; U32 repCode; for (repCode = ll0; repCode < lastR; repCode++) { U32 const repOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode]; U32 const repIndex = curr - repOffset; U32 repLen = 0; assert(curr >= dictLimit); if (repOffset-1 /* intentional overflow, discards 0 and -1 */ < curr-dictLimit) { /* equivalent to `curr > repIndex >= dictLimit` */ /* We must validate the repcode offset because when we're using a dictionary the * valid offset range shrinks when the dictionary goes out of bounds. */ if ((repIndex >= windowLow) & (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(ip - repOffset, minMatch))) { repLen = (U32)ZSTD_count(ip+minMatch, ip+minMatch-repOffset, iLimit) + minMatch; } } else { /* repIndex < dictLimit || repIndex >= curr */ const BYTE* const repMatch = dictMode == ZSTD_dictMatchState ? dmsBase + repIndex - dmsIndexDelta : dictBase + repIndex; assert(curr >= windowLow); if ( dictMode == ZSTD_extDict && ( ((repOffset-1) /*intentional overflow*/ < curr - windowLow) /* equivalent to `curr > repIndex >= windowLow` */ & (((U32)((dictLimit-1) - repIndex) >= 3) ) /* intentional overflow : do not test positions overlapping 2 memory segments */) && (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch)) ) { repLen = (U32)ZSTD_count_2segments(ip+minMatch, repMatch+minMatch, iLimit, dictEnd, prefixStart) + minMatch; } if (dictMode == ZSTD_dictMatchState && ( ((repOffset-1) /*intentional overflow*/ < curr - (dmsLowLimit + dmsIndexDelta)) /* equivalent to `curr > repIndex >= dmsLowLimit` */ & ((U32)((dictLimit-1) - repIndex) >= 3) ) /* intentional overflow : do not test positions overlapping 2 memory segments */ && (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch)) ) { repLen = (U32)ZSTD_count_2segments(ip+minMatch, repMatch+minMatch, iLimit, dmsEnd, prefixStart) + minMatch; } } /* save longer solution */ if (repLen > bestLength) { DEBUGLOG(8, "found repCode %u (ll0:%u, offset:%u) of length %u", repCode, ll0, repOffset, repLen); bestLength = repLen; matches[mnum].off = REPCODE_TO_OFFBASE(repCode - ll0 + 1); /* expect value between 1 and 3 */ matches[mnum].len = (U32)repLen; mnum++; if ( (repLen > sufficient_len) | (ip+repLen == iLimit) ) { /* best possible */ return mnum; } } } } /* HC3 match finder */ if ((mls == 3) /*static*/ && (bestLength < mls)) { U32 const matchIndex3 = ZSTD_insertAndFindFirstIndexHash3(ms, nextToUpdate3, ip); if ((matchIndex3 >= matchLow) & (curr - matchIndex3 < (1<<18)) /*heuristic : longer distance likely too expensive*/ ) { size_t mlen; if ((dictMode == ZSTD_noDict) /*static*/ || (dictMode == ZSTD_dictMatchState) /*static*/ || (matchIndex3 >= dictLimit)) { const BYTE* const match = base + matchIndex3; mlen = ZSTD_count(ip, match, iLimit); } else { const BYTE* const match = dictBase + matchIndex3; mlen = ZSTD_count_2segments(ip, match, iLimit, dictEnd, prefixStart); } /* save best solution */ if (mlen >= mls /* == 3 > bestLength */) { DEBUGLOG(8, "found small match with hlog3, of length %u", (U32)mlen); bestLength = mlen; assert(curr > matchIndex3); assert(mnum==0); /* no prior solution */ matches[0].off = OFFSET_TO_OFFBASE(curr - matchIndex3); matches[0].len = (U32)mlen; mnum = 1; if ( (mlen > sufficient_len) | (ip+mlen == iLimit) ) { /* best possible length */ ms->nextToUpdate = curr+1; /* skip insertion */ return 1; } } } /* no dictMatchState lookup: dicts don't have a populated HC3 table */ } /* if (mls == 3) */ hashTable[h] = curr; /* Update Hash Table */ for (; nbCompares && (matchIndex >= matchLow); --nbCompares) { U32* const nextPtr = bt + 2*(matchIndex & btMask); const BYTE* match; size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */ assert(curr > matchIndex); if ((dictMode == ZSTD_noDict) || (dictMode == ZSTD_dictMatchState) || (matchIndex+matchLength >= dictLimit)) { assert(matchIndex+matchLength >= dictLimit); /* ensure the condition is correct when !extDict */ match = base + matchIndex; if (matchIndex >= dictLimit) assert(memcmp(match, ip, matchLength) == 0); /* ensure early section of match is equal as expected */ matchLength += ZSTD_count(ip+matchLength, match+matchLength, iLimit); } else { match = dictBase + matchIndex; assert(memcmp(match, ip, matchLength) == 0); /* ensure early section of match is equal as expected */ matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iLimit, dictEnd, prefixStart); if (matchIndex+matchLength >= dictLimit) match = base + matchIndex; /* prepare for match[matchLength] read */ } if (matchLength > bestLength) { DEBUGLOG(8, "found match of length %u at distance %u (offBase=%u)", (U32)matchLength, curr - matchIndex, OFFSET_TO_OFFBASE(curr - matchIndex)); assert(matchEndIdx > matchIndex); if (matchLength > matchEndIdx - matchIndex) matchEndIdx = matchIndex + (U32)matchLength; bestLength = matchLength; matches[mnum].off = OFFSET_TO_OFFBASE(curr - matchIndex); matches[mnum].len = (U32)matchLength; mnum++; if ( (matchLength > ZSTD_OPT_NUM) | (ip+matchLength == iLimit) /* equal : no way to know if inf or sup */) { if (dictMode == ZSTD_dictMatchState) nbCompares = 0; /* break should also skip searching dms */ break; /* drop, to preserve bt consistency (miss a little bit of compression) */ } } if (match[matchLength] < ip[matchLength]) { /* match smaller than current */ *smallerPtr = matchIndex; /* update smaller idx */ commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */ if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */ smallerPtr = nextPtr+1; /* new candidate => larger than match, which was smaller than current */ matchIndex = nextPtr[1]; /* new matchIndex, larger than previous, closer to current */ } else { *largerPtr = matchIndex; commonLengthLarger = matchLength; if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */ largerPtr = nextPtr; matchIndex = nextPtr[0]; } } *smallerPtr = *largerPtr = 0; assert(nbCompares <= (1U << ZSTD_SEARCHLOG_MAX)); /* Check we haven't underflowed. */ if (dictMode == ZSTD_dictMatchState && nbCompares) { size_t const dmsH = ZSTD_hashPtr(ip, dmsHashLog, mls); U32 dictMatchIndex = dms->hashTable[dmsH]; const U32* const dmsBt = dms->chainTable; commonLengthSmaller = commonLengthLarger = 0; for (; nbCompares && (dictMatchIndex > dmsLowLimit); --nbCompares) { const U32* const nextPtr = dmsBt + 2*(dictMatchIndex & dmsBtMask); size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */ const BYTE* match = dmsBase + dictMatchIndex; matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iLimit, dmsEnd, prefixStart); if (dictMatchIndex+matchLength >= dmsHighLimit) match = base + dictMatchIndex + dmsIndexDelta; /* to prepare for next usage of match[matchLength] */ if (matchLength > bestLength) { matchIndex = dictMatchIndex + dmsIndexDelta; DEBUGLOG(8, "found dms match of length %u at distance %u (offBase=%u)", (U32)matchLength, curr - matchIndex, OFFSET_TO_OFFBASE(curr - matchIndex)); if (matchLength > matchEndIdx - matchIndex) matchEndIdx = matchIndex + (U32)matchLength; bestLength = matchLength; matches[mnum].off = OFFSET_TO_OFFBASE(curr - matchIndex); matches[mnum].len = (U32)matchLength; mnum++; if ( (matchLength > ZSTD_OPT_NUM) | (ip+matchLength == iLimit) /* equal : no way to know if inf or sup */) { break; /* drop, to guarantee consistency (miss a little bit of compression) */ } } if (dictMatchIndex <= dmsBtLow) { break; } /* beyond tree size, stop the search */ if (match[matchLength] < ip[matchLength]) { commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */ dictMatchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */ } else { /* match is larger than current */ commonLengthLarger = matchLength; dictMatchIndex = nextPtr[0]; } } } /* if (dictMode == ZSTD_dictMatchState) */ assert(matchEndIdx > curr+8); ms->nextToUpdate = matchEndIdx - 8; /* skip repetitive patterns */ return mnum; } typedef U32 (*ZSTD_getAllMatchesFn)( ZSTD_match_t*, ZSTD_matchState_t*, U32*, const BYTE*, const BYTE*, const U32 rep[ZSTD_REP_NUM], U32 const ll0, U32 const lengthToBeat); FORCE_INLINE_TEMPLATE U32 ZSTD_btGetAllMatches_internal( ZSTD_match_t* matches, ZSTD_matchState_t* ms, U32* nextToUpdate3, const BYTE* ip, const BYTE* const iHighLimit, const U32 rep[ZSTD_REP_NUM], U32 const ll0, U32 const lengthToBeat, const ZSTD_dictMode_e dictMode, const U32 mls) { assert(BOUNDED(3, ms->cParams.minMatch, 6) == mls); DEBUGLOG(8, "ZSTD_BtGetAllMatches(dictMode=%d, mls=%u)", (int)dictMode, mls); if (ip < ms->window.base + ms->nextToUpdate) return 0; /* skipped area */ ZSTD_updateTree_internal(ms, ip, iHighLimit, mls, dictMode); return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, mls); } #define ZSTD_BT_GET_ALL_MATCHES_FN(dictMode, mls) ZSTD_btGetAllMatches_##dictMode##_##mls #define GEN_ZSTD_BT_GET_ALL_MATCHES_(dictMode, mls) \ static U32 ZSTD_BT_GET_ALL_MATCHES_FN(dictMode, mls)( \ ZSTD_match_t* matches, \ ZSTD_matchState_t* ms, \ U32* nextToUpdate3, \ const BYTE* ip, \ const BYTE* const iHighLimit, \ const U32 rep[ZSTD_REP_NUM], \ U32 const ll0, \ U32 const lengthToBeat) \ { \ return ZSTD_btGetAllMatches_internal( \ matches, ms, nextToUpdate3, ip, iHighLimit, \ rep, ll0, lengthToBeat, ZSTD_##dictMode, mls); \ } #define GEN_ZSTD_BT_GET_ALL_MATCHES(dictMode) \ GEN_ZSTD_BT_GET_ALL_MATCHES_(dictMode, 3) \ GEN_ZSTD_BT_GET_ALL_MATCHES_(dictMode, 4) \ GEN_ZSTD_BT_GET_ALL_MATCHES_(dictMode, 5) \ GEN_ZSTD_BT_GET_ALL_MATCHES_(dictMode, 6) GEN_ZSTD_BT_GET_ALL_MATCHES(noDict) GEN_ZSTD_BT_GET_ALL_MATCHES(extDict) GEN_ZSTD_BT_GET_ALL_MATCHES(dictMatchState) #define ZSTD_BT_GET_ALL_MATCHES_ARRAY(dictMode) \ { \ ZSTD_BT_GET_ALL_MATCHES_FN(dictMode, 3), \ ZSTD_BT_GET_ALL_MATCHES_FN(dictMode, 4), \ ZSTD_BT_GET_ALL_MATCHES_FN(dictMode, 5), \ ZSTD_BT_GET_ALL_MATCHES_FN(dictMode, 6) \ } static ZSTD_getAllMatchesFn ZSTD_selectBtGetAllMatches(ZSTD_matchState_t const* ms, ZSTD_dictMode_e const dictMode) { ZSTD_getAllMatchesFn const getAllMatchesFns[3][4] = { ZSTD_BT_GET_ALL_MATCHES_ARRAY(noDict), ZSTD_BT_GET_ALL_MATCHES_ARRAY(extDict), ZSTD_BT_GET_ALL_MATCHES_ARRAY(dictMatchState) }; U32 const mls = BOUNDED(3, ms->cParams.minMatch, 6); assert((U32)dictMode < 3); assert(mls - 3 < 4); return getAllMatchesFns[(int)dictMode][mls - 3]; } /************************* * LDM helper functions * *************************/ /* Struct containing info needed to make decision about ldm inclusion */ typedef struct { rawSeqStore_t seqStore; /* External match candidates store for this block */ U32 startPosInBlock; /* Start position of the current match candidate */ U32 endPosInBlock; /* End position of the current match candidate */ U32 offset; /* Offset of the match candidate */ } ZSTD_optLdm_t; /* ZSTD_optLdm_skipRawSeqStoreBytes(): * Moves forward in @rawSeqStore by @nbBytes, * which will update the fields 'pos' and 'posInSequence'. */ static void ZSTD_optLdm_skipRawSeqStoreBytes(rawSeqStore_t* rawSeqStore, size_t nbBytes) { U32 currPos = (U32)(rawSeqStore->posInSequence + nbBytes); while (currPos && rawSeqStore->pos < rawSeqStore->size) { rawSeq currSeq = rawSeqStore->seq[rawSeqStore->pos]; if (currPos >= currSeq.litLength + currSeq.matchLength) { currPos -= currSeq.litLength + currSeq.matchLength; rawSeqStore->pos++; } else { rawSeqStore->posInSequence = currPos; break; } } if (currPos == 0 || rawSeqStore->pos == rawSeqStore->size) { rawSeqStore->posInSequence = 0; } } /* ZSTD_opt_getNextMatchAndUpdateSeqStore(): * Calculates the beginning and end of the next match in the current block. * Updates 'pos' and 'posInSequence' of the ldmSeqStore. */ static void ZSTD_opt_getNextMatchAndUpdateSeqStore(ZSTD_optLdm_t* optLdm, U32 currPosInBlock, U32 blockBytesRemaining) { rawSeq currSeq; U32 currBlockEndPos; U32 literalsBytesRemaining; U32 matchBytesRemaining; /* Setting match end position to MAX to ensure we never use an LDM during this block */ if (optLdm->seqStore.size == 0 || optLdm->seqStore.pos >= optLdm->seqStore.size) { optLdm->startPosInBlock = UINT_MAX; optLdm->endPosInBlock = UINT_MAX; return; } /* Calculate appropriate bytes left in matchLength and litLength * after adjusting based on ldmSeqStore->posInSequence */ currSeq = optLdm->seqStore.seq[optLdm->seqStore.pos]; assert(optLdm->seqStore.posInSequence <= currSeq.litLength + currSeq.matchLength); currBlockEndPos = currPosInBlock + blockBytesRemaining; literalsBytesRemaining = (optLdm->seqStore.posInSequence < currSeq.litLength) ? currSeq.litLength - (U32)optLdm->seqStore.posInSequence : 0; matchBytesRemaining = (literalsBytesRemaining == 0) ? currSeq.matchLength - ((U32)optLdm->seqStore.posInSequence - currSeq.litLength) : currSeq.matchLength; /* If there are more literal bytes than bytes remaining in block, no ldm is possible */ if (literalsBytesRemaining >= blockBytesRemaining) { optLdm->startPosInBlock = UINT_MAX; optLdm->endPosInBlock = UINT_MAX; ZSTD_optLdm_skipRawSeqStoreBytes(&optLdm->seqStore, blockBytesRemaining); return; } /* Matches may be < MINMATCH by this process. In that case, we will reject them when we are deciding whether or not to add the ldm */ optLdm->startPosInBlock = currPosInBlock + literalsBytesRemaining; optLdm->endPosInBlock = optLdm->startPosInBlock + matchBytesRemaining; optLdm->offset = currSeq.offset; if (optLdm->endPosInBlock > currBlockEndPos) { /* Match ends after the block ends, we can't use the whole match */ optLdm->endPosInBlock = currBlockEndPos; ZSTD_optLdm_skipRawSeqStoreBytes(&optLdm->seqStore, currBlockEndPos - currPosInBlock); } else { /* Consume nb of bytes equal to size of sequence left */ ZSTD_optLdm_skipRawSeqStoreBytes(&optLdm->seqStore, literalsBytesRemaining + matchBytesRemaining); } } /* ZSTD_optLdm_maybeAddMatch(): * Adds a match if it's long enough, * based on it's 'matchStartPosInBlock' and 'matchEndPosInBlock', * into 'matches'. Maintains the correct ordering of 'matches'. */ static void ZSTD_optLdm_maybeAddMatch(ZSTD_match_t* matches, U32* nbMatches, const ZSTD_optLdm_t* optLdm, U32 currPosInBlock) { U32 const posDiff = currPosInBlock - optLdm->startPosInBlock; /* Note: ZSTD_match_t actually contains offBase and matchLength (before subtracting MINMATCH) */ U32 const candidateMatchLength = optLdm->endPosInBlock - optLdm->startPosInBlock - posDiff; /* Ensure that current block position is not outside of the match */ if (currPosInBlock < optLdm->startPosInBlock || currPosInBlock >= optLdm->endPosInBlock || candidateMatchLength < MINMATCH) { return; } if (*nbMatches == 0 || ((candidateMatchLength > matches[*nbMatches-1].len) && *nbMatches < ZSTD_OPT_NUM)) { U32 const candidateOffBase = OFFSET_TO_OFFBASE(optLdm->offset); DEBUGLOG(6, "ZSTD_optLdm_maybeAddMatch(): Adding ldm candidate match (offBase: %u matchLength %u) at block position=%u", candidateOffBase, candidateMatchLength, currPosInBlock); matches[*nbMatches].len = candidateMatchLength; matches[*nbMatches].off = candidateOffBase; (*nbMatches)++; } } /* ZSTD_optLdm_processMatchCandidate(): * Wrapper function to update ldm seq store and call ldm functions as necessary. */ static void ZSTD_optLdm_processMatchCandidate(ZSTD_optLdm_t* optLdm, ZSTD_match_t* matches, U32* nbMatches, U32 currPosInBlock, U32 remainingBytes) { if (optLdm->seqStore.size == 0 || optLdm->seqStore.pos >= optLdm->seqStore.size) { return; } if (currPosInBlock >= optLdm->endPosInBlock) { if (currPosInBlock > optLdm->endPosInBlock) { /* The position at which ZSTD_optLdm_processMatchCandidate() is called is not necessarily * at the end of a match from the ldm seq store, and will often be some bytes * over beyond matchEndPosInBlock. As such, we need to correct for these "overshoots" */ U32 const posOvershoot = currPosInBlock - optLdm->endPosInBlock; ZSTD_optLdm_skipRawSeqStoreBytes(&optLdm->seqStore, posOvershoot); } ZSTD_opt_getNextMatchAndUpdateSeqStore(optLdm, currPosInBlock, remainingBytes); } ZSTD_optLdm_maybeAddMatch(matches, nbMatches, optLdm, currPosInBlock); } /*-******************************* * Optimal parser *********************************/ static U32 ZSTD_totalLen(ZSTD_optimal_t sol) { return sol.litlen + sol.mlen; } #if 0 /* debug */ static void listStats(const U32* table, int lastEltID) { int const nbElts = lastEltID + 1; int enb; for (enb=0; enb < nbElts; enb++) { (void)table; /* RAWLOG(2, "%3i:%3i, ", enb, table[enb]); */ RAWLOG(2, "%4i,", table[enb]); } RAWLOG(2, " \n"); } #endif FORCE_INLINE_TEMPLATE size_t ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize, const int optLevel, const ZSTD_dictMode_e dictMode) { optState_t* const optStatePtr = &ms->opt; const BYTE* const istart = (const BYTE*)src; const BYTE* ip = istart; const BYTE* anchor = istart; const BYTE* const iend = istart + srcSize; const BYTE* const ilimit = iend - 8; const BYTE* const base = ms->window.base; const BYTE* const prefixStart = base + ms->window.dictLimit; const ZSTD_compressionParameters* const cParams = &ms->cParams; ZSTD_getAllMatchesFn getAllMatches = ZSTD_selectBtGetAllMatches(ms, dictMode); U32 const sufficient_len = MIN(cParams->targetLength, ZSTD_OPT_NUM -1); U32 const minMatch = (cParams->minMatch == 3) ? 3 : 4; U32 nextToUpdate3 = ms->nextToUpdate; ZSTD_optimal_t* const opt = optStatePtr->priceTable; ZSTD_match_t* const matches = optStatePtr->matchTable; ZSTD_optimal_t lastSequence; ZSTD_optLdm_t optLdm; ZSTD_memset(&lastSequence, 0, sizeof(ZSTD_optimal_t)); optLdm.seqStore = ms->ldmSeqStore ? *ms->ldmSeqStore : kNullRawSeqStore; optLdm.endPosInBlock = optLdm.startPosInBlock = optLdm.offset = 0; ZSTD_opt_getNextMatchAndUpdateSeqStore(&optLdm, (U32)(ip-istart), (U32)(iend-ip)); /* init */ DEBUGLOG(5, "ZSTD_compressBlock_opt_generic: current=%u, prefix=%u, nextToUpdate=%u", (U32)(ip - base), ms->window.dictLimit, ms->nextToUpdate); assert(optLevel <= 2); ZSTD_rescaleFreqs(optStatePtr, (const BYTE*)src, srcSize, optLevel); ip += (ip==prefixStart); /* Match Loop */ while (ip < ilimit) { U32 cur, last_pos = 0; /* find first match */ { U32 const litlen = (U32)(ip - anchor); U32 const ll0 = !litlen; U32 nbMatches = getAllMatches(matches, ms, &nextToUpdate3, ip, iend, rep, ll0, minMatch); ZSTD_optLdm_processMatchCandidate(&optLdm, matches, &nbMatches, (U32)(ip-istart), (U32)(iend - ip)); if (!nbMatches) { ip++; continue; } /* initialize opt[0] */ { U32 i ; for (i=0; i immediate encoding */ { U32 const maxML = matches[nbMatches-1].len; U32 const maxOffBase = matches[nbMatches-1].off; DEBUGLOG(6, "found %u matches of maxLength=%u and maxOffBase=%u at cPos=%u => start new series", nbMatches, maxML, maxOffBase, (U32)(ip-prefixStart)); if (maxML > sufficient_len) { lastSequence.litlen = litlen; lastSequence.mlen = maxML; lastSequence.off = maxOffBase; DEBUGLOG(6, "large match (%u>%u), immediate encoding", maxML, sufficient_len); cur = 0; last_pos = ZSTD_totalLen(lastSequence); goto _shortestPath; } } /* set prices for first matches starting position == 0 */ assert(opt[0].price >= 0); { U32 const literalsPrice = (U32)opt[0].price + ZSTD_litLengthPrice(0, optStatePtr, optLevel); U32 pos; U32 matchNb; for (pos = 1; pos < minMatch; pos++) { opt[pos].price = ZSTD_MAX_PRICE; /* mlen, litlen and price will be fixed during forward scanning */ } for (matchNb = 0; matchNb < nbMatches; matchNb++) { U32 const offBase = matches[matchNb].off; U32 const end = matches[matchNb].len; for ( ; pos <= end ; pos++ ) { U32 const matchPrice = ZSTD_getMatchPrice(offBase, pos, optStatePtr, optLevel); U32 const sequencePrice = literalsPrice + matchPrice; DEBUGLOG(7, "rPos:%u => set initial price : %.2f", pos, ZSTD_fCost((int)sequencePrice)); opt[pos].mlen = pos; opt[pos].off = offBase; opt[pos].litlen = litlen; opt[pos].price = (int)sequencePrice; } } last_pos = pos-1; } } /* check further positions */ for (cur = 1; cur <= last_pos; cur++) { const BYTE* const inr = ip + cur; assert(cur < ZSTD_OPT_NUM); DEBUGLOG(7, "cPos:%zi==rPos:%u", inr-istart, cur) /* Fix current position with one literal if cheaper */ { U32 const litlen = (opt[cur-1].mlen == 0) ? opt[cur-1].litlen + 1 : 1; int const price = opt[cur-1].price + (int)ZSTD_rawLiteralsCost(ip+cur-1, 1, optStatePtr, optLevel) + (int)ZSTD_litLengthPrice(litlen, optStatePtr, optLevel) - (int)ZSTD_litLengthPrice(litlen-1, optStatePtr, optLevel); assert(price < 1000000000); /* overflow check */ if (price <= opt[cur].price) { DEBUGLOG(7, "cPos:%zi==rPos:%u : better price (%.2f<=%.2f) using literal (ll==%u) (hist:%u,%u,%u)", inr-istart, cur, ZSTD_fCost(price), ZSTD_fCost(opt[cur].price), litlen, opt[cur-1].rep[0], opt[cur-1].rep[1], opt[cur-1].rep[2]); opt[cur].mlen = 0; opt[cur].off = 0; opt[cur].litlen = litlen; opt[cur].price = price; } else { DEBUGLOG(7, "cPos:%zi==rPos:%u : literal would cost more (%.2f>%.2f) (hist:%u,%u,%u)", inr-istart, cur, ZSTD_fCost(price), ZSTD_fCost(opt[cur].price), opt[cur].rep[0], opt[cur].rep[1], opt[cur].rep[2]); } } /* Set the repcodes of the current position. We must do it here * because we rely on the repcodes of the 2nd to last sequence being * correct to set the next chunks repcodes during the backward * traversal. */ ZSTD_STATIC_ASSERT(sizeof(opt[cur].rep) == sizeof(repcodes_t)); assert(cur >= opt[cur].mlen); if (opt[cur].mlen != 0) { U32 const prev = cur - opt[cur].mlen; repcodes_t const newReps = ZSTD_newRep(opt[prev].rep, opt[cur].off, opt[cur].litlen==0); ZSTD_memcpy(opt[cur].rep, &newReps, sizeof(repcodes_t)); } else { ZSTD_memcpy(opt[cur].rep, opt[cur - 1].rep, sizeof(repcodes_t)); } /* last match must start at a minimum distance of 8 from oend */ if (inr > ilimit) continue; if (cur == last_pos) break; if ( (optLevel==0) /*static_test*/ && (opt[cur+1].price <= opt[cur].price + (BITCOST_MULTIPLIER/2)) ) { DEBUGLOG(7, "move to next rPos:%u : price is <=", cur+1); continue; /* skip unpromising positions; about ~+6% speed, -0.01 ratio */ } assert(opt[cur].price >= 0); { U32 const ll0 = (opt[cur].mlen != 0); U32 const litlen = (opt[cur].mlen == 0) ? opt[cur].litlen : 0; U32 const previousPrice = (U32)opt[cur].price; U32 const basePrice = previousPrice + ZSTD_litLengthPrice(0, optStatePtr, optLevel); U32 nbMatches = getAllMatches(matches, ms, &nextToUpdate3, inr, iend, opt[cur].rep, ll0, minMatch); U32 matchNb; ZSTD_optLdm_processMatchCandidate(&optLdm, matches, &nbMatches, (U32)(inr-istart), (U32)(iend-inr)); if (!nbMatches) { DEBUGLOG(7, "rPos:%u : no match found", cur); continue; } { U32 const maxML = matches[nbMatches-1].len; DEBUGLOG(7, "cPos:%zi==rPos:%u, found %u matches, of maxLength=%u", inr-istart, cur, nbMatches, maxML); if ( (maxML > sufficient_len) || (cur + maxML >= ZSTD_OPT_NUM) ) { lastSequence.mlen = maxML; lastSequence.off = matches[nbMatches-1].off; lastSequence.litlen = litlen; cur -= (opt[cur].mlen==0) ? opt[cur].litlen : 0; /* last sequence is actually only literals, fix cur to last match - note : may underflow, in which case, it's first sequence, and it's okay */ last_pos = cur + ZSTD_totalLen(lastSequence); if (cur > ZSTD_OPT_NUM) cur = 0; /* underflow => first match */ goto _shortestPath; } } /* set prices using matches found at position == cur */ for (matchNb = 0; matchNb < nbMatches; matchNb++) { U32 const offset = matches[matchNb].off; U32 const lastML = matches[matchNb].len; U32 const startML = (matchNb>0) ? matches[matchNb-1].len+1 : minMatch; U32 mlen; DEBUGLOG(7, "testing match %u => offBase=%4u, mlen=%2u, llen=%2u", matchNb, matches[matchNb].off, lastML, litlen); for (mlen = lastML; mlen >= startML; mlen--) { /* scan downward */ U32 const pos = cur + mlen; int const price = (int)basePrice + (int)ZSTD_getMatchPrice(offset, mlen, optStatePtr, optLevel); if ((pos > last_pos) || (price < opt[pos].price)) { DEBUGLOG(7, "rPos:%u (ml=%2u) => new better price (%.2f<%.2f)", pos, mlen, ZSTD_fCost(price), ZSTD_fCost(opt[pos].price)); while (last_pos < pos) { opt[last_pos+1].price = ZSTD_MAX_PRICE; last_pos++; } /* fill empty positions */ opt[pos].mlen = mlen; opt[pos].off = offset; opt[pos].litlen = litlen; opt[pos].price = price; } else { DEBUGLOG(7, "rPos:%u (ml=%2u) => new price is worse (%.2f>=%.2f)", pos, mlen, ZSTD_fCost(price), ZSTD_fCost(opt[pos].price)); if (optLevel==0) break; /* early update abort; gets ~+10% speed for about -0.01 ratio loss */ } } } } } /* for (cur = 1; cur <= last_pos; cur++) */ lastSequence = opt[last_pos]; cur = last_pos > ZSTD_totalLen(lastSequence) ? last_pos - ZSTD_totalLen(lastSequence) : 0; /* single sequence, and it starts before `ip` */ assert(cur < ZSTD_OPT_NUM); /* control overflow*/ _shortestPath: /* cur, last_pos, best_mlen, best_off have to be set */ assert(opt[0].mlen == 0); /* Set the next chunk's repcodes based on the repcodes of the beginning * of the last match, and the last sequence. This avoids us having to * update them while traversing the sequences. */ if (lastSequence.mlen != 0) { repcodes_t const reps = ZSTD_newRep(opt[cur].rep, lastSequence.off, lastSequence.litlen==0); ZSTD_memcpy(rep, &reps, sizeof(reps)); } else { ZSTD_memcpy(rep, opt[cur].rep, sizeof(repcodes_t)); } { U32 const storeEnd = cur + 1; U32 storeStart = storeEnd; U32 seqPos = cur; DEBUGLOG(6, "start reverse traversal (last_pos:%u, cur:%u)", last_pos, cur); (void)last_pos; assert(storeEnd < ZSTD_OPT_NUM); DEBUGLOG(6, "last sequence copied into pos=%u (llen=%u,mlen=%u,ofc=%u)", storeEnd, lastSequence.litlen, lastSequence.mlen, lastSequence.off); opt[storeEnd] = lastSequence; while (seqPos > 0) { U32 const backDist = ZSTD_totalLen(opt[seqPos]); storeStart--; DEBUGLOG(6, "sequence from rPos=%u copied into pos=%u (llen=%u,mlen=%u,ofc=%u)", seqPos, storeStart, opt[seqPos].litlen, opt[seqPos].mlen, opt[seqPos].off); opt[storeStart] = opt[seqPos]; seqPos = (seqPos > backDist) ? seqPos - backDist : 0; } /* save sequences */ DEBUGLOG(6, "sending selected sequences into seqStore") { U32 storePos; for (storePos=storeStart; storePos <= storeEnd; storePos++) { U32 const llen = opt[storePos].litlen; U32 const mlen = opt[storePos].mlen; U32 const offBase = opt[storePos].off; U32 const advance = llen + mlen; DEBUGLOG(6, "considering seq starting at %zi, llen=%u, mlen=%u", anchor - istart, (unsigned)llen, (unsigned)mlen); if (mlen==0) { /* only literals => must be last "sequence", actually starting a new stream of sequences */ assert(storePos == storeEnd); /* must be last sequence */ ip = anchor + llen; /* last "sequence" is a bunch of literals => don't progress anchor */ continue; /* will finish */ } assert(anchor + llen <= iend); ZSTD_updateStats(optStatePtr, llen, anchor, offBase, mlen); ZSTD_storeSeq(seqStore, llen, anchor, iend, offBase, mlen); anchor += advance; ip = anchor; } } ZSTD_setBasePrices(optStatePtr, optLevel); } } /* while (ip < ilimit) */ /* Return the last literals size */ return (size_t)(iend - anchor); } static size_t ZSTD_compressBlock_opt0( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize, const ZSTD_dictMode_e dictMode) { return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /* optLevel */, dictMode); } static size_t ZSTD_compressBlock_opt2( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize, const ZSTD_dictMode_e dictMode) { return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /* optLevel */, dictMode); } size_t ZSTD_compressBlock_btopt( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { DEBUGLOG(5, "ZSTD_compressBlock_btopt"); return ZSTD_compressBlock_opt0(ms, seqStore, rep, src, srcSize, ZSTD_noDict); } /* ZSTD_initStats_ultra(): * make a first compression pass, just to seed stats with more accurate starting values. * only works on first block, with no dictionary and no ldm. * this function cannot error out, its narrow contract must be respected. */ static void ZSTD_initStats_ultra(ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { U32 tmpRep[ZSTD_REP_NUM]; /* updated rep codes will sink here */ ZSTD_memcpy(tmpRep, rep, sizeof(tmpRep)); DEBUGLOG(4, "ZSTD_initStats_ultra (srcSize=%zu)", srcSize); assert(ms->opt.litLengthSum == 0); /* first block */ assert(seqStore->sequences == seqStore->sequencesStart); /* no ldm */ assert(ms->window.dictLimit == ms->window.lowLimit); /* no dictionary */ assert(ms->window.dictLimit - ms->nextToUpdate <= 1); /* no prefix (note: intentional overflow, defined as 2-complement) */ ZSTD_compressBlock_opt2(ms, seqStore, tmpRep, src, srcSize, ZSTD_noDict); /* generate stats into ms->opt*/ /* invalidate first scan from history, only keep entropy stats */ ZSTD_resetSeqStore(seqStore); ms->window.base -= srcSize; ms->window.dictLimit += (U32)srcSize; ms->window.lowLimit = ms->window.dictLimit; ms->nextToUpdate = ms->window.dictLimit; } size_t ZSTD_compressBlock_btultra( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { DEBUGLOG(5, "ZSTD_compressBlock_btultra (srcSize=%zu)", srcSize); return ZSTD_compressBlock_opt2(ms, seqStore, rep, src, srcSize, ZSTD_noDict); } size_t ZSTD_compressBlock_btultra2( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { U32 const curr = (U32)((const BYTE*)src - ms->window.base); DEBUGLOG(5, "ZSTD_compressBlock_btultra2 (srcSize=%zu)", srcSize); /* 2-passes strategy: * this strategy makes a first pass over first block to collect statistics * in order to seed next round's statistics with it. * After 1st pass, function forgets history, and starts a new block. * Consequently, this can only work if no data has been previously loaded in tables, * aka, no dictionary, no prefix, no ldm preprocessing. * The compression ratio gain is generally small (~0.5% on first block), ** the cost is 2x cpu time on first block. */ assert(srcSize <= ZSTD_BLOCKSIZE_MAX); if ( (ms->opt.litLengthSum==0) /* first block */ && (seqStore->sequences == seqStore->sequencesStart) /* no ldm */ && (ms->window.dictLimit == ms->window.lowLimit) /* no dictionary */ && (curr == ms->window.dictLimit) /* start of frame, nothing already loaded nor skipped */ && (srcSize > ZSTD_PREDEF_THRESHOLD) /* input large enough to not employ default stats */ ) { ZSTD_initStats_ultra(ms, seqStore, rep, src, srcSize); } return ZSTD_compressBlock_opt2(ms, seqStore, rep, src, srcSize, ZSTD_noDict); } size_t ZSTD_compressBlock_btopt_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { return ZSTD_compressBlock_opt0(ms, seqStore, rep, src, srcSize, ZSTD_dictMatchState); } size_t ZSTD_compressBlock_btultra_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { return ZSTD_compressBlock_opt2(ms, seqStore, rep, src, srcSize, ZSTD_dictMatchState); } size_t ZSTD_compressBlock_btopt_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { return ZSTD_compressBlock_opt0(ms, seqStore, rep, src, srcSize, ZSTD_extDict); } size_t ZSTD_compressBlock_btultra_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { return ZSTD_compressBlock_opt2(ms, seqStore, rep, src, srcSize, ZSTD_extDict); } /* note : no btultra2 variant for extDict nor dictMatchState, * because btultra2 is not meant to work with dictionaries * and is only specific for the first block (no prefix) */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_compress_sequences.h0000644000175200007730000000417714515254731030307 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_COMPRESS_SEQUENCES_H #define ZSTD_COMPRESS_SEQUENCES_H #include "../common/fse.h" /* FSE_repeat, FSE_CTable */ #include "../common/zstd_internal.h" /* symbolEncodingType_e, ZSTD_strategy */ typedef enum { ZSTD_defaultDisallowed = 0, ZSTD_defaultAllowed = 1 } ZSTD_defaultPolicy_e; symbolEncodingType_e ZSTD_selectEncodingType( FSE_repeat* repeatMode, unsigned const* count, unsigned const max, size_t const mostFrequent, size_t nbSeq, unsigned const FSELog, FSE_CTable const* prevCTable, short const* defaultNorm, U32 defaultNormLog, ZSTD_defaultPolicy_e const isDefaultAllowed, ZSTD_strategy const strategy); size_t ZSTD_buildCTable(void* dst, size_t dstCapacity, FSE_CTable* nextCTable, U32 FSELog, symbolEncodingType_e type, unsigned* count, U32 max, const BYTE* codeTable, size_t nbSeq, const S16* defaultNorm, U32 defaultNormLog, U32 defaultMax, const FSE_CTable* prevCTable, size_t prevCTableSize, void* entropyWorkspace, size_t entropyWorkspaceSize); size_t ZSTD_encodeSequences( void* dst, size_t dstCapacity, FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable, FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable, FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable, seqDef const* sequences, size_t nbSeq, int longOffsets, int bmi2); size_t ZSTD_fseBitCost( FSE_CTable const* ctable, unsigned const* count, unsigned const max); size_t ZSTD_crossEntropyCost(short const* norm, unsigned accuracyLog, unsigned const* count, unsigned const max); #endif /* ZSTD_COMPRESS_SEQUENCES_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/hist.c0000644000175200007730000001643014515254731024272 0ustar rlaboissrlaboiss/* ****************************************************************** * hist : Histogram functions * part of Finite State Entropy project * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy * - Public forum : https://groups.google.com/forum/#!forum/lz4c * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ /* --- dependencies --- */ #include "../common/mem.h" /* U32, BYTE, etc. */ #include "../common/debug.h" /* assert, DEBUGLOG */ #include "../common/error_private.h" /* ERROR */ #include "hist.h" /* --- Error management --- */ unsigned HIST_isError(size_t code) { return ERR_isError(code); } /*-************************************************************** * Histogram functions ****************************************************************/ unsigned HIST_count_simple(unsigned* count, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize) { const BYTE* ip = (const BYTE*)src; const BYTE* const end = ip + srcSize; unsigned maxSymbolValue = *maxSymbolValuePtr; unsigned largestCount=0; ZSTD_memset(count, 0, (maxSymbolValue+1) * sizeof(*count)); if (srcSize==0) { *maxSymbolValuePtr = 0; return 0; } while (ip largestCount) largestCount = count[s]; } return largestCount; } typedef enum { trustInput, checkMaxSymbolValue } HIST_checkInput_e; /* HIST_count_parallel_wksp() : * store histogram into 4 intermediate tables, recombined at the end. * this design makes better use of OoO cpus, * and is noticeably faster when some values are heavily repeated. * But it needs some additional workspace for intermediate tables. * `workSpace` must be a U32 table of size >= HIST_WKSP_SIZE_U32. * @return : largest histogram frequency, * or an error code (notably when histogram's alphabet is larger than *maxSymbolValuePtr) */ static size_t HIST_count_parallel_wksp( unsigned* count, unsigned* maxSymbolValuePtr, const void* source, size_t sourceSize, HIST_checkInput_e check, U32* const workSpace) { const BYTE* ip = (const BYTE*)source; const BYTE* const iend = ip+sourceSize; size_t const countSize = (*maxSymbolValuePtr + 1) * sizeof(*count); unsigned max=0; U32* const Counting1 = workSpace; U32* const Counting2 = Counting1 + 256; U32* const Counting3 = Counting2 + 256; U32* const Counting4 = Counting3 + 256; /* safety checks */ assert(*maxSymbolValuePtr <= 255); if (!sourceSize) { ZSTD_memset(count, 0, countSize); *maxSymbolValuePtr = 0; return 0; } ZSTD_memset(workSpace, 0, 4*256*sizeof(unsigned)); /* by stripes of 16 bytes */ { U32 cached = MEM_read32(ip); ip += 4; while (ip < iend-15) { U32 c = cached; cached = MEM_read32(ip); ip += 4; Counting1[(BYTE) c ]++; Counting2[(BYTE)(c>>8) ]++; Counting3[(BYTE)(c>>16)]++; Counting4[ c>>24 ]++; c = cached; cached = MEM_read32(ip); ip += 4; Counting1[(BYTE) c ]++; Counting2[(BYTE)(c>>8) ]++; Counting3[(BYTE)(c>>16)]++; Counting4[ c>>24 ]++; c = cached; cached = MEM_read32(ip); ip += 4; Counting1[(BYTE) c ]++; Counting2[(BYTE)(c>>8) ]++; Counting3[(BYTE)(c>>16)]++; Counting4[ c>>24 ]++; c = cached; cached = MEM_read32(ip); ip += 4; Counting1[(BYTE) c ]++; Counting2[(BYTE)(c>>8) ]++; Counting3[(BYTE)(c>>16)]++; Counting4[ c>>24 ]++; } ip-=4; } /* finish last symbols */ while (ip max) max = Counting1[s]; } } { unsigned maxSymbolValue = 255; while (!Counting1[maxSymbolValue]) maxSymbolValue--; if (check && maxSymbolValue > *maxSymbolValuePtr) return ERROR(maxSymbolValue_tooSmall); *maxSymbolValuePtr = maxSymbolValue; ZSTD_memmove(count, Counting1, countSize); /* in case count & Counting1 are overlapping */ } return (size_t)max; } /* HIST_countFast_wksp() : * Same as HIST_countFast(), but using an externally provided scratch buffer. * `workSpace` is a writable buffer which must be 4-bytes aligned, * `workSpaceSize` must be >= HIST_WKSP_SIZE */ size_t HIST_countFast_wksp(unsigned* count, unsigned* maxSymbolValuePtr, const void* source, size_t sourceSize, void* workSpace, size_t workSpaceSize) { if (sourceSize < 1500) /* heuristic threshold */ return HIST_count_simple(count, maxSymbolValuePtr, source, sourceSize); if ((size_t)workSpace & 3) return ERROR(GENERIC); /* must be aligned on 4-bytes boundaries */ if (workSpaceSize < HIST_WKSP_SIZE) return ERROR(workSpace_tooSmall); return HIST_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, trustInput, (U32*)workSpace); } /* HIST_count_wksp() : * Same as HIST_count(), but using an externally provided scratch buffer. * `workSpace` size must be table of >= HIST_WKSP_SIZE_U32 unsigned */ size_t HIST_count_wksp(unsigned* count, unsigned* maxSymbolValuePtr, const void* source, size_t sourceSize, void* workSpace, size_t workSpaceSize) { if ((size_t)workSpace & 3) return ERROR(GENERIC); /* must be aligned on 4-bytes boundaries */ if (workSpaceSize < HIST_WKSP_SIZE) return ERROR(workSpace_tooSmall); if (*maxSymbolValuePtr < 255) return HIST_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, checkMaxSymbolValue, (U32*)workSpace); *maxSymbolValuePtr = 255; return HIST_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, workSpace, workSpaceSize); } #ifndef ZSTD_NO_UNUSED_FUNCTIONS /* fast variant (unsafe : won't check if src contains values beyond count[] limit) */ size_t HIST_countFast(unsigned* count, unsigned* maxSymbolValuePtr, const void* source, size_t sourceSize) { unsigned tmpCounters[HIST_WKSP_SIZE_U32]; return HIST_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, tmpCounters, sizeof(tmpCounters)); } size_t HIST_count(unsigned* count, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize) { unsigned tmpCounters[HIST_WKSP_SIZE_U32]; return HIST_count_wksp(count, maxSymbolValuePtr, src, srcSize, tmpCounters, sizeof(tmpCounters)); } #endif zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_lazy.h0000644000175200007730000001313214515254731025347 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_LAZY_H #define ZSTD_LAZY_H #if defined (__cplusplus) extern "C" { #endif #include "zstd_compress_internal.h" /** * Dedicated Dictionary Search Structure bucket log. In the * ZSTD_dedicatedDictSearch mode, the hashTable has * 2 ** ZSTD_LAZY_DDSS_BUCKET_LOG entries in each bucket, rather than just * one. */ #define ZSTD_LAZY_DDSS_BUCKET_LOG 2 #define ZSTD_ROW_HASH_TAG_BITS 8 /* nb bits to use for the tag */ U32 ZSTD_insertAndFindFirstIndex(ZSTD_matchState_t* ms, const BYTE* ip); void ZSTD_row_update(ZSTD_matchState_t* const ms, const BYTE* ip); void ZSTD_dedicatedDictSearch_lazy_loadDictionary(ZSTD_matchState_t* ms, const BYTE* const ip); void ZSTD_preserveUnsortedMark (U32* const table, U32 const size, U32 const reducerValue); /*! used in ZSTD_reduceIndex(). preemptively increase value of ZSTD_DUBT_UNSORTED_MARK */ size_t ZSTD_compressBlock_btlazy2( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy2( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_greedy( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy2_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_greedy_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_btlazy2_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy2_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_greedy_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy2_dictMatchState_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy_dictMatchState_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_greedy_dictMatchState_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy2_dedicatedDictSearch( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy_dedicatedDictSearch( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_greedy_dedicatedDictSearch( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy2_dedicatedDictSearch_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy_dedicatedDictSearch_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_greedy_dedicatedDictSearch_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_greedy_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy2_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_greedy_extDict_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy_extDict_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_lazy2_extDict_row( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); size_t ZSTD_compressBlock_btlazy2_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); #if defined (__cplusplus) } #endif #endif /* ZSTD_LAZY_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_ldm.h0000644000175200007730000001054014515254731025144 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_LDM_H #define ZSTD_LDM_H #if defined (__cplusplus) extern "C" { #endif #include "zstd_compress_internal.h" /* ldmParams_t, U32 */ #include "../zstd.h" /* ZSTD_CCtx, size_t */ /*-************************************* * Long distance matching ***************************************/ #define ZSTD_LDM_DEFAULT_WINDOW_LOG ZSTD_WINDOWLOG_LIMIT_DEFAULT void ZSTD_ldm_fillHashTable( ldmState_t* state, const BYTE* ip, const BYTE* iend, ldmParams_t const* params); /** * ZSTD_ldm_generateSequences(): * * Generates the sequences using the long distance match finder. * Generates long range matching sequences in `sequences`, which parse a prefix * of the source. `sequences` must be large enough to store every sequence, * which can be checked with `ZSTD_ldm_getMaxNbSeq()`. * @returns 0 or an error code. * * NOTE: The user must have called ZSTD_window_update() for all of the input * they have, even if they pass it to ZSTD_ldm_generateSequences() in chunks. * NOTE: This function returns an error if it runs out of space to store * sequences. */ size_t ZSTD_ldm_generateSequences( ldmState_t* ldms, rawSeqStore_t* sequences, ldmParams_t const* params, void const* src, size_t srcSize); /** * ZSTD_ldm_blockCompress(): * * Compresses a block using the predefined sequences, along with a secondary * block compressor. The literals section of every sequence is passed to the * secondary block compressor, and those sequences are interspersed with the * predefined sequences. Returns the length of the last literals. * Updates `rawSeqStore.pos` to indicate how many sequences have been consumed. * `rawSeqStore.seq` may also be updated to split the last sequence between two * blocks. * @return The length of the last literals. * * NOTE: The source must be at most the maximum block size, but the predefined * sequences can be any size, and may be longer than the block. In the case that * they are longer than the block, the last sequences may need to be split into * two. We handle that case correctly, and update `rawSeqStore` appropriately. * NOTE: This function does not return any errors. */ size_t ZSTD_ldm_blockCompress(rawSeqStore_t* rawSeqStore, ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], ZSTD_paramSwitch_e useRowMatchFinder, void const* src, size_t srcSize); /** * ZSTD_ldm_skipSequences(): * * Skip past `srcSize` bytes worth of sequences in `rawSeqStore`. * Avoids emitting matches less than `minMatch` bytes. * Must be called for data that is not passed to ZSTD_ldm_blockCompress(). */ void ZSTD_ldm_skipSequences(rawSeqStore_t* rawSeqStore, size_t srcSize, U32 const minMatch); /* ZSTD_ldm_skipRawSeqStoreBytes(): * Moves forward in rawSeqStore by nbBytes, updating fields 'pos' and 'posInSequence'. * Not to be used in conjunction with ZSTD_ldm_skipSequences(). * Must be called for data with is not passed to ZSTD_ldm_blockCompress(). */ void ZSTD_ldm_skipRawSeqStoreBytes(rawSeqStore_t* rawSeqStore, size_t nbBytes); /** ZSTD_ldm_getTableSize() : * Estimate the space needed for long distance matching tables or 0 if LDM is * disabled. */ size_t ZSTD_ldm_getTableSize(ldmParams_t params); /** ZSTD_ldm_getSeqSpace() : * Return an upper bound on the number of sequences that can be produced by * the long distance matcher, or 0 if LDM is disabled. */ size_t ZSTD_ldm_getMaxNbSeq(ldmParams_t params, size_t maxChunkSize); /** ZSTD_ldm_adjustParameters() : * If the params->hashRateLog is not set, set it to its default value based on * windowLog and params->hashLog. * * Ensures that params->bucketSizeLog is <= params->hashLog (setting it to * params->hashLog if it is not). * * Ensures that the minMatchLength >= targetLength during optimal parsing. */ void ZSTD_ldm_adjustParameters(ldmParams_t* params, ZSTD_compressionParameters const* cParams); #if defined (__cplusplus) } #endif #endif /* ZSTD_FAST_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/compress/zstd_compress.c0000644000175200007730000114271014515254731026224 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /*-************************************* * Dependencies ***************************************/ #include "../common/allocations.h" /* ZSTD_customMalloc, ZSTD_customCalloc, ZSTD_customFree */ #include "../common/zstd_deps.h" /* INT_MAX, ZSTD_memset, ZSTD_memcpy */ #include "../common/mem.h" #include "hist.h" /* HIST_countFast_wksp */ #define FSE_STATIC_LINKING_ONLY /* FSE_encodeSymbol */ #include "../common/fse.h" #include "../common/huf.h" #include "zstd_compress_internal.h" #include "zstd_compress_sequences.h" #include "zstd_compress_literals.h" #include "zstd_fast.h" #include "zstd_double_fast.h" #include "zstd_lazy.h" #include "zstd_opt.h" #include "zstd_ldm.h" #include "zstd_compress_superblock.h" #include "../common/bits.h" /* ZSTD_highbit32, ZSTD_rotateRight_U64 */ /* *************************************************************** * Tuning parameters *****************************************************************/ /*! * COMPRESS_HEAPMODE : * Select how default decompression function ZSTD_compress() allocates its context, * on stack (0, default), or into heap (1). * Note that functions with explicit context such as ZSTD_compressCCtx() are unaffected. */ #ifndef ZSTD_COMPRESS_HEAPMODE # define ZSTD_COMPRESS_HEAPMODE 0 #endif /*! * ZSTD_HASHLOG3_MAX : * Maximum size of the hash table dedicated to find 3-bytes matches, * in log format, aka 17 => 1 << 17 == 128Ki positions. * This structure is only used in zstd_opt. * Since allocation is centralized for all strategies, it has to be known here. * The actual (selected) size of the hash table is then stored in ZSTD_matchState_t.hashLog3, * so that zstd_opt.c doesn't need to know about this constant. */ #ifndef ZSTD_HASHLOG3_MAX # define ZSTD_HASHLOG3_MAX 17 #endif /*-************************************* * Helper functions ***************************************/ /* ZSTD_compressBound() * Note that the result from this function is only valid for * the one-pass compression functions. * When employing the streaming mode, * if flushes are frequently altering the size of blocks, * the overhead from block headers can make the compressed data larger * than the return value of ZSTD_compressBound(). */ size_t ZSTD_compressBound(size_t srcSize) { size_t const r = ZSTD_COMPRESSBOUND(srcSize); if (r==0) return ERROR(srcSize_wrong); return r; } /*-************************************* * Context memory management ***************************************/ struct ZSTD_CDict_s { const void* dictContent; size_t dictContentSize; ZSTD_dictContentType_e dictContentType; /* The dictContentType the CDict was created with */ U32* entropyWorkspace; /* entropy workspace of HUF_WORKSPACE_SIZE bytes */ ZSTD_cwksp workspace; ZSTD_matchState_t matchState; ZSTD_compressedBlockState_t cBlockState; ZSTD_customMem customMem; U32 dictID; int compressionLevel; /* 0 indicates that advanced API was used to select CDict params */ ZSTD_paramSwitch_e useRowMatchFinder; /* Indicates whether the CDict was created with params that would use * row-based matchfinder. Unless the cdict is reloaded, we will use * the same greedy/lazy matchfinder at compression time. */ }; /* typedef'd to ZSTD_CDict within "zstd.h" */ ZSTD_CCtx* ZSTD_createCCtx(void) { return ZSTD_createCCtx_advanced(ZSTD_defaultCMem); } static void ZSTD_initCCtx(ZSTD_CCtx* cctx, ZSTD_customMem memManager) { assert(cctx != NULL); ZSTD_memset(cctx, 0, sizeof(*cctx)); cctx->customMem = memManager; cctx->bmi2 = ZSTD_cpuSupportsBmi2(); { size_t const err = ZSTD_CCtx_reset(cctx, ZSTD_reset_parameters); assert(!ZSTD_isError(err)); (void)err; } } ZSTD_CCtx* ZSTD_createCCtx_advanced(ZSTD_customMem customMem) { ZSTD_STATIC_ASSERT(zcss_init==0); ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_UNKNOWN==(0ULL - 1)); if ((!customMem.customAlloc) ^ (!customMem.customFree)) return NULL; { ZSTD_CCtx* const cctx = (ZSTD_CCtx*)ZSTD_customMalloc(sizeof(ZSTD_CCtx), customMem); if (!cctx) return NULL; ZSTD_initCCtx(cctx, customMem); return cctx; } } ZSTD_CCtx* ZSTD_initStaticCCtx(void* workspace, size_t workspaceSize) { ZSTD_cwksp ws; ZSTD_CCtx* cctx; if (workspaceSize <= sizeof(ZSTD_CCtx)) return NULL; /* minimum size */ if ((size_t)workspace & 7) return NULL; /* must be 8-aligned */ ZSTD_cwksp_init(&ws, workspace, workspaceSize, ZSTD_cwksp_static_alloc); cctx = (ZSTD_CCtx*)ZSTD_cwksp_reserve_object(&ws, sizeof(ZSTD_CCtx)); if (cctx == NULL) return NULL; ZSTD_memset(cctx, 0, sizeof(ZSTD_CCtx)); ZSTD_cwksp_move(&cctx->workspace, &ws); cctx->staticSize = workspaceSize; /* statically sized space. entropyWorkspace never moves (but prev/next block swap places) */ if (!ZSTD_cwksp_check_available(&cctx->workspace, ENTROPY_WORKSPACE_SIZE + 2 * sizeof(ZSTD_compressedBlockState_t))) return NULL; cctx->blockState.prevCBlock = (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object(&cctx->workspace, sizeof(ZSTD_compressedBlockState_t)); cctx->blockState.nextCBlock = (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object(&cctx->workspace, sizeof(ZSTD_compressedBlockState_t)); cctx->entropyWorkspace = (U32*)ZSTD_cwksp_reserve_object(&cctx->workspace, ENTROPY_WORKSPACE_SIZE); cctx->bmi2 = ZSTD_cpuid_bmi2(ZSTD_cpuid()); return cctx; } /** * Clears and frees all of the dictionaries in the CCtx. */ static void ZSTD_clearAllDicts(ZSTD_CCtx* cctx) { ZSTD_customFree(cctx->localDict.dictBuffer, cctx->customMem); ZSTD_freeCDict(cctx->localDict.cdict); ZSTD_memset(&cctx->localDict, 0, sizeof(cctx->localDict)); ZSTD_memset(&cctx->prefixDict, 0, sizeof(cctx->prefixDict)); cctx->cdict = NULL; } static size_t ZSTD_sizeof_localDict(ZSTD_localDict dict) { size_t const bufferSize = dict.dictBuffer != NULL ? dict.dictSize : 0; size_t const cdictSize = ZSTD_sizeof_CDict(dict.cdict); return bufferSize + cdictSize; } static void ZSTD_freeCCtxContent(ZSTD_CCtx* cctx) { assert(cctx != NULL); assert(cctx->staticSize == 0); ZSTD_clearAllDicts(cctx); #ifdef ZSTD_MULTITHREAD ZSTDMT_freeCCtx(cctx->mtctx); cctx->mtctx = NULL; #endif ZSTD_cwksp_free(&cctx->workspace, cctx->customMem); } size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx) { if (cctx==NULL) return 0; /* support free on NULL */ RETURN_ERROR_IF(cctx->staticSize, memory_allocation, "not compatible with static CCtx"); { int cctxInWorkspace = ZSTD_cwksp_owns_buffer(&cctx->workspace, cctx); ZSTD_freeCCtxContent(cctx); if (!cctxInWorkspace) ZSTD_customFree(cctx, cctx->customMem); } return 0; } static size_t ZSTD_sizeof_mtctx(const ZSTD_CCtx* cctx) { #ifdef ZSTD_MULTITHREAD return ZSTDMT_sizeof_CCtx(cctx->mtctx); #else (void)cctx; return 0; #endif } size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx) { if (cctx==NULL) return 0; /* support sizeof on NULL */ /* cctx may be in the workspace */ return (cctx->workspace.workspace == cctx ? 0 : sizeof(*cctx)) + ZSTD_cwksp_sizeof(&cctx->workspace) + ZSTD_sizeof_localDict(cctx->localDict) + ZSTD_sizeof_mtctx(cctx); } size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs) { return ZSTD_sizeof_CCtx(zcs); /* same object */ } /* private API call, for dictBuilder only */ const seqStore_t* ZSTD_getSeqStore(const ZSTD_CCtx* ctx) { return &(ctx->seqStore); } /* Returns true if the strategy supports using a row based matchfinder */ static int ZSTD_rowMatchFinderSupported(const ZSTD_strategy strategy) { return (strategy >= ZSTD_greedy && strategy <= ZSTD_lazy2); } /* Returns true if the strategy and useRowMatchFinder mode indicate that we will use the row based matchfinder * for this compression. */ static int ZSTD_rowMatchFinderUsed(const ZSTD_strategy strategy, const ZSTD_paramSwitch_e mode) { assert(mode != ZSTD_ps_auto); return ZSTD_rowMatchFinderSupported(strategy) && (mode == ZSTD_ps_enable); } /* Returns row matchfinder usage given an initial mode and cParams */ static ZSTD_paramSwitch_e ZSTD_resolveRowMatchFinderMode(ZSTD_paramSwitch_e mode, const ZSTD_compressionParameters* const cParams) { #if defined(ZSTD_ARCH_X86_SSE2) || defined(ZSTD_ARCH_ARM_NEON) int const kHasSIMD128 = 1; #else int const kHasSIMD128 = 0; #endif if (mode != ZSTD_ps_auto) return mode; /* if requested enabled, but no SIMD, we still will use row matchfinder */ mode = ZSTD_ps_disable; if (!ZSTD_rowMatchFinderSupported(cParams->strategy)) return mode; if (kHasSIMD128) { if (cParams->windowLog > 14) mode = ZSTD_ps_enable; } else { if (cParams->windowLog > 17) mode = ZSTD_ps_enable; } return mode; } /* Returns block splitter usage (generally speaking, when using slower/stronger compression modes) */ static ZSTD_paramSwitch_e ZSTD_resolveBlockSplitterMode(ZSTD_paramSwitch_e mode, const ZSTD_compressionParameters* const cParams) { if (mode != ZSTD_ps_auto) return mode; return (cParams->strategy >= ZSTD_btopt && cParams->windowLog >= 17) ? ZSTD_ps_enable : ZSTD_ps_disable; } /* Returns 1 if the arguments indicate that we should allocate a chainTable, 0 otherwise */ static int ZSTD_allocateChainTable(const ZSTD_strategy strategy, const ZSTD_paramSwitch_e useRowMatchFinder, const U32 forDDSDict) { assert(useRowMatchFinder != ZSTD_ps_auto); /* We always should allocate a chaintable if we are allocating a matchstate for a DDS dictionary matchstate. * We do not allocate a chaintable if we are using ZSTD_fast, or are using the row-based matchfinder. */ return forDDSDict || ((strategy != ZSTD_fast) && !ZSTD_rowMatchFinderUsed(strategy, useRowMatchFinder)); } /* Returns ZSTD_ps_enable if compression parameters are such that we should * enable long distance matching (wlog >= 27, strategy >= btopt). * Returns ZSTD_ps_disable otherwise. */ static ZSTD_paramSwitch_e ZSTD_resolveEnableLdm(ZSTD_paramSwitch_e mode, const ZSTD_compressionParameters* const cParams) { if (mode != ZSTD_ps_auto) return mode; return (cParams->strategy >= ZSTD_btopt && cParams->windowLog >= 27) ? ZSTD_ps_enable : ZSTD_ps_disable; } static int ZSTD_resolveExternalSequenceValidation(int mode) { return mode; } /* Resolves maxBlockSize to the default if no value is present. */ static size_t ZSTD_resolveMaxBlockSize(size_t maxBlockSize) { if (maxBlockSize == 0) { return ZSTD_BLOCKSIZE_MAX; } else { return maxBlockSize; } } static ZSTD_paramSwitch_e ZSTD_resolveExternalRepcodeSearch(ZSTD_paramSwitch_e value, int cLevel) { if (value != ZSTD_ps_auto) return value; if (cLevel < 10) { return ZSTD_ps_disable; } else { return ZSTD_ps_enable; } } /* Returns 1 if compression parameters are such that CDict hashtable and chaintable indices are tagged. * If so, the tags need to be removed in ZSTD_resetCCtx_byCopyingCDict. */ static int ZSTD_CDictIndicesAreTagged(const ZSTD_compressionParameters* const cParams) { return cParams->strategy == ZSTD_fast || cParams->strategy == ZSTD_dfast; } static ZSTD_CCtx_params ZSTD_makeCCtxParamsFromCParams( ZSTD_compressionParameters cParams) { ZSTD_CCtx_params cctxParams; /* should not matter, as all cParams are presumed properly defined */ ZSTD_CCtxParams_init(&cctxParams, ZSTD_CLEVEL_DEFAULT); cctxParams.cParams = cParams; /* Adjust advanced params according to cParams */ cctxParams.ldmParams.enableLdm = ZSTD_resolveEnableLdm(cctxParams.ldmParams.enableLdm, &cParams); if (cctxParams.ldmParams.enableLdm == ZSTD_ps_enable) { ZSTD_ldm_adjustParameters(&cctxParams.ldmParams, &cParams); assert(cctxParams.ldmParams.hashLog >= cctxParams.ldmParams.bucketSizeLog); assert(cctxParams.ldmParams.hashRateLog < 32); } cctxParams.useBlockSplitter = ZSTD_resolveBlockSplitterMode(cctxParams.useBlockSplitter, &cParams); cctxParams.useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(cctxParams.useRowMatchFinder, &cParams); cctxParams.validateSequences = ZSTD_resolveExternalSequenceValidation(cctxParams.validateSequences); cctxParams.maxBlockSize = ZSTD_resolveMaxBlockSize(cctxParams.maxBlockSize); cctxParams.searchForExternalRepcodes = ZSTD_resolveExternalRepcodeSearch(cctxParams.searchForExternalRepcodes, cctxParams.compressionLevel); assert(!ZSTD_checkCParams(cParams)); return cctxParams; } static ZSTD_CCtx_params* ZSTD_createCCtxParams_advanced( ZSTD_customMem customMem) { ZSTD_CCtx_params* params; if ((!customMem.customAlloc) ^ (!customMem.customFree)) return NULL; params = (ZSTD_CCtx_params*)ZSTD_customCalloc( sizeof(ZSTD_CCtx_params), customMem); if (!params) { return NULL; } ZSTD_CCtxParams_init(params, ZSTD_CLEVEL_DEFAULT); params->customMem = customMem; return params; } ZSTD_CCtx_params* ZSTD_createCCtxParams(void) { return ZSTD_createCCtxParams_advanced(ZSTD_defaultCMem); } size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params) { if (params == NULL) { return 0; } ZSTD_customFree(params, params->customMem); return 0; } size_t ZSTD_CCtxParams_reset(ZSTD_CCtx_params* params) { return ZSTD_CCtxParams_init(params, ZSTD_CLEVEL_DEFAULT); } size_t ZSTD_CCtxParams_init(ZSTD_CCtx_params* cctxParams, int compressionLevel) { RETURN_ERROR_IF(!cctxParams, GENERIC, "NULL pointer!"); ZSTD_memset(cctxParams, 0, sizeof(*cctxParams)); cctxParams->compressionLevel = compressionLevel; cctxParams->fParams.contentSizeFlag = 1; return 0; } #define ZSTD_NO_CLEVEL 0 /** * Initializes `cctxParams` from `params` and `compressionLevel`. * @param compressionLevel If params are derived from a compression level then that compression level, otherwise ZSTD_NO_CLEVEL. */ static void ZSTD_CCtxParams_init_internal(ZSTD_CCtx_params* cctxParams, const ZSTD_parameters* params, int compressionLevel) { assert(!ZSTD_checkCParams(params->cParams)); ZSTD_memset(cctxParams, 0, sizeof(*cctxParams)); cctxParams->cParams = params->cParams; cctxParams->fParams = params->fParams; /* Should not matter, as all cParams are presumed properly defined. * But, set it for tracing anyway. */ cctxParams->compressionLevel = compressionLevel; cctxParams->useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(cctxParams->useRowMatchFinder, ¶ms->cParams); cctxParams->useBlockSplitter = ZSTD_resolveBlockSplitterMode(cctxParams->useBlockSplitter, ¶ms->cParams); cctxParams->ldmParams.enableLdm = ZSTD_resolveEnableLdm(cctxParams->ldmParams.enableLdm, ¶ms->cParams); cctxParams->validateSequences = ZSTD_resolveExternalSequenceValidation(cctxParams->validateSequences); cctxParams->maxBlockSize = ZSTD_resolveMaxBlockSize(cctxParams->maxBlockSize); cctxParams->searchForExternalRepcodes = ZSTD_resolveExternalRepcodeSearch(cctxParams->searchForExternalRepcodes, compressionLevel); DEBUGLOG(4, "ZSTD_CCtxParams_init_internal: useRowMatchFinder=%d, useBlockSplitter=%d ldm=%d", cctxParams->useRowMatchFinder, cctxParams->useBlockSplitter, cctxParams->ldmParams.enableLdm); } size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params) { RETURN_ERROR_IF(!cctxParams, GENERIC, "NULL pointer!"); FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) , ""); ZSTD_CCtxParams_init_internal(cctxParams, ¶ms, ZSTD_NO_CLEVEL); return 0; } /** * Sets cctxParams' cParams and fParams from params, but otherwise leaves them alone. * @param params Validated zstd parameters. */ static void ZSTD_CCtxParams_setZstdParams( ZSTD_CCtx_params* cctxParams, const ZSTD_parameters* params) { assert(!ZSTD_checkCParams(params->cParams)); cctxParams->cParams = params->cParams; cctxParams->fParams = params->fParams; /* Should not matter, as all cParams are presumed properly defined. * But, set it for tracing anyway. */ cctxParams->compressionLevel = ZSTD_NO_CLEVEL; } ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter param) { ZSTD_bounds bounds = { 0, 0, 0 }; switch(param) { case ZSTD_c_compressionLevel: bounds.lowerBound = ZSTD_minCLevel(); bounds.upperBound = ZSTD_maxCLevel(); return bounds; case ZSTD_c_windowLog: bounds.lowerBound = ZSTD_WINDOWLOG_MIN; bounds.upperBound = ZSTD_WINDOWLOG_MAX; return bounds; case ZSTD_c_hashLog: bounds.lowerBound = ZSTD_HASHLOG_MIN; bounds.upperBound = ZSTD_HASHLOG_MAX; return bounds; case ZSTD_c_chainLog: bounds.lowerBound = ZSTD_CHAINLOG_MIN; bounds.upperBound = ZSTD_CHAINLOG_MAX; return bounds; case ZSTD_c_searchLog: bounds.lowerBound = ZSTD_SEARCHLOG_MIN; bounds.upperBound = ZSTD_SEARCHLOG_MAX; return bounds; case ZSTD_c_minMatch: bounds.lowerBound = ZSTD_MINMATCH_MIN; bounds.upperBound = ZSTD_MINMATCH_MAX; return bounds; case ZSTD_c_targetLength: bounds.lowerBound = ZSTD_TARGETLENGTH_MIN; bounds.upperBound = ZSTD_TARGETLENGTH_MAX; return bounds; case ZSTD_c_strategy: bounds.lowerBound = ZSTD_STRATEGY_MIN; bounds.upperBound = ZSTD_STRATEGY_MAX; return bounds; case ZSTD_c_contentSizeFlag: bounds.lowerBound = 0; bounds.upperBound = 1; return bounds; case ZSTD_c_checksumFlag: bounds.lowerBound = 0; bounds.upperBound = 1; return bounds; case ZSTD_c_dictIDFlag: bounds.lowerBound = 0; bounds.upperBound = 1; return bounds; case ZSTD_c_nbWorkers: bounds.lowerBound = 0; #ifdef ZSTD_MULTITHREAD bounds.upperBound = ZSTDMT_NBWORKERS_MAX; #else bounds.upperBound = 0; #endif return bounds; case ZSTD_c_jobSize: bounds.lowerBound = 0; #ifdef ZSTD_MULTITHREAD bounds.upperBound = ZSTDMT_JOBSIZE_MAX; #else bounds.upperBound = 0; #endif return bounds; case ZSTD_c_overlapLog: #ifdef ZSTD_MULTITHREAD bounds.lowerBound = ZSTD_OVERLAPLOG_MIN; bounds.upperBound = ZSTD_OVERLAPLOG_MAX; #else bounds.lowerBound = 0; bounds.upperBound = 0; #endif return bounds; case ZSTD_c_enableDedicatedDictSearch: bounds.lowerBound = 0; bounds.upperBound = 1; return bounds; case ZSTD_c_enableLongDistanceMatching: bounds.lowerBound = (int)ZSTD_ps_auto; bounds.upperBound = (int)ZSTD_ps_disable; return bounds; case ZSTD_c_ldmHashLog: bounds.lowerBound = ZSTD_LDM_HASHLOG_MIN; bounds.upperBound = ZSTD_LDM_HASHLOG_MAX; return bounds; case ZSTD_c_ldmMinMatch: bounds.lowerBound = ZSTD_LDM_MINMATCH_MIN; bounds.upperBound = ZSTD_LDM_MINMATCH_MAX; return bounds; case ZSTD_c_ldmBucketSizeLog: bounds.lowerBound = ZSTD_LDM_BUCKETSIZELOG_MIN; bounds.upperBound = ZSTD_LDM_BUCKETSIZELOG_MAX; return bounds; case ZSTD_c_ldmHashRateLog: bounds.lowerBound = ZSTD_LDM_HASHRATELOG_MIN; bounds.upperBound = ZSTD_LDM_HASHRATELOG_MAX; return bounds; /* experimental parameters */ case ZSTD_c_rsyncable: bounds.lowerBound = 0; bounds.upperBound = 1; return bounds; case ZSTD_c_forceMaxWindow : bounds.lowerBound = 0; bounds.upperBound = 1; return bounds; case ZSTD_c_format: ZSTD_STATIC_ASSERT(ZSTD_f_zstd1 < ZSTD_f_zstd1_magicless); bounds.lowerBound = ZSTD_f_zstd1; bounds.upperBound = ZSTD_f_zstd1_magicless; /* note : how to ensure at compile time that this is the highest value enum ? */ return bounds; case ZSTD_c_forceAttachDict: ZSTD_STATIC_ASSERT(ZSTD_dictDefaultAttach < ZSTD_dictForceLoad); bounds.lowerBound = ZSTD_dictDefaultAttach; bounds.upperBound = ZSTD_dictForceLoad; /* note : how to ensure at compile time that this is the highest value enum ? */ return bounds; case ZSTD_c_literalCompressionMode: ZSTD_STATIC_ASSERT(ZSTD_ps_auto < ZSTD_ps_enable && ZSTD_ps_enable < ZSTD_ps_disable); bounds.lowerBound = (int)ZSTD_ps_auto; bounds.upperBound = (int)ZSTD_ps_disable; return bounds; case ZSTD_c_targetCBlockSize: bounds.lowerBound = ZSTD_TARGETCBLOCKSIZE_MIN; bounds.upperBound = ZSTD_TARGETCBLOCKSIZE_MAX; return bounds; case ZSTD_c_srcSizeHint: bounds.lowerBound = ZSTD_SRCSIZEHINT_MIN; bounds.upperBound = ZSTD_SRCSIZEHINT_MAX; return bounds; case ZSTD_c_stableInBuffer: case ZSTD_c_stableOutBuffer: bounds.lowerBound = (int)ZSTD_bm_buffered; bounds.upperBound = (int)ZSTD_bm_stable; return bounds; case ZSTD_c_blockDelimiters: bounds.lowerBound = (int)ZSTD_sf_noBlockDelimiters; bounds.upperBound = (int)ZSTD_sf_explicitBlockDelimiters; return bounds; case ZSTD_c_validateSequences: bounds.lowerBound = 0; bounds.upperBound = 1; return bounds; case ZSTD_c_useBlockSplitter: bounds.lowerBound = (int)ZSTD_ps_auto; bounds.upperBound = (int)ZSTD_ps_disable; return bounds; case ZSTD_c_useRowMatchFinder: bounds.lowerBound = (int)ZSTD_ps_auto; bounds.upperBound = (int)ZSTD_ps_disable; return bounds; case ZSTD_c_deterministicRefPrefix: bounds.lowerBound = 0; bounds.upperBound = 1; return bounds; case ZSTD_c_prefetchCDictTables: bounds.lowerBound = (int)ZSTD_ps_auto; bounds.upperBound = (int)ZSTD_ps_disable; return bounds; case ZSTD_c_enableSeqProducerFallback: bounds.lowerBound = 0; bounds.upperBound = 1; return bounds; case ZSTD_c_maxBlockSize: bounds.lowerBound = ZSTD_BLOCKSIZE_MAX_MIN; bounds.upperBound = ZSTD_BLOCKSIZE_MAX; return bounds; case ZSTD_c_searchForExternalRepcodes: bounds.lowerBound = (int)ZSTD_ps_auto; bounds.upperBound = (int)ZSTD_ps_disable; return bounds; default: bounds.error = ERROR(parameter_unsupported); return bounds; } } /* ZSTD_cParam_clampBounds: * Clamps the value into the bounded range. */ static size_t ZSTD_cParam_clampBounds(ZSTD_cParameter cParam, int* value) { ZSTD_bounds const bounds = ZSTD_cParam_getBounds(cParam); if (ZSTD_isError(bounds.error)) return bounds.error; if (*value < bounds.lowerBound) *value = bounds.lowerBound; if (*value > bounds.upperBound) *value = bounds.upperBound; return 0; } #define BOUNDCHECK(cParam, val) { \ RETURN_ERROR_IF(!ZSTD_cParam_withinBounds(cParam,val), \ parameter_outOfBound, "Param out of bounds"); \ } static int ZSTD_isUpdateAuthorized(ZSTD_cParameter param) { switch(param) { case ZSTD_c_compressionLevel: case ZSTD_c_hashLog: case ZSTD_c_chainLog: case ZSTD_c_searchLog: case ZSTD_c_minMatch: case ZSTD_c_targetLength: case ZSTD_c_strategy: return 1; case ZSTD_c_format: case ZSTD_c_windowLog: case ZSTD_c_contentSizeFlag: case ZSTD_c_checksumFlag: case ZSTD_c_dictIDFlag: case ZSTD_c_forceMaxWindow : case ZSTD_c_nbWorkers: case ZSTD_c_jobSize: case ZSTD_c_overlapLog: case ZSTD_c_rsyncable: case ZSTD_c_enableDedicatedDictSearch: case ZSTD_c_enableLongDistanceMatching: case ZSTD_c_ldmHashLog: case ZSTD_c_ldmMinMatch: case ZSTD_c_ldmBucketSizeLog: case ZSTD_c_ldmHashRateLog: case ZSTD_c_forceAttachDict: case ZSTD_c_literalCompressionMode: case ZSTD_c_targetCBlockSize: case ZSTD_c_srcSizeHint: case ZSTD_c_stableInBuffer: case ZSTD_c_stableOutBuffer: case ZSTD_c_blockDelimiters: case ZSTD_c_validateSequences: case ZSTD_c_useBlockSplitter: case ZSTD_c_useRowMatchFinder: case ZSTD_c_deterministicRefPrefix: case ZSTD_c_prefetchCDictTables: case ZSTD_c_enableSeqProducerFallback: case ZSTD_c_maxBlockSize: case ZSTD_c_searchForExternalRepcodes: default: return 0; } } size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value) { DEBUGLOG(4, "ZSTD_CCtx_setParameter (%i, %i)", (int)param, value); if (cctx->streamStage != zcss_init) { if (ZSTD_isUpdateAuthorized(param)) { cctx->cParamsChanged = 1; } else { RETURN_ERROR(stage_wrong, "can only set params in cctx init stage"); } } switch(param) { case ZSTD_c_nbWorkers: RETURN_ERROR_IF((value!=0) && cctx->staticSize, parameter_unsupported, "MT not compatible with static alloc"); break; case ZSTD_c_compressionLevel: case ZSTD_c_windowLog: case ZSTD_c_hashLog: case ZSTD_c_chainLog: case ZSTD_c_searchLog: case ZSTD_c_minMatch: case ZSTD_c_targetLength: case ZSTD_c_strategy: case ZSTD_c_ldmHashRateLog: case ZSTD_c_format: case ZSTD_c_contentSizeFlag: case ZSTD_c_checksumFlag: case ZSTD_c_dictIDFlag: case ZSTD_c_forceMaxWindow: case ZSTD_c_forceAttachDict: case ZSTD_c_literalCompressionMode: case ZSTD_c_jobSize: case ZSTD_c_overlapLog: case ZSTD_c_rsyncable: case ZSTD_c_enableDedicatedDictSearch: case ZSTD_c_enableLongDistanceMatching: case ZSTD_c_ldmHashLog: case ZSTD_c_ldmMinMatch: case ZSTD_c_ldmBucketSizeLog: case ZSTD_c_targetCBlockSize: case ZSTD_c_srcSizeHint: case ZSTD_c_stableInBuffer: case ZSTD_c_stableOutBuffer: case ZSTD_c_blockDelimiters: case ZSTD_c_validateSequences: case ZSTD_c_useBlockSplitter: case ZSTD_c_useRowMatchFinder: case ZSTD_c_deterministicRefPrefix: case ZSTD_c_prefetchCDictTables: case ZSTD_c_enableSeqProducerFallback: case ZSTD_c_maxBlockSize: case ZSTD_c_searchForExternalRepcodes: break; default: RETURN_ERROR(parameter_unsupported, "unknown parameter"); } return ZSTD_CCtxParams_setParameter(&cctx->requestedParams, param, value); } size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams, ZSTD_cParameter param, int value) { DEBUGLOG(4, "ZSTD_CCtxParams_setParameter (%i, %i)", (int)param, value); switch(param) { case ZSTD_c_format : BOUNDCHECK(ZSTD_c_format, value); CCtxParams->format = (ZSTD_format_e)value; return (size_t)CCtxParams->format; case ZSTD_c_compressionLevel : { FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value), ""); if (value == 0) CCtxParams->compressionLevel = ZSTD_CLEVEL_DEFAULT; /* 0 == default */ else CCtxParams->compressionLevel = value; if (CCtxParams->compressionLevel >= 0) return (size_t)CCtxParams->compressionLevel; return 0; /* return type (size_t) cannot represent negative values */ } case ZSTD_c_windowLog : if (value!=0) /* 0 => use default */ BOUNDCHECK(ZSTD_c_windowLog, value); CCtxParams->cParams.windowLog = (U32)value; return CCtxParams->cParams.windowLog; case ZSTD_c_hashLog : if (value!=0) /* 0 => use default */ BOUNDCHECK(ZSTD_c_hashLog, value); CCtxParams->cParams.hashLog = (U32)value; return CCtxParams->cParams.hashLog; case ZSTD_c_chainLog : if (value!=0) /* 0 => use default */ BOUNDCHECK(ZSTD_c_chainLog, value); CCtxParams->cParams.chainLog = (U32)value; return CCtxParams->cParams.chainLog; case ZSTD_c_searchLog : if (value!=0) /* 0 => use default */ BOUNDCHECK(ZSTD_c_searchLog, value); CCtxParams->cParams.searchLog = (U32)value; return (size_t)value; case ZSTD_c_minMatch : if (value!=0) /* 0 => use default */ BOUNDCHECK(ZSTD_c_minMatch, value); CCtxParams->cParams.minMatch = (U32)value; return CCtxParams->cParams.minMatch; case ZSTD_c_targetLength : BOUNDCHECK(ZSTD_c_targetLength, value); CCtxParams->cParams.targetLength = (U32)value; return CCtxParams->cParams.targetLength; case ZSTD_c_strategy : if (value!=0) /* 0 => use default */ BOUNDCHECK(ZSTD_c_strategy, value); CCtxParams->cParams.strategy = (ZSTD_strategy)value; return (size_t)CCtxParams->cParams.strategy; case ZSTD_c_contentSizeFlag : /* Content size written in frame header _when known_ (default:1) */ DEBUGLOG(4, "set content size flag = %u", (value!=0)); CCtxParams->fParams.contentSizeFlag = value != 0; return (size_t)CCtxParams->fParams.contentSizeFlag; case ZSTD_c_checksumFlag : /* A 32-bits content checksum will be calculated and written at end of frame (default:0) */ CCtxParams->fParams.checksumFlag = value != 0; return (size_t)CCtxParams->fParams.checksumFlag; case ZSTD_c_dictIDFlag : /* When applicable, dictionary's dictID is provided in frame header (default:1) */ DEBUGLOG(4, "set dictIDFlag = %u", (value!=0)); CCtxParams->fParams.noDictIDFlag = !value; return !CCtxParams->fParams.noDictIDFlag; case ZSTD_c_forceMaxWindow : CCtxParams->forceWindow = (value != 0); return (size_t)CCtxParams->forceWindow; case ZSTD_c_forceAttachDict : { const ZSTD_dictAttachPref_e pref = (ZSTD_dictAttachPref_e)value; BOUNDCHECK(ZSTD_c_forceAttachDict, (int)pref); CCtxParams->attachDictPref = pref; return CCtxParams->attachDictPref; } case ZSTD_c_literalCompressionMode : { const ZSTD_paramSwitch_e lcm = (ZSTD_paramSwitch_e)value; BOUNDCHECK(ZSTD_c_literalCompressionMode, (int)lcm); CCtxParams->literalCompressionMode = lcm; return CCtxParams->literalCompressionMode; } case ZSTD_c_nbWorkers : #ifndef ZSTD_MULTITHREAD RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading"); return 0; #else FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value), ""); CCtxParams->nbWorkers = value; return CCtxParams->nbWorkers; #endif case ZSTD_c_jobSize : #ifndef ZSTD_MULTITHREAD RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading"); return 0; #else /* Adjust to the minimum non-default value. */ if (value != 0 && value < ZSTDMT_JOBSIZE_MIN) value = ZSTDMT_JOBSIZE_MIN; FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value), ""); assert(value >= 0); CCtxParams->jobSize = value; return CCtxParams->jobSize; #endif case ZSTD_c_overlapLog : #ifndef ZSTD_MULTITHREAD RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading"); return 0; #else FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(ZSTD_c_overlapLog, &value), ""); CCtxParams->overlapLog = value; return CCtxParams->overlapLog; #endif case ZSTD_c_rsyncable : #ifndef ZSTD_MULTITHREAD RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading"); return 0; #else FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(ZSTD_c_overlapLog, &value), ""); CCtxParams->rsyncable = value; return CCtxParams->rsyncable; #endif case ZSTD_c_enableDedicatedDictSearch : CCtxParams->enableDedicatedDictSearch = (value!=0); return (size_t)CCtxParams->enableDedicatedDictSearch; case ZSTD_c_enableLongDistanceMatching : BOUNDCHECK(ZSTD_c_enableLongDistanceMatching, value); CCtxParams->ldmParams.enableLdm = (ZSTD_paramSwitch_e)value; return CCtxParams->ldmParams.enableLdm; case ZSTD_c_ldmHashLog : if (value!=0) /* 0 ==> auto */ BOUNDCHECK(ZSTD_c_ldmHashLog, value); CCtxParams->ldmParams.hashLog = (U32)value; return CCtxParams->ldmParams.hashLog; case ZSTD_c_ldmMinMatch : if (value!=0) /* 0 ==> default */ BOUNDCHECK(ZSTD_c_ldmMinMatch, value); CCtxParams->ldmParams.minMatchLength = (U32)value; return CCtxParams->ldmParams.minMatchLength; case ZSTD_c_ldmBucketSizeLog : if (value!=0) /* 0 ==> default */ BOUNDCHECK(ZSTD_c_ldmBucketSizeLog, value); CCtxParams->ldmParams.bucketSizeLog = (U32)value; return CCtxParams->ldmParams.bucketSizeLog; case ZSTD_c_ldmHashRateLog : if (value!=0) /* 0 ==> default */ BOUNDCHECK(ZSTD_c_ldmHashRateLog, value); CCtxParams->ldmParams.hashRateLog = (U32)value; return CCtxParams->ldmParams.hashRateLog; case ZSTD_c_targetCBlockSize : if (value!=0) /* 0 ==> default */ BOUNDCHECK(ZSTD_c_targetCBlockSize, value); CCtxParams->targetCBlockSize = (U32)value; return CCtxParams->targetCBlockSize; case ZSTD_c_srcSizeHint : if (value!=0) /* 0 ==> default */ BOUNDCHECK(ZSTD_c_srcSizeHint, value); CCtxParams->srcSizeHint = value; return (size_t)CCtxParams->srcSizeHint; case ZSTD_c_stableInBuffer: BOUNDCHECK(ZSTD_c_stableInBuffer, value); CCtxParams->inBufferMode = (ZSTD_bufferMode_e)value; return CCtxParams->inBufferMode; case ZSTD_c_stableOutBuffer: BOUNDCHECK(ZSTD_c_stableOutBuffer, value); CCtxParams->outBufferMode = (ZSTD_bufferMode_e)value; return CCtxParams->outBufferMode; case ZSTD_c_blockDelimiters: BOUNDCHECK(ZSTD_c_blockDelimiters, value); CCtxParams->blockDelimiters = (ZSTD_sequenceFormat_e)value; return CCtxParams->blockDelimiters; case ZSTD_c_validateSequences: BOUNDCHECK(ZSTD_c_validateSequences, value); CCtxParams->validateSequences = value; return CCtxParams->validateSequences; case ZSTD_c_useBlockSplitter: BOUNDCHECK(ZSTD_c_useBlockSplitter, value); CCtxParams->useBlockSplitter = (ZSTD_paramSwitch_e)value; return CCtxParams->useBlockSplitter; case ZSTD_c_useRowMatchFinder: BOUNDCHECK(ZSTD_c_useRowMatchFinder, value); CCtxParams->useRowMatchFinder = (ZSTD_paramSwitch_e)value; return CCtxParams->useRowMatchFinder; case ZSTD_c_deterministicRefPrefix: BOUNDCHECK(ZSTD_c_deterministicRefPrefix, value); CCtxParams->deterministicRefPrefix = !!value; return CCtxParams->deterministicRefPrefix; case ZSTD_c_prefetchCDictTables: BOUNDCHECK(ZSTD_c_prefetchCDictTables, value); CCtxParams->prefetchCDictTables = (ZSTD_paramSwitch_e)value; return CCtxParams->prefetchCDictTables; case ZSTD_c_enableSeqProducerFallback: BOUNDCHECK(ZSTD_c_enableSeqProducerFallback, value); CCtxParams->enableMatchFinderFallback = value; return CCtxParams->enableMatchFinderFallback; case ZSTD_c_maxBlockSize: if (value!=0) /* 0 ==> default */ BOUNDCHECK(ZSTD_c_maxBlockSize, value); CCtxParams->maxBlockSize = value; return CCtxParams->maxBlockSize; case ZSTD_c_searchForExternalRepcodes: BOUNDCHECK(ZSTD_c_searchForExternalRepcodes, value); CCtxParams->searchForExternalRepcodes = (ZSTD_paramSwitch_e)value; return CCtxParams->searchForExternalRepcodes; default: RETURN_ERROR(parameter_unsupported, "unknown parameter"); } } size_t ZSTD_CCtx_getParameter(ZSTD_CCtx const* cctx, ZSTD_cParameter param, int* value) { return ZSTD_CCtxParams_getParameter(&cctx->requestedParams, param, value); } size_t ZSTD_CCtxParams_getParameter( ZSTD_CCtx_params const* CCtxParams, ZSTD_cParameter param, int* value) { switch(param) { case ZSTD_c_format : *value = CCtxParams->format; break; case ZSTD_c_compressionLevel : *value = CCtxParams->compressionLevel; break; case ZSTD_c_windowLog : *value = (int)CCtxParams->cParams.windowLog; break; case ZSTD_c_hashLog : *value = (int)CCtxParams->cParams.hashLog; break; case ZSTD_c_chainLog : *value = (int)CCtxParams->cParams.chainLog; break; case ZSTD_c_searchLog : *value = CCtxParams->cParams.searchLog; break; case ZSTD_c_minMatch : *value = CCtxParams->cParams.minMatch; break; case ZSTD_c_targetLength : *value = CCtxParams->cParams.targetLength; break; case ZSTD_c_strategy : *value = (unsigned)CCtxParams->cParams.strategy; break; case ZSTD_c_contentSizeFlag : *value = CCtxParams->fParams.contentSizeFlag; break; case ZSTD_c_checksumFlag : *value = CCtxParams->fParams.checksumFlag; break; case ZSTD_c_dictIDFlag : *value = !CCtxParams->fParams.noDictIDFlag; break; case ZSTD_c_forceMaxWindow : *value = CCtxParams->forceWindow; break; case ZSTD_c_forceAttachDict : *value = CCtxParams->attachDictPref; break; case ZSTD_c_literalCompressionMode : *value = CCtxParams->literalCompressionMode; break; case ZSTD_c_nbWorkers : #ifndef ZSTD_MULTITHREAD assert(CCtxParams->nbWorkers == 0); #endif *value = CCtxParams->nbWorkers; break; case ZSTD_c_jobSize : #ifndef ZSTD_MULTITHREAD RETURN_ERROR(parameter_unsupported, "not compiled with multithreading"); #else assert(CCtxParams->jobSize <= INT_MAX); *value = (int)CCtxParams->jobSize; break; #endif case ZSTD_c_overlapLog : #ifndef ZSTD_MULTITHREAD RETURN_ERROR(parameter_unsupported, "not compiled with multithreading"); #else *value = CCtxParams->overlapLog; break; #endif case ZSTD_c_rsyncable : #ifndef ZSTD_MULTITHREAD RETURN_ERROR(parameter_unsupported, "not compiled with multithreading"); #else *value = CCtxParams->rsyncable; break; #endif case ZSTD_c_enableDedicatedDictSearch : *value = CCtxParams->enableDedicatedDictSearch; break; case ZSTD_c_enableLongDistanceMatching : *value = CCtxParams->ldmParams.enableLdm; break; case ZSTD_c_ldmHashLog : *value = CCtxParams->ldmParams.hashLog; break; case ZSTD_c_ldmMinMatch : *value = CCtxParams->ldmParams.minMatchLength; break; case ZSTD_c_ldmBucketSizeLog : *value = CCtxParams->ldmParams.bucketSizeLog; break; case ZSTD_c_ldmHashRateLog : *value = CCtxParams->ldmParams.hashRateLog; break; case ZSTD_c_targetCBlockSize : *value = (int)CCtxParams->targetCBlockSize; break; case ZSTD_c_srcSizeHint : *value = (int)CCtxParams->srcSizeHint; break; case ZSTD_c_stableInBuffer : *value = (int)CCtxParams->inBufferMode; break; case ZSTD_c_stableOutBuffer : *value = (int)CCtxParams->outBufferMode; break; case ZSTD_c_blockDelimiters : *value = (int)CCtxParams->blockDelimiters; break; case ZSTD_c_validateSequences : *value = (int)CCtxParams->validateSequences; break; case ZSTD_c_useBlockSplitter : *value = (int)CCtxParams->useBlockSplitter; break; case ZSTD_c_useRowMatchFinder : *value = (int)CCtxParams->useRowMatchFinder; break; case ZSTD_c_deterministicRefPrefix: *value = (int)CCtxParams->deterministicRefPrefix; break; case ZSTD_c_prefetchCDictTables: *value = (int)CCtxParams->prefetchCDictTables; break; case ZSTD_c_enableSeqProducerFallback: *value = CCtxParams->enableMatchFinderFallback; break; case ZSTD_c_maxBlockSize: *value = (int)CCtxParams->maxBlockSize; break; case ZSTD_c_searchForExternalRepcodes: *value = (int)CCtxParams->searchForExternalRepcodes; break; default: RETURN_ERROR(parameter_unsupported, "unknown parameter"); } return 0; } /** ZSTD_CCtx_setParametersUsingCCtxParams() : * just applies `params` into `cctx` * no action is performed, parameters are merely stored. * If ZSTDMT is enabled, parameters are pushed to cctx->mtctx. * This is possible even if a compression is ongoing. * In which case, new parameters will be applied on the fly, starting with next compression job. */ size_t ZSTD_CCtx_setParametersUsingCCtxParams( ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params) { DEBUGLOG(4, "ZSTD_CCtx_setParametersUsingCCtxParams"); RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong, "The context is in the wrong stage!"); RETURN_ERROR_IF(cctx->cdict, stage_wrong, "Can't override parameters with cdict attached (some must " "be inherited from the cdict)."); cctx->requestedParams = *params; return 0; } size_t ZSTD_CCtx_setCParams(ZSTD_CCtx* cctx, ZSTD_compressionParameters cparams) { ZSTD_STATIC_ASSERT(sizeof(cparams) == 7 * 4 /* all params are listed below */); DEBUGLOG(4, "ZSTD_CCtx_setCParams"); /* only update if all parameters are valid */ FORWARD_IF_ERROR(ZSTD_checkCParams(cparams), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(cctx, ZSTD_c_windowLog, cparams.windowLog), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(cctx, ZSTD_c_chainLog, cparams.chainLog), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(cctx, ZSTD_c_hashLog, cparams.hashLog), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(cctx, ZSTD_c_searchLog, cparams.searchLog), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(cctx, ZSTD_c_minMatch, cparams.minMatch), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(cctx, ZSTD_c_targetLength, cparams.targetLength), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(cctx, ZSTD_c_strategy, cparams.strategy), ""); return 0; } size_t ZSTD_CCtx_setFParams(ZSTD_CCtx* cctx, ZSTD_frameParameters fparams) { ZSTD_STATIC_ASSERT(sizeof(fparams) == 3 * 4 /* all params are listed below */); DEBUGLOG(4, "ZSTD_CCtx_setFParams"); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(cctx, ZSTD_c_contentSizeFlag, fparams.contentSizeFlag != 0), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(cctx, ZSTD_c_checksumFlag, fparams.checksumFlag != 0), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(cctx, ZSTD_c_dictIDFlag, fparams.noDictIDFlag == 0), ""); return 0; } size_t ZSTD_CCtx_setParams(ZSTD_CCtx* cctx, ZSTD_parameters params) { DEBUGLOG(4, "ZSTD_CCtx_setParams"); /* First check cParams, because we want to update all or none. */ FORWARD_IF_ERROR(ZSTD_checkCParams(params.cParams), ""); /* Next set fParams, because this could fail if the cctx isn't in init stage. */ FORWARD_IF_ERROR(ZSTD_CCtx_setFParams(cctx, params.fParams), ""); /* Finally set cParams, which should succeed. */ FORWARD_IF_ERROR(ZSTD_CCtx_setCParams(cctx, params.cParams), ""); return 0; } size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long pledgedSrcSize) { DEBUGLOG(4, "ZSTD_CCtx_setPledgedSrcSize to %llu bytes", pledgedSrcSize); RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong, "Can't set pledgedSrcSize when not in init stage."); cctx->pledgedSrcSizePlusOne = pledgedSrcSize+1; return 0; } static ZSTD_compressionParameters ZSTD_dedicatedDictSearch_getCParams( int const compressionLevel, size_t const dictSize); static int ZSTD_dedicatedDictSearch_isSupported( const ZSTD_compressionParameters* cParams); static void ZSTD_dedicatedDictSearch_revertCParams( ZSTD_compressionParameters* cParams); /** * Initializes the local dictionary using requested parameters. * NOTE: Initialization does not employ the pledged src size, * because the dictionary may be used for multiple compressions. */ static size_t ZSTD_initLocalDict(ZSTD_CCtx* cctx) { ZSTD_localDict* const dl = &cctx->localDict; if (dl->dict == NULL) { /* No local dictionary. */ assert(dl->dictBuffer == NULL); assert(dl->cdict == NULL); assert(dl->dictSize == 0); return 0; } if (dl->cdict != NULL) { /* Local dictionary already initialized. */ assert(cctx->cdict == dl->cdict); return 0; } assert(dl->dictSize > 0); assert(cctx->cdict == NULL); assert(cctx->prefixDict.dict == NULL); dl->cdict = ZSTD_createCDict_advanced2( dl->dict, dl->dictSize, ZSTD_dlm_byRef, dl->dictContentType, &cctx->requestedParams, cctx->customMem); RETURN_ERROR_IF(!dl->cdict, memory_allocation, "ZSTD_createCDict_advanced failed"); cctx->cdict = dl->cdict; return 0; } size_t ZSTD_CCtx_loadDictionary_advanced( ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType) { DEBUGLOG(4, "ZSTD_CCtx_loadDictionary_advanced (size: %u)", (U32)dictSize); RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong, "Can't load a dictionary when cctx is not in init stage."); ZSTD_clearAllDicts(cctx); /* erase any previously set dictionary */ if (dict == NULL || dictSize == 0) /* no dictionary */ return 0; if (dictLoadMethod == ZSTD_dlm_byRef) { cctx->localDict.dict = dict; } else { /* copy dictionary content inside CCtx to own its lifetime */ void* dictBuffer; RETURN_ERROR_IF(cctx->staticSize, memory_allocation, "static CCtx can't allocate for an internal copy of dictionary"); dictBuffer = ZSTD_customMalloc(dictSize, cctx->customMem); RETURN_ERROR_IF(dictBuffer==NULL, memory_allocation, "allocation failed for dictionary content"); ZSTD_memcpy(dictBuffer, dict, dictSize); cctx->localDict.dictBuffer = dictBuffer; /* owned ptr to free */ cctx->localDict.dict = dictBuffer; /* read-only reference */ } cctx->localDict.dictSize = dictSize; cctx->localDict.dictContentType = dictContentType; return 0; } size_t ZSTD_CCtx_loadDictionary_byReference( ZSTD_CCtx* cctx, const void* dict, size_t dictSize) { return ZSTD_CCtx_loadDictionary_advanced( cctx, dict, dictSize, ZSTD_dlm_byRef, ZSTD_dct_auto); } size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize) { return ZSTD_CCtx_loadDictionary_advanced( cctx, dict, dictSize, ZSTD_dlm_byCopy, ZSTD_dct_auto); } size_t ZSTD_CCtx_refCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict) { RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong, "Can't ref a dict when ctx not in init stage."); /* Free the existing local cdict (if any) to save memory. */ ZSTD_clearAllDicts(cctx); cctx->cdict = cdict; return 0; } size_t ZSTD_CCtx_refThreadPool(ZSTD_CCtx* cctx, ZSTD_threadPool* pool) { RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong, "Can't ref a pool when ctx not in init stage."); cctx->pool = pool; return 0; } size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize) { return ZSTD_CCtx_refPrefix_advanced(cctx, prefix, prefixSize, ZSTD_dct_rawContent); } size_t ZSTD_CCtx_refPrefix_advanced( ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType) { RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong, "Can't ref a prefix when ctx not in init stage."); ZSTD_clearAllDicts(cctx); if (prefix != NULL && prefixSize > 0) { cctx->prefixDict.dict = prefix; cctx->prefixDict.dictSize = prefixSize; cctx->prefixDict.dictContentType = dictContentType; } return 0; } /*! ZSTD_CCtx_reset() : * Also dumps dictionary */ size_t ZSTD_CCtx_reset(ZSTD_CCtx* cctx, ZSTD_ResetDirective reset) { if ( (reset == ZSTD_reset_session_only) || (reset == ZSTD_reset_session_and_parameters) ) { cctx->streamStage = zcss_init; cctx->pledgedSrcSizePlusOne = 0; } if ( (reset == ZSTD_reset_parameters) || (reset == ZSTD_reset_session_and_parameters) ) { RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong, "Reset parameters is only possible during init stage."); ZSTD_clearAllDicts(cctx); ZSTD_memset(&cctx->externalMatchCtx, 0, sizeof(cctx->externalMatchCtx)); return ZSTD_CCtxParams_reset(&cctx->requestedParams); } return 0; } /** ZSTD_checkCParams() : control CParam values remain within authorized range. @return : 0, or an error code if one value is beyond authorized range */ size_t ZSTD_checkCParams(ZSTD_compressionParameters cParams) { BOUNDCHECK(ZSTD_c_windowLog, (int)cParams.windowLog); BOUNDCHECK(ZSTD_c_chainLog, (int)cParams.chainLog); BOUNDCHECK(ZSTD_c_hashLog, (int)cParams.hashLog); BOUNDCHECK(ZSTD_c_searchLog, (int)cParams.searchLog); BOUNDCHECK(ZSTD_c_minMatch, (int)cParams.minMatch); BOUNDCHECK(ZSTD_c_targetLength,(int)cParams.targetLength); BOUNDCHECK(ZSTD_c_strategy, cParams.strategy); return 0; } /** ZSTD_clampCParams() : * make CParam values within valid range. * @return : valid CParams */ static ZSTD_compressionParameters ZSTD_clampCParams(ZSTD_compressionParameters cParams) { # define CLAMP_TYPE(cParam, val, type) { \ ZSTD_bounds const bounds = ZSTD_cParam_getBounds(cParam); \ if ((int)valbounds.upperBound) val=(type)bounds.upperBound; \ } # define CLAMP(cParam, val) CLAMP_TYPE(cParam, val, unsigned) CLAMP(ZSTD_c_windowLog, cParams.windowLog); CLAMP(ZSTD_c_chainLog, cParams.chainLog); CLAMP(ZSTD_c_hashLog, cParams.hashLog); CLAMP(ZSTD_c_searchLog, cParams.searchLog); CLAMP(ZSTD_c_minMatch, cParams.minMatch); CLAMP(ZSTD_c_targetLength,cParams.targetLength); CLAMP_TYPE(ZSTD_c_strategy,cParams.strategy, ZSTD_strategy); return cParams; } /** ZSTD_cycleLog() : * condition for correct operation : hashLog > 1 */ U32 ZSTD_cycleLog(U32 hashLog, ZSTD_strategy strat) { U32 const btScale = ((U32)strat >= (U32)ZSTD_btlazy2); return hashLog - btScale; } /** ZSTD_dictAndWindowLog() : * Returns an adjusted window log that is large enough to fit the source and the dictionary. * The zstd format says that the entire dictionary is valid if one byte of the dictionary * is within the window. So the hashLog and chainLog should be large enough to reference both * the dictionary and the window. So we must use this adjusted dictAndWindowLog when downsizing * the hashLog and windowLog. * NOTE: srcSize must not be ZSTD_CONTENTSIZE_UNKNOWN. */ static U32 ZSTD_dictAndWindowLog(U32 windowLog, U64 srcSize, U64 dictSize) { const U64 maxWindowSize = 1ULL << ZSTD_WINDOWLOG_MAX; /* No dictionary ==> No change */ if (dictSize == 0) { return windowLog; } assert(windowLog <= ZSTD_WINDOWLOG_MAX); assert(srcSize != ZSTD_CONTENTSIZE_UNKNOWN); /* Handled in ZSTD_adjustCParams_internal() */ { U64 const windowSize = 1ULL << windowLog; U64 const dictAndWindowSize = dictSize + windowSize; /* If the window size is already large enough to fit both the source and the dictionary * then just use the window size. Otherwise adjust so that it fits the dictionary and * the window. */ if (windowSize >= dictSize + srcSize) { return windowLog; /* Window size large enough already */ } else if (dictAndWindowSize >= maxWindowSize) { return ZSTD_WINDOWLOG_MAX; /* Larger than max window log */ } else { return ZSTD_highbit32((U32)dictAndWindowSize - 1) + 1; } } } /** ZSTD_adjustCParams_internal() : * optimize `cPar` for a specified input (`srcSize` and `dictSize`). * mostly downsize to reduce memory consumption and initialization latency. * `srcSize` can be ZSTD_CONTENTSIZE_UNKNOWN when not known. * `mode` is the mode for parameter adjustment. See docs for `ZSTD_cParamMode_e`. * note : `srcSize==0` means 0! * condition : cPar is presumed validated (can be checked using ZSTD_checkCParams()). */ static ZSTD_compressionParameters ZSTD_adjustCParams_internal(ZSTD_compressionParameters cPar, unsigned long long srcSize, size_t dictSize, ZSTD_cParamMode_e mode, ZSTD_paramSwitch_e useRowMatchFinder) { const U64 minSrcSize = 513; /* (1<<9) + 1 */ const U64 maxWindowResize = 1ULL << (ZSTD_WINDOWLOG_MAX-1); assert(ZSTD_checkCParams(cPar)==0); switch (mode) { case ZSTD_cpm_unknown: case ZSTD_cpm_noAttachDict: /* If we don't know the source size, don't make any * assumptions about it. We will already have selected * smaller parameters if a dictionary is in use. */ break; case ZSTD_cpm_createCDict: /* Assume a small source size when creating a dictionary * with an unknown source size. */ if (dictSize && srcSize == ZSTD_CONTENTSIZE_UNKNOWN) srcSize = minSrcSize; break; case ZSTD_cpm_attachDict: /* Dictionary has its own dedicated parameters which have * already been selected. We are selecting parameters * for only the source. */ dictSize = 0; break; default: assert(0); break; } /* resize windowLog if input is small enough, to use less memory */ if ( (srcSize <= maxWindowResize) && (dictSize <= maxWindowResize) ) { U32 const tSize = (U32)(srcSize + dictSize); static U32 const hashSizeMin = 1 << ZSTD_HASHLOG_MIN; U32 const srcLog = (tSize < hashSizeMin) ? ZSTD_HASHLOG_MIN : ZSTD_highbit32(tSize-1) + 1; if (cPar.windowLog > srcLog) cPar.windowLog = srcLog; } if (srcSize != ZSTD_CONTENTSIZE_UNKNOWN) { U32 const dictAndWindowLog = ZSTD_dictAndWindowLog(cPar.windowLog, (U64)srcSize, (U64)dictSize); U32 const cycleLog = ZSTD_cycleLog(cPar.chainLog, cPar.strategy); if (cPar.hashLog > dictAndWindowLog+1) cPar.hashLog = dictAndWindowLog+1; if (cycleLog > dictAndWindowLog) cPar.chainLog -= (cycleLog - dictAndWindowLog); } if (cPar.windowLog < ZSTD_WINDOWLOG_ABSOLUTEMIN) cPar.windowLog = ZSTD_WINDOWLOG_ABSOLUTEMIN; /* minimum wlog required for valid frame header */ /* We can't use more than 32 bits of hash in total, so that means that we require: * (hashLog + 8) <= 32 && (chainLog + 8) <= 32 */ if (mode == ZSTD_cpm_createCDict && ZSTD_CDictIndicesAreTagged(&cPar)) { U32 const maxShortCacheHashLog = 32 - ZSTD_SHORT_CACHE_TAG_BITS; if (cPar.hashLog > maxShortCacheHashLog) { cPar.hashLog = maxShortCacheHashLog; } if (cPar.chainLog > maxShortCacheHashLog) { cPar.chainLog = maxShortCacheHashLog; } } /* At this point, we aren't 100% sure if we are using the row match finder. * Unless it is explicitly disabled, conservatively assume that it is enabled. * In this case it will only be disabled for small sources, so shrinking the * hash log a little bit shouldn't result in any ratio loss. */ if (useRowMatchFinder == ZSTD_ps_auto) useRowMatchFinder = ZSTD_ps_enable; /* We can't hash more than 32-bits in total. So that means that we require: * (hashLog - rowLog + 8) <= 32 */ if (ZSTD_rowMatchFinderUsed(cPar.strategy, useRowMatchFinder)) { /* Switch to 32-entry rows if searchLog is 5 (or more) */ U32 const rowLog = BOUNDED(4, cPar.searchLog, 6); U32 const maxRowHashLog = 32 - ZSTD_ROW_HASH_TAG_BITS; U32 const maxHashLog = maxRowHashLog + rowLog; assert(cPar.hashLog >= rowLog); if (cPar.hashLog > maxHashLog) { cPar.hashLog = maxHashLog; } } return cPar; } ZSTD_compressionParameters ZSTD_adjustCParams(ZSTD_compressionParameters cPar, unsigned long long srcSize, size_t dictSize) { cPar = ZSTD_clampCParams(cPar); /* resulting cPar is necessarily valid (all parameters within range) */ if (srcSize == 0) srcSize = ZSTD_CONTENTSIZE_UNKNOWN; return ZSTD_adjustCParams_internal(cPar, srcSize, dictSize, ZSTD_cpm_unknown, ZSTD_ps_auto); } static ZSTD_compressionParameters ZSTD_getCParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode); static ZSTD_parameters ZSTD_getParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode); static void ZSTD_overrideCParams( ZSTD_compressionParameters* cParams, const ZSTD_compressionParameters* overrides) { if (overrides->windowLog) cParams->windowLog = overrides->windowLog; if (overrides->hashLog) cParams->hashLog = overrides->hashLog; if (overrides->chainLog) cParams->chainLog = overrides->chainLog; if (overrides->searchLog) cParams->searchLog = overrides->searchLog; if (overrides->minMatch) cParams->minMatch = overrides->minMatch; if (overrides->targetLength) cParams->targetLength = overrides->targetLength; if (overrides->strategy) cParams->strategy = overrides->strategy; } ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams( const ZSTD_CCtx_params* CCtxParams, U64 srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode) { ZSTD_compressionParameters cParams; if (srcSizeHint == ZSTD_CONTENTSIZE_UNKNOWN && CCtxParams->srcSizeHint > 0) { srcSizeHint = CCtxParams->srcSizeHint; } cParams = ZSTD_getCParams_internal(CCtxParams->compressionLevel, srcSizeHint, dictSize, mode); if (CCtxParams->ldmParams.enableLdm == ZSTD_ps_enable) cParams.windowLog = ZSTD_LDM_DEFAULT_WINDOW_LOG; ZSTD_overrideCParams(&cParams, &CCtxParams->cParams); assert(!ZSTD_checkCParams(cParams)); /* srcSizeHint == 0 means 0 */ return ZSTD_adjustCParams_internal(cParams, srcSizeHint, dictSize, mode, CCtxParams->useRowMatchFinder); } static size_t ZSTD_sizeof_matchState(const ZSTD_compressionParameters* const cParams, const ZSTD_paramSwitch_e useRowMatchFinder, const U32 enableDedicatedDictSearch, const U32 forCCtx) { /* chain table size should be 0 for fast or row-hash strategies */ size_t const chainSize = ZSTD_allocateChainTable(cParams->strategy, useRowMatchFinder, enableDedicatedDictSearch && !forCCtx) ? ((size_t)1 << cParams->chainLog) : 0; size_t const hSize = ((size_t)1) << cParams->hashLog; U32 const hashLog3 = (forCCtx && cParams->minMatch==3) ? MIN(ZSTD_HASHLOG3_MAX, cParams->windowLog) : 0; size_t const h3Size = hashLog3 ? ((size_t)1) << hashLog3 : 0; /* We don't use ZSTD_cwksp_alloc_size() here because the tables aren't * surrounded by redzones in ASAN. */ size_t const tableSpace = chainSize * sizeof(U32) + hSize * sizeof(U32) + h3Size * sizeof(U32); size_t const optPotentialSpace = ZSTD_cwksp_aligned_alloc_size((MaxML+1) * sizeof(U32)) + ZSTD_cwksp_aligned_alloc_size((MaxLL+1) * sizeof(U32)) + ZSTD_cwksp_aligned_alloc_size((MaxOff+1) * sizeof(U32)) + ZSTD_cwksp_aligned_alloc_size((1<strategy, useRowMatchFinder) ? ZSTD_cwksp_aligned_alloc_size(hSize) : 0; size_t const optSpace = (forCCtx && (cParams->strategy >= ZSTD_btopt)) ? optPotentialSpace : 0; size_t const slackSpace = ZSTD_cwksp_slack_space_required(); /* tables are guaranteed to be sized in multiples of 64 bytes (or 16 uint32_t) */ ZSTD_STATIC_ASSERT(ZSTD_HASHLOG_MIN >= 4 && ZSTD_WINDOWLOG_MIN >= 4 && ZSTD_CHAINLOG_MIN >= 4); assert(useRowMatchFinder != ZSTD_ps_auto); DEBUGLOG(4, "chainSize: %u - hSize: %u - h3Size: %u", (U32)chainSize, (U32)hSize, (U32)h3Size); return tableSpace + optSpace + slackSpace + lazyAdditionalSpace; } /* Helper function for calculating memory requirements. * Gives a tighter bound than ZSTD_sequenceBound() by taking minMatch into account. */ static size_t ZSTD_maxNbSeq(size_t blockSize, unsigned minMatch, int useSequenceProducer) { U32 const divider = (minMatch==3 || useSequenceProducer) ? 3 : 4; return blockSize / divider; } static size_t ZSTD_estimateCCtxSize_usingCCtxParams_internal( const ZSTD_compressionParameters* cParams, const ldmParams_t* ldmParams, const int isStatic, const ZSTD_paramSwitch_e useRowMatchFinder, const size_t buffInSize, const size_t buffOutSize, const U64 pledgedSrcSize, int useSequenceProducer, size_t maxBlockSize) { size_t const windowSize = (size_t) BOUNDED(1ULL, 1ULL << cParams->windowLog, pledgedSrcSize); size_t const blockSize = MIN(ZSTD_resolveMaxBlockSize(maxBlockSize), windowSize); size_t const maxNbSeq = ZSTD_maxNbSeq(blockSize, cParams->minMatch, useSequenceProducer); size_t const tokenSpace = ZSTD_cwksp_alloc_size(WILDCOPY_OVERLENGTH + blockSize) + ZSTD_cwksp_aligned_alloc_size(maxNbSeq * sizeof(seqDef)) + 3 * ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(BYTE)); size_t const entropySpace = ZSTD_cwksp_alloc_size(ENTROPY_WORKSPACE_SIZE); size_t const blockStateSpace = 2 * ZSTD_cwksp_alloc_size(sizeof(ZSTD_compressedBlockState_t)); size_t const matchStateSize = ZSTD_sizeof_matchState(cParams, useRowMatchFinder, /* enableDedicatedDictSearch */ 0, /* forCCtx */ 1); size_t const ldmSpace = ZSTD_ldm_getTableSize(*ldmParams); size_t const maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(*ldmParams, blockSize); size_t const ldmSeqSpace = ldmParams->enableLdm == ZSTD_ps_enable ? ZSTD_cwksp_aligned_alloc_size(maxNbLdmSeq * sizeof(rawSeq)) : 0; size_t const bufferSpace = ZSTD_cwksp_alloc_size(buffInSize) + ZSTD_cwksp_alloc_size(buffOutSize); size_t const cctxSpace = isStatic ? ZSTD_cwksp_alloc_size(sizeof(ZSTD_CCtx)) : 0; size_t const maxNbExternalSeq = ZSTD_sequenceBound(blockSize); size_t const externalSeqSpace = useSequenceProducer ? ZSTD_cwksp_aligned_alloc_size(maxNbExternalSeq * sizeof(ZSTD_Sequence)) : 0; size_t const neededSpace = cctxSpace + entropySpace + blockStateSpace + ldmSpace + ldmSeqSpace + matchStateSize + tokenSpace + bufferSpace + externalSeqSpace; DEBUGLOG(5, "estimate workspace : %u", (U32)neededSpace); return neededSpace; } size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params) { ZSTD_compressionParameters const cParams = ZSTD_getCParamsFromCCtxParams(params, ZSTD_CONTENTSIZE_UNKNOWN, 0, ZSTD_cpm_noAttachDict); ZSTD_paramSwitch_e const useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(params->useRowMatchFinder, &cParams); RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only."); /* estimateCCtxSize is for one-shot compression. So no buffers should * be needed. However, we still allocate two 0-sized buffers, which can * take space under ASAN. */ return ZSTD_estimateCCtxSize_usingCCtxParams_internal( &cParams, ¶ms->ldmParams, 1, useRowMatchFinder, 0, 0, ZSTD_CONTENTSIZE_UNKNOWN, params->useSequenceProducer, params->maxBlockSize); } size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams) { ZSTD_CCtx_params initialParams = ZSTD_makeCCtxParamsFromCParams(cParams); if (ZSTD_rowMatchFinderSupported(cParams.strategy)) { /* Pick bigger of not using and using row-based matchfinder for greedy and lazy strategies */ size_t noRowCCtxSize; size_t rowCCtxSize; initialParams.useRowMatchFinder = ZSTD_ps_disable; noRowCCtxSize = ZSTD_estimateCCtxSize_usingCCtxParams(&initialParams); initialParams.useRowMatchFinder = ZSTD_ps_enable; rowCCtxSize = ZSTD_estimateCCtxSize_usingCCtxParams(&initialParams); return MAX(noRowCCtxSize, rowCCtxSize); } else { return ZSTD_estimateCCtxSize_usingCCtxParams(&initialParams); } } static size_t ZSTD_estimateCCtxSize_internal(int compressionLevel) { int tier = 0; size_t largestSize = 0; static const unsigned long long srcSizeTiers[4] = {16 KB, 128 KB, 256 KB, ZSTD_CONTENTSIZE_UNKNOWN}; for (; tier < 4; ++tier) { /* Choose the set of cParams for a given level across all srcSizes that give the largest cctxSize */ ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, srcSizeTiers[tier], 0, ZSTD_cpm_noAttachDict); largestSize = MAX(ZSTD_estimateCCtxSize_usingCParams(cParams), largestSize); } return largestSize; } size_t ZSTD_estimateCCtxSize(int compressionLevel) { int level; size_t memBudget = 0; for (level=MIN(compressionLevel, 1); level<=compressionLevel; level++) { /* Ensure monotonically increasing memory usage as compression level increases */ size_t const newMB = ZSTD_estimateCCtxSize_internal(level); if (newMB > memBudget) memBudget = newMB; } return memBudget; } size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params) { RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only."); { ZSTD_compressionParameters const cParams = ZSTD_getCParamsFromCCtxParams(params, ZSTD_CONTENTSIZE_UNKNOWN, 0, ZSTD_cpm_noAttachDict); size_t const blockSize = MIN(ZSTD_resolveMaxBlockSize(params->maxBlockSize), (size_t)1 << cParams.windowLog); size_t const inBuffSize = (params->inBufferMode == ZSTD_bm_buffered) ? ((size_t)1 << cParams.windowLog) + blockSize : 0; size_t const outBuffSize = (params->outBufferMode == ZSTD_bm_buffered) ? ZSTD_compressBound(blockSize) + 1 : 0; ZSTD_paramSwitch_e const useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(params->useRowMatchFinder, ¶ms->cParams); return ZSTD_estimateCCtxSize_usingCCtxParams_internal( &cParams, ¶ms->ldmParams, 1, useRowMatchFinder, inBuffSize, outBuffSize, ZSTD_CONTENTSIZE_UNKNOWN, params->useSequenceProducer, params->maxBlockSize); } } size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams) { ZSTD_CCtx_params initialParams = ZSTD_makeCCtxParamsFromCParams(cParams); if (ZSTD_rowMatchFinderSupported(cParams.strategy)) { /* Pick bigger of not using and using row-based matchfinder for greedy and lazy strategies */ size_t noRowCCtxSize; size_t rowCCtxSize; initialParams.useRowMatchFinder = ZSTD_ps_disable; noRowCCtxSize = ZSTD_estimateCStreamSize_usingCCtxParams(&initialParams); initialParams.useRowMatchFinder = ZSTD_ps_enable; rowCCtxSize = ZSTD_estimateCStreamSize_usingCCtxParams(&initialParams); return MAX(noRowCCtxSize, rowCCtxSize); } else { return ZSTD_estimateCStreamSize_usingCCtxParams(&initialParams); } } static size_t ZSTD_estimateCStreamSize_internal(int compressionLevel) { ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, 0, ZSTD_cpm_noAttachDict); return ZSTD_estimateCStreamSize_usingCParams(cParams); } size_t ZSTD_estimateCStreamSize(int compressionLevel) { int level; size_t memBudget = 0; for (level=MIN(compressionLevel, 1); level<=compressionLevel; level++) { size_t const newMB = ZSTD_estimateCStreamSize_internal(level); if (newMB > memBudget) memBudget = newMB; } return memBudget; } /* ZSTD_getFrameProgression(): * tells how much data has been consumed (input) and produced (output) for current frame. * able to count progression inside worker threads (non-blocking mode). */ ZSTD_frameProgression ZSTD_getFrameProgression(const ZSTD_CCtx* cctx) { #ifdef ZSTD_MULTITHREAD if (cctx->appliedParams.nbWorkers > 0) { return ZSTDMT_getFrameProgression(cctx->mtctx); } #endif { ZSTD_frameProgression fp; size_t const buffered = (cctx->inBuff == NULL) ? 0 : cctx->inBuffPos - cctx->inToCompress; if (buffered) assert(cctx->inBuffPos >= cctx->inToCompress); assert(buffered <= ZSTD_BLOCKSIZE_MAX); fp.ingested = cctx->consumedSrcSize + buffered; fp.consumed = cctx->consumedSrcSize; fp.produced = cctx->producedCSize; fp.flushed = cctx->producedCSize; /* simplified; some data might still be left within streaming output buffer */ fp.currentJobID = 0; fp.nbActiveWorkers = 0; return fp; } } /*! ZSTD_toFlushNow() * Only useful for multithreading scenarios currently (nbWorkers >= 1). */ size_t ZSTD_toFlushNow(ZSTD_CCtx* cctx) { #ifdef ZSTD_MULTITHREAD if (cctx->appliedParams.nbWorkers > 0) { return ZSTDMT_toFlushNow(cctx->mtctx); } #endif (void)cctx; return 0; /* over-simplification; could also check if context is currently running in streaming mode, and in which case, report how many bytes are left to be flushed within output buffer */ } static void ZSTD_assertEqualCParams(ZSTD_compressionParameters cParams1, ZSTD_compressionParameters cParams2) { (void)cParams1; (void)cParams2; assert(cParams1.windowLog == cParams2.windowLog); assert(cParams1.chainLog == cParams2.chainLog); assert(cParams1.hashLog == cParams2.hashLog); assert(cParams1.searchLog == cParams2.searchLog); assert(cParams1.minMatch == cParams2.minMatch); assert(cParams1.targetLength == cParams2.targetLength); assert(cParams1.strategy == cParams2.strategy); } void ZSTD_reset_compressedBlockState(ZSTD_compressedBlockState_t* bs) { int i; for (i = 0; i < ZSTD_REP_NUM; ++i) bs->rep[i] = repStartValue[i]; bs->entropy.huf.repeatMode = HUF_repeat_none; bs->entropy.fse.offcode_repeatMode = FSE_repeat_none; bs->entropy.fse.matchlength_repeatMode = FSE_repeat_none; bs->entropy.fse.litlength_repeatMode = FSE_repeat_none; } /*! ZSTD_invalidateMatchState() * Invalidate all the matches in the match finder tables. * Requires nextSrc and base to be set (can be NULL). */ static void ZSTD_invalidateMatchState(ZSTD_matchState_t* ms) { ZSTD_window_clear(&ms->window); ms->nextToUpdate = ms->window.dictLimit; ms->loadedDictEnd = 0; ms->opt.litLengthSum = 0; /* force reset of btopt stats */ ms->dictMatchState = NULL; } /** * Controls, for this matchState reset, whether the tables need to be cleared / * prepared for the coming compression (ZSTDcrp_makeClean), or whether the * tables can be left unclean (ZSTDcrp_leaveDirty), because we know that a * subsequent operation will overwrite the table space anyways (e.g., copying * the matchState contents in from a CDict). */ typedef enum { ZSTDcrp_makeClean, ZSTDcrp_leaveDirty } ZSTD_compResetPolicy_e; /** * Controls, for this matchState reset, whether indexing can continue where it * left off (ZSTDirp_continue), or whether it needs to be restarted from zero * (ZSTDirp_reset). */ typedef enum { ZSTDirp_continue, ZSTDirp_reset } ZSTD_indexResetPolicy_e; typedef enum { ZSTD_resetTarget_CDict, ZSTD_resetTarget_CCtx } ZSTD_resetTarget_e; /* Mixes bits in a 64 bits in a value, based on XXH3_rrmxmx */ static U64 ZSTD_bitmix(U64 val, U64 len) { val ^= ZSTD_rotateRight_U64(val, 49) ^ ZSTD_rotateRight_U64(val, 24); val *= 0x9FB21C651E98DF25ULL; val ^= (val >> 35) + len ; val *= 0x9FB21C651E98DF25ULL; return val ^ (val >> 28); } /* Mixes in the hashSalt and hashSaltEntropy to create a new hashSalt */ static void ZSTD_advanceHashSalt(ZSTD_matchState_t* ms) { ms->hashSalt = ZSTD_bitmix(ms->hashSalt, 8) ^ ZSTD_bitmix((U64) ms->hashSaltEntropy, 4); } static size_t ZSTD_reset_matchState(ZSTD_matchState_t* ms, ZSTD_cwksp* ws, const ZSTD_compressionParameters* cParams, const ZSTD_paramSwitch_e useRowMatchFinder, const ZSTD_compResetPolicy_e crp, const ZSTD_indexResetPolicy_e forceResetIndex, const ZSTD_resetTarget_e forWho) { /* disable chain table allocation for fast or row-based strategies */ size_t const chainSize = ZSTD_allocateChainTable(cParams->strategy, useRowMatchFinder, ms->dedicatedDictSearch && (forWho == ZSTD_resetTarget_CDict)) ? ((size_t)1 << cParams->chainLog) : 0; size_t const hSize = ((size_t)1) << cParams->hashLog; U32 const hashLog3 = ((forWho == ZSTD_resetTarget_CCtx) && cParams->minMatch==3) ? MIN(ZSTD_HASHLOG3_MAX, cParams->windowLog) : 0; size_t const h3Size = hashLog3 ? ((size_t)1) << hashLog3 : 0; DEBUGLOG(4, "reset indices : %u", forceResetIndex == ZSTDirp_reset); assert(useRowMatchFinder != ZSTD_ps_auto); if (forceResetIndex == ZSTDirp_reset) { ZSTD_window_init(&ms->window); ZSTD_cwksp_mark_tables_dirty(ws); } ms->hashLog3 = hashLog3; ms->lazySkipping = 0; ZSTD_invalidateMatchState(ms); assert(!ZSTD_cwksp_reserve_failed(ws)); /* check that allocation hasn't already failed */ ZSTD_cwksp_clear_tables(ws); DEBUGLOG(5, "reserving table space"); /* table Space */ ms->hashTable = (U32*)ZSTD_cwksp_reserve_table(ws, hSize * sizeof(U32)); ms->chainTable = (U32*)ZSTD_cwksp_reserve_table(ws, chainSize * sizeof(U32)); ms->hashTable3 = (U32*)ZSTD_cwksp_reserve_table(ws, h3Size * sizeof(U32)); RETURN_ERROR_IF(ZSTD_cwksp_reserve_failed(ws), memory_allocation, "failed a workspace allocation in ZSTD_reset_matchState"); DEBUGLOG(4, "reset table : %u", crp!=ZSTDcrp_leaveDirty); if (crp!=ZSTDcrp_leaveDirty) { /* reset tables only */ ZSTD_cwksp_clean_tables(ws); } if (ZSTD_rowMatchFinderUsed(cParams->strategy, useRowMatchFinder)) { /* Row match finder needs an additional table of hashes ("tags") */ size_t const tagTableSize = hSize; /* We want to generate a new salt in case we reset a Cctx, but we always want to use * 0 when we reset a Cdict */ if(forWho == ZSTD_resetTarget_CCtx) { ms->tagTable = (BYTE*) ZSTD_cwksp_reserve_aligned_init_once(ws, tagTableSize); ZSTD_advanceHashSalt(ms); } else { /* When we are not salting we want to always memset the memory */ ms->tagTable = (BYTE*) ZSTD_cwksp_reserve_aligned(ws, tagTableSize); ZSTD_memset(ms->tagTable, 0, tagTableSize); ms->hashSalt = 0; } { /* Switch to 32-entry rows if searchLog is 5 (or more) */ U32 const rowLog = BOUNDED(4, cParams->searchLog, 6); assert(cParams->hashLog >= rowLog); ms->rowHashLog = cParams->hashLog - rowLog; } } /* opt parser space */ if ((forWho == ZSTD_resetTarget_CCtx) && (cParams->strategy >= ZSTD_btopt)) { DEBUGLOG(4, "reserving optimal parser space"); ms->opt.litFreq = (unsigned*)ZSTD_cwksp_reserve_aligned(ws, (1<opt.litLengthFreq = (unsigned*)ZSTD_cwksp_reserve_aligned(ws, (MaxLL+1) * sizeof(unsigned)); ms->opt.matchLengthFreq = (unsigned*)ZSTD_cwksp_reserve_aligned(ws, (MaxML+1) * sizeof(unsigned)); ms->opt.offCodeFreq = (unsigned*)ZSTD_cwksp_reserve_aligned(ws, (MaxOff+1) * sizeof(unsigned)); ms->opt.matchTable = (ZSTD_match_t*)ZSTD_cwksp_reserve_aligned(ws, (ZSTD_OPT_NUM+1) * sizeof(ZSTD_match_t)); ms->opt.priceTable = (ZSTD_optimal_t*)ZSTD_cwksp_reserve_aligned(ws, (ZSTD_OPT_NUM+1) * sizeof(ZSTD_optimal_t)); } ms->cParams = *cParams; RETURN_ERROR_IF(ZSTD_cwksp_reserve_failed(ws), memory_allocation, "failed a workspace allocation in ZSTD_reset_matchState"); return 0; } /* ZSTD_indexTooCloseToMax() : * minor optimization : prefer memset() rather than reduceIndex() * which is measurably slow in some circumstances (reported for Visual Studio). * Works when re-using a context for a lot of smallish inputs : * if all inputs are smaller than ZSTD_INDEXOVERFLOW_MARGIN, * memset() will be triggered before reduceIndex(). */ #define ZSTD_INDEXOVERFLOW_MARGIN (16 MB) static int ZSTD_indexTooCloseToMax(ZSTD_window_t w) { return (size_t)(w.nextSrc - w.base) > (ZSTD_CURRENT_MAX - ZSTD_INDEXOVERFLOW_MARGIN); } /** ZSTD_dictTooBig(): * When dictionaries are larger than ZSTD_CHUNKSIZE_MAX they can't be loaded in * one go generically. So we ensure that in that case we reset the tables to zero, * so that we can load as much of the dictionary as possible. */ static int ZSTD_dictTooBig(size_t const loadedDictSize) { return loadedDictSize > ZSTD_CHUNKSIZE_MAX; } /*! ZSTD_resetCCtx_internal() : * @param loadedDictSize The size of the dictionary to be loaded * into the context, if any. If no dictionary is used, or the * dictionary is being attached / copied, then pass 0. * note : `params` are assumed fully validated at this stage. */ static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc, ZSTD_CCtx_params const* params, U64 const pledgedSrcSize, size_t const loadedDictSize, ZSTD_compResetPolicy_e const crp, ZSTD_buffered_policy_e const zbuff) { ZSTD_cwksp* const ws = &zc->workspace; DEBUGLOG(4, "ZSTD_resetCCtx_internal: pledgedSrcSize=%u, wlog=%u, useRowMatchFinder=%d useBlockSplitter=%d", (U32)pledgedSrcSize, params->cParams.windowLog, (int)params->useRowMatchFinder, (int)params->useBlockSplitter); assert(!ZSTD_isError(ZSTD_checkCParams(params->cParams))); zc->isFirstBlock = 1; /* Set applied params early so we can modify them for LDM, * and point params at the applied params. */ zc->appliedParams = *params; params = &zc->appliedParams; assert(params->useRowMatchFinder != ZSTD_ps_auto); assert(params->useBlockSplitter != ZSTD_ps_auto); assert(params->ldmParams.enableLdm != ZSTD_ps_auto); assert(params->maxBlockSize != 0); if (params->ldmParams.enableLdm == ZSTD_ps_enable) { /* Adjust long distance matching parameters */ ZSTD_ldm_adjustParameters(&zc->appliedParams.ldmParams, ¶ms->cParams); assert(params->ldmParams.hashLog >= params->ldmParams.bucketSizeLog); assert(params->ldmParams.hashRateLog < 32); } { size_t const windowSize = MAX(1, (size_t)MIN(((U64)1 << params->cParams.windowLog), pledgedSrcSize)); size_t const blockSize = MIN(params->maxBlockSize, windowSize); size_t const maxNbSeq = ZSTD_maxNbSeq(blockSize, params->cParams.minMatch, params->useSequenceProducer); size_t const buffOutSize = (zbuff == ZSTDb_buffered && params->outBufferMode == ZSTD_bm_buffered) ? ZSTD_compressBound(blockSize) + 1 : 0; size_t const buffInSize = (zbuff == ZSTDb_buffered && params->inBufferMode == ZSTD_bm_buffered) ? windowSize + blockSize : 0; size_t const maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(params->ldmParams, blockSize); int const indexTooClose = ZSTD_indexTooCloseToMax(zc->blockState.matchState.window); int const dictTooBig = ZSTD_dictTooBig(loadedDictSize); ZSTD_indexResetPolicy_e needsIndexReset = (indexTooClose || dictTooBig || !zc->initialized) ? ZSTDirp_reset : ZSTDirp_continue; size_t const neededSpace = ZSTD_estimateCCtxSize_usingCCtxParams_internal( ¶ms->cParams, ¶ms->ldmParams, zc->staticSize != 0, params->useRowMatchFinder, buffInSize, buffOutSize, pledgedSrcSize, params->useSequenceProducer, params->maxBlockSize); int resizeWorkspace; FORWARD_IF_ERROR(neededSpace, "cctx size estimate failed!"); if (!zc->staticSize) ZSTD_cwksp_bump_oversized_duration(ws, 0); { /* Check if workspace is large enough, alloc a new one if needed */ int const workspaceTooSmall = ZSTD_cwksp_sizeof(ws) < neededSpace; int const workspaceWasteful = ZSTD_cwksp_check_wasteful(ws, neededSpace); resizeWorkspace = workspaceTooSmall || workspaceWasteful; DEBUGLOG(4, "Need %zu B workspace", neededSpace); DEBUGLOG(4, "windowSize: %zu - blockSize: %zu", windowSize, blockSize); if (resizeWorkspace) { DEBUGLOG(4, "Resize workspaceSize from %zuKB to %zuKB", ZSTD_cwksp_sizeof(ws) >> 10, neededSpace >> 10); RETURN_ERROR_IF(zc->staticSize, memory_allocation, "static cctx : no resize"); needsIndexReset = ZSTDirp_reset; ZSTD_cwksp_free(ws, zc->customMem); FORWARD_IF_ERROR(ZSTD_cwksp_create(ws, neededSpace, zc->customMem), ""); DEBUGLOG(5, "reserving object space"); /* Statically sized space. * entropyWorkspace never moves, * though prev/next block swap places */ assert(ZSTD_cwksp_check_available(ws, 2 * sizeof(ZSTD_compressedBlockState_t))); zc->blockState.prevCBlock = (ZSTD_compressedBlockState_t*) ZSTD_cwksp_reserve_object(ws, sizeof(ZSTD_compressedBlockState_t)); RETURN_ERROR_IF(zc->blockState.prevCBlock == NULL, memory_allocation, "couldn't allocate prevCBlock"); zc->blockState.nextCBlock = (ZSTD_compressedBlockState_t*) ZSTD_cwksp_reserve_object(ws, sizeof(ZSTD_compressedBlockState_t)); RETURN_ERROR_IF(zc->blockState.nextCBlock == NULL, memory_allocation, "couldn't allocate nextCBlock"); zc->entropyWorkspace = (U32*) ZSTD_cwksp_reserve_object(ws, ENTROPY_WORKSPACE_SIZE); RETURN_ERROR_IF(zc->entropyWorkspace == NULL, memory_allocation, "couldn't allocate entropyWorkspace"); } } ZSTD_cwksp_clear(ws); /* init params */ zc->blockState.matchState.cParams = params->cParams; zc->blockState.matchState.prefetchCDictTables = params->prefetchCDictTables == ZSTD_ps_enable; zc->pledgedSrcSizePlusOne = pledgedSrcSize+1; zc->consumedSrcSize = 0; zc->producedCSize = 0; if (pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN) zc->appliedParams.fParams.contentSizeFlag = 0; DEBUGLOG(4, "pledged content size : %u ; flag : %u", (unsigned)pledgedSrcSize, zc->appliedParams.fParams.contentSizeFlag); zc->blockSize = blockSize; XXH64_reset(&zc->xxhState, 0); zc->stage = ZSTDcs_init; zc->dictID = 0; zc->dictContentSize = 0; ZSTD_reset_compressedBlockState(zc->blockState.prevCBlock); FORWARD_IF_ERROR(ZSTD_reset_matchState( &zc->blockState.matchState, ws, ¶ms->cParams, params->useRowMatchFinder, crp, needsIndexReset, ZSTD_resetTarget_CCtx), ""); zc->seqStore.sequencesStart = (seqDef*)ZSTD_cwksp_reserve_aligned(ws, maxNbSeq * sizeof(seqDef)); /* ldm hash table */ if (params->ldmParams.enableLdm == ZSTD_ps_enable) { /* TODO: avoid memset? */ size_t const ldmHSize = ((size_t)1) << params->ldmParams.hashLog; zc->ldmState.hashTable = (ldmEntry_t*)ZSTD_cwksp_reserve_aligned(ws, ldmHSize * sizeof(ldmEntry_t)); ZSTD_memset(zc->ldmState.hashTable, 0, ldmHSize * sizeof(ldmEntry_t)); zc->ldmSequences = (rawSeq*)ZSTD_cwksp_reserve_aligned(ws, maxNbLdmSeq * sizeof(rawSeq)); zc->maxNbLdmSequences = maxNbLdmSeq; ZSTD_window_init(&zc->ldmState.window); zc->ldmState.loadedDictEnd = 0; } /* reserve space for block-level external sequences */ if (params->useSequenceProducer) { size_t const maxNbExternalSeq = ZSTD_sequenceBound(blockSize); zc->externalMatchCtx.seqBufferCapacity = maxNbExternalSeq; zc->externalMatchCtx.seqBuffer = (ZSTD_Sequence*)ZSTD_cwksp_reserve_aligned(ws, maxNbExternalSeq * sizeof(ZSTD_Sequence)); } /* buffers */ /* ZSTD_wildcopy() is used to copy into the literals buffer, * so we have to oversize the buffer by WILDCOPY_OVERLENGTH bytes. */ zc->seqStore.litStart = ZSTD_cwksp_reserve_buffer(ws, blockSize + WILDCOPY_OVERLENGTH); zc->seqStore.maxNbLit = blockSize; zc->bufferedPolicy = zbuff; zc->inBuffSize = buffInSize; zc->inBuff = (char*)ZSTD_cwksp_reserve_buffer(ws, buffInSize); zc->outBuffSize = buffOutSize; zc->outBuff = (char*)ZSTD_cwksp_reserve_buffer(ws, buffOutSize); /* ldm bucketOffsets table */ if (params->ldmParams.enableLdm == ZSTD_ps_enable) { /* TODO: avoid memset? */ size_t const numBuckets = ((size_t)1) << (params->ldmParams.hashLog - params->ldmParams.bucketSizeLog); zc->ldmState.bucketOffsets = ZSTD_cwksp_reserve_buffer(ws, numBuckets); ZSTD_memset(zc->ldmState.bucketOffsets, 0, numBuckets); } /* sequences storage */ ZSTD_referenceExternalSequences(zc, NULL, 0); zc->seqStore.maxNbSeq = maxNbSeq; zc->seqStore.llCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(BYTE)); zc->seqStore.mlCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(BYTE)); zc->seqStore.ofCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(BYTE)); DEBUGLOG(3, "wksp: finished allocating, %zd bytes remain available", ZSTD_cwksp_available_space(ws)); assert(ZSTD_cwksp_estimated_space_within_bounds(ws, neededSpace)); zc->initialized = 1; return 0; } } /* ZSTD_invalidateRepCodes() : * ensures next compression will not use repcodes from previous block. * Note : only works with regular variant; * do not use with extDict variant ! */ void ZSTD_invalidateRepCodes(ZSTD_CCtx* cctx) { int i; for (i=0; iblockState.prevCBlock->rep[i] = 0; assert(!ZSTD_window_hasExtDict(cctx->blockState.matchState.window)); } /* These are the approximate sizes for each strategy past which copying the * dictionary tables into the working context is faster than using them * in-place. */ static const size_t attachDictSizeCutoffs[ZSTD_STRATEGY_MAX+1] = { 8 KB, /* unused */ 8 KB, /* ZSTD_fast */ 16 KB, /* ZSTD_dfast */ 32 KB, /* ZSTD_greedy */ 32 KB, /* ZSTD_lazy */ 32 KB, /* ZSTD_lazy2 */ 32 KB, /* ZSTD_btlazy2 */ 32 KB, /* ZSTD_btopt */ 8 KB, /* ZSTD_btultra */ 8 KB /* ZSTD_btultra2 */ }; static int ZSTD_shouldAttachDict(const ZSTD_CDict* cdict, const ZSTD_CCtx_params* params, U64 pledgedSrcSize) { size_t cutoff = attachDictSizeCutoffs[cdict->matchState.cParams.strategy]; int const dedicatedDictSearch = cdict->matchState.dedicatedDictSearch; return dedicatedDictSearch || ( ( pledgedSrcSize <= cutoff || pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN || params->attachDictPref == ZSTD_dictForceAttach ) && params->attachDictPref != ZSTD_dictForceCopy && !params->forceWindow ); /* dictMatchState isn't correctly * handled in _enforceMaxDist */ } static size_t ZSTD_resetCCtx_byAttachingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict, ZSTD_CCtx_params params, U64 pledgedSrcSize, ZSTD_buffered_policy_e zbuff) { DEBUGLOG(4, "ZSTD_resetCCtx_byAttachingCDict() pledgedSrcSize=%llu", (unsigned long long)pledgedSrcSize); { ZSTD_compressionParameters adjusted_cdict_cParams = cdict->matchState.cParams; unsigned const windowLog = params.cParams.windowLog; assert(windowLog != 0); /* Resize working context table params for input only, since the dict * has its own tables. */ /* pledgedSrcSize == 0 means 0! */ if (cdict->matchState.dedicatedDictSearch) { ZSTD_dedicatedDictSearch_revertCParams(&adjusted_cdict_cParams); } params.cParams = ZSTD_adjustCParams_internal(adjusted_cdict_cParams, pledgedSrcSize, cdict->dictContentSize, ZSTD_cpm_attachDict, params.useRowMatchFinder); params.cParams.windowLog = windowLog; params.useRowMatchFinder = cdict->useRowMatchFinder; /* cdict overrides */ FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, ¶ms, pledgedSrcSize, /* loadedDictSize */ 0, ZSTDcrp_makeClean, zbuff), ""); assert(cctx->appliedParams.cParams.strategy == adjusted_cdict_cParams.strategy); } { const U32 cdictEnd = (U32)( cdict->matchState.window.nextSrc - cdict->matchState.window.base); const U32 cdictLen = cdictEnd - cdict->matchState.window.dictLimit; if (cdictLen == 0) { /* don't even attach dictionaries with no contents */ DEBUGLOG(4, "skipping attaching empty dictionary"); } else { DEBUGLOG(4, "attaching dictionary into context"); cctx->blockState.matchState.dictMatchState = &cdict->matchState; /* prep working match state so dict matches never have negative indices * when they are translated to the working context's index space. */ if (cctx->blockState.matchState.window.dictLimit < cdictEnd) { cctx->blockState.matchState.window.nextSrc = cctx->blockState.matchState.window.base + cdictEnd; ZSTD_window_clear(&cctx->blockState.matchState.window); } /* loadedDictEnd is expressed within the referential of the active context */ cctx->blockState.matchState.loadedDictEnd = cctx->blockState.matchState.window.dictLimit; } } cctx->dictID = cdict->dictID; cctx->dictContentSize = cdict->dictContentSize; /* copy block state */ ZSTD_memcpy(cctx->blockState.prevCBlock, &cdict->cBlockState, sizeof(cdict->cBlockState)); return 0; } static void ZSTD_copyCDictTableIntoCCtx(U32* dst, U32 const* src, size_t tableSize, ZSTD_compressionParameters const* cParams) { if (ZSTD_CDictIndicesAreTagged(cParams)){ /* Remove tags from the CDict table if they are present. * See docs on "short cache" in zstd_compress_internal.h for context. */ size_t i; for (i = 0; i < tableSize; i++) { U32 const taggedIndex = src[i]; U32 const index = taggedIndex >> ZSTD_SHORT_CACHE_TAG_BITS; dst[i] = index; } } else { ZSTD_memcpy(dst, src, tableSize * sizeof(U32)); } } static size_t ZSTD_resetCCtx_byCopyingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict, ZSTD_CCtx_params params, U64 pledgedSrcSize, ZSTD_buffered_policy_e zbuff) { const ZSTD_compressionParameters *cdict_cParams = &cdict->matchState.cParams; assert(!cdict->matchState.dedicatedDictSearch); DEBUGLOG(4, "ZSTD_resetCCtx_byCopyingCDict() pledgedSrcSize=%llu", (unsigned long long)pledgedSrcSize); { unsigned const windowLog = params.cParams.windowLog; assert(windowLog != 0); /* Copy only compression parameters related to tables. */ params.cParams = *cdict_cParams; params.cParams.windowLog = windowLog; params.useRowMatchFinder = cdict->useRowMatchFinder; FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, ¶ms, pledgedSrcSize, /* loadedDictSize */ 0, ZSTDcrp_leaveDirty, zbuff), ""); assert(cctx->appliedParams.cParams.strategy == cdict_cParams->strategy); assert(cctx->appliedParams.cParams.hashLog == cdict_cParams->hashLog); assert(cctx->appliedParams.cParams.chainLog == cdict_cParams->chainLog); } ZSTD_cwksp_mark_tables_dirty(&cctx->workspace); assert(params.useRowMatchFinder != ZSTD_ps_auto); /* copy tables */ { size_t const chainSize = ZSTD_allocateChainTable(cdict_cParams->strategy, cdict->useRowMatchFinder, 0 /* DDS guaranteed disabled */) ? ((size_t)1 << cdict_cParams->chainLog) : 0; size_t const hSize = (size_t)1 << cdict_cParams->hashLog; ZSTD_copyCDictTableIntoCCtx(cctx->blockState.matchState.hashTable, cdict->matchState.hashTable, hSize, cdict_cParams); /* Do not copy cdict's chainTable if cctx has parameters such that it would not use chainTable */ if (ZSTD_allocateChainTable(cctx->appliedParams.cParams.strategy, cctx->appliedParams.useRowMatchFinder, 0 /* forDDSDict */)) { ZSTD_copyCDictTableIntoCCtx(cctx->blockState.matchState.chainTable, cdict->matchState.chainTable, chainSize, cdict_cParams); } /* copy tag table */ if (ZSTD_rowMatchFinderUsed(cdict_cParams->strategy, cdict->useRowMatchFinder)) { size_t const tagTableSize = hSize; ZSTD_memcpy(cctx->blockState.matchState.tagTable, cdict->matchState.tagTable, tagTableSize); cctx->blockState.matchState.hashSalt = cdict->matchState.hashSalt; } } /* Zero the hashTable3, since the cdict never fills it */ { int const h3log = cctx->blockState.matchState.hashLog3; size_t const h3Size = h3log ? ((size_t)1 << h3log) : 0; assert(cdict->matchState.hashLog3 == 0); ZSTD_memset(cctx->blockState.matchState.hashTable3, 0, h3Size * sizeof(U32)); } ZSTD_cwksp_mark_tables_clean(&cctx->workspace); /* copy dictionary offsets */ { ZSTD_matchState_t const* srcMatchState = &cdict->matchState; ZSTD_matchState_t* dstMatchState = &cctx->blockState.matchState; dstMatchState->window = srcMatchState->window; dstMatchState->nextToUpdate = srcMatchState->nextToUpdate; dstMatchState->loadedDictEnd= srcMatchState->loadedDictEnd; } cctx->dictID = cdict->dictID; cctx->dictContentSize = cdict->dictContentSize; /* copy block state */ ZSTD_memcpy(cctx->blockState.prevCBlock, &cdict->cBlockState, sizeof(cdict->cBlockState)); return 0; } /* We have a choice between copying the dictionary context into the working * context, or referencing the dictionary context from the working context * in-place. We decide here which strategy to use. */ static size_t ZSTD_resetCCtx_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict, const ZSTD_CCtx_params* params, U64 pledgedSrcSize, ZSTD_buffered_policy_e zbuff) { DEBUGLOG(4, "ZSTD_resetCCtx_usingCDict (pledgedSrcSize=%u)", (unsigned)pledgedSrcSize); if (ZSTD_shouldAttachDict(cdict, params, pledgedSrcSize)) { return ZSTD_resetCCtx_byAttachingCDict( cctx, cdict, *params, pledgedSrcSize, zbuff); } else { return ZSTD_resetCCtx_byCopyingCDict( cctx, cdict, *params, pledgedSrcSize, zbuff); } } /*! ZSTD_copyCCtx_internal() : * Duplicate an existing context `srcCCtx` into another one `dstCCtx`. * Only works during stage ZSTDcs_init (i.e. after creation, but before first call to ZSTD_compressContinue()). * The "context", in this case, refers to the hash and chain tables, * entropy tables, and dictionary references. * `windowLog` value is enforced if != 0, otherwise value is copied from srcCCtx. * @return : 0, or an error code */ static size_t ZSTD_copyCCtx_internal(ZSTD_CCtx* dstCCtx, const ZSTD_CCtx* srcCCtx, ZSTD_frameParameters fParams, U64 pledgedSrcSize, ZSTD_buffered_policy_e zbuff) { RETURN_ERROR_IF(srcCCtx->stage!=ZSTDcs_init, stage_wrong, "Can't copy a ctx that's not in init stage."); DEBUGLOG(5, "ZSTD_copyCCtx_internal"); ZSTD_memcpy(&dstCCtx->customMem, &srcCCtx->customMem, sizeof(ZSTD_customMem)); { ZSTD_CCtx_params params = dstCCtx->requestedParams; /* Copy only compression parameters related to tables. */ params.cParams = srcCCtx->appliedParams.cParams; assert(srcCCtx->appliedParams.useRowMatchFinder != ZSTD_ps_auto); assert(srcCCtx->appliedParams.useBlockSplitter != ZSTD_ps_auto); assert(srcCCtx->appliedParams.ldmParams.enableLdm != ZSTD_ps_auto); params.useRowMatchFinder = srcCCtx->appliedParams.useRowMatchFinder; params.useBlockSplitter = srcCCtx->appliedParams.useBlockSplitter; params.ldmParams = srcCCtx->appliedParams.ldmParams; params.fParams = fParams; params.maxBlockSize = srcCCtx->appliedParams.maxBlockSize; ZSTD_resetCCtx_internal(dstCCtx, ¶ms, pledgedSrcSize, /* loadedDictSize */ 0, ZSTDcrp_leaveDirty, zbuff); assert(dstCCtx->appliedParams.cParams.windowLog == srcCCtx->appliedParams.cParams.windowLog); assert(dstCCtx->appliedParams.cParams.strategy == srcCCtx->appliedParams.cParams.strategy); assert(dstCCtx->appliedParams.cParams.hashLog == srcCCtx->appliedParams.cParams.hashLog); assert(dstCCtx->appliedParams.cParams.chainLog == srcCCtx->appliedParams.cParams.chainLog); assert(dstCCtx->blockState.matchState.hashLog3 == srcCCtx->blockState.matchState.hashLog3); } ZSTD_cwksp_mark_tables_dirty(&dstCCtx->workspace); /* copy tables */ { size_t const chainSize = ZSTD_allocateChainTable(srcCCtx->appliedParams.cParams.strategy, srcCCtx->appliedParams.useRowMatchFinder, 0 /* forDDSDict */) ? ((size_t)1 << srcCCtx->appliedParams.cParams.chainLog) : 0; size_t const hSize = (size_t)1 << srcCCtx->appliedParams.cParams.hashLog; int const h3log = srcCCtx->blockState.matchState.hashLog3; size_t const h3Size = h3log ? ((size_t)1 << h3log) : 0; ZSTD_memcpy(dstCCtx->blockState.matchState.hashTable, srcCCtx->blockState.matchState.hashTable, hSize * sizeof(U32)); ZSTD_memcpy(dstCCtx->blockState.matchState.chainTable, srcCCtx->blockState.matchState.chainTable, chainSize * sizeof(U32)); ZSTD_memcpy(dstCCtx->blockState.matchState.hashTable3, srcCCtx->blockState.matchState.hashTable3, h3Size * sizeof(U32)); } ZSTD_cwksp_mark_tables_clean(&dstCCtx->workspace); /* copy dictionary offsets */ { const ZSTD_matchState_t* srcMatchState = &srcCCtx->blockState.matchState; ZSTD_matchState_t* dstMatchState = &dstCCtx->blockState.matchState; dstMatchState->window = srcMatchState->window; dstMatchState->nextToUpdate = srcMatchState->nextToUpdate; dstMatchState->loadedDictEnd= srcMatchState->loadedDictEnd; } dstCCtx->dictID = srcCCtx->dictID; dstCCtx->dictContentSize = srcCCtx->dictContentSize; /* copy block state */ ZSTD_memcpy(dstCCtx->blockState.prevCBlock, srcCCtx->blockState.prevCBlock, sizeof(*srcCCtx->blockState.prevCBlock)); return 0; } /*! ZSTD_copyCCtx() : * Duplicate an existing context `srcCCtx` into another one `dstCCtx`. * Only works during stage ZSTDcs_init (i.e. after creation, but before first call to ZSTD_compressContinue()). * pledgedSrcSize==0 means "unknown". * @return : 0, or an error code */ size_t ZSTD_copyCCtx(ZSTD_CCtx* dstCCtx, const ZSTD_CCtx* srcCCtx, unsigned long long pledgedSrcSize) { ZSTD_frameParameters fParams = { 1 /*content*/, 0 /*checksum*/, 0 /*noDictID*/ }; ZSTD_buffered_policy_e const zbuff = srcCCtx->bufferedPolicy; ZSTD_STATIC_ASSERT((U32)ZSTDb_buffered==1); if (pledgedSrcSize==0) pledgedSrcSize = ZSTD_CONTENTSIZE_UNKNOWN; fParams.contentSizeFlag = (pledgedSrcSize != ZSTD_CONTENTSIZE_UNKNOWN); return ZSTD_copyCCtx_internal(dstCCtx, srcCCtx, fParams, pledgedSrcSize, zbuff); } #define ZSTD_ROWSIZE 16 /*! ZSTD_reduceTable() : * reduce table indexes by `reducerValue`, or squash to zero. * PreserveMark preserves "unsorted mark" for btlazy2 strategy. * It must be set to a clear 0/1 value, to remove branch during inlining. * Presume table size is a multiple of ZSTD_ROWSIZE * to help auto-vectorization */ FORCE_INLINE_TEMPLATE void ZSTD_reduceTable_internal (U32* const table, U32 const size, U32 const reducerValue, int const preserveMark) { int const nbRows = (int)size / ZSTD_ROWSIZE; int cellNb = 0; int rowNb; /* Protect special index values < ZSTD_WINDOW_START_INDEX. */ U32 const reducerThreshold = reducerValue + ZSTD_WINDOW_START_INDEX; assert((size & (ZSTD_ROWSIZE-1)) == 0); /* multiple of ZSTD_ROWSIZE */ assert(size < (1U<<31)); /* can be casted to int */ #if ZSTD_MEMORY_SANITIZER && !defined (ZSTD_MSAN_DONT_POISON_WORKSPACE) /* To validate that the table re-use logic is sound, and that we don't * access table space that we haven't cleaned, we re-"poison" the table * space every time we mark it dirty. * * This function however is intended to operate on those dirty tables and * re-clean them. So when this function is used correctly, we can unpoison * the memory it operated on. This introduces a blind spot though, since * if we now try to operate on __actually__ poisoned memory, we will not * detect that. */ __msan_unpoison(table, size * sizeof(U32)); #endif for (rowNb=0 ; rowNb < nbRows ; rowNb++) { int column; for (column=0; columncParams.hashLog; ZSTD_reduceTable(ms->hashTable, hSize, reducerValue); } if (ZSTD_allocateChainTable(params->cParams.strategy, params->useRowMatchFinder, (U32)ms->dedicatedDictSearch)) { U32 const chainSize = (U32)1 << params->cParams.chainLog; if (params->cParams.strategy == ZSTD_btlazy2) ZSTD_reduceTable_btlazy2(ms->chainTable, chainSize, reducerValue); else ZSTD_reduceTable(ms->chainTable, chainSize, reducerValue); } if (ms->hashLog3) { U32 const h3Size = (U32)1 << ms->hashLog3; ZSTD_reduceTable(ms->hashTable3, h3Size, reducerValue); } } /*-******************************************************* * Block entropic compression *********************************************************/ /* See doc/zstd_compression_format.md for detailed format description */ int ZSTD_seqToCodes(const seqStore_t* seqStorePtr) { const seqDef* const sequences = seqStorePtr->sequencesStart; BYTE* const llCodeTable = seqStorePtr->llCode; BYTE* const ofCodeTable = seqStorePtr->ofCode; BYTE* const mlCodeTable = seqStorePtr->mlCode; U32 const nbSeq = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart); U32 u; int longOffsets = 0; assert(nbSeq <= seqStorePtr->maxNbSeq); for (u=0; u= STREAM_ACCUMULATOR_MIN)); if (MEM_32bits() && ofCode >= STREAM_ACCUMULATOR_MIN) longOffsets = 1; } if (seqStorePtr->longLengthType==ZSTD_llt_literalLength) llCodeTable[seqStorePtr->longLengthPos] = MaxLL; if (seqStorePtr->longLengthType==ZSTD_llt_matchLength) mlCodeTable[seqStorePtr->longLengthPos] = MaxML; return longOffsets; } /* ZSTD_useTargetCBlockSize(): * Returns if target compressed block size param is being used. * If used, compression will do best effort to make a compressed block size to be around targetCBlockSize. * Returns 1 if true, 0 otherwise. */ static int ZSTD_useTargetCBlockSize(const ZSTD_CCtx_params* cctxParams) { DEBUGLOG(5, "ZSTD_useTargetCBlockSize (targetCBlockSize=%zu)", cctxParams->targetCBlockSize); return (cctxParams->targetCBlockSize != 0); } /* ZSTD_blockSplitterEnabled(): * Returns if block splitting param is being used * If used, compression will do best effort to split a block in order to improve compression ratio. * At the time this function is called, the parameter must be finalized. * Returns 1 if true, 0 otherwise. */ static int ZSTD_blockSplitterEnabled(ZSTD_CCtx_params* cctxParams) { DEBUGLOG(5, "ZSTD_blockSplitterEnabled (useBlockSplitter=%d)", cctxParams->useBlockSplitter); assert(cctxParams->useBlockSplitter != ZSTD_ps_auto); return (cctxParams->useBlockSplitter == ZSTD_ps_enable); } /* Type returned by ZSTD_buildSequencesStatistics containing finalized symbol encoding types * and size of the sequences statistics */ typedef struct { U32 LLtype; U32 Offtype; U32 MLtype; size_t size; size_t lastCountSize; /* Accounts for bug in 1.3.4. More detail in ZSTD_entropyCompressSeqStore_internal() */ int longOffsets; } ZSTD_symbolEncodingTypeStats_t; /* ZSTD_buildSequencesStatistics(): * Returns a ZSTD_symbolEncodingTypeStats_t, or a zstd error code in the `size` field. * Modifies `nextEntropy` to have the appropriate values as a side effect. * nbSeq must be greater than 0. * * entropyWkspSize must be of size at least ENTROPY_WORKSPACE_SIZE - (MaxSeq + 1)*sizeof(U32) */ static ZSTD_symbolEncodingTypeStats_t ZSTD_buildSequencesStatistics( const seqStore_t* seqStorePtr, size_t nbSeq, const ZSTD_fseCTables_t* prevEntropy, ZSTD_fseCTables_t* nextEntropy, BYTE* dst, const BYTE* const dstEnd, ZSTD_strategy strategy, unsigned* countWorkspace, void* entropyWorkspace, size_t entropyWkspSize) { BYTE* const ostart = dst; const BYTE* const oend = dstEnd; BYTE* op = ostart; FSE_CTable* CTable_LitLength = nextEntropy->litlengthCTable; FSE_CTable* CTable_OffsetBits = nextEntropy->offcodeCTable; FSE_CTable* CTable_MatchLength = nextEntropy->matchlengthCTable; const BYTE* const ofCodeTable = seqStorePtr->ofCode; const BYTE* const llCodeTable = seqStorePtr->llCode; const BYTE* const mlCodeTable = seqStorePtr->mlCode; ZSTD_symbolEncodingTypeStats_t stats; stats.lastCountSize = 0; /* convert length/distances into codes */ stats.longOffsets = ZSTD_seqToCodes(seqStorePtr); assert(op <= oend); assert(nbSeq != 0); /* ZSTD_selectEncodingType() divides by nbSeq */ /* build CTable for Literal Lengths */ { unsigned max = MaxLL; size_t const mostFrequent = HIST_countFast_wksp(countWorkspace, &max, llCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */ DEBUGLOG(5, "Building LL table"); nextEntropy->litlength_repeatMode = prevEntropy->litlength_repeatMode; stats.LLtype = ZSTD_selectEncodingType(&nextEntropy->litlength_repeatMode, countWorkspace, max, mostFrequent, nbSeq, LLFSELog, prevEntropy->litlengthCTable, LL_defaultNorm, LL_defaultNormLog, ZSTD_defaultAllowed, strategy); assert(set_basic < set_compressed && set_rle < set_compressed); assert(!(stats.LLtype < set_compressed && nextEntropy->litlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */ { size_t const countSize = ZSTD_buildCTable( op, (size_t)(oend - op), CTable_LitLength, LLFSELog, (symbolEncodingType_e)stats.LLtype, countWorkspace, max, llCodeTable, nbSeq, LL_defaultNorm, LL_defaultNormLog, MaxLL, prevEntropy->litlengthCTable, sizeof(prevEntropy->litlengthCTable), entropyWorkspace, entropyWkspSize); if (ZSTD_isError(countSize)) { DEBUGLOG(3, "ZSTD_buildCTable for LitLens failed"); stats.size = countSize; return stats; } if (stats.LLtype == set_compressed) stats.lastCountSize = countSize; op += countSize; assert(op <= oend); } } /* build CTable for Offsets */ { unsigned max = MaxOff; size_t const mostFrequent = HIST_countFast_wksp( countWorkspace, &max, ofCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */ /* We can only use the basic table if max <= DefaultMaxOff, otherwise the offsets are too large */ ZSTD_defaultPolicy_e const defaultPolicy = (max <= DefaultMaxOff) ? ZSTD_defaultAllowed : ZSTD_defaultDisallowed; DEBUGLOG(5, "Building OF table"); nextEntropy->offcode_repeatMode = prevEntropy->offcode_repeatMode; stats.Offtype = ZSTD_selectEncodingType(&nextEntropy->offcode_repeatMode, countWorkspace, max, mostFrequent, nbSeq, OffFSELog, prevEntropy->offcodeCTable, OF_defaultNorm, OF_defaultNormLog, defaultPolicy, strategy); assert(!(stats.Offtype < set_compressed && nextEntropy->offcode_repeatMode != FSE_repeat_none)); /* We don't copy tables */ { size_t const countSize = ZSTD_buildCTable( op, (size_t)(oend - op), CTable_OffsetBits, OffFSELog, (symbolEncodingType_e)stats.Offtype, countWorkspace, max, ofCodeTable, nbSeq, OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff, prevEntropy->offcodeCTable, sizeof(prevEntropy->offcodeCTable), entropyWorkspace, entropyWkspSize); if (ZSTD_isError(countSize)) { DEBUGLOG(3, "ZSTD_buildCTable for Offsets failed"); stats.size = countSize; return stats; } if (stats.Offtype == set_compressed) stats.lastCountSize = countSize; op += countSize; assert(op <= oend); } } /* build CTable for MatchLengths */ { unsigned max = MaxML; size_t const mostFrequent = HIST_countFast_wksp( countWorkspace, &max, mlCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */ DEBUGLOG(5, "Building ML table (remaining space : %i)", (int)(oend-op)); nextEntropy->matchlength_repeatMode = prevEntropy->matchlength_repeatMode; stats.MLtype = ZSTD_selectEncodingType(&nextEntropy->matchlength_repeatMode, countWorkspace, max, mostFrequent, nbSeq, MLFSELog, prevEntropy->matchlengthCTable, ML_defaultNorm, ML_defaultNormLog, ZSTD_defaultAllowed, strategy); assert(!(stats.MLtype < set_compressed && nextEntropy->matchlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */ { size_t const countSize = ZSTD_buildCTable( op, (size_t)(oend - op), CTable_MatchLength, MLFSELog, (symbolEncodingType_e)stats.MLtype, countWorkspace, max, mlCodeTable, nbSeq, ML_defaultNorm, ML_defaultNormLog, MaxML, prevEntropy->matchlengthCTable, sizeof(prevEntropy->matchlengthCTable), entropyWorkspace, entropyWkspSize); if (ZSTD_isError(countSize)) { DEBUGLOG(3, "ZSTD_buildCTable for MatchLengths failed"); stats.size = countSize; return stats; } if (stats.MLtype == set_compressed) stats.lastCountSize = countSize; op += countSize; assert(op <= oend); } } stats.size = (size_t)(op-ostart); return stats; } /* ZSTD_entropyCompressSeqStore_internal(): * compresses both literals and sequences * Returns compressed size of block, or a zstd error. */ #define SUSPECT_UNCOMPRESSIBLE_LITERAL_RATIO 20 MEM_STATIC size_t ZSTD_entropyCompressSeqStore_internal( const seqStore_t* seqStorePtr, const ZSTD_entropyCTables_t* prevEntropy, ZSTD_entropyCTables_t* nextEntropy, const ZSTD_CCtx_params* cctxParams, void* dst, size_t dstCapacity, void* entropyWorkspace, size_t entropyWkspSize, const int bmi2) { ZSTD_strategy const strategy = cctxParams->cParams.strategy; unsigned* count = (unsigned*)entropyWorkspace; FSE_CTable* CTable_LitLength = nextEntropy->fse.litlengthCTable; FSE_CTable* CTable_OffsetBits = nextEntropy->fse.offcodeCTable; FSE_CTable* CTable_MatchLength = nextEntropy->fse.matchlengthCTable; const seqDef* const sequences = seqStorePtr->sequencesStart; const size_t nbSeq = (size_t)(seqStorePtr->sequences - seqStorePtr->sequencesStart); const BYTE* const ofCodeTable = seqStorePtr->ofCode; const BYTE* const llCodeTable = seqStorePtr->llCode; const BYTE* const mlCodeTable = seqStorePtr->mlCode; BYTE* const ostart = (BYTE*)dst; BYTE* const oend = ostart + dstCapacity; BYTE* op = ostart; size_t lastCountSize; int longOffsets = 0; entropyWorkspace = count + (MaxSeq + 1); entropyWkspSize -= (MaxSeq + 1) * sizeof(*count); DEBUGLOG(5, "ZSTD_entropyCompressSeqStore_internal (nbSeq=%zu, dstCapacity=%zu)", nbSeq, dstCapacity); ZSTD_STATIC_ASSERT(HUF_WORKSPACE_SIZE >= (1<= HUF_WORKSPACE_SIZE); /* Compress literals */ { const BYTE* const literals = seqStorePtr->litStart; size_t const numSequences = (size_t)(seqStorePtr->sequences - seqStorePtr->sequencesStart); size_t const numLiterals = (size_t)(seqStorePtr->lit - seqStorePtr->litStart); /* Base suspicion of uncompressibility on ratio of literals to sequences */ unsigned const suspectUncompressible = (numSequences == 0) || (numLiterals / numSequences >= SUSPECT_UNCOMPRESSIBLE_LITERAL_RATIO); size_t const litSize = (size_t)(seqStorePtr->lit - literals); size_t const cSize = ZSTD_compressLiterals( op, dstCapacity, literals, litSize, entropyWorkspace, entropyWkspSize, &prevEntropy->huf, &nextEntropy->huf, cctxParams->cParams.strategy, ZSTD_literalsCompressionIsDisabled(cctxParams), suspectUncompressible, bmi2); FORWARD_IF_ERROR(cSize, "ZSTD_compressLiterals failed"); assert(cSize <= dstCapacity); op += cSize; } /* Sequences Header */ RETURN_ERROR_IF((oend-op) < 3 /*max nbSeq Size*/ + 1 /*seqHead*/, dstSize_tooSmall, "Can't fit seq hdr in output buf!"); if (nbSeq < 128) { *op++ = (BYTE)nbSeq; } else if (nbSeq < LONGNBSEQ) { op[0] = (BYTE)((nbSeq>>8) + 0x80); op[1] = (BYTE)nbSeq; op+=2; } else { op[0]=0xFF; MEM_writeLE16(op+1, (U16)(nbSeq - LONGNBSEQ)); op+=3; } assert(op <= oend); if (nbSeq==0) { /* Copy the old tables over as if we repeated them */ ZSTD_memcpy(&nextEntropy->fse, &prevEntropy->fse, sizeof(prevEntropy->fse)); return (size_t)(op - ostart); } { BYTE* const seqHead = op++; /* build stats for sequences */ const ZSTD_symbolEncodingTypeStats_t stats = ZSTD_buildSequencesStatistics(seqStorePtr, nbSeq, &prevEntropy->fse, &nextEntropy->fse, op, oend, strategy, count, entropyWorkspace, entropyWkspSize); FORWARD_IF_ERROR(stats.size, "ZSTD_buildSequencesStatistics failed!"); *seqHead = (BYTE)((stats.LLtype<<6) + (stats.Offtype<<4) + (stats.MLtype<<2)); lastCountSize = stats.lastCountSize; op += stats.size; longOffsets = stats.longOffsets; } { size_t const bitstreamSize = ZSTD_encodeSequences( op, (size_t)(oend - op), CTable_MatchLength, mlCodeTable, CTable_OffsetBits, ofCodeTable, CTable_LitLength, llCodeTable, sequences, nbSeq, longOffsets, bmi2); FORWARD_IF_ERROR(bitstreamSize, "ZSTD_encodeSequences failed"); op += bitstreamSize; assert(op <= oend); /* zstd versions <= 1.3.4 mistakenly report corruption when * FSE_readNCount() receives a buffer < 4 bytes. * Fixed by https://github.com/facebook/zstd/pull/1146. * This can happen when the last set_compressed table present is 2 * bytes and the bitstream is only one byte. * In this exceedingly rare case, we will simply emit an uncompressed * block, since it isn't worth optimizing. */ if (lastCountSize && (lastCountSize + bitstreamSize) < 4) { /* lastCountSize >= 2 && bitstreamSize > 0 ==> lastCountSize == 3 */ assert(lastCountSize + bitstreamSize == 3); DEBUGLOG(5, "Avoiding bug in zstd decoder in versions <= 1.3.4 by " "emitting an uncompressed block."); return 0; } } DEBUGLOG(5, "compressed block size : %u", (unsigned)(op - ostart)); return (size_t)(op - ostart); } MEM_STATIC size_t ZSTD_entropyCompressSeqStore( const seqStore_t* seqStorePtr, const ZSTD_entropyCTables_t* prevEntropy, ZSTD_entropyCTables_t* nextEntropy, const ZSTD_CCtx_params* cctxParams, void* dst, size_t dstCapacity, size_t srcSize, void* entropyWorkspace, size_t entropyWkspSize, int bmi2) { size_t const cSize = ZSTD_entropyCompressSeqStore_internal( seqStorePtr, prevEntropy, nextEntropy, cctxParams, dst, dstCapacity, entropyWorkspace, entropyWkspSize, bmi2); if (cSize == 0) return 0; /* When srcSize <= dstCapacity, there is enough space to write a raw uncompressed block. * Since we ran out of space, block must be not compressible, so fall back to raw uncompressed block. */ if ((cSize == ERROR(dstSize_tooSmall)) & (srcSize <= dstCapacity)) { DEBUGLOG(4, "not enough dstCapacity (%zu) for ZSTD_entropyCompressSeqStore_internal()=> do not compress block", dstCapacity); return 0; /* block not compressed */ } FORWARD_IF_ERROR(cSize, "ZSTD_entropyCompressSeqStore_internal failed"); /* Check compressibility */ { size_t const maxCSize = srcSize - ZSTD_minGain(srcSize, cctxParams->cParams.strategy); if (cSize >= maxCSize) return 0; /* block not compressed */ } DEBUGLOG(5, "ZSTD_entropyCompressSeqStore() cSize: %zu", cSize); /* libzstd decoder before > v1.5.4 is not compatible with compressed blocks of size ZSTD_BLOCKSIZE_MAX exactly. * This restriction is indirectly already fulfilled by respecting ZSTD_minGain() condition above. */ assert(cSize < ZSTD_BLOCKSIZE_MAX); return cSize; } /* ZSTD_selectBlockCompressor() : * Not static, but internal use only (used by long distance matcher) * assumption : strat is a valid strategy */ ZSTD_blockCompressor ZSTD_selectBlockCompressor(ZSTD_strategy strat, ZSTD_paramSwitch_e useRowMatchFinder, ZSTD_dictMode_e dictMode) { static const ZSTD_blockCompressor blockCompressor[4][ZSTD_STRATEGY_MAX+1] = { { ZSTD_compressBlock_fast /* default for 0 */, ZSTD_compressBlock_fast, ZSTD_compressBlock_doubleFast, ZSTD_compressBlock_greedy, ZSTD_compressBlock_lazy, ZSTD_compressBlock_lazy2, ZSTD_compressBlock_btlazy2, ZSTD_compressBlock_btopt, ZSTD_compressBlock_btultra, ZSTD_compressBlock_btultra2 }, { ZSTD_compressBlock_fast_extDict /* default for 0 */, ZSTD_compressBlock_fast_extDict, ZSTD_compressBlock_doubleFast_extDict, ZSTD_compressBlock_greedy_extDict, ZSTD_compressBlock_lazy_extDict, ZSTD_compressBlock_lazy2_extDict, ZSTD_compressBlock_btlazy2_extDict, ZSTD_compressBlock_btopt_extDict, ZSTD_compressBlock_btultra_extDict, ZSTD_compressBlock_btultra_extDict }, { ZSTD_compressBlock_fast_dictMatchState /* default for 0 */, ZSTD_compressBlock_fast_dictMatchState, ZSTD_compressBlock_doubleFast_dictMatchState, ZSTD_compressBlock_greedy_dictMatchState, ZSTD_compressBlock_lazy_dictMatchState, ZSTD_compressBlock_lazy2_dictMatchState, ZSTD_compressBlock_btlazy2_dictMatchState, ZSTD_compressBlock_btopt_dictMatchState, ZSTD_compressBlock_btultra_dictMatchState, ZSTD_compressBlock_btultra_dictMatchState }, { NULL /* default for 0 */, NULL, NULL, ZSTD_compressBlock_greedy_dedicatedDictSearch, ZSTD_compressBlock_lazy_dedicatedDictSearch, ZSTD_compressBlock_lazy2_dedicatedDictSearch, NULL, NULL, NULL, NULL } }; ZSTD_blockCompressor selectedCompressor; ZSTD_STATIC_ASSERT((unsigned)ZSTD_fast == 1); assert(ZSTD_cParam_withinBounds(ZSTD_c_strategy, strat)); DEBUGLOG(4, "Selected block compressor: dictMode=%d strat=%d rowMatchfinder=%d", (int)dictMode, (int)strat, (int)useRowMatchFinder); if (ZSTD_rowMatchFinderUsed(strat, useRowMatchFinder)) { static const ZSTD_blockCompressor rowBasedBlockCompressors[4][3] = { { ZSTD_compressBlock_greedy_row, ZSTD_compressBlock_lazy_row, ZSTD_compressBlock_lazy2_row }, { ZSTD_compressBlock_greedy_extDict_row, ZSTD_compressBlock_lazy_extDict_row, ZSTD_compressBlock_lazy2_extDict_row }, { ZSTD_compressBlock_greedy_dictMatchState_row, ZSTD_compressBlock_lazy_dictMatchState_row, ZSTD_compressBlock_lazy2_dictMatchState_row }, { ZSTD_compressBlock_greedy_dedicatedDictSearch_row, ZSTD_compressBlock_lazy_dedicatedDictSearch_row, ZSTD_compressBlock_lazy2_dedicatedDictSearch_row } }; DEBUGLOG(4, "Selecting a row-based matchfinder"); assert(useRowMatchFinder != ZSTD_ps_auto); selectedCompressor = rowBasedBlockCompressors[(int)dictMode][(int)strat - (int)ZSTD_greedy]; } else { selectedCompressor = blockCompressor[(int)dictMode][(int)strat]; } assert(selectedCompressor != NULL); return selectedCompressor; } static void ZSTD_storeLastLiterals(seqStore_t* seqStorePtr, const BYTE* anchor, size_t lastLLSize) { ZSTD_memcpy(seqStorePtr->lit, anchor, lastLLSize); seqStorePtr->lit += lastLLSize; } void ZSTD_resetSeqStore(seqStore_t* ssPtr) { ssPtr->lit = ssPtr->litStart; ssPtr->sequences = ssPtr->sequencesStart; ssPtr->longLengthType = ZSTD_llt_none; } /* ZSTD_postProcessSequenceProducerResult() : * Validates and post-processes sequences obtained through the external matchfinder API: * - Checks whether nbExternalSeqs represents an error condition. * - Appends a block delimiter to outSeqs if one is not already present. * See zstd.h for context regarding block delimiters. * Returns the number of sequences after post-processing, or an error code. */ static size_t ZSTD_postProcessSequenceProducerResult( ZSTD_Sequence* outSeqs, size_t nbExternalSeqs, size_t outSeqsCapacity, size_t srcSize ) { RETURN_ERROR_IF( nbExternalSeqs > outSeqsCapacity, sequenceProducer_failed, "External sequence producer returned error code %lu", (unsigned long)nbExternalSeqs ); RETURN_ERROR_IF( nbExternalSeqs == 0 && srcSize > 0, sequenceProducer_failed, "Got zero sequences from external sequence producer for a non-empty src buffer!" ); if (srcSize == 0) { ZSTD_memset(&outSeqs[0], 0, sizeof(ZSTD_Sequence)); return 1; } { ZSTD_Sequence const lastSeq = outSeqs[nbExternalSeqs - 1]; /* We can return early if lastSeq is already a block delimiter. */ if (lastSeq.offset == 0 && lastSeq.matchLength == 0) { return nbExternalSeqs; } /* This error condition is only possible if the external matchfinder * produced an invalid parse, by definition of ZSTD_sequenceBound(). */ RETURN_ERROR_IF( nbExternalSeqs == outSeqsCapacity, sequenceProducer_failed, "nbExternalSeqs == outSeqsCapacity but lastSeq is not a block delimiter!" ); /* lastSeq is not a block delimiter, so we need to append one. */ ZSTD_memset(&outSeqs[nbExternalSeqs], 0, sizeof(ZSTD_Sequence)); return nbExternalSeqs + 1; } } /* ZSTD_fastSequenceLengthSum() : * Returns sum(litLen) + sum(matchLen) + lastLits for *seqBuf*. * Similar to another function in zstd_compress.c (determine_blockSize), * except it doesn't check for a block delimiter to end summation. * Removing the early exit allows the compiler to auto-vectorize (https://godbolt.org/z/cY1cajz9P). * This function can be deleted and replaced by determine_blockSize after we resolve issue #3456. */ static size_t ZSTD_fastSequenceLengthSum(ZSTD_Sequence const* seqBuf, size_t seqBufSize) { size_t matchLenSum, litLenSum, i; matchLenSum = 0; litLenSum = 0; for (i = 0; i < seqBufSize; i++) { litLenSum += seqBuf[i].litLength; matchLenSum += seqBuf[i].matchLength; } return litLenSum + matchLenSum; } typedef enum { ZSTDbss_compress, ZSTDbss_noCompress } ZSTD_buildSeqStore_e; static size_t ZSTD_buildSeqStore(ZSTD_CCtx* zc, const void* src, size_t srcSize) { ZSTD_matchState_t* const ms = &zc->blockState.matchState; DEBUGLOG(5, "ZSTD_buildSeqStore (srcSize=%zu)", srcSize); assert(srcSize <= ZSTD_BLOCKSIZE_MAX); /* Assert that we have correctly flushed the ctx params into the ms's copy */ ZSTD_assertEqualCParams(zc->appliedParams.cParams, ms->cParams); /* TODO: See 3090. We reduced MIN_CBLOCK_SIZE from 3 to 2 so to compensate we are adding * additional 1. We need to revisit and change this logic to be more consistent */ if (srcSize < MIN_CBLOCK_SIZE+ZSTD_blockHeaderSize+1+1) { if (zc->appliedParams.cParams.strategy >= ZSTD_btopt) { ZSTD_ldm_skipRawSeqStoreBytes(&zc->externSeqStore, srcSize); } else { ZSTD_ldm_skipSequences(&zc->externSeqStore, srcSize, zc->appliedParams.cParams.minMatch); } return ZSTDbss_noCompress; /* don't even attempt compression below a certain srcSize */ } ZSTD_resetSeqStore(&(zc->seqStore)); /* required for optimal parser to read stats from dictionary */ ms->opt.symbolCosts = &zc->blockState.prevCBlock->entropy; /* tell the optimal parser how we expect to compress literals */ ms->opt.literalCompressionMode = zc->appliedParams.literalCompressionMode; /* a gap between an attached dict and the current window is not safe, * they must remain adjacent, * and when that stops being the case, the dict must be unset */ assert(ms->dictMatchState == NULL || ms->loadedDictEnd == ms->window.dictLimit); /* limited update after a very long match */ { const BYTE* const base = ms->window.base; const BYTE* const istart = (const BYTE*)src; const U32 curr = (U32)(istart-base); if (sizeof(ptrdiff_t)==8) assert(istart - base < (ptrdiff_t)(U32)(-1)); /* ensure no overflow */ if (curr > ms->nextToUpdate + 384) ms->nextToUpdate = curr - MIN(192, (U32)(curr - ms->nextToUpdate - 384)); } /* select and store sequences */ { ZSTD_dictMode_e const dictMode = ZSTD_matchState_dictMode(ms); size_t lastLLSize; { int i; for (i = 0; i < ZSTD_REP_NUM; ++i) zc->blockState.nextCBlock->rep[i] = zc->blockState.prevCBlock->rep[i]; } if (zc->externSeqStore.pos < zc->externSeqStore.size) { assert(zc->appliedParams.ldmParams.enableLdm == ZSTD_ps_disable); /* External matchfinder + LDM is technically possible, just not implemented yet. * We need to revisit soon and implement it. */ RETURN_ERROR_IF( zc->appliedParams.useSequenceProducer, parameter_combination_unsupported, "Long-distance matching with external sequence producer enabled is not currently supported." ); /* Updates ldmSeqStore.pos */ lastLLSize = ZSTD_ldm_blockCompress(&zc->externSeqStore, ms, &zc->seqStore, zc->blockState.nextCBlock->rep, zc->appliedParams.useRowMatchFinder, src, srcSize); assert(zc->externSeqStore.pos <= zc->externSeqStore.size); } else if (zc->appliedParams.ldmParams.enableLdm == ZSTD_ps_enable) { rawSeqStore_t ldmSeqStore = kNullRawSeqStore; /* External matchfinder + LDM is technically possible, just not implemented yet. * We need to revisit soon and implement it. */ RETURN_ERROR_IF( zc->appliedParams.useSequenceProducer, parameter_combination_unsupported, "Long-distance matching with external sequence producer enabled is not currently supported." ); ldmSeqStore.seq = zc->ldmSequences; ldmSeqStore.capacity = zc->maxNbLdmSequences; /* Updates ldmSeqStore.size */ FORWARD_IF_ERROR(ZSTD_ldm_generateSequences(&zc->ldmState, &ldmSeqStore, &zc->appliedParams.ldmParams, src, srcSize), ""); /* Updates ldmSeqStore.pos */ lastLLSize = ZSTD_ldm_blockCompress(&ldmSeqStore, ms, &zc->seqStore, zc->blockState.nextCBlock->rep, zc->appliedParams.useRowMatchFinder, src, srcSize); assert(ldmSeqStore.pos == ldmSeqStore.size); } else if (zc->appliedParams.useSequenceProducer) { assert( zc->externalMatchCtx.seqBufferCapacity >= ZSTD_sequenceBound(srcSize) ); assert(zc->externalMatchCtx.mFinder != NULL); { U32 const windowSize = (U32)1 << zc->appliedParams.cParams.windowLog; size_t const nbExternalSeqs = (zc->externalMatchCtx.mFinder)( zc->externalMatchCtx.mState, zc->externalMatchCtx.seqBuffer, zc->externalMatchCtx.seqBufferCapacity, src, srcSize, NULL, 0, /* dict and dictSize, currently not supported */ zc->appliedParams.compressionLevel, windowSize ); size_t const nbPostProcessedSeqs = ZSTD_postProcessSequenceProducerResult( zc->externalMatchCtx.seqBuffer, nbExternalSeqs, zc->externalMatchCtx.seqBufferCapacity, srcSize ); /* Return early if there is no error, since we don't need to worry about last literals */ if (!ZSTD_isError(nbPostProcessedSeqs)) { ZSTD_sequencePosition seqPos = {0,0,0}; size_t const seqLenSum = ZSTD_fastSequenceLengthSum(zc->externalMatchCtx.seqBuffer, nbPostProcessedSeqs); RETURN_ERROR_IF(seqLenSum > srcSize, externalSequences_invalid, "External sequences imply too large a block!"); FORWARD_IF_ERROR( ZSTD_copySequencesToSeqStoreExplicitBlockDelim( zc, &seqPos, zc->externalMatchCtx.seqBuffer, nbPostProcessedSeqs, src, srcSize, zc->appliedParams.searchForExternalRepcodes ), "Failed to copy external sequences to seqStore!" ); ms->ldmSeqStore = NULL; DEBUGLOG(5, "Copied %lu sequences from external sequence producer to internal seqStore.", (unsigned long)nbExternalSeqs); return ZSTDbss_compress; } /* Propagate the error if fallback is disabled */ if (!zc->appliedParams.enableMatchFinderFallback) { return nbPostProcessedSeqs; } /* Fallback to software matchfinder */ { ZSTD_blockCompressor const blockCompressor = ZSTD_selectBlockCompressor(zc->appliedParams.cParams.strategy, zc->appliedParams.useRowMatchFinder, dictMode); ms->ldmSeqStore = NULL; DEBUGLOG( 5, "External sequence producer returned error code %lu. Falling back to internal parser.", (unsigned long)nbExternalSeqs ); lastLLSize = blockCompressor(ms, &zc->seqStore, zc->blockState.nextCBlock->rep, src, srcSize); } } } else { /* not long range mode and no external matchfinder */ ZSTD_blockCompressor const blockCompressor = ZSTD_selectBlockCompressor(zc->appliedParams.cParams.strategy, zc->appliedParams.useRowMatchFinder, dictMode); ms->ldmSeqStore = NULL; lastLLSize = blockCompressor(ms, &zc->seqStore, zc->blockState.nextCBlock->rep, src, srcSize); } { const BYTE* const lastLiterals = (const BYTE*)src + srcSize - lastLLSize; ZSTD_storeLastLiterals(&zc->seqStore, lastLiterals, lastLLSize); } } return ZSTDbss_compress; } static void ZSTD_copyBlockSequences(ZSTD_CCtx* zc) { const seqStore_t* seqStore = ZSTD_getSeqStore(zc); const seqDef* seqStoreSeqs = seqStore->sequencesStart; size_t seqStoreSeqSize = seqStore->sequences - seqStoreSeqs; size_t seqStoreLiteralsSize = (size_t)(seqStore->lit - seqStore->litStart); size_t literalsRead = 0; size_t lastLLSize; ZSTD_Sequence* outSeqs = &zc->seqCollector.seqStart[zc->seqCollector.seqIndex]; size_t i; repcodes_t updatedRepcodes; assert(zc->seqCollector.seqIndex + 1 < zc->seqCollector.maxSequences); /* Ensure we have enough space for last literals "sequence" */ assert(zc->seqCollector.maxSequences >= seqStoreSeqSize + 1); ZSTD_memcpy(updatedRepcodes.rep, zc->blockState.prevCBlock->rep, sizeof(repcodes_t)); for (i = 0; i < seqStoreSeqSize; ++i) { U32 rawOffset = seqStoreSeqs[i].offBase - ZSTD_REP_NUM; outSeqs[i].litLength = seqStoreSeqs[i].litLength; outSeqs[i].matchLength = seqStoreSeqs[i].mlBase + MINMATCH; outSeqs[i].rep = 0; if (i == seqStore->longLengthPos) { if (seqStore->longLengthType == ZSTD_llt_literalLength) { outSeqs[i].litLength += 0x10000; } else if (seqStore->longLengthType == ZSTD_llt_matchLength) { outSeqs[i].matchLength += 0x10000; } } if (seqStoreSeqs[i].offBase <= ZSTD_REP_NUM) { /* Derive the correct offset corresponding to a repcode */ outSeqs[i].rep = seqStoreSeqs[i].offBase; if (outSeqs[i].litLength != 0) { rawOffset = updatedRepcodes.rep[outSeqs[i].rep - 1]; } else { if (outSeqs[i].rep == 3) { rawOffset = updatedRepcodes.rep[0] - 1; } else { rawOffset = updatedRepcodes.rep[outSeqs[i].rep]; } } } outSeqs[i].offset = rawOffset; /* seqStoreSeqs[i].offset == offCode+1, and ZSTD_updateRep() expects offCode so we provide seqStoreSeqs[i].offset - 1 */ ZSTD_updateRep(updatedRepcodes.rep, seqStoreSeqs[i].offBase, seqStoreSeqs[i].litLength == 0); literalsRead += outSeqs[i].litLength; } /* Insert last literals (if any exist) in the block as a sequence with ml == off == 0. * If there are no last literals, then we'll emit (of: 0, ml: 0, ll: 0), which is a marker * for the block boundary, according to the API. */ assert(seqStoreLiteralsSize >= literalsRead); lastLLSize = seqStoreLiteralsSize - literalsRead; outSeqs[i].litLength = (U32)lastLLSize; outSeqs[i].matchLength = outSeqs[i].offset = outSeqs[i].rep = 0; seqStoreSeqSize++; zc->seqCollector.seqIndex += seqStoreSeqSize; } size_t ZSTD_sequenceBound(size_t srcSize) { return (srcSize / ZSTD_MINMATCH_MIN) + 1; } size_t ZSTD_generateSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs, size_t outSeqsSize, const void* src, size_t srcSize) { const size_t dstCapacity = ZSTD_compressBound(srcSize); void* dst = ZSTD_customMalloc(dstCapacity, ZSTD_defaultCMem); SeqCollector seqCollector; RETURN_ERROR_IF(dst == NULL, memory_allocation, "NULL pointer!"); seqCollector.collectSequences = 1; seqCollector.seqStart = outSeqs; seqCollector.seqIndex = 0; seqCollector.maxSequences = outSeqsSize; zc->seqCollector = seqCollector; ZSTD_compress2(zc, dst, dstCapacity, src, srcSize); ZSTD_customFree(dst, ZSTD_defaultCMem); return zc->seqCollector.seqIndex; } size_t ZSTD_mergeBlockDelimiters(ZSTD_Sequence* sequences, size_t seqsSize) { size_t in = 0; size_t out = 0; for (; in < seqsSize; ++in) { if (sequences[in].offset == 0 && sequences[in].matchLength == 0) { if (in != seqsSize - 1) { sequences[in+1].litLength += sequences[in].litLength; } } else { sequences[out] = sequences[in]; ++out; } } return out; } /* Unrolled loop to read four size_ts of input at a time. Returns 1 if is RLE, 0 if not. */ static int ZSTD_isRLE(const BYTE* src, size_t length) { const BYTE* ip = src; const BYTE value = ip[0]; const size_t valueST = (size_t)((U64)value * 0x0101010101010101ULL); const size_t unrollSize = sizeof(size_t) * 4; const size_t unrollMask = unrollSize - 1; const size_t prefixLength = length & unrollMask; size_t i; if (length == 1) return 1; /* Check if prefix is RLE first before using unrolled loop */ if (prefixLength && ZSTD_count(ip+1, ip, ip+prefixLength) != prefixLength-1) { return 0; } for (i = prefixLength; i != length; i += unrollSize) { size_t u; for (u = 0; u < unrollSize; u += sizeof(size_t)) { if (MEM_readST(ip + i + u) != valueST) { return 0; } } } return 1; } /* Returns true if the given block may be RLE. * This is just a heuristic based on the compressibility. * It may return both false positives and false negatives. */ static int ZSTD_maybeRLE(seqStore_t const* seqStore) { size_t const nbSeqs = (size_t)(seqStore->sequences - seqStore->sequencesStart); size_t const nbLits = (size_t)(seqStore->lit - seqStore->litStart); return nbSeqs < 4 && nbLits < 10; } static void ZSTD_blockState_confirmRepcodesAndEntropyTables(ZSTD_blockState_t* const bs) { ZSTD_compressedBlockState_t* const tmp = bs->prevCBlock; bs->prevCBlock = bs->nextCBlock; bs->nextCBlock = tmp; } /* Writes the block header */ static void writeBlockHeader(void* op, size_t cSize, size_t blockSize, U32 lastBlock) { U32 const cBlockHeader = cSize == 1 ? lastBlock + (((U32)bt_rle)<<1) + (U32)(blockSize << 3) : lastBlock + (((U32)bt_compressed)<<1) + (U32)(cSize << 3); MEM_writeLE24(op, cBlockHeader); DEBUGLOG(3, "writeBlockHeader: cSize: %zu blockSize: %zu lastBlock: %u", cSize, blockSize, lastBlock); } /** ZSTD_buildBlockEntropyStats_literals() : * Builds entropy for the literals. * Stores literals block type (raw, rle, compressed, repeat) and * huffman description table to hufMetadata. * Requires ENTROPY_WORKSPACE_SIZE workspace * @return : size of huffman description table, or an error code */ static size_t ZSTD_buildBlockEntropyStats_literals(void* const src, size_t srcSize, const ZSTD_hufCTables_t* prevHuf, ZSTD_hufCTables_t* nextHuf, ZSTD_hufCTablesMetadata_t* hufMetadata, const int literalsCompressionIsDisabled, void* workspace, size_t wkspSize, int hufFlags) { BYTE* const wkspStart = (BYTE*)workspace; BYTE* const wkspEnd = wkspStart + wkspSize; BYTE* const countWkspStart = wkspStart; unsigned* const countWksp = (unsigned*)workspace; const size_t countWkspSize = (HUF_SYMBOLVALUE_MAX + 1) * sizeof(unsigned); BYTE* const nodeWksp = countWkspStart + countWkspSize; const size_t nodeWkspSize = (size_t)(wkspEnd - nodeWksp); unsigned maxSymbolValue = HUF_SYMBOLVALUE_MAX; unsigned huffLog = LitHufLog; HUF_repeat repeat = prevHuf->repeatMode; DEBUGLOG(5, "ZSTD_buildBlockEntropyStats_literals (srcSize=%zu)", srcSize); /* Prepare nextEntropy assuming reusing the existing table */ ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf)); if (literalsCompressionIsDisabled) { DEBUGLOG(5, "set_basic - disabled"); hufMetadata->hType = set_basic; return 0; } /* small ? don't even attempt compression (speed opt) */ #ifndef COMPRESS_LITERALS_SIZE_MIN # define COMPRESS_LITERALS_SIZE_MIN 63 /* heuristic */ #endif { size_t const minLitSize = (prevHuf->repeatMode == HUF_repeat_valid) ? 6 : COMPRESS_LITERALS_SIZE_MIN; if (srcSize <= minLitSize) { DEBUGLOG(5, "set_basic - too small"); hufMetadata->hType = set_basic; return 0; } } /* Scan input and build symbol stats */ { size_t const largest = HIST_count_wksp (countWksp, &maxSymbolValue, (const BYTE*)src, srcSize, workspace, wkspSize); FORWARD_IF_ERROR(largest, "HIST_count_wksp failed"); if (largest == srcSize) { /* only one literal symbol */ DEBUGLOG(5, "set_rle"); hufMetadata->hType = set_rle; return 0; } if (largest <= (srcSize >> 7)+4) { /* heuristic: likely not compressible */ DEBUGLOG(5, "set_basic - no gain"); hufMetadata->hType = set_basic; return 0; } } /* Validate the previous Huffman table */ if (repeat == HUF_repeat_check && !HUF_validateCTable((HUF_CElt const*)prevHuf->CTable, countWksp, maxSymbolValue)) { repeat = HUF_repeat_none; } /* Build Huffman Tree */ ZSTD_memset(nextHuf->CTable, 0, sizeof(nextHuf->CTable)); huffLog = HUF_optimalTableLog(huffLog, srcSize, maxSymbolValue, nodeWksp, nodeWkspSize, nextHuf->CTable, countWksp, hufFlags); assert(huffLog <= LitHufLog); { size_t const maxBits = HUF_buildCTable_wksp((HUF_CElt*)nextHuf->CTable, countWksp, maxSymbolValue, huffLog, nodeWksp, nodeWkspSize); FORWARD_IF_ERROR(maxBits, "HUF_buildCTable_wksp"); huffLog = (U32)maxBits; } { /* Build and write the CTable */ size_t const newCSize = HUF_estimateCompressedSize( (HUF_CElt*)nextHuf->CTable, countWksp, maxSymbolValue); size_t const hSize = HUF_writeCTable_wksp( hufMetadata->hufDesBuffer, sizeof(hufMetadata->hufDesBuffer), (HUF_CElt*)nextHuf->CTable, maxSymbolValue, huffLog, nodeWksp, nodeWkspSize); /* Check against repeating the previous CTable */ if (repeat != HUF_repeat_none) { size_t const oldCSize = HUF_estimateCompressedSize( (HUF_CElt const*)prevHuf->CTable, countWksp, maxSymbolValue); if (oldCSize < srcSize && (oldCSize <= hSize + newCSize || hSize + 12 >= srcSize)) { DEBUGLOG(5, "set_repeat - smaller"); ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf)); hufMetadata->hType = set_repeat; return 0; } } if (newCSize + hSize >= srcSize) { DEBUGLOG(5, "set_basic - no gains"); ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf)); hufMetadata->hType = set_basic; return 0; } DEBUGLOG(5, "set_compressed (hSize=%u)", (U32)hSize); hufMetadata->hType = set_compressed; nextHuf->repeatMode = HUF_repeat_check; return hSize; } } /* ZSTD_buildDummySequencesStatistics(): * Returns a ZSTD_symbolEncodingTypeStats_t with all encoding types as set_basic, * and updates nextEntropy to the appropriate repeatMode. */ static ZSTD_symbolEncodingTypeStats_t ZSTD_buildDummySequencesStatistics(ZSTD_fseCTables_t* nextEntropy) { ZSTD_symbolEncodingTypeStats_t stats = {set_basic, set_basic, set_basic, 0, 0, 0}; nextEntropy->litlength_repeatMode = FSE_repeat_none; nextEntropy->offcode_repeatMode = FSE_repeat_none; nextEntropy->matchlength_repeatMode = FSE_repeat_none; return stats; } /** ZSTD_buildBlockEntropyStats_sequences() : * Builds entropy for the sequences. * Stores symbol compression modes and fse table to fseMetadata. * Requires ENTROPY_WORKSPACE_SIZE wksp. * @return : size of fse tables or error code */ static size_t ZSTD_buildBlockEntropyStats_sequences( const seqStore_t* seqStorePtr, const ZSTD_fseCTables_t* prevEntropy, ZSTD_fseCTables_t* nextEntropy, const ZSTD_CCtx_params* cctxParams, ZSTD_fseCTablesMetadata_t* fseMetadata, void* workspace, size_t wkspSize) { ZSTD_strategy const strategy = cctxParams->cParams.strategy; size_t const nbSeq = (size_t)(seqStorePtr->sequences - seqStorePtr->sequencesStart); BYTE* const ostart = fseMetadata->fseTablesBuffer; BYTE* const oend = ostart + sizeof(fseMetadata->fseTablesBuffer); BYTE* op = ostart; unsigned* countWorkspace = (unsigned*)workspace; unsigned* entropyWorkspace = countWorkspace + (MaxSeq + 1); size_t entropyWorkspaceSize = wkspSize - (MaxSeq + 1) * sizeof(*countWorkspace); ZSTD_symbolEncodingTypeStats_t stats; DEBUGLOG(5, "ZSTD_buildBlockEntropyStats_sequences (nbSeq=%zu)", nbSeq); stats = nbSeq != 0 ? ZSTD_buildSequencesStatistics(seqStorePtr, nbSeq, prevEntropy, nextEntropy, op, oend, strategy, countWorkspace, entropyWorkspace, entropyWorkspaceSize) : ZSTD_buildDummySequencesStatistics(nextEntropy); FORWARD_IF_ERROR(stats.size, "ZSTD_buildSequencesStatistics failed!"); fseMetadata->llType = (symbolEncodingType_e) stats.LLtype; fseMetadata->ofType = (symbolEncodingType_e) stats.Offtype; fseMetadata->mlType = (symbolEncodingType_e) stats.MLtype; fseMetadata->lastCountSize = stats.lastCountSize; return stats.size; } /** ZSTD_buildBlockEntropyStats() : * Builds entropy for the block. * Requires workspace size ENTROPY_WORKSPACE_SIZE * @return : 0 on success, or an error code * Note : also employed in superblock */ size_t ZSTD_buildBlockEntropyStats( const seqStore_t* seqStorePtr, const ZSTD_entropyCTables_t* prevEntropy, ZSTD_entropyCTables_t* nextEntropy, const ZSTD_CCtx_params* cctxParams, ZSTD_entropyCTablesMetadata_t* entropyMetadata, void* workspace, size_t wkspSize) { size_t const litSize = (size_t)(seqStorePtr->lit - seqStorePtr->litStart); int const huf_useOptDepth = (cctxParams->cParams.strategy >= HUF_OPTIMAL_DEPTH_THRESHOLD); int const hufFlags = huf_useOptDepth ? HUF_flags_optimalDepth : 0; entropyMetadata->hufMetadata.hufDesSize = ZSTD_buildBlockEntropyStats_literals(seqStorePtr->litStart, litSize, &prevEntropy->huf, &nextEntropy->huf, &entropyMetadata->hufMetadata, ZSTD_literalsCompressionIsDisabled(cctxParams), workspace, wkspSize, hufFlags); FORWARD_IF_ERROR(entropyMetadata->hufMetadata.hufDesSize, "ZSTD_buildBlockEntropyStats_literals failed"); entropyMetadata->fseMetadata.fseTablesSize = ZSTD_buildBlockEntropyStats_sequences(seqStorePtr, &prevEntropy->fse, &nextEntropy->fse, cctxParams, &entropyMetadata->fseMetadata, workspace, wkspSize); FORWARD_IF_ERROR(entropyMetadata->fseMetadata.fseTablesSize, "ZSTD_buildBlockEntropyStats_sequences failed"); return 0; } /* Returns the size estimate for the literals section (header + content) of a block */ static size_t ZSTD_estimateBlockSize_literal(const BYTE* literals, size_t litSize, const ZSTD_hufCTables_t* huf, const ZSTD_hufCTablesMetadata_t* hufMetadata, void* workspace, size_t wkspSize, int writeEntropy) { unsigned* const countWksp = (unsigned*)workspace; unsigned maxSymbolValue = HUF_SYMBOLVALUE_MAX; size_t literalSectionHeaderSize = 3 + (litSize >= 1 KB) + (litSize >= 16 KB); U32 singleStream = litSize < 256; if (hufMetadata->hType == set_basic) return litSize; else if (hufMetadata->hType == set_rle) return 1; else if (hufMetadata->hType == set_compressed || hufMetadata->hType == set_repeat) { size_t const largest = HIST_count_wksp (countWksp, &maxSymbolValue, (const BYTE*)literals, litSize, workspace, wkspSize); if (ZSTD_isError(largest)) return litSize; { size_t cLitSizeEstimate = HUF_estimateCompressedSize((const HUF_CElt*)huf->CTable, countWksp, maxSymbolValue); if (writeEntropy) cLitSizeEstimate += hufMetadata->hufDesSize; if (!singleStream) cLitSizeEstimate += 6; /* multi-stream huffman uses 6-byte jump table */ return cLitSizeEstimate + literalSectionHeaderSize; } } assert(0); /* impossible */ return 0; } /* Returns the size estimate for the FSE-compressed symbols (of, ml, ll) of a block */ static size_t ZSTD_estimateBlockSize_symbolType(symbolEncodingType_e type, const BYTE* codeTable, size_t nbSeq, unsigned maxCode, const FSE_CTable* fseCTable, const U8* additionalBits, short const* defaultNorm, U32 defaultNormLog, U32 defaultMax, void* workspace, size_t wkspSize) { unsigned* const countWksp = (unsigned*)workspace; const BYTE* ctp = codeTable; const BYTE* const ctStart = ctp; const BYTE* const ctEnd = ctStart + nbSeq; size_t cSymbolTypeSizeEstimateInBits = 0; unsigned max = maxCode; HIST_countFast_wksp(countWksp, &max, codeTable, nbSeq, workspace, wkspSize); /* can't fail */ if (type == set_basic) { /* We selected this encoding type, so it must be valid. */ assert(max <= defaultMax); (void)defaultMax; cSymbolTypeSizeEstimateInBits = ZSTD_crossEntropyCost(defaultNorm, defaultNormLog, countWksp, max); } else if (type == set_rle) { cSymbolTypeSizeEstimateInBits = 0; } else if (type == set_compressed || type == set_repeat) { cSymbolTypeSizeEstimateInBits = ZSTD_fseBitCost(fseCTable, countWksp, max); } if (ZSTD_isError(cSymbolTypeSizeEstimateInBits)) { return nbSeq * 10; } while (ctp < ctEnd) { if (additionalBits) cSymbolTypeSizeEstimateInBits += additionalBits[*ctp]; else cSymbolTypeSizeEstimateInBits += *ctp; /* for offset, offset code is also the number of additional bits */ ctp++; } return cSymbolTypeSizeEstimateInBits >> 3; } /* Returns the size estimate for the sequences section (header + content) of a block */ static size_t ZSTD_estimateBlockSize_sequences(const BYTE* ofCodeTable, const BYTE* llCodeTable, const BYTE* mlCodeTable, size_t nbSeq, const ZSTD_fseCTables_t* fseTables, const ZSTD_fseCTablesMetadata_t* fseMetadata, void* workspace, size_t wkspSize, int writeEntropy) { size_t sequencesSectionHeaderSize = 1 /* seqHead */ + 1 /* min seqSize size */ + (nbSeq >= 128) + (nbSeq >= LONGNBSEQ); size_t cSeqSizeEstimate = 0; cSeqSizeEstimate += ZSTD_estimateBlockSize_symbolType(fseMetadata->ofType, ofCodeTable, nbSeq, MaxOff, fseTables->offcodeCTable, NULL, OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff, workspace, wkspSize); cSeqSizeEstimate += ZSTD_estimateBlockSize_symbolType(fseMetadata->llType, llCodeTable, nbSeq, MaxLL, fseTables->litlengthCTable, LL_bits, LL_defaultNorm, LL_defaultNormLog, MaxLL, workspace, wkspSize); cSeqSizeEstimate += ZSTD_estimateBlockSize_symbolType(fseMetadata->mlType, mlCodeTable, nbSeq, MaxML, fseTables->matchlengthCTable, ML_bits, ML_defaultNorm, ML_defaultNormLog, MaxML, workspace, wkspSize); if (writeEntropy) cSeqSizeEstimate += fseMetadata->fseTablesSize; return cSeqSizeEstimate + sequencesSectionHeaderSize; } /* Returns the size estimate for a given stream of literals, of, ll, ml */ static size_t ZSTD_estimateBlockSize(const BYTE* literals, size_t litSize, const BYTE* ofCodeTable, const BYTE* llCodeTable, const BYTE* mlCodeTable, size_t nbSeq, const ZSTD_entropyCTables_t* entropy, const ZSTD_entropyCTablesMetadata_t* entropyMetadata, void* workspace, size_t wkspSize, int writeLitEntropy, int writeSeqEntropy) { size_t const literalsSize = ZSTD_estimateBlockSize_literal(literals, litSize, &entropy->huf, &entropyMetadata->hufMetadata, workspace, wkspSize, writeLitEntropy); size_t const seqSize = ZSTD_estimateBlockSize_sequences(ofCodeTable, llCodeTable, mlCodeTable, nbSeq, &entropy->fse, &entropyMetadata->fseMetadata, workspace, wkspSize, writeSeqEntropy); return seqSize + literalsSize + ZSTD_blockHeaderSize; } /* Builds entropy statistics and uses them for blocksize estimation. * * @return: estimated compressed size of the seqStore, or a zstd error. */ static size_t ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize(seqStore_t* seqStore, ZSTD_CCtx* zc) { ZSTD_entropyCTablesMetadata_t* const entropyMetadata = &zc->blockSplitCtx.entropyMetadata; DEBUGLOG(6, "ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize()"); FORWARD_IF_ERROR(ZSTD_buildBlockEntropyStats(seqStore, &zc->blockState.prevCBlock->entropy, &zc->blockState.nextCBlock->entropy, &zc->appliedParams, entropyMetadata, zc->entropyWorkspace, ENTROPY_WORKSPACE_SIZE), ""); return ZSTD_estimateBlockSize( seqStore->litStart, (size_t)(seqStore->lit - seqStore->litStart), seqStore->ofCode, seqStore->llCode, seqStore->mlCode, (size_t)(seqStore->sequences - seqStore->sequencesStart), &zc->blockState.nextCBlock->entropy, entropyMetadata, zc->entropyWorkspace, ENTROPY_WORKSPACE_SIZE, (int)(entropyMetadata->hufMetadata.hType == set_compressed), 1); } /* Returns literals bytes represented in a seqStore */ static size_t ZSTD_countSeqStoreLiteralsBytes(const seqStore_t* const seqStore) { size_t literalsBytes = 0; size_t const nbSeqs = (size_t)(seqStore->sequences - seqStore->sequencesStart); size_t i; for (i = 0; i < nbSeqs; ++i) { seqDef const seq = seqStore->sequencesStart[i]; literalsBytes += seq.litLength; if (i == seqStore->longLengthPos && seqStore->longLengthType == ZSTD_llt_literalLength) { literalsBytes += 0x10000; } } return literalsBytes; } /* Returns match bytes represented in a seqStore */ static size_t ZSTD_countSeqStoreMatchBytes(const seqStore_t* const seqStore) { size_t matchBytes = 0; size_t const nbSeqs = (size_t)(seqStore->sequences - seqStore->sequencesStart); size_t i; for (i = 0; i < nbSeqs; ++i) { seqDef seq = seqStore->sequencesStart[i]; matchBytes += seq.mlBase + MINMATCH; if (i == seqStore->longLengthPos && seqStore->longLengthType == ZSTD_llt_matchLength) { matchBytes += 0x10000; } } return matchBytes; } /* Derives the seqStore that is a chunk of the originalSeqStore from [startIdx, endIdx). * Stores the result in resultSeqStore. */ static void ZSTD_deriveSeqStoreChunk(seqStore_t* resultSeqStore, const seqStore_t* originalSeqStore, size_t startIdx, size_t endIdx) { *resultSeqStore = *originalSeqStore; if (startIdx > 0) { resultSeqStore->sequences = originalSeqStore->sequencesStart + startIdx; resultSeqStore->litStart += ZSTD_countSeqStoreLiteralsBytes(resultSeqStore); } /* Move longLengthPos into the correct position if necessary */ if (originalSeqStore->longLengthType != ZSTD_llt_none) { if (originalSeqStore->longLengthPos < startIdx || originalSeqStore->longLengthPos > endIdx) { resultSeqStore->longLengthType = ZSTD_llt_none; } else { resultSeqStore->longLengthPos -= (U32)startIdx; } } resultSeqStore->sequencesStart = originalSeqStore->sequencesStart + startIdx; resultSeqStore->sequences = originalSeqStore->sequencesStart + endIdx; if (endIdx == (size_t)(originalSeqStore->sequences - originalSeqStore->sequencesStart)) { /* This accounts for possible last literals if the derived chunk reaches the end of the block */ assert(resultSeqStore->lit == originalSeqStore->lit); } else { size_t const literalsBytes = ZSTD_countSeqStoreLiteralsBytes(resultSeqStore); resultSeqStore->lit = resultSeqStore->litStart + literalsBytes; } resultSeqStore->llCode += startIdx; resultSeqStore->mlCode += startIdx; resultSeqStore->ofCode += startIdx; } /** * Returns the raw offset represented by the combination of offBase, ll0, and repcode history. * offBase must represent a repcode in the numeric representation of ZSTD_storeSeq(). */ static U32 ZSTD_resolveRepcodeToRawOffset(const U32 rep[ZSTD_REP_NUM], const U32 offBase, const U32 ll0) { U32 const adjustedRepCode = OFFBASE_TO_REPCODE(offBase) - 1 + ll0; /* [ 0 - 3 ] */ assert(OFFBASE_IS_REPCODE(offBase)); if (adjustedRepCode == ZSTD_REP_NUM) { assert(ll0); /* litlength == 0 and offCode == 2 implies selection of first repcode - 1 * This is only valid if it results in a valid offset value, aka > 0. * Note : it may happen that `rep[0]==1` in exceptional circumstances. * In which case this function will return 0, which is an invalid offset. * It's not an issue though, since this value will be * compared and discarded within ZSTD_seqStore_resolveOffCodes(). */ return rep[0] - 1; } return rep[adjustedRepCode]; } /** * ZSTD_seqStore_resolveOffCodes() reconciles any possible divergences in offset history that may arise * due to emission of RLE/raw blocks that disturb the offset history, * and replaces any repcodes within the seqStore that may be invalid. * * dRepcodes are updated as would be on the decompression side. * cRepcodes are updated exactly in accordance with the seqStore. * * Note : this function assumes seq->offBase respects the following numbering scheme : * 0 : invalid * 1-3 : repcode 1-3 * 4+ : real_offset+3 */ static void ZSTD_seqStore_resolveOffCodes(repcodes_t* const dRepcodes, repcodes_t* const cRepcodes, const seqStore_t* const seqStore, U32 const nbSeq) { U32 idx = 0; U32 const longLitLenIdx = seqStore->longLengthType == ZSTD_llt_literalLength ? seqStore->longLengthPos : nbSeq; for (; idx < nbSeq; ++idx) { seqDef* const seq = seqStore->sequencesStart + idx; U32 const ll0 = (seq->litLength == 0) && (idx != longLitLenIdx); U32 const offBase = seq->offBase; assert(offBase > 0); if (OFFBASE_IS_REPCODE(offBase)) { U32 const dRawOffset = ZSTD_resolveRepcodeToRawOffset(dRepcodes->rep, offBase, ll0); U32 const cRawOffset = ZSTD_resolveRepcodeToRawOffset(cRepcodes->rep, offBase, ll0); /* Adjust simulated decompression repcode history if we come across a mismatch. Replace * the repcode with the offset it actually references, determined by the compression * repcode history. */ if (dRawOffset != cRawOffset) { seq->offBase = OFFSET_TO_OFFBASE(cRawOffset); } } /* Compression repcode history is always updated with values directly from the unmodified seqStore. * Decompression repcode history may use modified seq->offset value taken from compression repcode history. */ ZSTD_updateRep(dRepcodes->rep, seq->offBase, ll0); ZSTD_updateRep(cRepcodes->rep, offBase, ll0); } } /* ZSTD_compressSeqStore_singleBlock(): * Compresses a seqStore into a block with a block header, into the buffer dst. * * Returns the total size of that block (including header) or a ZSTD error code. */ static size_t ZSTD_compressSeqStore_singleBlock(ZSTD_CCtx* zc, const seqStore_t* const seqStore, repcodes_t* const dRep, repcodes_t* const cRep, void* dst, size_t dstCapacity, const void* src, size_t srcSize, U32 lastBlock, U32 isPartition) { const U32 rleMaxLength = 25; BYTE* op = (BYTE*)dst; const BYTE* ip = (const BYTE*)src; size_t cSize; size_t cSeqsSize; /* In case of an RLE or raw block, the simulated decompression repcode history must be reset */ repcodes_t const dRepOriginal = *dRep; DEBUGLOG(5, "ZSTD_compressSeqStore_singleBlock"); if (isPartition) ZSTD_seqStore_resolveOffCodes(dRep, cRep, seqStore, (U32)(seqStore->sequences - seqStore->sequencesStart)); RETURN_ERROR_IF(dstCapacity < ZSTD_blockHeaderSize, dstSize_tooSmall, "Block header doesn't fit"); cSeqsSize = ZSTD_entropyCompressSeqStore(seqStore, &zc->blockState.prevCBlock->entropy, &zc->blockState.nextCBlock->entropy, &zc->appliedParams, op + ZSTD_blockHeaderSize, dstCapacity - ZSTD_blockHeaderSize, srcSize, zc->entropyWorkspace, ENTROPY_WORKSPACE_SIZE /* statically allocated in resetCCtx */, zc->bmi2); FORWARD_IF_ERROR(cSeqsSize, "ZSTD_entropyCompressSeqStore failed!"); if (!zc->isFirstBlock && cSeqsSize < rleMaxLength && ZSTD_isRLE((BYTE const*)src, srcSize)) { /* We don't want to emit our first block as a RLE even if it qualifies because * doing so will cause the decoder (cli only) to throw a "should consume all input error." * This is only an issue for zstd <= v1.4.3 */ cSeqsSize = 1; } if (zc->seqCollector.collectSequences) { ZSTD_copyBlockSequences(zc); ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState); return 0; } if (cSeqsSize == 0) { cSize = ZSTD_noCompressBlock(op, dstCapacity, ip, srcSize, lastBlock); FORWARD_IF_ERROR(cSize, "Nocompress block failed"); DEBUGLOG(4, "Writing out nocompress block, size: %zu", cSize); *dRep = dRepOriginal; /* reset simulated decompression repcode history */ } else if (cSeqsSize == 1) { cSize = ZSTD_rleCompressBlock(op, dstCapacity, *ip, srcSize, lastBlock); FORWARD_IF_ERROR(cSize, "RLE compress block failed"); DEBUGLOG(4, "Writing out RLE block, size: %zu", cSize); *dRep = dRepOriginal; /* reset simulated decompression repcode history */ } else { ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState); writeBlockHeader(op, cSeqsSize, srcSize, lastBlock); cSize = ZSTD_blockHeaderSize + cSeqsSize; DEBUGLOG(4, "Writing out compressed block, size: %zu", cSize); } if (zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid) zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check; return cSize; } /* Struct to keep track of where we are in our recursive calls. */ typedef struct { U32* splitLocations; /* Array of split indices */ size_t idx; /* The current index within splitLocations being worked on */ } seqStoreSplits; #define MIN_SEQUENCES_BLOCK_SPLITTING 300 /* Helper function to perform the recursive search for block splits. * Estimates the cost of seqStore prior to split, and estimates the cost of splitting the sequences in half. * If advantageous to split, then we recurse down the two sub-blocks. * If not, or if an error occurred in estimation, then we do not recurse. * * Note: The recursion depth is capped by a heuristic minimum number of sequences, * defined by MIN_SEQUENCES_BLOCK_SPLITTING. * In theory, this means the absolute largest recursion depth is 10 == log2(maxNbSeqInBlock/MIN_SEQUENCES_BLOCK_SPLITTING). * In practice, recursion depth usually doesn't go beyond 4. * * Furthermore, the number of splits is capped by ZSTD_MAX_NB_BLOCK_SPLITS. * At ZSTD_MAX_NB_BLOCK_SPLITS == 196 with the current existing blockSize * maximum of 128 KB, this value is actually impossible to reach. */ static void ZSTD_deriveBlockSplitsHelper(seqStoreSplits* splits, size_t startIdx, size_t endIdx, ZSTD_CCtx* zc, const seqStore_t* origSeqStore) { seqStore_t* const fullSeqStoreChunk = &zc->blockSplitCtx.fullSeqStoreChunk; seqStore_t* const firstHalfSeqStore = &zc->blockSplitCtx.firstHalfSeqStore; seqStore_t* const secondHalfSeqStore = &zc->blockSplitCtx.secondHalfSeqStore; size_t estimatedOriginalSize; size_t estimatedFirstHalfSize; size_t estimatedSecondHalfSize; size_t midIdx = (startIdx + endIdx)/2; DEBUGLOG(5, "ZSTD_deriveBlockSplitsHelper: startIdx=%zu endIdx=%zu", startIdx, endIdx); assert(endIdx >= startIdx); if (endIdx - startIdx < MIN_SEQUENCES_BLOCK_SPLITTING || splits->idx >= ZSTD_MAX_NB_BLOCK_SPLITS) { DEBUGLOG(6, "ZSTD_deriveBlockSplitsHelper: Too few sequences (%zu)", endIdx - startIdx); return; } ZSTD_deriveSeqStoreChunk(fullSeqStoreChunk, origSeqStore, startIdx, endIdx); ZSTD_deriveSeqStoreChunk(firstHalfSeqStore, origSeqStore, startIdx, midIdx); ZSTD_deriveSeqStoreChunk(secondHalfSeqStore, origSeqStore, midIdx, endIdx); estimatedOriginalSize = ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize(fullSeqStoreChunk, zc); estimatedFirstHalfSize = ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize(firstHalfSeqStore, zc); estimatedSecondHalfSize = ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize(secondHalfSeqStore, zc); DEBUGLOG(5, "Estimated original block size: %zu -- First half split: %zu -- Second half split: %zu", estimatedOriginalSize, estimatedFirstHalfSize, estimatedSecondHalfSize); if (ZSTD_isError(estimatedOriginalSize) || ZSTD_isError(estimatedFirstHalfSize) || ZSTD_isError(estimatedSecondHalfSize)) { return; } if (estimatedFirstHalfSize + estimatedSecondHalfSize < estimatedOriginalSize) { DEBUGLOG(5, "split decided at seqNb:%zu", midIdx); ZSTD_deriveBlockSplitsHelper(splits, startIdx, midIdx, zc, origSeqStore); splits->splitLocations[splits->idx] = (U32)midIdx; splits->idx++; ZSTD_deriveBlockSplitsHelper(splits, midIdx, endIdx, zc, origSeqStore); } } /* Base recursive function. * Populates a table with intra-block partition indices that can improve compression ratio. * * @return: number of splits made (which equals the size of the partition table - 1). */ static size_t ZSTD_deriveBlockSplits(ZSTD_CCtx* zc, U32 partitions[], U32 nbSeq) { seqStoreSplits splits; splits.splitLocations = partitions; splits.idx = 0; if (nbSeq <= 4) { DEBUGLOG(5, "ZSTD_deriveBlockSplits: Too few sequences to split (%u <= 4)", nbSeq); /* Refuse to try and split anything with less than 4 sequences */ return 0; } ZSTD_deriveBlockSplitsHelper(&splits, 0, nbSeq, zc, &zc->seqStore); splits.splitLocations[splits.idx] = nbSeq; DEBUGLOG(5, "ZSTD_deriveBlockSplits: final nb partitions: %zu", splits.idx+1); return splits.idx; } /* ZSTD_compressBlock_splitBlock(): * Attempts to split a given block into multiple blocks to improve compression ratio. * * Returns combined size of all blocks (which includes headers), or a ZSTD error code. */ static size_t ZSTD_compressBlock_splitBlock_internal(ZSTD_CCtx* zc, void* dst, size_t dstCapacity, const void* src, size_t blockSize, U32 lastBlock, U32 nbSeq) { size_t cSize = 0; const BYTE* ip = (const BYTE*)src; BYTE* op = (BYTE*)dst; size_t i = 0; size_t srcBytesTotal = 0; U32* const partitions = zc->blockSplitCtx.partitions; /* size == ZSTD_MAX_NB_BLOCK_SPLITS */ seqStore_t* const nextSeqStore = &zc->blockSplitCtx.nextSeqStore; seqStore_t* const currSeqStore = &zc->blockSplitCtx.currSeqStore; size_t const numSplits = ZSTD_deriveBlockSplits(zc, partitions, nbSeq); /* If a block is split and some partitions are emitted as RLE/uncompressed, then repcode history * may become invalid. In order to reconcile potentially invalid repcodes, we keep track of two * separate repcode histories that simulate repcode history on compression and decompression side, * and use the histories to determine whether we must replace a particular repcode with its raw offset. * * 1) cRep gets updated for each partition, regardless of whether the block was emitted as uncompressed * or RLE. This allows us to retrieve the offset value that an invalid repcode references within * a nocompress/RLE block. * 2) dRep gets updated only for compressed partitions, and when a repcode gets replaced, will use * the replacement offset value rather than the original repcode to update the repcode history. * dRep also will be the final repcode history sent to the next block. * * See ZSTD_seqStore_resolveOffCodes() for more details. */ repcodes_t dRep; repcodes_t cRep; ZSTD_memcpy(dRep.rep, zc->blockState.prevCBlock->rep, sizeof(repcodes_t)); ZSTD_memcpy(cRep.rep, zc->blockState.prevCBlock->rep, sizeof(repcodes_t)); ZSTD_memset(nextSeqStore, 0, sizeof(seqStore_t)); DEBUGLOG(5, "ZSTD_compressBlock_splitBlock_internal (dstCapacity=%u, dictLimit=%u, nextToUpdate=%u)", (unsigned)dstCapacity, (unsigned)zc->blockState.matchState.window.dictLimit, (unsigned)zc->blockState.matchState.nextToUpdate); if (numSplits == 0) { size_t cSizeSingleBlock = ZSTD_compressSeqStore_singleBlock(zc, &zc->seqStore, &dRep, &cRep, op, dstCapacity, ip, blockSize, lastBlock, 0 /* isPartition */); FORWARD_IF_ERROR(cSizeSingleBlock, "Compressing single block from splitBlock_internal() failed!"); DEBUGLOG(5, "ZSTD_compressBlock_splitBlock_internal: No splits"); assert(zc->blockSize <= ZSTD_BLOCKSIZE_MAX); assert(cSizeSingleBlock <= zc->blockSize + ZSTD_blockHeaderSize); return cSizeSingleBlock; } ZSTD_deriveSeqStoreChunk(currSeqStore, &zc->seqStore, 0, partitions[0]); for (i = 0; i <= numSplits; ++i) { size_t cSizeChunk; U32 const lastPartition = (i == numSplits); U32 lastBlockEntireSrc = 0; size_t srcBytes = ZSTD_countSeqStoreLiteralsBytes(currSeqStore) + ZSTD_countSeqStoreMatchBytes(currSeqStore); srcBytesTotal += srcBytes; if (lastPartition) { /* This is the final partition, need to account for possible last literals */ srcBytes += blockSize - srcBytesTotal; lastBlockEntireSrc = lastBlock; } else { ZSTD_deriveSeqStoreChunk(nextSeqStore, &zc->seqStore, partitions[i], partitions[i+1]); } cSizeChunk = ZSTD_compressSeqStore_singleBlock(zc, currSeqStore, &dRep, &cRep, op, dstCapacity, ip, srcBytes, lastBlockEntireSrc, 1 /* isPartition */); DEBUGLOG(5, "Estimated size: %zu vs %zu : actual size", ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize(currSeqStore, zc), cSizeChunk); FORWARD_IF_ERROR(cSizeChunk, "Compressing chunk failed!"); ip += srcBytes; op += cSizeChunk; dstCapacity -= cSizeChunk; cSize += cSizeChunk; *currSeqStore = *nextSeqStore; assert(cSizeChunk <= zc->blockSize + ZSTD_blockHeaderSize); } /* cRep and dRep may have diverged during the compression. * If so, we use the dRep repcodes for the next block. */ ZSTD_memcpy(zc->blockState.prevCBlock->rep, dRep.rep, sizeof(repcodes_t)); return cSize; } static size_t ZSTD_compressBlock_splitBlock(ZSTD_CCtx* zc, void* dst, size_t dstCapacity, const void* src, size_t srcSize, U32 lastBlock) { U32 nbSeq; size_t cSize; DEBUGLOG(4, "ZSTD_compressBlock_splitBlock"); assert(zc->appliedParams.useBlockSplitter == ZSTD_ps_enable); { const size_t bss = ZSTD_buildSeqStore(zc, src, srcSize); FORWARD_IF_ERROR(bss, "ZSTD_buildSeqStore failed"); if (bss == ZSTDbss_noCompress) { if (zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid) zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check; cSize = ZSTD_noCompressBlock(dst, dstCapacity, src, srcSize, lastBlock); FORWARD_IF_ERROR(cSize, "ZSTD_noCompressBlock failed"); DEBUGLOG(4, "ZSTD_compressBlock_splitBlock: Nocompress block"); return cSize; } nbSeq = (U32)(zc->seqStore.sequences - zc->seqStore.sequencesStart); } cSize = ZSTD_compressBlock_splitBlock_internal(zc, dst, dstCapacity, src, srcSize, lastBlock, nbSeq); FORWARD_IF_ERROR(cSize, "Splitting blocks failed!"); return cSize; } static size_t ZSTD_compressBlock_internal(ZSTD_CCtx* zc, void* dst, size_t dstCapacity, const void* src, size_t srcSize, U32 frame) { /* This is an estimated upper bound for the length of an rle block. * This isn't the actual upper bound. * Finding the real threshold needs further investigation. */ const U32 rleMaxLength = 25; size_t cSize; const BYTE* ip = (const BYTE*)src; BYTE* op = (BYTE*)dst; DEBUGLOG(5, "ZSTD_compressBlock_internal (dstCapacity=%u, dictLimit=%u, nextToUpdate=%u)", (unsigned)dstCapacity, (unsigned)zc->blockState.matchState.window.dictLimit, (unsigned)zc->blockState.matchState.nextToUpdate); { const size_t bss = ZSTD_buildSeqStore(zc, src, srcSize); FORWARD_IF_ERROR(bss, "ZSTD_buildSeqStore failed"); if (bss == ZSTDbss_noCompress) { cSize = 0; goto out; } } if (zc->seqCollector.collectSequences) { ZSTD_copyBlockSequences(zc); ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState); return 0; } /* encode sequences and literals */ cSize = ZSTD_entropyCompressSeqStore(&zc->seqStore, &zc->blockState.prevCBlock->entropy, &zc->blockState.nextCBlock->entropy, &zc->appliedParams, dst, dstCapacity, srcSize, zc->entropyWorkspace, ENTROPY_WORKSPACE_SIZE /* statically allocated in resetCCtx */, zc->bmi2); if (frame && /* We don't want to emit our first block as a RLE even if it qualifies because * doing so will cause the decoder (cli only) to throw a "should consume all input error." * This is only an issue for zstd <= v1.4.3 */ !zc->isFirstBlock && cSize < rleMaxLength && ZSTD_isRLE(ip, srcSize)) { cSize = 1; op[0] = ip[0]; } out: if (!ZSTD_isError(cSize) && cSize > 1) { ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState); } /* We check that dictionaries have offset codes available for the first * block. After the first block, the offcode table might not have large * enough codes to represent the offsets in the data. */ if (zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid) zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check; return cSize; } static size_t ZSTD_compressBlock_targetCBlockSize_body(ZSTD_CCtx* zc, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const size_t bss, U32 lastBlock) { DEBUGLOG(6, "Attempting ZSTD_compressSuperBlock()"); if (bss == ZSTDbss_compress) { if (/* We don't want to emit our first block as a RLE even if it qualifies because * doing so will cause the decoder (cli only) to throw a "should consume all input error." * This is only an issue for zstd <= v1.4.3 */ !zc->isFirstBlock && ZSTD_maybeRLE(&zc->seqStore) && ZSTD_isRLE((BYTE const*)src, srcSize)) { return ZSTD_rleCompressBlock(dst, dstCapacity, *(BYTE const*)src, srcSize, lastBlock); } /* Attempt superblock compression. * * Note that compressed size of ZSTD_compressSuperBlock() is not bound by the * standard ZSTD_compressBound(). This is a problem, because even if we have * space now, taking an extra byte now could cause us to run out of space later * and violate ZSTD_compressBound(). * * Define blockBound(blockSize) = blockSize + ZSTD_blockHeaderSize. * * In order to respect ZSTD_compressBound() we must attempt to emit a raw * uncompressed block in these cases: * * cSize == 0: Return code for an uncompressed block. * * cSize == dstSize_tooSmall: We may have expanded beyond blockBound(srcSize). * ZSTD_noCompressBlock() will return dstSize_tooSmall if we are really out of * output space. * * cSize >= blockBound(srcSize): We have expanded the block too much so * emit an uncompressed block. */ { size_t const cSize = ZSTD_compressSuperBlock(zc, dst, dstCapacity, src, srcSize, lastBlock); if (cSize != ERROR(dstSize_tooSmall)) { size_t const maxCSize = srcSize - ZSTD_minGain(srcSize, zc->appliedParams.cParams.strategy); FORWARD_IF_ERROR(cSize, "ZSTD_compressSuperBlock failed"); if (cSize != 0 && cSize < maxCSize + ZSTD_blockHeaderSize) { ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState); return cSize; } } } } /* if (bss == ZSTDbss_compress)*/ DEBUGLOG(6, "Resorting to ZSTD_noCompressBlock()"); /* Superblock compression failed, attempt to emit a single no compress block. * The decoder will be able to stream this block since it is uncompressed. */ return ZSTD_noCompressBlock(dst, dstCapacity, src, srcSize, lastBlock); } static size_t ZSTD_compressBlock_targetCBlockSize(ZSTD_CCtx* zc, void* dst, size_t dstCapacity, const void* src, size_t srcSize, U32 lastBlock) { size_t cSize = 0; const size_t bss = ZSTD_buildSeqStore(zc, src, srcSize); DEBUGLOG(5, "ZSTD_compressBlock_targetCBlockSize (dstCapacity=%u, dictLimit=%u, nextToUpdate=%u, srcSize=%zu)", (unsigned)dstCapacity, (unsigned)zc->blockState.matchState.window.dictLimit, (unsigned)zc->blockState.matchState.nextToUpdate, srcSize); FORWARD_IF_ERROR(bss, "ZSTD_buildSeqStore failed"); cSize = ZSTD_compressBlock_targetCBlockSize_body(zc, dst, dstCapacity, src, srcSize, bss, lastBlock); FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_targetCBlockSize_body failed"); if (zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid) zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check; return cSize; } static void ZSTD_overflowCorrectIfNeeded(ZSTD_matchState_t* ms, ZSTD_cwksp* ws, ZSTD_CCtx_params const* params, void const* ip, void const* iend) { U32 const cycleLog = ZSTD_cycleLog(params->cParams.chainLog, params->cParams.strategy); U32 const maxDist = (U32)1 << params->cParams.windowLog; if (ZSTD_window_needOverflowCorrection(ms->window, cycleLog, maxDist, ms->loadedDictEnd, ip, iend)) { U32 const correction = ZSTD_window_correctOverflow(&ms->window, cycleLog, maxDist, ip); ZSTD_STATIC_ASSERT(ZSTD_CHAINLOG_MAX <= 30); ZSTD_STATIC_ASSERT(ZSTD_WINDOWLOG_MAX_32 <= 30); ZSTD_STATIC_ASSERT(ZSTD_WINDOWLOG_MAX <= 31); ZSTD_cwksp_mark_tables_dirty(ws); ZSTD_reduceIndex(ms, params, correction); ZSTD_cwksp_mark_tables_clean(ws); if (ms->nextToUpdate < correction) ms->nextToUpdate = 0; else ms->nextToUpdate -= correction; /* invalidate dictionaries on overflow correction */ ms->loadedDictEnd = 0; ms->dictMatchState = NULL; } } /*! ZSTD_compress_frameChunk() : * Compress a chunk of data into one or multiple blocks. * All blocks will be terminated, all input will be consumed. * Function will issue an error if there is not enough `dstCapacity` to hold the compressed content. * Frame is supposed already started (header already produced) * @return : compressed size, or an error code */ static size_t ZSTD_compress_frameChunk(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, U32 lastFrameChunk) { size_t blockSize = cctx->blockSize; size_t remaining = srcSize; const BYTE* ip = (const BYTE*)src; BYTE* const ostart = (BYTE*)dst; BYTE* op = ostart; U32 const maxDist = (U32)1 << cctx->appliedParams.cParams.windowLog; assert(cctx->appliedParams.cParams.windowLog <= ZSTD_WINDOWLOG_MAX); DEBUGLOG(4, "ZSTD_compress_frameChunk (blockSize=%u)", (unsigned)blockSize); if (cctx->appliedParams.fParams.checksumFlag && srcSize) XXH64_update(&cctx->xxhState, src, srcSize); while (remaining) { ZSTD_matchState_t* const ms = &cctx->blockState.matchState; U32 const lastBlock = lastFrameChunk & (blockSize >= remaining); /* TODO: See 3090. We reduced MIN_CBLOCK_SIZE from 3 to 2 so to compensate we are adding * additional 1. We need to revisit and change this logic to be more consistent */ RETURN_ERROR_IF(dstCapacity < ZSTD_blockHeaderSize + MIN_CBLOCK_SIZE + 1, dstSize_tooSmall, "not enough space to store compressed block"); if (remaining < blockSize) blockSize = remaining; ZSTD_overflowCorrectIfNeeded( ms, &cctx->workspace, &cctx->appliedParams, ip, ip + blockSize); ZSTD_checkDictValidity(&ms->window, ip + blockSize, maxDist, &ms->loadedDictEnd, &ms->dictMatchState); ZSTD_window_enforceMaxDist(&ms->window, ip, maxDist, &ms->loadedDictEnd, &ms->dictMatchState); /* Ensure hash/chain table insertion resumes no sooner than lowlimit */ if (ms->nextToUpdate < ms->window.lowLimit) ms->nextToUpdate = ms->window.lowLimit; { size_t cSize; if (ZSTD_useTargetCBlockSize(&cctx->appliedParams)) { cSize = ZSTD_compressBlock_targetCBlockSize(cctx, op, dstCapacity, ip, blockSize, lastBlock); FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_targetCBlockSize failed"); assert(cSize > 0); assert(cSize <= blockSize + ZSTD_blockHeaderSize); } else if (ZSTD_blockSplitterEnabled(&cctx->appliedParams)) { cSize = ZSTD_compressBlock_splitBlock(cctx, op, dstCapacity, ip, blockSize, lastBlock); FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_splitBlock failed"); assert(cSize > 0 || cctx->seqCollector.collectSequences == 1); } else { cSize = ZSTD_compressBlock_internal(cctx, op+ZSTD_blockHeaderSize, dstCapacity-ZSTD_blockHeaderSize, ip, blockSize, 1 /* frame */); FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_internal failed"); if (cSize == 0) { /* block is not compressible */ cSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock); FORWARD_IF_ERROR(cSize, "ZSTD_noCompressBlock failed"); } else { U32 const cBlockHeader = cSize == 1 ? lastBlock + (((U32)bt_rle)<<1) + (U32)(blockSize << 3) : lastBlock + (((U32)bt_compressed)<<1) + (U32)(cSize << 3); MEM_writeLE24(op, cBlockHeader); cSize += ZSTD_blockHeaderSize; } } /* if (ZSTD_useTargetCBlockSize(&cctx->appliedParams))*/ ip += blockSize; assert(remaining >= blockSize); remaining -= blockSize; op += cSize; assert(dstCapacity >= cSize); dstCapacity -= cSize; cctx->isFirstBlock = 0; DEBUGLOG(5, "ZSTD_compress_frameChunk: adding a block of size %u", (unsigned)cSize); } } if (lastFrameChunk && (op>ostart)) cctx->stage = ZSTDcs_ending; return (size_t)(op-ostart); } static size_t ZSTD_writeFrameHeader(void* dst, size_t dstCapacity, const ZSTD_CCtx_params* params, U64 pledgedSrcSize, U32 dictID) { BYTE* const op = (BYTE*)dst; U32 const dictIDSizeCodeLength = (dictID>0) + (dictID>=256) + (dictID>=65536); /* 0-3 */ U32 const dictIDSizeCode = params->fParams.noDictIDFlag ? 0 : dictIDSizeCodeLength; /* 0-3 */ U32 const checksumFlag = params->fParams.checksumFlag>0; U32 const windowSize = (U32)1 << params->cParams.windowLog; U32 const singleSegment = params->fParams.contentSizeFlag && (windowSize >= pledgedSrcSize); BYTE const windowLogByte = (BYTE)((params->cParams.windowLog - ZSTD_WINDOWLOG_ABSOLUTEMIN) << 3); U32 const fcsCode = params->fParams.contentSizeFlag ? (pledgedSrcSize>=256) + (pledgedSrcSize>=65536+256) + (pledgedSrcSize>=0xFFFFFFFFU) : 0; /* 0-3 */ BYTE const frameHeaderDescriptionByte = (BYTE)(dictIDSizeCode + (checksumFlag<<2) + (singleSegment<<5) + (fcsCode<<6) ); size_t pos=0; assert(!(params->fParams.contentSizeFlag && pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN)); RETURN_ERROR_IF(dstCapacity < ZSTD_FRAMEHEADERSIZE_MAX, dstSize_tooSmall, "dst buf is too small to fit worst-case frame header size."); DEBUGLOG(4, "ZSTD_writeFrameHeader : dictIDFlag : %u ; dictID : %u ; dictIDSizeCode : %u", !params->fParams.noDictIDFlag, (unsigned)dictID, (unsigned)dictIDSizeCode); if (params->format == ZSTD_f_zstd1) { MEM_writeLE32(dst, ZSTD_MAGICNUMBER); pos = 4; } op[pos++] = frameHeaderDescriptionByte; if (!singleSegment) op[pos++] = windowLogByte; switch(dictIDSizeCode) { default: assert(0); /* impossible */ ZSTD_FALLTHROUGH; case 0 : break; case 1 : op[pos] = (BYTE)(dictID); pos++; break; case 2 : MEM_writeLE16(op+pos, (U16)dictID); pos+=2; break; case 3 : MEM_writeLE32(op+pos, dictID); pos+=4; break; } switch(fcsCode) { default: assert(0); /* impossible */ ZSTD_FALLTHROUGH; case 0 : if (singleSegment) op[pos++] = (BYTE)(pledgedSrcSize); break; case 1 : MEM_writeLE16(op+pos, (U16)(pledgedSrcSize-256)); pos+=2; break; case 2 : MEM_writeLE32(op+pos, (U32)(pledgedSrcSize)); pos+=4; break; case 3 : MEM_writeLE64(op+pos, (U64)(pledgedSrcSize)); pos+=8; break; } return pos; } /* ZSTD_writeSkippableFrame_advanced() : * Writes out a skippable frame with the specified magic number variant (16 are supported), * from ZSTD_MAGIC_SKIPPABLE_START to ZSTD_MAGIC_SKIPPABLE_START+15, and the desired source data. * * Returns the total number of bytes written, or a ZSTD error code. */ size_t ZSTD_writeSkippableFrame(void* dst, size_t dstCapacity, const void* src, size_t srcSize, unsigned magicVariant) { BYTE* op = (BYTE*)dst; RETURN_ERROR_IF(dstCapacity < srcSize + ZSTD_SKIPPABLEHEADERSIZE /* Skippable frame overhead */, dstSize_tooSmall, "Not enough room for skippable frame"); RETURN_ERROR_IF(srcSize > (unsigned)0xFFFFFFFF, srcSize_wrong, "Src size too large for skippable frame"); RETURN_ERROR_IF(magicVariant > 15, parameter_outOfBound, "Skippable frame magic number variant not supported"); MEM_writeLE32(op, (U32)(ZSTD_MAGIC_SKIPPABLE_START + magicVariant)); MEM_writeLE32(op+4, (U32)srcSize); ZSTD_memcpy(op+8, src, srcSize); return srcSize + ZSTD_SKIPPABLEHEADERSIZE; } /* ZSTD_writeLastEmptyBlock() : * output an empty Block with end-of-frame mark to complete a frame * @return : size of data written into `dst` (== ZSTD_blockHeaderSize (defined in zstd_internal.h)) * or an error code if `dstCapacity` is too small (stage != ZSTDcs_init, stage_wrong, "wrong cctx stage"); RETURN_ERROR_IF(cctx->appliedParams.ldmParams.enableLdm == ZSTD_ps_enable, parameter_unsupported, "incompatible with ldm"); cctx->externSeqStore.seq = seq; cctx->externSeqStore.size = nbSeq; cctx->externSeqStore.capacity = nbSeq; cctx->externSeqStore.pos = 0; cctx->externSeqStore.posInSequence = 0; return 0; } static size_t ZSTD_compressContinue_internal (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, U32 frame, U32 lastFrameChunk) { ZSTD_matchState_t* const ms = &cctx->blockState.matchState; size_t fhSize = 0; DEBUGLOG(5, "ZSTD_compressContinue_internal, stage: %u, srcSize: %u", cctx->stage, (unsigned)srcSize); RETURN_ERROR_IF(cctx->stage==ZSTDcs_created, stage_wrong, "missing init (ZSTD_compressBegin)"); if (frame && (cctx->stage==ZSTDcs_init)) { fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, &cctx->appliedParams, cctx->pledgedSrcSizePlusOne-1, cctx->dictID); FORWARD_IF_ERROR(fhSize, "ZSTD_writeFrameHeader failed"); assert(fhSize <= dstCapacity); dstCapacity -= fhSize; dst = (char*)dst + fhSize; cctx->stage = ZSTDcs_ongoing; } if (!srcSize) return fhSize; /* do not generate an empty block if no input */ if (!ZSTD_window_update(&ms->window, src, srcSize, ms->forceNonContiguous)) { ms->forceNonContiguous = 0; ms->nextToUpdate = ms->window.dictLimit; } if (cctx->appliedParams.ldmParams.enableLdm == ZSTD_ps_enable) { ZSTD_window_update(&cctx->ldmState.window, src, srcSize, /* forceNonContiguous */ 0); } if (!frame) { /* overflow check and correction for block mode */ ZSTD_overflowCorrectIfNeeded( ms, &cctx->workspace, &cctx->appliedParams, src, (BYTE const*)src + srcSize); } DEBUGLOG(5, "ZSTD_compressContinue_internal (blockSize=%u)", (unsigned)cctx->blockSize); { size_t const cSize = frame ? ZSTD_compress_frameChunk (cctx, dst, dstCapacity, src, srcSize, lastFrameChunk) : ZSTD_compressBlock_internal (cctx, dst, dstCapacity, src, srcSize, 0 /* frame */); FORWARD_IF_ERROR(cSize, "%s", frame ? "ZSTD_compress_frameChunk failed" : "ZSTD_compressBlock_internal failed"); cctx->consumedSrcSize += srcSize; cctx->producedCSize += (cSize + fhSize); assert(!(cctx->appliedParams.fParams.contentSizeFlag && cctx->pledgedSrcSizePlusOne == 0)); if (cctx->pledgedSrcSizePlusOne != 0) { /* control src size */ ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_UNKNOWN == (unsigned long long)-1); RETURN_ERROR_IF( cctx->consumedSrcSize+1 > cctx->pledgedSrcSizePlusOne, srcSize_wrong, "error : pledgedSrcSize = %u, while realSrcSize >= %u", (unsigned)cctx->pledgedSrcSizePlusOne-1, (unsigned)cctx->consumedSrcSize); } return cSize + fhSize; } } size_t ZSTD_compressContinue_public(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize) { DEBUGLOG(5, "ZSTD_compressContinue (srcSize=%u)", (unsigned)srcSize); return ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 1 /* frame mode */, 0 /* last chunk */); } /* NOTE: Must just wrap ZSTD_compressContinue_public() */ size_t ZSTD_compressContinue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize) { return ZSTD_compressContinue_public(cctx, dst, dstCapacity, src, srcSize); } static size_t ZSTD_getBlockSize_deprecated(const ZSTD_CCtx* cctx) { ZSTD_compressionParameters const cParams = cctx->appliedParams.cParams; assert(!ZSTD_checkCParams(cParams)); return MIN(cctx->appliedParams.maxBlockSize, (size_t)1 << cParams.windowLog); } /* NOTE: Must just wrap ZSTD_getBlockSize_deprecated() */ size_t ZSTD_getBlockSize(const ZSTD_CCtx* cctx) { return ZSTD_getBlockSize_deprecated(cctx); } /* NOTE: Must just wrap ZSTD_compressBlock_deprecated() */ size_t ZSTD_compressBlock_deprecated(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize) { DEBUGLOG(5, "ZSTD_compressBlock: srcSize = %u", (unsigned)srcSize); { size_t const blockSizeMax = ZSTD_getBlockSize_deprecated(cctx); RETURN_ERROR_IF(srcSize > blockSizeMax, srcSize_wrong, "input is larger than a block"); } return ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 0 /* frame mode */, 0 /* last chunk */); } /* NOTE: Must just wrap ZSTD_compressBlock_deprecated() */ size_t ZSTD_compressBlock(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize) { return ZSTD_compressBlock_deprecated(cctx, dst, dstCapacity, src, srcSize); } /*! ZSTD_loadDictionaryContent() : * @return : 0, or an error code */ static size_t ZSTD_loadDictionaryContent(ZSTD_matchState_t* ms, ldmState_t* ls, ZSTD_cwksp* ws, ZSTD_CCtx_params const* params, const void* src, size_t srcSize, ZSTD_dictTableLoadMethod_e dtlm, ZSTD_tableFillPurpose_e tfp) { const BYTE* ip = (const BYTE*) src; const BYTE* const iend = ip + srcSize; int const loadLdmDict = params->ldmParams.enableLdm == ZSTD_ps_enable && ls != NULL; /* Assert that the ms params match the params we're being given */ ZSTD_assertEqualCParams(params->cParams, ms->cParams); { /* Ensure large dictionaries can't cause index overflow */ /* Allow the dictionary to set indices up to exactly ZSTD_CURRENT_MAX. * Dictionaries right at the edge will immediately trigger overflow * correction, but I don't want to insert extra constraints here. */ U32 maxDictSize = ZSTD_CURRENT_MAX - ZSTD_WINDOW_START_INDEX; int const CDictTaggedIndices = ZSTD_CDictIndicesAreTagged(¶ms->cParams); if (CDictTaggedIndices && tfp == ZSTD_tfp_forCDict) { /* Some dictionary matchfinders in zstd use "short cache", * which treats the lower ZSTD_SHORT_CACHE_TAG_BITS of each * CDict hashtable entry as a tag rather than as part of an index. * When short cache is used, we need to truncate the dictionary * so that its indices don't overlap with the tag. */ U32 const shortCacheMaxDictSize = (1u << (32 - ZSTD_SHORT_CACHE_TAG_BITS)) - ZSTD_WINDOW_START_INDEX; maxDictSize = MIN(maxDictSize, shortCacheMaxDictSize); assert(!loadLdmDict); } /* If the dictionary is too large, only load the suffix of the dictionary. */ if (srcSize > maxDictSize) { ip = iend - maxDictSize; src = ip; srcSize = maxDictSize; } } if (srcSize > ZSTD_CHUNKSIZE_MAX) { /* We must have cleared our windows when our source is this large. */ assert(ZSTD_window_isEmpty(ms->window)); if (loadLdmDict) assert(ZSTD_window_isEmpty(ls->window)); } ZSTD_window_update(&ms->window, src, srcSize, /* forceNonContiguous */ 0); DEBUGLOG(4, "ZSTD_loadDictionaryContent(): useRowMatchFinder=%d", (int)params->useRowMatchFinder); if (loadLdmDict) { /* Load the entire dict into LDM matchfinders. */ ZSTD_window_update(&ls->window, src, srcSize, /* forceNonContiguous */ 0); ls->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ls->window.base); ZSTD_ldm_fillHashTable(ls, ip, iend, ¶ms->ldmParams); } /* If the dict is larger than we can reasonably index in our tables, only load the suffix. */ if (params->cParams.strategy < ZSTD_btultra) { U32 maxDictSize = 8U << MIN(MAX(params->cParams.hashLog, params->cParams.chainLog), 28); if (srcSize > maxDictSize) { ip = iend - maxDictSize; src = ip; srcSize = maxDictSize; } } ms->nextToUpdate = (U32)(ip - ms->window.base); ms->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ms->window.base); ms->forceNonContiguous = params->deterministicRefPrefix; if (srcSize <= HASH_READ_SIZE) return 0; ZSTD_overflowCorrectIfNeeded(ms, ws, params, ip, iend); switch(params->cParams.strategy) { case ZSTD_fast: ZSTD_fillHashTable(ms, iend, dtlm, tfp); break; case ZSTD_dfast: ZSTD_fillDoubleHashTable(ms, iend, dtlm, tfp); break; case ZSTD_greedy: case ZSTD_lazy: case ZSTD_lazy2: assert(srcSize >= HASH_READ_SIZE); if (ms->dedicatedDictSearch) { assert(ms->chainTable != NULL); ZSTD_dedicatedDictSearch_lazy_loadDictionary(ms, iend-HASH_READ_SIZE); } else { assert(params->useRowMatchFinder != ZSTD_ps_auto); if (params->useRowMatchFinder == ZSTD_ps_enable) { size_t const tagTableSize = ((size_t)1 << params->cParams.hashLog); ZSTD_memset(ms->tagTable, 0, tagTableSize); ZSTD_row_update(ms, iend-HASH_READ_SIZE); DEBUGLOG(4, "Using row-based hash table for lazy dict"); } else { ZSTD_insertAndFindFirstIndex(ms, iend-HASH_READ_SIZE); DEBUGLOG(4, "Using chain-based hash table for lazy dict"); } } break; case ZSTD_btlazy2: /* we want the dictionary table fully sorted */ case ZSTD_btopt: case ZSTD_btultra: case ZSTD_btultra2: assert(srcSize >= HASH_READ_SIZE); ZSTD_updateTree(ms, iend-HASH_READ_SIZE, iend); break; default: assert(0); /* not possible : not a valid strategy id */ } ms->nextToUpdate = (U32)(iend - ms->window.base); return 0; } /* Dictionaries that assign zero probability to symbols that show up causes problems * when FSE encoding. Mark dictionaries with zero probability symbols as FSE_repeat_check * and only dictionaries with 100% valid symbols can be assumed valid. */ static FSE_repeat ZSTD_dictNCountRepeat(short* normalizedCounter, unsigned dictMaxSymbolValue, unsigned maxSymbolValue) { U32 s; if (dictMaxSymbolValue < maxSymbolValue) { return FSE_repeat_check; } for (s = 0; s <= maxSymbolValue; ++s) { if (normalizedCounter[s] == 0) { return FSE_repeat_check; } } return FSE_repeat_valid; } size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace, const void* const dict, size_t dictSize) { short offcodeNCount[MaxOff+1]; unsigned offcodeMaxValue = MaxOff; const BYTE* dictPtr = (const BYTE*)dict; /* skip magic num and dict ID */ const BYTE* const dictEnd = dictPtr + dictSize; dictPtr += 8; bs->entropy.huf.repeatMode = HUF_repeat_check; { unsigned maxSymbolValue = 255; unsigned hasZeroWeights = 1; size_t const hufHeaderSize = HUF_readCTable((HUF_CElt*)bs->entropy.huf.CTable, &maxSymbolValue, dictPtr, dictEnd-dictPtr, &hasZeroWeights); /* We only set the loaded table as valid if it contains all non-zero * weights. Otherwise, we set it to check */ if (!hasZeroWeights) bs->entropy.huf.repeatMode = HUF_repeat_valid; RETURN_ERROR_IF(HUF_isError(hufHeaderSize), dictionary_corrupted, ""); RETURN_ERROR_IF(maxSymbolValue < 255, dictionary_corrupted, ""); dictPtr += hufHeaderSize; } { unsigned offcodeLog; size_t const offcodeHeaderSize = FSE_readNCount(offcodeNCount, &offcodeMaxValue, &offcodeLog, dictPtr, dictEnd-dictPtr); RETURN_ERROR_IF(FSE_isError(offcodeHeaderSize), dictionary_corrupted, ""); RETURN_ERROR_IF(offcodeLog > OffFSELog, dictionary_corrupted, ""); /* fill all offset symbols to avoid garbage at end of table */ RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp( bs->entropy.fse.offcodeCTable, offcodeNCount, MaxOff, offcodeLog, workspace, HUF_WORKSPACE_SIZE)), dictionary_corrupted, ""); /* Defer checking offcodeMaxValue because we need to know the size of the dictionary content */ dictPtr += offcodeHeaderSize; } { short matchlengthNCount[MaxML+1]; unsigned matchlengthMaxValue = MaxML, matchlengthLog; size_t const matchlengthHeaderSize = FSE_readNCount(matchlengthNCount, &matchlengthMaxValue, &matchlengthLog, dictPtr, dictEnd-dictPtr); RETURN_ERROR_IF(FSE_isError(matchlengthHeaderSize), dictionary_corrupted, ""); RETURN_ERROR_IF(matchlengthLog > MLFSELog, dictionary_corrupted, ""); RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp( bs->entropy.fse.matchlengthCTable, matchlengthNCount, matchlengthMaxValue, matchlengthLog, workspace, HUF_WORKSPACE_SIZE)), dictionary_corrupted, ""); bs->entropy.fse.matchlength_repeatMode = ZSTD_dictNCountRepeat(matchlengthNCount, matchlengthMaxValue, MaxML); dictPtr += matchlengthHeaderSize; } { short litlengthNCount[MaxLL+1]; unsigned litlengthMaxValue = MaxLL, litlengthLog; size_t const litlengthHeaderSize = FSE_readNCount(litlengthNCount, &litlengthMaxValue, &litlengthLog, dictPtr, dictEnd-dictPtr); RETURN_ERROR_IF(FSE_isError(litlengthHeaderSize), dictionary_corrupted, ""); RETURN_ERROR_IF(litlengthLog > LLFSELog, dictionary_corrupted, ""); RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp( bs->entropy.fse.litlengthCTable, litlengthNCount, litlengthMaxValue, litlengthLog, workspace, HUF_WORKSPACE_SIZE)), dictionary_corrupted, ""); bs->entropy.fse.litlength_repeatMode = ZSTD_dictNCountRepeat(litlengthNCount, litlengthMaxValue, MaxLL); dictPtr += litlengthHeaderSize; } RETURN_ERROR_IF(dictPtr+12 > dictEnd, dictionary_corrupted, ""); bs->rep[0] = MEM_readLE32(dictPtr+0); bs->rep[1] = MEM_readLE32(dictPtr+4); bs->rep[2] = MEM_readLE32(dictPtr+8); dictPtr += 12; { size_t const dictContentSize = (size_t)(dictEnd - dictPtr); U32 offcodeMax = MaxOff; if (dictContentSize <= ((U32)-1) - 128 KB) { U32 const maxOffset = (U32)dictContentSize + 128 KB; /* The maximum offset that must be supported */ offcodeMax = ZSTD_highbit32(maxOffset); /* Calculate minimum offset code required to represent maxOffset */ } /* All offset values <= dictContentSize + 128 KB must be representable for a valid table */ bs->entropy.fse.offcode_repeatMode = ZSTD_dictNCountRepeat(offcodeNCount, offcodeMaxValue, MIN(offcodeMax, MaxOff)); /* All repCodes must be <= dictContentSize and != 0 */ { U32 u; for (u=0; u<3; u++) { RETURN_ERROR_IF(bs->rep[u] == 0, dictionary_corrupted, ""); RETURN_ERROR_IF(bs->rep[u] > dictContentSize, dictionary_corrupted, ""); } } } return dictPtr - (const BYTE*)dict; } /* Dictionary format : * See : * https://github.com/facebook/zstd/blob/release/doc/zstd_compression_format.md#dictionary-format */ /*! ZSTD_loadZstdDictionary() : * @return : dictID, or an error code * assumptions : magic number supposed already checked * dictSize supposed >= 8 */ static size_t ZSTD_loadZstdDictionary(ZSTD_compressedBlockState_t* bs, ZSTD_matchState_t* ms, ZSTD_cwksp* ws, ZSTD_CCtx_params const* params, const void* dict, size_t dictSize, ZSTD_dictTableLoadMethod_e dtlm, ZSTD_tableFillPurpose_e tfp, void* workspace) { const BYTE* dictPtr = (const BYTE*)dict; const BYTE* const dictEnd = dictPtr + dictSize; size_t dictID; size_t eSize; ZSTD_STATIC_ASSERT(HUF_WORKSPACE_SIZE >= (1<= 8); assert(MEM_readLE32(dictPtr) == ZSTD_MAGIC_DICTIONARY); dictID = params->fParams.noDictIDFlag ? 0 : MEM_readLE32(dictPtr + 4 /* skip magic number */ ); eSize = ZSTD_loadCEntropy(bs, workspace, dict, dictSize); FORWARD_IF_ERROR(eSize, "ZSTD_loadCEntropy failed"); dictPtr += eSize; { size_t const dictContentSize = (size_t)(dictEnd - dictPtr); FORWARD_IF_ERROR(ZSTD_loadDictionaryContent( ms, NULL, ws, params, dictPtr, dictContentSize, dtlm, tfp), ""); } return dictID; } /** ZSTD_compress_insertDictionary() : * @return : dictID, or an error code */ static size_t ZSTD_compress_insertDictionary(ZSTD_compressedBlockState_t* bs, ZSTD_matchState_t* ms, ldmState_t* ls, ZSTD_cwksp* ws, const ZSTD_CCtx_params* params, const void* dict, size_t dictSize, ZSTD_dictContentType_e dictContentType, ZSTD_dictTableLoadMethod_e dtlm, ZSTD_tableFillPurpose_e tfp, void* workspace) { DEBUGLOG(4, "ZSTD_compress_insertDictionary (dictSize=%u)", (U32)dictSize); if ((dict==NULL) || (dictSize<8)) { RETURN_ERROR_IF(dictContentType == ZSTD_dct_fullDict, dictionary_wrong, ""); return 0; } ZSTD_reset_compressedBlockState(bs); /* dict restricted modes */ if (dictContentType == ZSTD_dct_rawContent) return ZSTD_loadDictionaryContent(ms, ls, ws, params, dict, dictSize, dtlm, tfp); if (MEM_readLE32(dict) != ZSTD_MAGIC_DICTIONARY) { if (dictContentType == ZSTD_dct_auto) { DEBUGLOG(4, "raw content dictionary detected"); return ZSTD_loadDictionaryContent( ms, ls, ws, params, dict, dictSize, dtlm, tfp); } RETURN_ERROR_IF(dictContentType == ZSTD_dct_fullDict, dictionary_wrong, ""); assert(0); /* impossible */ } /* dict as full zstd dictionary */ return ZSTD_loadZstdDictionary( bs, ms, ws, params, dict, dictSize, dtlm, tfp, workspace); } #define ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF (128 KB) #define ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER (6ULL) /*! ZSTD_compressBegin_internal() : * Assumption : either @dict OR @cdict (or none) is non-NULL, never both * @return : 0, or an error code */ static size_t ZSTD_compressBegin_internal(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_dictContentType_e dictContentType, ZSTD_dictTableLoadMethod_e dtlm, const ZSTD_CDict* cdict, const ZSTD_CCtx_params* params, U64 pledgedSrcSize, ZSTD_buffered_policy_e zbuff) { size_t const dictContentSize = cdict ? cdict->dictContentSize : dictSize; #if ZSTD_TRACE cctx->traceCtx = (ZSTD_trace_compress_begin != NULL) ? ZSTD_trace_compress_begin(cctx) : 0; #endif DEBUGLOG(4, "ZSTD_compressBegin_internal: wlog=%u", params->cParams.windowLog); /* params are supposed to be fully validated at this point */ assert(!ZSTD_isError(ZSTD_checkCParams(params->cParams))); assert(!((dict) && (cdict))); /* either dict or cdict, not both */ if ( (cdict) && (cdict->dictContentSize > 0) && ( pledgedSrcSize < ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF || pledgedSrcSize < cdict->dictContentSize * ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER || pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN || cdict->compressionLevel == 0) && (params->attachDictPref != ZSTD_dictForceLoad) ) { return ZSTD_resetCCtx_usingCDict(cctx, cdict, params, pledgedSrcSize, zbuff); } FORWARD_IF_ERROR( ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize, dictContentSize, ZSTDcrp_makeClean, zbuff) , ""); { size_t const dictID = cdict ? ZSTD_compress_insertDictionary( cctx->blockState.prevCBlock, &cctx->blockState.matchState, &cctx->ldmState, &cctx->workspace, &cctx->appliedParams, cdict->dictContent, cdict->dictContentSize, cdict->dictContentType, dtlm, ZSTD_tfp_forCCtx, cctx->entropyWorkspace) : ZSTD_compress_insertDictionary( cctx->blockState.prevCBlock, &cctx->blockState.matchState, &cctx->ldmState, &cctx->workspace, &cctx->appliedParams, dict, dictSize, dictContentType, dtlm, ZSTD_tfp_forCCtx, cctx->entropyWorkspace); FORWARD_IF_ERROR(dictID, "ZSTD_compress_insertDictionary failed"); assert(dictID <= UINT_MAX); cctx->dictID = (U32)dictID; cctx->dictContentSize = dictContentSize; } return 0; } size_t ZSTD_compressBegin_advanced_internal(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_dictContentType_e dictContentType, ZSTD_dictTableLoadMethod_e dtlm, const ZSTD_CDict* cdict, const ZSTD_CCtx_params* params, unsigned long long pledgedSrcSize) { DEBUGLOG(4, "ZSTD_compressBegin_advanced_internal: wlog=%u", params->cParams.windowLog); /* compression parameters verification and optimization */ FORWARD_IF_ERROR( ZSTD_checkCParams(params->cParams) , ""); return ZSTD_compressBegin_internal(cctx, dict, dictSize, dictContentType, dtlm, cdict, params, pledgedSrcSize, ZSTDb_not_buffered); } /*! ZSTD_compressBegin_advanced() : * @return : 0, or an error code */ size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize) { ZSTD_CCtx_params cctxParams; ZSTD_CCtxParams_init_internal(&cctxParams, ¶ms, ZSTD_NO_CLEVEL); return ZSTD_compressBegin_advanced_internal(cctx, dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL /*cdict*/, &cctxParams, pledgedSrcSize); } static size_t ZSTD_compressBegin_usingDict_deprecated(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel) { ZSTD_CCtx_params cctxParams; { ZSTD_parameters const params = ZSTD_getParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize, ZSTD_cpm_noAttachDict); ZSTD_CCtxParams_init_internal(&cctxParams, ¶ms, (compressionLevel == 0) ? ZSTD_CLEVEL_DEFAULT : compressionLevel); } DEBUGLOG(4, "ZSTD_compressBegin_usingDict (dictSize=%u)", (unsigned)dictSize); return ZSTD_compressBegin_internal(cctx, dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL, &cctxParams, ZSTD_CONTENTSIZE_UNKNOWN, ZSTDb_not_buffered); } size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel) { return ZSTD_compressBegin_usingDict_deprecated(cctx, dict, dictSize, compressionLevel); } size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel) { return ZSTD_compressBegin_usingDict_deprecated(cctx, NULL, 0, compressionLevel); } /*! ZSTD_writeEpilogue() : * Ends a frame. * @return : nb of bytes written into dst (or an error code) */ static size_t ZSTD_writeEpilogue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity) { BYTE* const ostart = (BYTE*)dst; BYTE* op = ostart; size_t fhSize = 0; DEBUGLOG(4, "ZSTD_writeEpilogue"); RETURN_ERROR_IF(cctx->stage == ZSTDcs_created, stage_wrong, "init missing"); /* special case : empty frame */ if (cctx->stage == ZSTDcs_init) { fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, &cctx->appliedParams, 0, 0); FORWARD_IF_ERROR(fhSize, "ZSTD_writeFrameHeader failed"); dstCapacity -= fhSize; op += fhSize; cctx->stage = ZSTDcs_ongoing; } if (cctx->stage != ZSTDcs_ending) { /* write one last empty block, make it the "last" block */ U32 const cBlockHeader24 = 1 /* last block */ + (((U32)bt_raw)<<1) + 0; RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall, "no room for epilogue"); MEM_writeLE32(op, cBlockHeader24); op += ZSTD_blockHeaderSize; dstCapacity -= ZSTD_blockHeaderSize; } if (cctx->appliedParams.fParams.checksumFlag) { U32 const checksum = (U32) XXH64_digest(&cctx->xxhState); RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall, "no room for checksum"); DEBUGLOG(4, "ZSTD_writeEpilogue: write checksum : %08X", (unsigned)checksum); MEM_writeLE32(op, checksum); op += 4; } cctx->stage = ZSTDcs_created; /* return to "created but no init" status */ return op-ostart; } void ZSTD_CCtx_trace(ZSTD_CCtx* cctx, size_t extraCSize) { #if ZSTD_TRACE if (cctx->traceCtx && ZSTD_trace_compress_end != NULL) { int const streaming = cctx->inBuffSize > 0 || cctx->outBuffSize > 0 || cctx->appliedParams.nbWorkers > 0; ZSTD_Trace trace; ZSTD_memset(&trace, 0, sizeof(trace)); trace.version = ZSTD_VERSION_NUMBER; trace.streaming = streaming; trace.dictionaryID = cctx->dictID; trace.dictionarySize = cctx->dictContentSize; trace.uncompressedSize = cctx->consumedSrcSize; trace.compressedSize = cctx->producedCSize + extraCSize; trace.params = &cctx->appliedParams; trace.cctx = cctx; ZSTD_trace_compress_end(cctx->traceCtx, &trace); } cctx->traceCtx = 0; #else (void)cctx; (void)extraCSize; #endif } size_t ZSTD_compressEnd_public(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize) { size_t endResult; size_t const cSize = ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 1 /* frame mode */, 1 /* last chunk */); FORWARD_IF_ERROR(cSize, "ZSTD_compressContinue_internal failed"); endResult = ZSTD_writeEpilogue(cctx, (char*)dst + cSize, dstCapacity-cSize); FORWARD_IF_ERROR(endResult, "ZSTD_writeEpilogue failed"); assert(!(cctx->appliedParams.fParams.contentSizeFlag && cctx->pledgedSrcSizePlusOne == 0)); if (cctx->pledgedSrcSizePlusOne != 0) { /* control src size */ ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_UNKNOWN == (unsigned long long)-1); DEBUGLOG(4, "end of frame : controlling src size"); RETURN_ERROR_IF( cctx->pledgedSrcSizePlusOne != cctx->consumedSrcSize+1, srcSize_wrong, "error : pledgedSrcSize = %u, while realSrcSize = %u", (unsigned)cctx->pledgedSrcSizePlusOne-1, (unsigned)cctx->consumedSrcSize); } ZSTD_CCtx_trace(cctx, endResult); return cSize + endResult; } /* NOTE: Must just wrap ZSTD_compressEnd_public() */ size_t ZSTD_compressEnd(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize) { return ZSTD_compressEnd_public(cctx, dst, dstCapacity, src, srcSize); } size_t ZSTD_compress_advanced (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const void* dict,size_t dictSize, ZSTD_parameters params) { DEBUGLOG(4, "ZSTD_compress_advanced"); FORWARD_IF_ERROR(ZSTD_checkCParams(params.cParams), ""); ZSTD_CCtxParams_init_internal(&cctx->simpleApiParams, ¶ms, ZSTD_NO_CLEVEL); return ZSTD_compress_advanced_internal(cctx, dst, dstCapacity, src, srcSize, dict, dictSize, &cctx->simpleApiParams); } /* Internal */ size_t ZSTD_compress_advanced_internal( ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const void* dict,size_t dictSize, const ZSTD_CCtx_params* params) { DEBUGLOG(4, "ZSTD_compress_advanced_internal (srcSize:%u)", (unsigned)srcSize); FORWARD_IF_ERROR( ZSTD_compressBegin_internal(cctx, dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL, params, srcSize, ZSTDb_not_buffered) , ""); return ZSTD_compressEnd_public(cctx, dst, dstCapacity, src, srcSize); } size_t ZSTD_compress_usingDict(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const void* dict, size_t dictSize, int compressionLevel) { { ZSTD_parameters const params = ZSTD_getParams_internal(compressionLevel, srcSize, dict ? dictSize : 0, ZSTD_cpm_noAttachDict); assert(params.fParams.contentSizeFlag == 1); ZSTD_CCtxParams_init_internal(&cctx->simpleApiParams, ¶ms, (compressionLevel == 0) ? ZSTD_CLEVEL_DEFAULT: compressionLevel); } DEBUGLOG(4, "ZSTD_compress_usingDict (srcSize=%u)", (unsigned)srcSize); return ZSTD_compress_advanced_internal(cctx, dst, dstCapacity, src, srcSize, dict, dictSize, &cctx->simpleApiParams); } size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, int compressionLevel) { DEBUGLOG(4, "ZSTD_compressCCtx (srcSize=%u)", (unsigned)srcSize); assert(cctx != NULL); return ZSTD_compress_usingDict(cctx, dst, dstCapacity, src, srcSize, NULL, 0, compressionLevel); } size_t ZSTD_compress(void* dst, size_t dstCapacity, const void* src, size_t srcSize, int compressionLevel) { size_t result; #if ZSTD_COMPRESS_HEAPMODE ZSTD_CCtx* cctx = ZSTD_createCCtx(); RETURN_ERROR_IF(!cctx, memory_allocation, "ZSTD_createCCtx failed"); result = ZSTD_compressCCtx(cctx, dst, dstCapacity, src, srcSize, compressionLevel); ZSTD_freeCCtx(cctx); #else ZSTD_CCtx ctxBody; ZSTD_initCCtx(&ctxBody, ZSTD_defaultCMem); result = ZSTD_compressCCtx(&ctxBody, dst, dstCapacity, src, srcSize, compressionLevel); ZSTD_freeCCtxContent(&ctxBody); /* can't free ctxBody itself, as it's on stack; free only heap content */ #endif return result; } /* ===== Dictionary API ===== */ /*! ZSTD_estimateCDictSize_advanced() : * Estimate amount of memory that will be needed to create a dictionary with following arguments */ size_t ZSTD_estimateCDictSize_advanced( size_t dictSize, ZSTD_compressionParameters cParams, ZSTD_dictLoadMethod_e dictLoadMethod) { DEBUGLOG(5, "sizeof(ZSTD_CDict) : %u", (unsigned)sizeof(ZSTD_CDict)); return ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict)) + ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE) /* enableDedicatedDictSearch == 1 ensures that CDict estimation will not be too small * in case we are using DDS with row-hash. */ + ZSTD_sizeof_matchState(&cParams, ZSTD_resolveRowMatchFinderMode(ZSTD_ps_auto, &cParams), /* enableDedicatedDictSearch */ 1, /* forCCtx */ 0) + (dictLoadMethod == ZSTD_dlm_byRef ? 0 : ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void *)))); } size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel) { ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize, ZSTD_cpm_createCDict); return ZSTD_estimateCDictSize_advanced(dictSize, cParams, ZSTD_dlm_byCopy); } size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict) { if (cdict==NULL) return 0; /* support sizeof on NULL */ DEBUGLOG(5, "sizeof(*cdict) : %u", (unsigned)sizeof(*cdict)); /* cdict may be in the workspace */ return (cdict->workspace.workspace == cdict ? 0 : sizeof(*cdict)) + ZSTD_cwksp_sizeof(&cdict->workspace); } static size_t ZSTD_initCDict_internal( ZSTD_CDict* cdict, const void* dictBuffer, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType, ZSTD_CCtx_params params) { DEBUGLOG(3, "ZSTD_initCDict_internal (dictContentType:%u)", (unsigned)dictContentType); assert(!ZSTD_checkCParams(params.cParams)); cdict->matchState.cParams = params.cParams; cdict->matchState.dedicatedDictSearch = params.enableDedicatedDictSearch; if ((dictLoadMethod == ZSTD_dlm_byRef) || (!dictBuffer) || (!dictSize)) { cdict->dictContent = dictBuffer; } else { void *internalBuffer = ZSTD_cwksp_reserve_object(&cdict->workspace, ZSTD_cwksp_align(dictSize, sizeof(void*))); RETURN_ERROR_IF(!internalBuffer, memory_allocation, "NULL pointer!"); cdict->dictContent = internalBuffer; ZSTD_memcpy(internalBuffer, dictBuffer, dictSize); } cdict->dictContentSize = dictSize; cdict->dictContentType = dictContentType; cdict->entropyWorkspace = (U32*)ZSTD_cwksp_reserve_object(&cdict->workspace, HUF_WORKSPACE_SIZE); /* Reset the state to no dictionary */ ZSTD_reset_compressedBlockState(&cdict->cBlockState); FORWARD_IF_ERROR(ZSTD_reset_matchState( &cdict->matchState, &cdict->workspace, ¶ms.cParams, params.useRowMatchFinder, ZSTDcrp_makeClean, ZSTDirp_reset, ZSTD_resetTarget_CDict), ""); /* (Maybe) load the dictionary * Skips loading the dictionary if it is < 8 bytes. */ { params.compressionLevel = ZSTD_CLEVEL_DEFAULT; params.fParams.contentSizeFlag = 1; { size_t const dictID = ZSTD_compress_insertDictionary( &cdict->cBlockState, &cdict->matchState, NULL, &cdict->workspace, ¶ms, cdict->dictContent, cdict->dictContentSize, dictContentType, ZSTD_dtlm_full, ZSTD_tfp_forCDict, cdict->entropyWorkspace); FORWARD_IF_ERROR(dictID, "ZSTD_compress_insertDictionary failed"); assert(dictID <= (size_t)(U32)-1); cdict->dictID = (U32)dictID; } } return 0; } static ZSTD_CDict* ZSTD_createCDict_advanced_internal(size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_compressionParameters cParams, ZSTD_paramSwitch_e useRowMatchFinder, U32 enableDedicatedDictSearch, ZSTD_customMem customMem) { if ((!customMem.customAlloc) ^ (!customMem.customFree)) return NULL; { size_t const workspaceSize = ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict)) + ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE) + ZSTD_sizeof_matchState(&cParams, useRowMatchFinder, enableDedicatedDictSearch, /* forCCtx */ 0) + (dictLoadMethod == ZSTD_dlm_byRef ? 0 : ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void*)))); void* const workspace = ZSTD_customMalloc(workspaceSize, customMem); ZSTD_cwksp ws; ZSTD_CDict* cdict; if (!workspace) { ZSTD_customFree(workspace, customMem); return NULL; } ZSTD_cwksp_init(&ws, workspace, workspaceSize, ZSTD_cwksp_dynamic_alloc); cdict = (ZSTD_CDict*)ZSTD_cwksp_reserve_object(&ws, sizeof(ZSTD_CDict)); assert(cdict != NULL); ZSTD_cwksp_move(&cdict->workspace, &ws); cdict->customMem = customMem; cdict->compressionLevel = ZSTD_NO_CLEVEL; /* signals advanced API usage */ cdict->useRowMatchFinder = useRowMatchFinder; return cdict; } } ZSTD_CDict* ZSTD_createCDict_advanced(const void* dictBuffer, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType, ZSTD_compressionParameters cParams, ZSTD_customMem customMem) { ZSTD_CCtx_params cctxParams; ZSTD_memset(&cctxParams, 0, sizeof(cctxParams)); ZSTD_CCtxParams_init(&cctxParams, 0); cctxParams.cParams = cParams; cctxParams.customMem = customMem; return ZSTD_createCDict_advanced2( dictBuffer, dictSize, dictLoadMethod, dictContentType, &cctxParams, customMem); } ZSTD_CDict* ZSTD_createCDict_advanced2( const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType, const ZSTD_CCtx_params* originalCctxParams, ZSTD_customMem customMem) { ZSTD_CCtx_params cctxParams = *originalCctxParams; ZSTD_compressionParameters cParams; ZSTD_CDict* cdict; DEBUGLOG(3, "ZSTD_createCDict_advanced2, mode %u", (unsigned)dictContentType); if (!customMem.customAlloc ^ !customMem.customFree) return NULL; if (cctxParams.enableDedicatedDictSearch) { cParams = ZSTD_dedicatedDictSearch_getCParams( cctxParams.compressionLevel, dictSize); ZSTD_overrideCParams(&cParams, &cctxParams.cParams); } else { cParams = ZSTD_getCParamsFromCCtxParams( &cctxParams, ZSTD_CONTENTSIZE_UNKNOWN, dictSize, ZSTD_cpm_createCDict); } if (!ZSTD_dedicatedDictSearch_isSupported(&cParams)) { /* Fall back to non-DDSS params */ cctxParams.enableDedicatedDictSearch = 0; cParams = ZSTD_getCParamsFromCCtxParams( &cctxParams, ZSTD_CONTENTSIZE_UNKNOWN, dictSize, ZSTD_cpm_createCDict); } DEBUGLOG(3, "ZSTD_createCDict_advanced2: DDS: %u", cctxParams.enableDedicatedDictSearch); cctxParams.cParams = cParams; cctxParams.useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(cctxParams.useRowMatchFinder, &cParams); cdict = ZSTD_createCDict_advanced_internal(dictSize, dictLoadMethod, cctxParams.cParams, cctxParams.useRowMatchFinder, cctxParams.enableDedicatedDictSearch, customMem); if (ZSTD_isError( ZSTD_initCDict_internal(cdict, dict, dictSize, dictLoadMethod, dictContentType, cctxParams) )) { ZSTD_freeCDict(cdict); return NULL; } return cdict; } ZSTD_CDict* ZSTD_createCDict(const void* dict, size_t dictSize, int compressionLevel) { ZSTD_compressionParameters cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize, ZSTD_cpm_createCDict); ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(dict, dictSize, ZSTD_dlm_byCopy, ZSTD_dct_auto, cParams, ZSTD_defaultCMem); if (cdict) cdict->compressionLevel = (compressionLevel == 0) ? ZSTD_CLEVEL_DEFAULT : compressionLevel; return cdict; } ZSTD_CDict* ZSTD_createCDict_byReference(const void* dict, size_t dictSize, int compressionLevel) { ZSTD_compressionParameters cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize, ZSTD_cpm_createCDict); ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(dict, dictSize, ZSTD_dlm_byRef, ZSTD_dct_auto, cParams, ZSTD_defaultCMem); if (cdict) cdict->compressionLevel = (compressionLevel == 0) ? ZSTD_CLEVEL_DEFAULT : compressionLevel; return cdict; } size_t ZSTD_freeCDict(ZSTD_CDict* cdict) { if (cdict==NULL) return 0; /* support free on NULL */ { ZSTD_customMem const cMem = cdict->customMem; int cdictInWorkspace = ZSTD_cwksp_owns_buffer(&cdict->workspace, cdict); ZSTD_cwksp_free(&cdict->workspace, cMem); if (!cdictInWorkspace) { ZSTD_customFree(cdict, cMem); } return 0; } } /*! ZSTD_initStaticCDict_advanced() : * Generate a digested dictionary in provided memory area. * workspace: The memory area to emplace the dictionary into. * Provided pointer must 8-bytes aligned. * It must outlive dictionary usage. * workspaceSize: Use ZSTD_estimateCDictSize() * to determine how large workspace must be. * cParams : use ZSTD_getCParams() to transform a compression level * into its relevants cParams. * @return : pointer to ZSTD_CDict*, or NULL if error (size too small) * Note : there is no corresponding "free" function. * Since workspace was allocated externally, it must be freed externally. */ const ZSTD_CDict* ZSTD_initStaticCDict( void* workspace, size_t workspaceSize, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType, ZSTD_compressionParameters cParams) { ZSTD_paramSwitch_e const useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(ZSTD_ps_auto, &cParams); /* enableDedicatedDictSearch == 1 ensures matchstate is not too small in case this CDict will be used for DDS + row hash */ size_t const matchStateSize = ZSTD_sizeof_matchState(&cParams, useRowMatchFinder, /* enableDedicatedDictSearch */ 1, /* forCCtx */ 0); size_t const neededSize = ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict)) + (dictLoadMethod == ZSTD_dlm_byRef ? 0 : ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void*)))) + ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE) + matchStateSize; ZSTD_CDict* cdict; ZSTD_CCtx_params params; if ((size_t)workspace & 7) return NULL; /* 8-aligned */ { ZSTD_cwksp ws; ZSTD_cwksp_init(&ws, workspace, workspaceSize, ZSTD_cwksp_static_alloc); cdict = (ZSTD_CDict*)ZSTD_cwksp_reserve_object(&ws, sizeof(ZSTD_CDict)); if (cdict == NULL) return NULL; ZSTD_cwksp_move(&cdict->workspace, &ws); } DEBUGLOG(4, "(workspaceSize < neededSize) : (%u < %u) => %u", (unsigned)workspaceSize, (unsigned)neededSize, (unsigned)(workspaceSize < neededSize)); if (workspaceSize < neededSize) return NULL; ZSTD_CCtxParams_init(¶ms, 0); params.cParams = cParams; params.useRowMatchFinder = useRowMatchFinder; cdict->useRowMatchFinder = useRowMatchFinder; cdict->compressionLevel = ZSTD_NO_CLEVEL; if (ZSTD_isError( ZSTD_initCDict_internal(cdict, dict, dictSize, dictLoadMethod, dictContentType, params) )) return NULL; return cdict; } ZSTD_compressionParameters ZSTD_getCParamsFromCDict(const ZSTD_CDict* cdict) { assert(cdict != NULL); return cdict->matchState.cParams; } /*! ZSTD_getDictID_fromCDict() : * Provides the dictID of the dictionary loaded into `cdict`. * If @return == 0, the dictionary is not conformant to Zstandard specification, or empty. * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */ unsigned ZSTD_getDictID_fromCDict(const ZSTD_CDict* cdict) { if (cdict==NULL) return 0; return cdict->dictID; } /* ZSTD_compressBegin_usingCDict_internal() : * Implementation of various ZSTD_compressBegin_usingCDict* functions. */ static size_t ZSTD_compressBegin_usingCDict_internal( ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize) { ZSTD_CCtx_params cctxParams; DEBUGLOG(4, "ZSTD_compressBegin_usingCDict_internal"); RETURN_ERROR_IF(cdict==NULL, dictionary_wrong, "NULL pointer!"); /* Initialize the cctxParams from the cdict */ { ZSTD_parameters params; params.fParams = fParams; params.cParams = ( pledgedSrcSize < ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF || pledgedSrcSize < cdict->dictContentSize * ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER || pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN || cdict->compressionLevel == 0 ) ? ZSTD_getCParamsFromCDict(cdict) : ZSTD_getCParams(cdict->compressionLevel, pledgedSrcSize, cdict->dictContentSize); ZSTD_CCtxParams_init_internal(&cctxParams, ¶ms, cdict->compressionLevel); } /* Increase window log to fit the entire dictionary and source if the * source size is known. Limit the increase to 19, which is the * window log for compression level 1 with the largest source size. */ if (pledgedSrcSize != ZSTD_CONTENTSIZE_UNKNOWN) { U32 const limitedSrcSize = (U32)MIN(pledgedSrcSize, 1U << 19); U32 const limitedSrcLog = limitedSrcSize > 1 ? ZSTD_highbit32(limitedSrcSize - 1) + 1 : 1; cctxParams.cParams.windowLog = MAX(cctxParams.cParams.windowLog, limitedSrcLog); } return ZSTD_compressBegin_internal(cctx, NULL, 0, ZSTD_dct_auto, ZSTD_dtlm_fast, cdict, &cctxParams, pledgedSrcSize, ZSTDb_not_buffered); } /* ZSTD_compressBegin_usingCDict_advanced() : * This function is DEPRECATED. * cdict must be != NULL */ size_t ZSTD_compressBegin_usingCDict_advanced( ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize) { return ZSTD_compressBegin_usingCDict_internal(cctx, cdict, fParams, pledgedSrcSize); } /* ZSTD_compressBegin_usingCDict() : * cdict must be != NULL */ size_t ZSTD_compressBegin_usingCDict_deprecated(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict) { ZSTD_frameParameters const fParams = { 0 /*content*/, 0 /*checksum*/, 0 /*noDictID*/ }; return ZSTD_compressBegin_usingCDict_internal(cctx, cdict, fParams, ZSTD_CONTENTSIZE_UNKNOWN); } size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict) { return ZSTD_compressBegin_usingCDict_deprecated(cctx, cdict); } /*! ZSTD_compress_usingCDict_internal(): * Implementation of various ZSTD_compress_usingCDict* functions. */ static size_t ZSTD_compress_usingCDict_internal(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const ZSTD_CDict* cdict, ZSTD_frameParameters fParams) { FORWARD_IF_ERROR(ZSTD_compressBegin_usingCDict_internal(cctx, cdict, fParams, srcSize), ""); /* will check if cdict != NULL */ return ZSTD_compressEnd_public(cctx, dst, dstCapacity, src, srcSize); } /*! ZSTD_compress_usingCDict_advanced(): * This function is DEPRECATED. */ size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const ZSTD_CDict* cdict, ZSTD_frameParameters fParams) { return ZSTD_compress_usingCDict_internal(cctx, dst, dstCapacity, src, srcSize, cdict, fParams); } /*! ZSTD_compress_usingCDict() : * Compression using a digested Dictionary. * Faster startup than ZSTD_compress_usingDict(), recommended when same dictionary is used multiple times. * Note that compression parameters are decided at CDict creation time * while frame parameters are hardcoded */ size_t ZSTD_compress_usingCDict(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const ZSTD_CDict* cdict) { ZSTD_frameParameters const fParams = { 1 /*content*/, 0 /*checksum*/, 0 /*noDictID*/ }; return ZSTD_compress_usingCDict_internal(cctx, dst, dstCapacity, src, srcSize, cdict, fParams); } /* ****************************************************************** * Streaming ********************************************************************/ ZSTD_CStream* ZSTD_createCStream(void) { DEBUGLOG(3, "ZSTD_createCStream"); return ZSTD_createCStream_advanced(ZSTD_defaultCMem); } ZSTD_CStream* ZSTD_initStaticCStream(void *workspace, size_t workspaceSize) { return ZSTD_initStaticCCtx(workspace, workspaceSize); } ZSTD_CStream* ZSTD_createCStream_advanced(ZSTD_customMem customMem) { /* CStream and CCtx are now same object */ return ZSTD_createCCtx_advanced(customMem); } size_t ZSTD_freeCStream(ZSTD_CStream* zcs) { return ZSTD_freeCCtx(zcs); /* same object */ } /*====== Initialization ======*/ size_t ZSTD_CStreamInSize(void) { return ZSTD_BLOCKSIZE_MAX; } size_t ZSTD_CStreamOutSize(void) { return ZSTD_compressBound(ZSTD_BLOCKSIZE_MAX) + ZSTD_blockHeaderSize + 4 /* 32-bits hash */ ; } static ZSTD_cParamMode_e ZSTD_getCParamMode(ZSTD_CDict const* cdict, ZSTD_CCtx_params const* params, U64 pledgedSrcSize) { if (cdict != NULL && ZSTD_shouldAttachDict(cdict, params, pledgedSrcSize)) return ZSTD_cpm_attachDict; else return ZSTD_cpm_noAttachDict; } /* ZSTD_resetCStream(): * pledgedSrcSize == 0 means "unknown" */ size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pss) { /* temporary : 0 interpreted as "unknown" during transition period. * Users willing to specify "unknown" **must** use ZSTD_CONTENTSIZE_UNKNOWN. * 0 will be interpreted as "empty" in the future. */ U64 const pledgedSrcSize = (pss==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss; DEBUGLOG(4, "ZSTD_resetCStream: pledgedSrcSize = %u", (unsigned)pledgedSrcSize); FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , ""); FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , ""); return 0; } /*! ZSTD_initCStream_internal() : * Note : for lib/compress only. Used by zstdmt_compress.c. * Assumption 1 : params are valid * Assumption 2 : either dict, or cdict, is defined, not both */ size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs, const void* dict, size_t dictSize, const ZSTD_CDict* cdict, const ZSTD_CCtx_params* params, unsigned long long pledgedSrcSize) { DEBUGLOG(4, "ZSTD_initCStream_internal"); FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , ""); FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , ""); assert(!ZSTD_isError(ZSTD_checkCParams(params->cParams))); zcs->requestedParams = *params; assert(!((dict) && (cdict))); /* either dict or cdict, not both */ if (dict) { FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) , ""); } else { /* Dictionary is cleared if !cdict */ FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) , ""); } return 0; } /* ZSTD_initCStream_usingCDict_advanced() : * same as ZSTD_initCStream_usingCDict(), with control over frame parameters */ size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs, const ZSTD_CDict* cdict, ZSTD_frameParameters fParams, unsigned long long pledgedSrcSize) { DEBUGLOG(4, "ZSTD_initCStream_usingCDict_advanced"); FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , ""); FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , ""); zcs->requestedParams.fParams = fParams; FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) , ""); return 0; } /* note : cdict must outlive compression session */ size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict) { DEBUGLOG(4, "ZSTD_initCStream_usingCDict"); FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , ""); FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) , ""); return 0; } /* ZSTD_initCStream_advanced() : * pledgedSrcSize must be exact. * if srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN. * dict is loaded with default parameters ZSTD_dct_auto and ZSTD_dlm_byCopy. */ size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pss) { /* for compatibility with older programs relying on this behavior. * Users should now specify ZSTD_CONTENTSIZE_UNKNOWN. * This line will be removed in the future. */ U64 const pledgedSrcSize = (pss==0 && params.fParams.contentSizeFlag==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss; DEBUGLOG(4, "ZSTD_initCStream_advanced"); FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , ""); FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , ""); FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) , ""); ZSTD_CCtxParams_setZstdParams(&zcs->requestedParams, ¶ms); FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) , ""); return 0; } size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel) { DEBUGLOG(4, "ZSTD_initCStream_usingDict"); FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , ""); FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) , ""); FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) , ""); return 0; } size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs, int compressionLevel, unsigned long long pss) { /* temporary : 0 interpreted as "unknown" during transition period. * Users willing to specify "unknown" **must** use ZSTD_CONTENTSIZE_UNKNOWN. * 0 will be interpreted as "empty" in the future. */ U64 const pledgedSrcSize = (pss==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss; DEBUGLOG(4, "ZSTD_initCStream_srcSize"); FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , ""); FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, NULL) , ""); FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) , ""); FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , ""); return 0; } size_t ZSTD_initCStream(ZSTD_CStream* zcs, int compressionLevel) { DEBUGLOG(4, "ZSTD_initCStream"); FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , ""); FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, NULL) , ""); FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) , ""); return 0; } /*====== Compression ======*/ static size_t ZSTD_nextInputSizeHint(const ZSTD_CCtx* cctx) { if (cctx->appliedParams.inBufferMode == ZSTD_bm_stable) { return cctx->blockSize - cctx->stableIn_notConsumed; } assert(cctx->appliedParams.inBufferMode == ZSTD_bm_buffered); { size_t hintInSize = cctx->inBuffTarget - cctx->inBuffPos; if (hintInSize==0) hintInSize = cctx->blockSize; return hintInSize; } } /** ZSTD_compressStream_generic(): * internal function for all *compressStream*() variants * @return : hint size for next input to complete ongoing block */ static size_t ZSTD_compressStream_generic(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input, ZSTD_EndDirective const flushMode) { const char* const istart = (assert(input != NULL), (const char*)input->src); const char* const iend = (istart != NULL) ? istart + input->size : istart; const char* ip = (istart != NULL) ? istart + input->pos : istart; char* const ostart = (assert(output != NULL), (char*)output->dst); char* const oend = (ostart != NULL) ? ostart + output->size : ostart; char* op = (ostart != NULL) ? ostart + output->pos : ostart; U32 someMoreWork = 1; /* check expectations */ DEBUGLOG(5, "ZSTD_compressStream_generic, flush=%i, srcSize = %zu", (int)flushMode, input->size - input->pos); assert(zcs != NULL); if (zcs->appliedParams.inBufferMode == ZSTD_bm_stable) { assert(input->pos >= zcs->stableIn_notConsumed); input->pos -= zcs->stableIn_notConsumed; ip -= zcs->stableIn_notConsumed; zcs->stableIn_notConsumed = 0; } if (zcs->appliedParams.inBufferMode == ZSTD_bm_buffered) { assert(zcs->inBuff != NULL); assert(zcs->inBuffSize > 0); } if (zcs->appliedParams.outBufferMode == ZSTD_bm_buffered) { assert(zcs->outBuff != NULL); assert(zcs->outBuffSize > 0); } if (input->src == NULL) assert(input->size == 0); assert(input->pos <= input->size); if (output->dst == NULL) assert(output->size == 0); assert(output->pos <= output->size); assert((U32)flushMode <= (U32)ZSTD_e_end); while (someMoreWork) { switch(zcs->streamStage) { case zcss_init: RETURN_ERROR(init_missing, "call ZSTD_initCStream() first!"); case zcss_load: if ( (flushMode == ZSTD_e_end) && ( (size_t)(oend-op) >= ZSTD_compressBound(iend-ip) /* Enough output space */ || zcs->appliedParams.outBufferMode == ZSTD_bm_stable) /* OR we are allowed to return dstSizeTooSmall */ && (zcs->inBuffPos == 0) ) { /* shortcut to compression pass directly into output buffer */ size_t const cSize = ZSTD_compressEnd_public(zcs, op, oend-op, ip, iend-ip); DEBUGLOG(4, "ZSTD_compressEnd : cSize=%u", (unsigned)cSize); FORWARD_IF_ERROR(cSize, "ZSTD_compressEnd failed"); ip = iend; op += cSize; zcs->frameEnded = 1; ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); someMoreWork = 0; break; } /* complete loading into inBuffer in buffered mode */ if (zcs->appliedParams.inBufferMode == ZSTD_bm_buffered) { size_t const toLoad = zcs->inBuffTarget - zcs->inBuffPos; size_t const loaded = ZSTD_limitCopy( zcs->inBuff + zcs->inBuffPos, toLoad, ip, iend-ip); zcs->inBuffPos += loaded; if (ip) ip += loaded; if ( (flushMode == ZSTD_e_continue) && (zcs->inBuffPos < zcs->inBuffTarget) ) { /* not enough input to fill full block : stop here */ someMoreWork = 0; break; } if ( (flushMode == ZSTD_e_flush) && (zcs->inBuffPos == zcs->inToCompress) ) { /* empty */ someMoreWork = 0; break; } } else { assert(zcs->appliedParams.inBufferMode == ZSTD_bm_stable); if ( (flushMode == ZSTD_e_continue) && ( (size_t)(iend - ip) < zcs->blockSize) ) { /* can't compress a full block : stop here */ zcs->stableIn_notConsumed = (size_t)(iend - ip); ip = iend; /* pretend to have consumed input */ someMoreWork = 0; break; } if ( (flushMode == ZSTD_e_flush) && (ip == iend) ) { /* empty */ someMoreWork = 0; break; } } /* compress current block (note : this stage cannot be stopped in the middle) */ DEBUGLOG(5, "stream compression stage (flushMode==%u)", flushMode); { int const inputBuffered = (zcs->appliedParams.inBufferMode == ZSTD_bm_buffered); void* cDst; size_t cSize; size_t oSize = oend-op; size_t const iSize = inputBuffered ? zcs->inBuffPos - zcs->inToCompress : MIN((size_t)(iend - ip), zcs->blockSize); if (oSize >= ZSTD_compressBound(iSize) || zcs->appliedParams.outBufferMode == ZSTD_bm_stable) cDst = op; /* compress into output buffer, to skip flush stage */ else cDst = zcs->outBuff, oSize = zcs->outBuffSize; if (inputBuffered) { unsigned const lastBlock = (flushMode == ZSTD_e_end) && (ip==iend); cSize = lastBlock ? ZSTD_compressEnd_public(zcs, cDst, oSize, zcs->inBuff + zcs->inToCompress, iSize) : ZSTD_compressContinue_public(zcs, cDst, oSize, zcs->inBuff + zcs->inToCompress, iSize); FORWARD_IF_ERROR(cSize, "%s", lastBlock ? "ZSTD_compressEnd failed" : "ZSTD_compressContinue failed"); zcs->frameEnded = lastBlock; /* prepare next block */ zcs->inBuffTarget = zcs->inBuffPos + zcs->blockSize; if (zcs->inBuffTarget > zcs->inBuffSize) zcs->inBuffPos = 0, zcs->inBuffTarget = zcs->blockSize; DEBUGLOG(5, "inBuffTarget:%u / inBuffSize:%u", (unsigned)zcs->inBuffTarget, (unsigned)zcs->inBuffSize); if (!lastBlock) assert(zcs->inBuffTarget <= zcs->inBuffSize); zcs->inToCompress = zcs->inBuffPos; } else { /* !inputBuffered, hence ZSTD_bm_stable */ unsigned const lastBlock = (flushMode == ZSTD_e_end) && (ip + iSize == iend); cSize = lastBlock ? ZSTD_compressEnd_public(zcs, cDst, oSize, ip, iSize) : ZSTD_compressContinue_public(zcs, cDst, oSize, ip, iSize); /* Consume the input prior to error checking to mirror buffered mode. */ if (ip) ip += iSize; FORWARD_IF_ERROR(cSize, "%s", lastBlock ? "ZSTD_compressEnd failed" : "ZSTD_compressContinue failed"); zcs->frameEnded = lastBlock; if (lastBlock) assert(ip == iend); } if (cDst == op) { /* no need to flush */ op += cSize; if (zcs->frameEnded) { DEBUGLOG(5, "Frame completed directly in outBuffer"); someMoreWork = 0; ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); } break; } zcs->outBuffContentSize = cSize; zcs->outBuffFlushedSize = 0; zcs->streamStage = zcss_flush; /* pass-through to flush stage */ } ZSTD_FALLTHROUGH; case zcss_flush: DEBUGLOG(5, "flush stage"); assert(zcs->appliedParams.outBufferMode == ZSTD_bm_buffered); { size_t const toFlush = zcs->outBuffContentSize - zcs->outBuffFlushedSize; size_t const flushed = ZSTD_limitCopy(op, (size_t)(oend-op), zcs->outBuff + zcs->outBuffFlushedSize, toFlush); DEBUGLOG(5, "toFlush: %u into %u ==> flushed: %u", (unsigned)toFlush, (unsigned)(oend-op), (unsigned)flushed); if (flushed) op += flushed; zcs->outBuffFlushedSize += flushed; if (toFlush!=flushed) { /* flush not fully completed, presumably because dst is too small */ assert(op==oend); someMoreWork = 0; break; } zcs->outBuffContentSize = zcs->outBuffFlushedSize = 0; if (zcs->frameEnded) { DEBUGLOG(5, "Frame completed on flush"); someMoreWork = 0; ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); break; } zcs->streamStage = zcss_load; break; } default: /* impossible */ assert(0); } } input->pos = ip - istart; output->pos = op - ostart; if (zcs->frameEnded) return 0; return ZSTD_nextInputSizeHint(zcs); } static size_t ZSTD_nextInputSizeHint_MTorST(const ZSTD_CCtx* cctx) { #ifdef ZSTD_MULTITHREAD if (cctx->appliedParams.nbWorkers >= 1) { assert(cctx->mtctx != NULL); return ZSTDMT_nextInputSizeHint(cctx->mtctx); } #endif return ZSTD_nextInputSizeHint(cctx); } size_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input) { FORWARD_IF_ERROR( ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue) , ""); return ZSTD_nextInputSizeHint_MTorST(zcs); } /* After a compression call set the expected input/output buffer. * This is validated at the start of the next compression call. */ static void ZSTD_setBufferExpectations(ZSTD_CCtx* cctx, const ZSTD_outBuffer* output, const ZSTD_inBuffer* input) { DEBUGLOG(5, "ZSTD_setBufferExpectations (for advanced stable in/out modes)"); if (cctx->appliedParams.inBufferMode == ZSTD_bm_stable) { cctx->expectedInBuffer = *input; } if (cctx->appliedParams.outBufferMode == ZSTD_bm_stable) { cctx->expectedOutBufferSize = output->size - output->pos; } } /* Validate that the input/output buffers match the expectations set by * ZSTD_setBufferExpectations. */ static size_t ZSTD_checkBufferStability(ZSTD_CCtx const* cctx, ZSTD_outBuffer const* output, ZSTD_inBuffer const* input, ZSTD_EndDirective endOp) { if (cctx->appliedParams.inBufferMode == ZSTD_bm_stable) { ZSTD_inBuffer const expect = cctx->expectedInBuffer; if (expect.src != input->src || expect.pos != input->pos) RETURN_ERROR(stabilityCondition_notRespected, "ZSTD_c_stableInBuffer enabled but input differs!"); } (void)endOp; if (cctx->appliedParams.outBufferMode == ZSTD_bm_stable) { size_t const outBufferSize = output->size - output->pos; if (cctx->expectedOutBufferSize != outBufferSize) RETURN_ERROR(stabilityCondition_notRespected, "ZSTD_c_stableOutBuffer enabled but output size differs!"); } return 0; } static size_t ZSTD_CCtx_init_compressStream2(ZSTD_CCtx* cctx, ZSTD_EndDirective endOp, size_t inSize) { ZSTD_CCtx_params params = cctx->requestedParams; ZSTD_prefixDict const prefixDict = cctx->prefixDict; FORWARD_IF_ERROR( ZSTD_initLocalDict(cctx) , ""); /* Init the local dict if present. */ ZSTD_memset(&cctx->prefixDict, 0, sizeof(cctx->prefixDict)); /* single usage */ assert(prefixDict.dict==NULL || cctx->cdict==NULL); /* only one can be set */ if (cctx->cdict && !cctx->localDict.cdict) { /* Let the cdict's compression level take priority over the requested params. * But do not take the cdict's compression level if the "cdict" is actually a localDict * generated from ZSTD_initLocalDict(). */ params.compressionLevel = cctx->cdict->compressionLevel; } DEBUGLOG(4, "ZSTD_compressStream2 : transparent init stage"); if (endOp == ZSTD_e_end) cctx->pledgedSrcSizePlusOne = inSize + 1; /* auto-determine pledgedSrcSize */ { size_t const dictSize = prefixDict.dict ? prefixDict.dictSize : (cctx->cdict ? cctx->cdict->dictContentSize : 0); ZSTD_cParamMode_e const mode = ZSTD_getCParamMode(cctx->cdict, ¶ms, cctx->pledgedSrcSizePlusOne - 1); params.cParams = ZSTD_getCParamsFromCCtxParams( ¶ms, cctx->pledgedSrcSizePlusOne-1, dictSize, mode); } params.useBlockSplitter = ZSTD_resolveBlockSplitterMode(params.useBlockSplitter, ¶ms.cParams); params.ldmParams.enableLdm = ZSTD_resolveEnableLdm(params.ldmParams.enableLdm, ¶ms.cParams); params.useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(params.useRowMatchFinder, ¶ms.cParams); params.validateSequences = ZSTD_resolveExternalSequenceValidation(params.validateSequences); params.maxBlockSize = ZSTD_resolveMaxBlockSize(params.maxBlockSize); params.searchForExternalRepcodes = ZSTD_resolveExternalRepcodeSearch(params.searchForExternalRepcodes, params.compressionLevel); #ifdef ZSTD_MULTITHREAD /* If external matchfinder is enabled, make sure to fail before checking job size (for consistency) */ RETURN_ERROR_IF( params.useSequenceProducer == 1 && params.nbWorkers >= 1, parameter_combination_unsupported, "External sequence producer isn't supported with nbWorkers >= 1" ); if ((cctx->pledgedSrcSizePlusOne-1) <= ZSTDMT_JOBSIZE_MIN) { params.nbWorkers = 0; /* do not invoke multi-threading when src size is too small */ } if (params.nbWorkers > 0) { #if ZSTD_TRACE cctx->traceCtx = (ZSTD_trace_compress_begin != NULL) ? ZSTD_trace_compress_begin(cctx) : 0; #endif /* mt context creation */ if (cctx->mtctx == NULL) { DEBUGLOG(4, "ZSTD_compressStream2: creating new mtctx for nbWorkers=%u", params.nbWorkers); cctx->mtctx = ZSTDMT_createCCtx_advanced((U32)params.nbWorkers, cctx->customMem, cctx->pool); RETURN_ERROR_IF(cctx->mtctx == NULL, memory_allocation, "NULL pointer!"); } /* mt compression */ DEBUGLOG(4, "call ZSTDMT_initCStream_internal as nbWorkers=%u", params.nbWorkers); FORWARD_IF_ERROR( ZSTDMT_initCStream_internal( cctx->mtctx, prefixDict.dict, prefixDict.dictSize, prefixDict.dictContentType, cctx->cdict, params, cctx->pledgedSrcSizePlusOne-1) , ""); cctx->dictID = cctx->cdict ? cctx->cdict->dictID : 0; cctx->dictContentSize = cctx->cdict ? cctx->cdict->dictContentSize : prefixDict.dictSize; cctx->consumedSrcSize = 0; cctx->producedCSize = 0; cctx->streamStage = zcss_load; cctx->appliedParams = params; } else #endif /* ZSTD_MULTITHREAD */ { U64 const pledgedSrcSize = cctx->pledgedSrcSizePlusOne - 1; assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams))); FORWARD_IF_ERROR( ZSTD_compressBegin_internal(cctx, prefixDict.dict, prefixDict.dictSize, prefixDict.dictContentType, ZSTD_dtlm_fast, cctx->cdict, ¶ms, pledgedSrcSize, ZSTDb_buffered) , ""); assert(cctx->appliedParams.nbWorkers == 0); cctx->inToCompress = 0; cctx->inBuffPos = 0; if (cctx->appliedParams.inBufferMode == ZSTD_bm_buffered) { /* for small input: avoid automatic flush on reaching end of block, since * it would require to add a 3-bytes null block to end frame */ cctx->inBuffTarget = cctx->blockSize + (cctx->blockSize == pledgedSrcSize); } else { cctx->inBuffTarget = 0; } cctx->outBuffContentSize = cctx->outBuffFlushedSize = 0; cctx->streamStage = zcss_load; cctx->frameEnded = 0; } return 0; } /* @return provides a minimum amount of data remaining to be flushed from internal buffers */ size_t ZSTD_compressStream2( ZSTD_CCtx* cctx, ZSTD_outBuffer* output, ZSTD_inBuffer* input, ZSTD_EndDirective endOp) { DEBUGLOG(5, "ZSTD_compressStream2, endOp=%u ", (unsigned)endOp); /* check conditions */ RETURN_ERROR_IF(output->pos > output->size, dstSize_tooSmall, "invalid output buffer"); RETURN_ERROR_IF(input->pos > input->size, srcSize_wrong, "invalid input buffer"); RETURN_ERROR_IF((U32)endOp > (U32)ZSTD_e_end, parameter_outOfBound, "invalid endDirective"); assert(cctx != NULL); /* transparent initialization stage */ if (cctx->streamStage == zcss_init) { size_t const inputSize = input->size - input->pos; /* no obligation to start from pos==0 */ size_t const totalInputSize = inputSize + cctx->stableIn_notConsumed; if ( (cctx->requestedParams.inBufferMode == ZSTD_bm_stable) /* input is presumed stable, across invocations */ && (endOp == ZSTD_e_continue) /* no flush requested, more input to come */ && (totalInputSize < ZSTD_BLOCKSIZE_MAX) ) { /* not even reached one block yet */ if (cctx->stableIn_notConsumed) { /* not the first time */ /* check stable source guarantees */ RETURN_ERROR_IF(input->src != cctx->expectedInBuffer.src, stabilityCondition_notRespected, "stableInBuffer condition not respected: wrong src pointer"); RETURN_ERROR_IF(input->pos != cctx->expectedInBuffer.size, stabilityCondition_notRespected, "stableInBuffer condition not respected: externally modified pos"); } /* pretend input was consumed, to give a sense forward progress */ input->pos = input->size; /* save stable inBuffer, for later control, and flush/end */ cctx->expectedInBuffer = *input; /* but actually input wasn't consumed, so keep track of position from where compression shall resume */ cctx->stableIn_notConsumed += inputSize; /* don't initialize yet, wait for the first block of flush() order, for better parameters adaptation */ return ZSTD_FRAMEHEADERSIZE_MIN(cctx->requestedParams.format); /* at least some header to produce */ } FORWARD_IF_ERROR(ZSTD_CCtx_init_compressStream2(cctx, endOp, totalInputSize), "compressStream2 initialization failed"); ZSTD_setBufferExpectations(cctx, output, input); /* Set initial buffer expectations now that we've initialized */ } /* end of transparent initialization stage */ FORWARD_IF_ERROR(ZSTD_checkBufferStability(cctx, output, input, endOp), "invalid buffers"); /* compression stage */ #ifdef ZSTD_MULTITHREAD if (cctx->appliedParams.nbWorkers > 0) { size_t flushMin; if (cctx->cParamsChanged) { ZSTDMT_updateCParams_whileCompressing(cctx->mtctx, &cctx->requestedParams); cctx->cParamsChanged = 0; } if (cctx->stableIn_notConsumed) { assert(cctx->appliedParams.inBufferMode == ZSTD_bm_stable); /* some early data was skipped - make it available for consumption */ assert(input->pos >= cctx->stableIn_notConsumed); input->pos -= cctx->stableIn_notConsumed; cctx->stableIn_notConsumed = 0; } for (;;) { size_t const ipos = input->pos; size_t const opos = output->pos; flushMin = ZSTDMT_compressStream_generic(cctx->mtctx, output, input, endOp); cctx->consumedSrcSize += (U64)(input->pos - ipos); cctx->producedCSize += (U64)(output->pos - opos); if ( ZSTD_isError(flushMin) || (endOp == ZSTD_e_end && flushMin == 0) ) { /* compression completed */ if (flushMin == 0) ZSTD_CCtx_trace(cctx, 0); ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only); } FORWARD_IF_ERROR(flushMin, "ZSTDMT_compressStream_generic failed"); if (endOp == ZSTD_e_continue) { /* We only require some progress with ZSTD_e_continue, not maximal progress. * We're done if we've consumed or produced any bytes, or either buffer is * full. */ if (input->pos != ipos || output->pos != opos || input->pos == input->size || output->pos == output->size) break; } else { assert(endOp == ZSTD_e_flush || endOp == ZSTD_e_end); /* We require maximal progress. We're done when the flush is complete or the * output buffer is full. */ if (flushMin == 0 || output->pos == output->size) break; } } DEBUGLOG(5, "completed ZSTD_compressStream2 delegating to ZSTDMT_compressStream_generic"); /* Either we don't require maximum forward progress, we've finished the * flush, or we are out of output space. */ assert(endOp == ZSTD_e_continue || flushMin == 0 || output->pos == output->size); ZSTD_setBufferExpectations(cctx, output, input); return flushMin; } #endif /* ZSTD_MULTITHREAD */ FORWARD_IF_ERROR( ZSTD_compressStream_generic(cctx, output, input, endOp) , ""); DEBUGLOG(5, "completed ZSTD_compressStream2"); ZSTD_setBufferExpectations(cctx, output, input); return cctx->outBuffContentSize - cctx->outBuffFlushedSize; /* remaining to flush */ } size_t ZSTD_compressStream2_simpleArgs ( ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, size_t* dstPos, const void* src, size_t srcSize, size_t* srcPos, ZSTD_EndDirective endOp) { ZSTD_outBuffer output; ZSTD_inBuffer input; output.dst = dst; output.size = dstCapacity; output.pos = *dstPos; input.src = src; input.size = srcSize; input.pos = *srcPos; /* ZSTD_compressStream2() will check validity of dstPos and srcPos */ { size_t const cErr = ZSTD_compressStream2(cctx, &output, &input, endOp); *dstPos = output.pos; *srcPos = input.pos; return cErr; } } size_t ZSTD_compress2(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize) { ZSTD_bufferMode_e const originalInBufferMode = cctx->requestedParams.inBufferMode; ZSTD_bufferMode_e const originalOutBufferMode = cctx->requestedParams.outBufferMode; DEBUGLOG(4, "ZSTD_compress2 (srcSize=%u)", (unsigned)srcSize); ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only); /* Enable stable input/output buffers. */ cctx->requestedParams.inBufferMode = ZSTD_bm_stable; cctx->requestedParams.outBufferMode = ZSTD_bm_stable; { size_t oPos = 0; size_t iPos = 0; size_t const result = ZSTD_compressStream2_simpleArgs(cctx, dst, dstCapacity, &oPos, src, srcSize, &iPos, ZSTD_e_end); /* Reset to the original values. */ cctx->requestedParams.inBufferMode = originalInBufferMode; cctx->requestedParams.outBufferMode = originalOutBufferMode; FORWARD_IF_ERROR(result, "ZSTD_compressStream2_simpleArgs failed"); if (result != 0) { /* compression not completed, due to lack of output space */ assert(oPos == dstCapacity); RETURN_ERROR(dstSize_tooSmall, ""); } assert(iPos == srcSize); /* all input is expected consumed */ return oPos; } } /* ZSTD_validateSequence() : * @offCode : is presumed to follow format required by ZSTD_storeSeq() * @returns a ZSTD error code if sequence is not valid */ static size_t ZSTD_validateSequence(U32 offCode, U32 matchLength, U32 minMatch, size_t posInSrc, U32 windowLog, size_t dictSize, int useSequenceProducer) { U32 const windowSize = 1u << windowLog; /* posInSrc represents the amount of data the decoder would decode up to this point. * As long as the amount of data decoded is less than or equal to window size, offsets may be * larger than the total length of output decoded in order to reference the dict, even larger than * window size. After output surpasses windowSize, we're limited to windowSize offsets again. */ size_t const offsetBound = posInSrc > windowSize ? (size_t)windowSize : posInSrc + (size_t)dictSize; size_t const matchLenLowerBound = (minMatch == 3 || useSequenceProducer) ? 3 : 4; RETURN_ERROR_IF(offCode > OFFSET_TO_OFFBASE(offsetBound), externalSequences_invalid, "Offset too large!"); /* Validate maxNbSeq is large enough for the given matchLength and minMatch */ RETURN_ERROR_IF(matchLength < matchLenLowerBound, externalSequences_invalid, "Matchlength too small for the minMatch"); return 0; } /* Returns an offset code, given a sequence's raw offset, the ongoing repcode array, and whether litLength == 0 */ static U32 ZSTD_finalizeOffBase(U32 rawOffset, const U32 rep[ZSTD_REP_NUM], U32 ll0) { U32 offBase = OFFSET_TO_OFFBASE(rawOffset); if (!ll0 && rawOffset == rep[0]) { offBase = REPCODE1_TO_OFFBASE; } else if (rawOffset == rep[1]) { offBase = REPCODE_TO_OFFBASE(2 - ll0); } else if (rawOffset == rep[2]) { offBase = REPCODE_TO_OFFBASE(3 - ll0); } else if (ll0 && rawOffset == rep[0] - 1) { offBase = REPCODE3_TO_OFFBASE; } return offBase; } size_t ZSTD_copySequencesToSeqStoreExplicitBlockDelim(ZSTD_CCtx* cctx, ZSTD_sequencePosition* seqPos, const ZSTD_Sequence* const inSeqs, size_t inSeqsSize, const void* src, size_t blockSize, ZSTD_paramSwitch_e externalRepSearch) { U32 idx = seqPos->idx; U32 const startIdx = idx; BYTE const* ip = (BYTE const*)(src); const BYTE* const iend = ip + blockSize; repcodes_t updatedRepcodes; U32 dictSize; DEBUGLOG(5, "ZSTD_copySequencesToSeqStoreExplicitBlockDelim (blockSize = %zu)", blockSize); if (cctx->cdict) { dictSize = (U32)cctx->cdict->dictContentSize; } else if (cctx->prefixDict.dict) { dictSize = (U32)cctx->prefixDict.dictSize; } else { dictSize = 0; } ZSTD_memcpy(updatedRepcodes.rep, cctx->blockState.prevCBlock->rep, sizeof(repcodes_t)); for (; idx < inSeqsSize && (inSeqs[idx].matchLength != 0 || inSeqs[idx].offset != 0); ++idx) { U32 const litLength = inSeqs[idx].litLength; U32 const matchLength = inSeqs[idx].matchLength; U32 offBase; if (externalRepSearch == ZSTD_ps_disable) { offBase = OFFSET_TO_OFFBASE(inSeqs[idx].offset); } else { U32 const ll0 = (litLength == 0); offBase = ZSTD_finalizeOffBase(inSeqs[idx].offset, updatedRepcodes.rep, ll0); ZSTD_updateRep(updatedRepcodes.rep, offBase, ll0); } DEBUGLOG(6, "Storing sequence: (of: %u, ml: %u, ll: %u)", offBase, matchLength, litLength); if (cctx->appliedParams.validateSequences) { seqPos->posInSrc += litLength + matchLength; FORWARD_IF_ERROR(ZSTD_validateSequence(offBase, matchLength, cctx->appliedParams.cParams.minMatch, seqPos->posInSrc, cctx->appliedParams.cParams.windowLog, dictSize, cctx->appliedParams.useSequenceProducer), "Sequence validation failed"); } RETURN_ERROR_IF(idx - seqPos->idx >= cctx->seqStore.maxNbSeq, externalSequences_invalid, "Not enough memory allocated. Try adjusting ZSTD_c_minMatch."); ZSTD_storeSeq(&cctx->seqStore, litLength, ip, iend, offBase, matchLength); ip += matchLength + litLength; } /* If we skipped repcode search while parsing, we need to update repcodes now */ assert(externalRepSearch != ZSTD_ps_auto); assert(idx >= startIdx); if (externalRepSearch == ZSTD_ps_disable && idx != startIdx) { U32* const rep = updatedRepcodes.rep; U32 lastSeqIdx = idx - 1; /* index of last non-block-delimiter sequence */ if (lastSeqIdx >= startIdx + 2) { rep[2] = inSeqs[lastSeqIdx - 2].offset; rep[1] = inSeqs[lastSeqIdx - 1].offset; rep[0] = inSeqs[lastSeqIdx].offset; } else if (lastSeqIdx == startIdx + 1) { rep[2] = rep[0]; rep[1] = inSeqs[lastSeqIdx - 1].offset; rep[0] = inSeqs[lastSeqIdx].offset; } else { assert(lastSeqIdx == startIdx); rep[2] = rep[1]; rep[1] = rep[0]; rep[0] = inSeqs[lastSeqIdx].offset; } } ZSTD_memcpy(cctx->blockState.nextCBlock->rep, updatedRepcodes.rep, sizeof(repcodes_t)); if (inSeqs[idx].litLength) { DEBUGLOG(6, "Storing last literals of size: %u", inSeqs[idx].litLength); ZSTD_storeLastLiterals(&cctx->seqStore, ip, inSeqs[idx].litLength); ip += inSeqs[idx].litLength; seqPos->posInSrc += inSeqs[idx].litLength; } RETURN_ERROR_IF(ip != iend, externalSequences_invalid, "Blocksize doesn't agree with block delimiter!"); seqPos->idx = idx+1; return 0; } size_t ZSTD_copySequencesToSeqStoreNoBlockDelim(ZSTD_CCtx* cctx, ZSTD_sequencePosition* seqPos, const ZSTD_Sequence* const inSeqs, size_t inSeqsSize, const void* src, size_t blockSize, ZSTD_paramSwitch_e externalRepSearch) { U32 idx = seqPos->idx; U32 startPosInSequence = seqPos->posInSequence; U32 endPosInSequence = seqPos->posInSequence + (U32)blockSize; size_t dictSize; BYTE const* ip = (BYTE const*)(src); BYTE const* iend = ip + blockSize; /* May be adjusted if we decide to process fewer than blockSize bytes */ repcodes_t updatedRepcodes; U32 bytesAdjustment = 0; U32 finalMatchSplit = 0; /* TODO(embg) support fast parsing mode in noBlockDelim mode */ (void)externalRepSearch; if (cctx->cdict) { dictSize = cctx->cdict->dictContentSize; } else if (cctx->prefixDict.dict) { dictSize = cctx->prefixDict.dictSize; } else { dictSize = 0; } DEBUGLOG(5, "ZSTD_copySequencesToSeqStoreNoBlockDelim: idx: %u PIS: %u blockSize: %zu", idx, startPosInSequence, blockSize); DEBUGLOG(5, "Start seq: idx: %u (of: %u ml: %u ll: %u)", idx, inSeqs[idx].offset, inSeqs[idx].matchLength, inSeqs[idx].litLength); ZSTD_memcpy(updatedRepcodes.rep, cctx->blockState.prevCBlock->rep, sizeof(repcodes_t)); while (endPosInSequence && idx < inSeqsSize && !finalMatchSplit) { const ZSTD_Sequence currSeq = inSeqs[idx]; U32 litLength = currSeq.litLength; U32 matchLength = currSeq.matchLength; U32 const rawOffset = currSeq.offset; U32 offBase; /* Modify the sequence depending on where endPosInSequence lies */ if (endPosInSequence >= currSeq.litLength + currSeq.matchLength) { if (startPosInSequence >= litLength) { startPosInSequence -= litLength; litLength = 0; matchLength -= startPosInSequence; } else { litLength -= startPosInSequence; } /* Move to the next sequence */ endPosInSequence -= currSeq.litLength + currSeq.matchLength; startPosInSequence = 0; } else { /* This is the final (partial) sequence we're adding from inSeqs, and endPosInSequence does not reach the end of the match. So, we have to split the sequence */ DEBUGLOG(6, "Require a split: diff: %u, idx: %u PIS: %u", currSeq.litLength + currSeq.matchLength - endPosInSequence, idx, endPosInSequence); if (endPosInSequence > litLength) { U32 firstHalfMatchLength; litLength = startPosInSequence >= litLength ? 0 : litLength - startPosInSequence; firstHalfMatchLength = endPosInSequence - startPosInSequence - litLength; if (matchLength > blockSize && firstHalfMatchLength >= cctx->appliedParams.cParams.minMatch) { /* Only ever split the match if it is larger than the block size */ U32 secondHalfMatchLength = currSeq.matchLength + currSeq.litLength - endPosInSequence; if (secondHalfMatchLength < cctx->appliedParams.cParams.minMatch) { /* Move the endPosInSequence backward so that it creates match of minMatch length */ endPosInSequence -= cctx->appliedParams.cParams.minMatch - secondHalfMatchLength; bytesAdjustment = cctx->appliedParams.cParams.minMatch - secondHalfMatchLength; firstHalfMatchLength -= bytesAdjustment; } matchLength = firstHalfMatchLength; /* Flag that we split the last match - after storing the sequence, exit the loop, but keep the value of endPosInSequence */ finalMatchSplit = 1; } else { /* Move the position in sequence backwards so that we don't split match, and break to store * the last literals. We use the original currSeq.litLength as a marker for where endPosInSequence * should go. We prefer to do this whenever it is not necessary to split the match, or if doing so * would cause the first half of the match to be too small */ bytesAdjustment = endPosInSequence - currSeq.litLength; endPosInSequence = currSeq.litLength; break; } } else { /* This sequence ends inside the literals, break to store the last literals */ break; } } /* Check if this offset can be represented with a repcode */ { U32 const ll0 = (litLength == 0); offBase = ZSTD_finalizeOffBase(rawOffset, updatedRepcodes.rep, ll0); ZSTD_updateRep(updatedRepcodes.rep, offBase, ll0); } if (cctx->appliedParams.validateSequences) { seqPos->posInSrc += litLength + matchLength; FORWARD_IF_ERROR(ZSTD_validateSequence(offBase, matchLength, cctx->appliedParams.cParams.minMatch, seqPos->posInSrc, cctx->appliedParams.cParams.windowLog, dictSize, cctx->appliedParams.useSequenceProducer), "Sequence validation failed"); } DEBUGLOG(6, "Storing sequence: (of: %u, ml: %u, ll: %u)", offBase, matchLength, litLength); RETURN_ERROR_IF(idx - seqPos->idx >= cctx->seqStore.maxNbSeq, externalSequences_invalid, "Not enough memory allocated. Try adjusting ZSTD_c_minMatch."); ZSTD_storeSeq(&cctx->seqStore, litLength, ip, iend, offBase, matchLength); ip += matchLength + litLength; if (!finalMatchSplit) idx++; /* Next Sequence */ } DEBUGLOG(5, "Ending seq: idx: %u (of: %u ml: %u ll: %u)", idx, inSeqs[idx].offset, inSeqs[idx].matchLength, inSeqs[idx].litLength); assert(idx == inSeqsSize || endPosInSequence <= inSeqs[idx].litLength + inSeqs[idx].matchLength); seqPos->idx = idx; seqPos->posInSequence = endPosInSequence; ZSTD_memcpy(cctx->blockState.nextCBlock->rep, updatedRepcodes.rep, sizeof(repcodes_t)); iend -= bytesAdjustment; if (ip != iend) { /* Store any last literals */ U32 lastLLSize = (U32)(iend - ip); assert(ip <= iend); DEBUGLOG(6, "Storing last literals of size: %u", lastLLSize); ZSTD_storeLastLiterals(&cctx->seqStore, ip, lastLLSize); seqPos->posInSrc += lastLLSize; } return bytesAdjustment; } typedef size_t (*ZSTD_sequenceCopier) (ZSTD_CCtx* cctx, ZSTD_sequencePosition* seqPos, const ZSTD_Sequence* const inSeqs, size_t inSeqsSize, const void* src, size_t blockSize, ZSTD_paramSwitch_e externalRepSearch); static ZSTD_sequenceCopier ZSTD_selectSequenceCopier(ZSTD_sequenceFormat_e mode) { ZSTD_sequenceCopier sequenceCopier = NULL; assert(ZSTD_cParam_withinBounds(ZSTD_c_blockDelimiters, mode)); if (mode == ZSTD_sf_explicitBlockDelimiters) { return ZSTD_copySequencesToSeqStoreExplicitBlockDelim; } else if (mode == ZSTD_sf_noBlockDelimiters) { return ZSTD_copySequencesToSeqStoreNoBlockDelim; } assert(sequenceCopier != NULL); return sequenceCopier; } /* Discover the size of next block by searching for the delimiter. * Note that a block delimiter **must** exist in this mode, * otherwise it's an input error. * The block size retrieved will be later compared to ensure it remains within bounds */ static size_t blockSize_explicitDelimiter(const ZSTD_Sequence* inSeqs, size_t inSeqsSize, ZSTD_sequencePosition seqPos) { int end = 0; size_t blockSize = 0; size_t spos = seqPos.idx; DEBUGLOG(6, "blockSize_explicitDelimiter : seq %zu / %zu", spos, inSeqsSize); assert(spos <= inSeqsSize); while (spos < inSeqsSize) { end = (inSeqs[spos].offset == 0); blockSize += inSeqs[spos].litLength + inSeqs[spos].matchLength; if (end) { if (inSeqs[spos].matchLength != 0) RETURN_ERROR(externalSequences_invalid, "delimiter format error : both matchlength and offset must be == 0"); break; } spos++; } if (!end) RETURN_ERROR(externalSequences_invalid, "Reached end of sequences without finding a block delimiter"); return blockSize; } /* More a "target" block size */ static size_t blockSize_noDelimiter(size_t blockSize, size_t remaining) { int const lastBlock = (remaining <= blockSize); return lastBlock ? remaining : blockSize; } static size_t determine_blockSize(ZSTD_sequenceFormat_e mode, size_t blockSize, size_t remaining, const ZSTD_Sequence* inSeqs, size_t inSeqsSize, ZSTD_sequencePosition seqPos) { DEBUGLOG(6, "determine_blockSize : remainingSize = %zu", remaining); if (mode == ZSTD_sf_noBlockDelimiters) return blockSize_noDelimiter(blockSize, remaining); { size_t const explicitBlockSize = blockSize_explicitDelimiter(inSeqs, inSeqsSize, seqPos); FORWARD_IF_ERROR(explicitBlockSize, "Error while determining block size with explicit delimiters"); if (explicitBlockSize > blockSize) RETURN_ERROR(externalSequences_invalid, "sequences incorrectly define a too large block"); if (explicitBlockSize > remaining) RETURN_ERROR(externalSequences_invalid, "sequences define a frame longer than source"); return explicitBlockSize; } } /* Compress, block-by-block, all of the sequences given. * * Returns the cumulative size of all compressed blocks (including their headers), * otherwise a ZSTD error. */ static size_t ZSTD_compressSequences_internal(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const ZSTD_Sequence* inSeqs, size_t inSeqsSize, const void* src, size_t srcSize) { size_t cSize = 0; size_t remaining = srcSize; ZSTD_sequencePosition seqPos = {0, 0, 0}; BYTE const* ip = (BYTE const*)src; BYTE* op = (BYTE*)dst; ZSTD_sequenceCopier const sequenceCopier = ZSTD_selectSequenceCopier(cctx->appliedParams.blockDelimiters); DEBUGLOG(4, "ZSTD_compressSequences_internal srcSize: %zu, inSeqsSize: %zu", srcSize, inSeqsSize); /* Special case: empty frame */ if (remaining == 0) { U32 const cBlockHeader24 = 1 /* last block */ + (((U32)bt_raw)<<1); RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall, "No room for empty frame block header"); MEM_writeLE32(op, cBlockHeader24); op += ZSTD_blockHeaderSize; dstCapacity -= ZSTD_blockHeaderSize; cSize += ZSTD_blockHeaderSize; } while (remaining) { size_t compressedSeqsSize; size_t cBlockSize; size_t additionalByteAdjustment; size_t blockSize = determine_blockSize(cctx->appliedParams.blockDelimiters, cctx->blockSize, remaining, inSeqs, inSeqsSize, seqPos); U32 const lastBlock = (blockSize == remaining); FORWARD_IF_ERROR(blockSize, "Error while trying to determine block size"); assert(blockSize <= remaining); ZSTD_resetSeqStore(&cctx->seqStore); DEBUGLOG(5, "Working on new block. Blocksize: %zu (total:%zu)", blockSize, (ip - (const BYTE*)src) + blockSize); additionalByteAdjustment = sequenceCopier(cctx, &seqPos, inSeqs, inSeqsSize, ip, blockSize, cctx->appliedParams.searchForExternalRepcodes); FORWARD_IF_ERROR(additionalByteAdjustment, "Bad sequence copy"); blockSize -= additionalByteAdjustment; /* If blocks are too small, emit as a nocompress block */ /* TODO: See 3090. We reduced MIN_CBLOCK_SIZE from 3 to 2 so to compensate we are adding * additional 1. We need to revisit and change this logic to be more consistent */ if (blockSize < MIN_CBLOCK_SIZE+ZSTD_blockHeaderSize+1+1) { cBlockSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock); FORWARD_IF_ERROR(cBlockSize, "Nocompress block failed"); DEBUGLOG(5, "Block too small, writing out nocompress block: cSize: %zu", cBlockSize); cSize += cBlockSize; ip += blockSize; op += cBlockSize; remaining -= blockSize; dstCapacity -= cBlockSize; continue; } RETURN_ERROR_IF(dstCapacity < ZSTD_blockHeaderSize, dstSize_tooSmall, "not enough dstCapacity to write a new compressed block"); compressedSeqsSize = ZSTD_entropyCompressSeqStore(&cctx->seqStore, &cctx->blockState.prevCBlock->entropy, &cctx->blockState.nextCBlock->entropy, &cctx->appliedParams, op + ZSTD_blockHeaderSize /* Leave space for block header */, dstCapacity - ZSTD_blockHeaderSize, blockSize, cctx->entropyWorkspace, ENTROPY_WORKSPACE_SIZE /* statically allocated in resetCCtx */, cctx->bmi2); FORWARD_IF_ERROR(compressedSeqsSize, "Compressing sequences of block failed"); DEBUGLOG(5, "Compressed sequences size: %zu", compressedSeqsSize); if (!cctx->isFirstBlock && ZSTD_maybeRLE(&cctx->seqStore) && ZSTD_isRLE(ip, blockSize)) { /* We don't want to emit our first block as a RLE even if it qualifies because * doing so will cause the decoder (cli only) to throw a "should consume all input error." * This is only an issue for zstd <= v1.4.3 */ compressedSeqsSize = 1; } if (compressedSeqsSize == 0) { /* ZSTD_noCompressBlock writes the block header as well */ cBlockSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock); FORWARD_IF_ERROR(cBlockSize, "ZSTD_noCompressBlock failed"); DEBUGLOG(5, "Writing out nocompress block, size: %zu", cBlockSize); } else if (compressedSeqsSize == 1) { cBlockSize = ZSTD_rleCompressBlock(op, dstCapacity, *ip, blockSize, lastBlock); FORWARD_IF_ERROR(cBlockSize, "ZSTD_rleCompressBlock failed"); DEBUGLOG(5, "Writing out RLE block, size: %zu", cBlockSize); } else { U32 cBlockHeader; /* Error checking and repcodes update */ ZSTD_blockState_confirmRepcodesAndEntropyTables(&cctx->blockState); if (cctx->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid) cctx->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check; /* Write block header into beginning of block*/ cBlockHeader = lastBlock + (((U32)bt_compressed)<<1) + (U32)(compressedSeqsSize << 3); MEM_writeLE24(op, cBlockHeader); cBlockSize = ZSTD_blockHeaderSize + compressedSeqsSize; DEBUGLOG(5, "Writing out compressed block, size: %zu", cBlockSize); } cSize += cBlockSize; if (lastBlock) { break; } else { ip += blockSize; op += cBlockSize; remaining -= blockSize; dstCapacity -= cBlockSize; cctx->isFirstBlock = 0; } DEBUGLOG(5, "cSize running total: %zu (remaining dstCapacity=%zu)", cSize, dstCapacity); } DEBUGLOG(4, "cSize final total: %zu", cSize); return cSize; } size_t ZSTD_compressSequences(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const ZSTD_Sequence* inSeqs, size_t inSeqsSize, const void* src, size_t srcSize) { BYTE* op = (BYTE*)dst; size_t cSize = 0; size_t compressedBlocksSize = 0; size_t frameHeaderSize = 0; /* Transparent initialization stage, same as compressStream2() */ DEBUGLOG(4, "ZSTD_compressSequences (dstCapacity=%zu)", dstCapacity); assert(cctx != NULL); FORWARD_IF_ERROR(ZSTD_CCtx_init_compressStream2(cctx, ZSTD_e_end, srcSize), "CCtx initialization failed"); /* Begin writing output, starting with frame header */ frameHeaderSize = ZSTD_writeFrameHeader(op, dstCapacity, &cctx->appliedParams, srcSize, cctx->dictID); op += frameHeaderSize; dstCapacity -= frameHeaderSize; cSize += frameHeaderSize; if (cctx->appliedParams.fParams.checksumFlag && srcSize) { XXH64_update(&cctx->xxhState, src, srcSize); } /* cSize includes block header size and compressed sequences size */ compressedBlocksSize = ZSTD_compressSequences_internal(cctx, op, dstCapacity, inSeqs, inSeqsSize, src, srcSize); FORWARD_IF_ERROR(compressedBlocksSize, "Compressing blocks failed!"); cSize += compressedBlocksSize; dstCapacity -= compressedBlocksSize; if (cctx->appliedParams.fParams.checksumFlag) { U32 const checksum = (U32) XXH64_digest(&cctx->xxhState); RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall, "no room for checksum"); DEBUGLOG(4, "Write checksum : %08X", (unsigned)checksum); MEM_writeLE32((char*)dst + cSize, checksum); cSize += 4; } DEBUGLOG(4, "Final compressed size: %zu", cSize); return cSize; } /*====== Finalize ======*/ static ZSTD_inBuffer inBuffer_forEndFlush(const ZSTD_CStream* zcs) { const ZSTD_inBuffer nullInput = { NULL, 0, 0 }; const int stableInput = (zcs->appliedParams.inBufferMode == ZSTD_bm_stable); return stableInput ? zcs->expectedInBuffer : nullInput; } /*! ZSTD_flushStream() : * @return : amount of data remaining to flush */ size_t ZSTD_flushStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output) { ZSTD_inBuffer input = inBuffer_forEndFlush(zcs); input.size = input.pos; /* do not ingest more input during flush */ return ZSTD_compressStream2(zcs, output, &input, ZSTD_e_flush); } size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output) { ZSTD_inBuffer input = inBuffer_forEndFlush(zcs); size_t const remainingToFlush = ZSTD_compressStream2(zcs, output, &input, ZSTD_e_end); FORWARD_IF_ERROR(remainingToFlush , "ZSTD_compressStream2(,,ZSTD_e_end) failed"); if (zcs->appliedParams.nbWorkers > 0) return remainingToFlush; /* minimal estimation */ /* single thread mode : attempt to calculate remaining to flush more precisely */ { size_t const lastBlockSize = zcs->frameEnded ? 0 : ZSTD_BLOCKHEADERSIZE; size_t const checksumSize = (size_t)(zcs->frameEnded ? 0 : zcs->appliedParams.fParams.checksumFlag * 4); size_t const toFlush = remainingToFlush + lastBlockSize + checksumSize; DEBUGLOG(4, "ZSTD_endStream : remaining to flush : %u", (unsigned)toFlush); return toFlush; } } /*-===== Pre-defined compression levels =====-*/ #include "clevels.h" int ZSTD_maxCLevel(void) { return ZSTD_MAX_CLEVEL; } int ZSTD_minCLevel(void) { return (int)-ZSTD_TARGETLENGTH_MAX; } int ZSTD_defaultCLevel(void) { return ZSTD_CLEVEL_DEFAULT; } static ZSTD_compressionParameters ZSTD_dedicatedDictSearch_getCParams(int const compressionLevel, size_t const dictSize) { ZSTD_compressionParameters cParams = ZSTD_getCParams_internal(compressionLevel, 0, dictSize, ZSTD_cpm_createCDict); switch (cParams.strategy) { case ZSTD_fast: case ZSTD_dfast: break; case ZSTD_greedy: case ZSTD_lazy: case ZSTD_lazy2: cParams.hashLog += ZSTD_LAZY_DDSS_BUCKET_LOG; break; case ZSTD_btlazy2: case ZSTD_btopt: case ZSTD_btultra: case ZSTD_btultra2: break; } return cParams; } static int ZSTD_dedicatedDictSearch_isSupported( ZSTD_compressionParameters const* cParams) { return (cParams->strategy >= ZSTD_greedy) && (cParams->strategy <= ZSTD_lazy2) && (cParams->hashLog > cParams->chainLog) && (cParams->chainLog <= 24); } /** * Reverses the adjustment applied to cparams when enabling dedicated dict * search. This is used to recover the params set to be used in the working * context. (Otherwise, those tables would also grow.) */ static void ZSTD_dedicatedDictSearch_revertCParams( ZSTD_compressionParameters* cParams) { switch (cParams->strategy) { case ZSTD_fast: case ZSTD_dfast: break; case ZSTD_greedy: case ZSTD_lazy: case ZSTD_lazy2: cParams->hashLog -= ZSTD_LAZY_DDSS_BUCKET_LOG; if (cParams->hashLog < ZSTD_HASHLOG_MIN) { cParams->hashLog = ZSTD_HASHLOG_MIN; } break; case ZSTD_btlazy2: case ZSTD_btopt: case ZSTD_btultra: case ZSTD_btultra2: break; } } static U64 ZSTD_getCParamRowSize(U64 srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode) { switch (mode) { case ZSTD_cpm_unknown: case ZSTD_cpm_noAttachDict: case ZSTD_cpm_createCDict: break; case ZSTD_cpm_attachDict: dictSize = 0; break; default: assert(0); break; } { int const unknown = srcSizeHint == ZSTD_CONTENTSIZE_UNKNOWN; size_t const addedSize = unknown && dictSize > 0 ? 500 : 0; return unknown && dictSize == 0 ? ZSTD_CONTENTSIZE_UNKNOWN : srcSizeHint+dictSize+addedSize; } } /*! ZSTD_getCParams_internal() : * @return ZSTD_compressionParameters structure for a selected compression level, srcSize and dictSize. * Note: srcSizeHint 0 means 0, use ZSTD_CONTENTSIZE_UNKNOWN for unknown. * Use dictSize == 0 for unknown or unused. * Note: `mode` controls how we treat the `dictSize`. See docs for `ZSTD_cParamMode_e`. */ static ZSTD_compressionParameters ZSTD_getCParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode) { U64 const rSize = ZSTD_getCParamRowSize(srcSizeHint, dictSize, mode); U32 const tableID = (rSize <= 256 KB) + (rSize <= 128 KB) + (rSize <= 16 KB); int row; DEBUGLOG(5, "ZSTD_getCParams_internal (cLevel=%i)", compressionLevel); /* row */ if (compressionLevel == 0) row = ZSTD_CLEVEL_DEFAULT; /* 0 == default */ else if (compressionLevel < 0) row = 0; /* entry 0 is baseline for fast mode */ else if (compressionLevel > ZSTD_MAX_CLEVEL) row = ZSTD_MAX_CLEVEL; else row = compressionLevel; { ZSTD_compressionParameters cp = ZSTD_defaultCParameters[tableID][row]; DEBUGLOG(5, "ZSTD_getCParams_internal selected tableID: %u row: %u strat: %u", tableID, row, (U32)cp.strategy); /* acceleration factor */ if (compressionLevel < 0) { int const clampedCompressionLevel = MAX(ZSTD_minCLevel(), compressionLevel); cp.targetLength = (unsigned)(-clampedCompressionLevel); } /* refine parameters based on srcSize & dictSize */ return ZSTD_adjustCParams_internal(cp, srcSizeHint, dictSize, mode, ZSTD_ps_auto); } } /*! ZSTD_getCParams() : * @return ZSTD_compressionParameters structure for a selected compression level, srcSize and dictSize. * Size values are optional, provide 0 if not known or unused */ ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize) { if (srcSizeHint == 0) srcSizeHint = ZSTD_CONTENTSIZE_UNKNOWN; return ZSTD_getCParams_internal(compressionLevel, srcSizeHint, dictSize, ZSTD_cpm_unknown); } /*! ZSTD_getParams() : * same idea as ZSTD_getCParams() * @return a `ZSTD_parameters` structure (instead of `ZSTD_compressionParameters`). * Fields of `ZSTD_frameParameters` are set to default values */ static ZSTD_parameters ZSTD_getParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode) { ZSTD_parameters params; ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, srcSizeHint, dictSize, mode); DEBUGLOG(5, "ZSTD_getParams (cLevel=%i)", compressionLevel); ZSTD_memset(¶ms, 0, sizeof(params)); params.cParams = cParams; params.fParams.contentSizeFlag = 1; return params; } /*! ZSTD_getParams() : * same idea as ZSTD_getCParams() * @return a `ZSTD_parameters` structure (instead of `ZSTD_compressionParameters`). * Fields of `ZSTD_frameParameters` are set to default values */ ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize) { if (srcSizeHint == 0) srcSizeHint = ZSTD_CONTENTSIZE_UNKNOWN; return ZSTD_getParams_internal(compressionLevel, srcSizeHint, dictSize, ZSTD_cpm_unknown); } void ZSTD_registerSequenceProducer( ZSTD_CCtx* zc, void* mState, ZSTD_sequenceProducer_F* mFinder ) { if (mFinder != NULL) { ZSTD_externalMatchCtx emctx; emctx.mState = mState; emctx.mFinder = mFinder; emctx.seqBuffer = NULL; emctx.seqBufferCapacity = 0; zc->externalMatchCtx = emctx; zc->requestedParams.useSequenceProducer = 1; } else { ZSTD_memset(&zc->externalMatchCtx, 0, sizeof(zc->externalMatchCtx)); zc->requestedParams.useSequenceProducer = 0; } } zmat-0.9.9/src/blosc2/internal-complibs/zstd/deprecated/0000755000175200007730000000000014515254731023420 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/internal-complibs/zstd/deprecated/zbuff_common.c0000644000175200007730000000176014515254731026254 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /*-************************************* * Dependencies ***************************************/ #include "../common/error_private.h" #include "zbuff.h" /*-**************************************** * ZBUFF Error Management (deprecated) ******************************************/ /*! ZBUFF_isError() : * tells if a return value is an error code */ unsigned ZBUFF_isError(size_t errorCode) { return ERR_isError(errorCode); } /*! ZBUFF_getErrorName() : * provides error code string from function result (useful for debugging) */ const char* ZBUFF_getErrorName(size_t errorCode) { return ERR_getErrorName(errorCode); } zmat-0.9.9/src/blosc2/internal-complibs/zstd/deprecated/zbuff_compress.c0000644000175200007730000001516214515254731026620 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* ************************************* * Dependencies ***************************************/ #define ZBUFF_STATIC_LINKING_ONLY #include "zbuff.h" #include "../common/error_private.h" /*-*********************************************************** * Streaming compression * * A ZBUFF_CCtx object is required to track streaming operation. * Use ZBUFF_createCCtx() and ZBUFF_freeCCtx() to create/release resources. * Use ZBUFF_compressInit() to start a new compression operation. * ZBUFF_CCtx objects can be reused multiple times. * * Use ZBUFF_compressContinue() repetitively to consume your input. * *srcSizePtr and *dstCapacityPtr can be any size. * The function will report how many bytes were read or written by modifying *srcSizePtr and *dstCapacityPtr. * Note that it may not consume the entire input, in which case it's up to the caller to call again the function with remaining input. * The content of dst will be overwritten (up to *dstCapacityPtr) at each function call, so save its content if it matters or change dst . * @return : a hint to preferred nb of bytes to use as input for next function call (it's only a hint, to improve latency) * or an error code, which can be tested using ZBUFF_isError(). * * ZBUFF_compressFlush() can be used to instruct ZBUFF to compress and output whatever remains within its buffer. * Note that it will not output more than *dstCapacityPtr. * Therefore, some content might still be left into its internal buffer if dst buffer is too small. * @return : nb of bytes still present into internal buffer (0 if it's empty) * or an error code, which can be tested using ZBUFF_isError(). * * ZBUFF_compressEnd() instructs to finish a frame. * It will perform a flush and write frame epilogue. * Similar to ZBUFF_compressFlush(), it may not be able to output the entire internal buffer content if *dstCapacityPtr is too small. * @return : nb of bytes still present into internal buffer (0 if it's empty) * or an error code, which can be tested using ZBUFF_isError(). * * Hint : recommended buffer sizes (not compulsory) * input : ZSTD_BLOCKSIZE_MAX (128 KB), internal unit size, it improves latency to use this value. * output : ZSTD_compressBound(ZSTD_BLOCKSIZE_MAX) + ZSTD_blockHeaderSize + ZBUFF_endFrameSize : ensures it's always possible to write/flush/end a full block at best speed. * ***********************************************************/ ZBUFF_CCtx* ZBUFF_createCCtx(void) { return ZSTD_createCStream(); } ZBUFF_CCtx* ZBUFF_createCCtx_advanced(ZSTD_customMem customMem) { return ZSTD_createCStream_advanced(customMem); } size_t ZBUFF_freeCCtx(ZBUFF_CCtx* zbc) { return ZSTD_freeCStream(zbc); } /* ====== Initialization ====== */ size_t ZBUFF_compressInit_advanced(ZBUFF_CCtx* zbc, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize) { if (pledgedSrcSize==0) pledgedSrcSize = ZSTD_CONTENTSIZE_UNKNOWN; /* preserve "0 == unknown" behavior */ FORWARD_IF_ERROR(ZSTD_CCtx_reset(zbc, ZSTD_reset_session_only), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setPledgedSrcSize(zbc, pledgedSrcSize), ""); FORWARD_IF_ERROR(ZSTD_checkCParams(params.cParams), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(zbc, ZSTD_c_windowLog, params.cParams.windowLog), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(zbc, ZSTD_c_hashLog, params.cParams.hashLog), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(zbc, ZSTD_c_chainLog, params.cParams.chainLog), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(zbc, ZSTD_c_searchLog, params.cParams.searchLog), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(zbc, ZSTD_c_minMatch, params.cParams.minMatch), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(zbc, ZSTD_c_targetLength, params.cParams.targetLength), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(zbc, ZSTD_c_strategy, params.cParams.strategy), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(zbc, ZSTD_c_contentSizeFlag, params.fParams.contentSizeFlag), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(zbc, ZSTD_c_checksumFlag, params.fParams.checksumFlag), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(zbc, ZSTD_c_dictIDFlag, params.fParams.noDictIDFlag), ""); FORWARD_IF_ERROR(ZSTD_CCtx_loadDictionary(zbc, dict, dictSize), ""); return 0; } size_t ZBUFF_compressInitDictionary(ZBUFF_CCtx* zbc, const void* dict, size_t dictSize, int compressionLevel) { FORWARD_IF_ERROR(ZSTD_CCtx_reset(zbc, ZSTD_reset_session_only), ""); FORWARD_IF_ERROR(ZSTD_CCtx_setParameter(zbc, ZSTD_c_compressionLevel, compressionLevel), ""); FORWARD_IF_ERROR(ZSTD_CCtx_loadDictionary(zbc, dict, dictSize), ""); return 0; } size_t ZBUFF_compressInit(ZBUFF_CCtx* zbc, int compressionLevel) { return ZSTD_initCStream(zbc, compressionLevel); } /* ====== Compression ====== */ size_t ZBUFF_compressContinue(ZBUFF_CCtx* zbc, void* dst, size_t* dstCapacityPtr, const void* src, size_t* srcSizePtr) { size_t result; ZSTD_outBuffer outBuff; ZSTD_inBuffer inBuff; outBuff.dst = dst; outBuff.pos = 0; outBuff.size = *dstCapacityPtr; inBuff.src = src; inBuff.pos = 0; inBuff.size = *srcSizePtr; result = ZSTD_compressStream(zbc, &outBuff, &inBuff); *dstCapacityPtr = outBuff.pos; *srcSizePtr = inBuff.pos; return result; } /* ====== Finalize ====== */ size_t ZBUFF_compressFlush(ZBUFF_CCtx* zbc, void* dst, size_t* dstCapacityPtr) { size_t result; ZSTD_outBuffer outBuff; outBuff.dst = dst; outBuff.pos = 0; outBuff.size = *dstCapacityPtr; result = ZSTD_flushStream(zbc, &outBuff); *dstCapacityPtr = outBuff.pos; return result; } size_t ZBUFF_compressEnd(ZBUFF_CCtx* zbc, void* dst, size_t* dstCapacityPtr) { size_t result; ZSTD_outBuffer outBuff; outBuff.dst = dst; outBuff.pos = 0; outBuff.size = *dstCapacityPtr; result = ZSTD_endStream(zbc, &outBuff); *dstCapacityPtr = outBuff.pos; return result; } /* ************************************* * Tool functions ***************************************/ size_t ZBUFF_recommendedCInSize(void) { return ZSTD_CStreamInSize(); } size_t ZBUFF_recommendedCOutSize(void) { return ZSTD_CStreamOutSize(); } zmat-0.9.9/src/blosc2/internal-complibs/zstd/deprecated/zbuff_decompress.c0000644000175200007730000000404214515254731027124 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* ************************************* * Dependencies ***************************************/ #define ZSTD_DISABLE_DEPRECATE_WARNINGS /* suppress warning on ZSTD_initDStream_usingDict */ #include "../zstd.h" /* ZSTD_CStream, ZSTD_DStream, ZSTDLIB_API */ #define ZBUFF_STATIC_LINKING_ONLY #include "zbuff.h" ZBUFF_DCtx* ZBUFF_createDCtx(void) { return ZSTD_createDStream(); } ZBUFF_DCtx* ZBUFF_createDCtx_advanced(ZSTD_customMem customMem) { return ZSTD_createDStream_advanced(customMem); } size_t ZBUFF_freeDCtx(ZBUFF_DCtx* zbd) { return ZSTD_freeDStream(zbd); } /* *** Initialization *** */ size_t ZBUFF_decompressInitDictionary(ZBUFF_DCtx* zbd, const void* dict, size_t dictSize) { return ZSTD_initDStream_usingDict(zbd, dict, dictSize); } size_t ZBUFF_decompressInit(ZBUFF_DCtx* zbd) { return ZSTD_initDStream(zbd); } /* *** Decompression *** */ size_t ZBUFF_decompressContinue(ZBUFF_DCtx* zbd, void* dst, size_t* dstCapacityPtr, const void* src, size_t* srcSizePtr) { ZSTD_outBuffer outBuff; ZSTD_inBuffer inBuff; size_t result; outBuff.dst = dst; outBuff.pos = 0; outBuff.size = *dstCapacityPtr; inBuff.src = src; inBuff.pos = 0; inBuff.size = *srcSizePtr; result = ZSTD_decompressStream(zbd, &outBuff, &inBuff); *dstCapacityPtr = outBuff.pos; *srcSizePtr = inBuff.pos; return result; } /* ************************************* * Tool functions ***************************************/ size_t ZBUFF_recommendedDInSize(void) { return ZSTD_DStreamInSize(); } size_t ZBUFF_recommendedDOutSize(void) { return ZSTD_DStreamOutSize(); } zmat-0.9.9/src/blosc2/internal-complibs/zstd/deprecated/zbuff.h0000644000175200007730000002635514515254731024720 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* *************************************************************** * NOTES/WARNINGS ******************************************************************/ /* The streaming API defined here is deprecated. * Consider migrating towards ZSTD_compressStream() API in `zstd.h` * See 'lib/README.md'. *****************************************************************/ #if defined (__cplusplus) extern "C" { #endif #ifndef ZSTD_BUFFERED_H_23987 #define ZSTD_BUFFERED_H_23987 /* ************************************* * Dependencies ***************************************/ #include /* size_t */ #include "../zstd.h" /* ZSTD_CStream, ZSTD_DStream, ZSTDLIB_API */ /* *************************************************************** * Compiler specifics *****************************************************************/ /* Deprecation warnings */ /* Should these warnings be a problem, * it is generally possible to disable them, * typically with -Wno-deprecated-declarations for gcc * or _CRT_SECURE_NO_WARNINGS in Visual. * Otherwise, it's also possible to define ZBUFF_DISABLE_DEPRECATE_WARNINGS */ #ifdef ZBUFF_DISABLE_DEPRECATE_WARNINGS # define ZBUFF_DEPRECATED(message) ZSTDLIB_API /* disable deprecation warnings */ #else # if defined (__cplusplus) && (__cplusplus >= 201402) /* C++14 or greater */ # define ZBUFF_DEPRECATED(message) [[deprecated(message)]] ZSTDLIB_API # elif (defined(GNUC) && (GNUC > 4 || (GNUC == 4 && GNUC_MINOR >= 5))) || defined(__clang__) # define ZBUFF_DEPRECATED(message) ZSTDLIB_API __attribute__((deprecated(message))) # elif defined(__GNUC__) && (__GNUC__ >= 3) # define ZBUFF_DEPRECATED(message) ZSTDLIB_API __attribute__((deprecated)) # elif defined(_MSC_VER) # define ZBUFF_DEPRECATED(message) ZSTDLIB_API __declspec(deprecated(message)) # else # pragma message("WARNING: You need to implement ZBUFF_DEPRECATED for this compiler") # define ZBUFF_DEPRECATED(message) ZSTDLIB_API # endif #endif /* ZBUFF_DISABLE_DEPRECATE_WARNINGS */ /* ************************************* * Streaming functions ***************************************/ /* This is the easier "buffered" streaming API, * using an internal buffer to lift all restrictions on user-provided buffers * which can be any size, any place, for both input and output. * ZBUFF and ZSTD are 100% interoperable, * frames created by one can be decoded by the other one */ typedef ZSTD_CStream ZBUFF_CCtx; ZBUFF_DEPRECATED("use ZSTD_createCStream") ZBUFF_CCtx* ZBUFF_createCCtx(void); ZBUFF_DEPRECATED("use ZSTD_freeCStream") size_t ZBUFF_freeCCtx(ZBUFF_CCtx* cctx); ZBUFF_DEPRECATED("use ZSTD_initCStream") size_t ZBUFF_compressInit(ZBUFF_CCtx* cctx, int compressionLevel); ZBUFF_DEPRECATED("use ZSTD_initCStream_usingDict") size_t ZBUFF_compressInitDictionary(ZBUFF_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel); ZBUFF_DEPRECATED("use ZSTD_compressStream") size_t ZBUFF_compressContinue(ZBUFF_CCtx* cctx, void* dst, size_t* dstCapacityPtr, const void* src, size_t* srcSizePtr); ZBUFF_DEPRECATED("use ZSTD_flushStream") size_t ZBUFF_compressFlush(ZBUFF_CCtx* cctx, void* dst, size_t* dstCapacityPtr); ZBUFF_DEPRECATED("use ZSTD_endStream") size_t ZBUFF_compressEnd(ZBUFF_CCtx* cctx, void* dst, size_t* dstCapacityPtr); /*-************************************************* * Streaming compression - howto * * A ZBUFF_CCtx object is required to track streaming operation. * Use ZBUFF_createCCtx() and ZBUFF_freeCCtx() to create/release resources. * ZBUFF_CCtx objects can be reused multiple times. * * Start by initializing ZBUF_CCtx. * Use ZBUFF_compressInit() to start a new compression operation. * Use ZBUFF_compressInitDictionary() for a compression which requires a dictionary. * * Use ZBUFF_compressContinue() repetitively to consume input stream. * *srcSizePtr and *dstCapacityPtr can be any size. * The function will report how many bytes were read or written within *srcSizePtr and *dstCapacityPtr. * Note that it may not consume the entire input, in which case it's up to the caller to present again remaining data. * The content of `dst` will be overwritten (up to *dstCapacityPtr) at each call, so save its content if it matters or change @dst . * @return : a hint to preferred nb of bytes to use as input for next function call (it's just a hint, to improve latency) * or an error code, which can be tested using ZBUFF_isError(). * * At any moment, it's possible to flush whatever data remains within buffer, using ZBUFF_compressFlush(). * The nb of bytes written into `dst` will be reported into *dstCapacityPtr. * Note that the function cannot output more than *dstCapacityPtr, * therefore, some content might still be left into internal buffer if *dstCapacityPtr is too small. * @return : nb of bytes still present into internal buffer (0 if it's empty) * or an error code, which can be tested using ZBUFF_isError(). * * ZBUFF_compressEnd() instructs to finish a frame. * It will perform a flush and write frame epilogue. * The epilogue is required for decoders to consider a frame completed. * Similar to ZBUFF_compressFlush(), it may not be able to output the entire internal buffer content if *dstCapacityPtr is too small. * In which case, call again ZBUFF_compressFlush() to complete the flush. * @return : nb of bytes still present into internal buffer (0 if it's empty) * or an error code, which can be tested using ZBUFF_isError(). * * Hint : _recommended buffer_ sizes (not compulsory) : ZBUFF_recommendedCInSize() / ZBUFF_recommendedCOutSize() * input : ZBUFF_recommendedCInSize==128 KB block size is the internal unit, use this value to reduce intermediate stages (better latency) * output : ZBUFF_recommendedCOutSize==ZSTD_compressBound(128 KB) + 3 + 3 : ensures it's always possible to write/flush/end a full block. Skip some buffering. * By using both, it ensures that input will be entirely consumed, and output will always contain the result, reducing intermediate buffering. * **************************************************/ typedef ZSTD_DStream ZBUFF_DCtx; ZBUFF_DEPRECATED("use ZSTD_createDStream") ZBUFF_DCtx* ZBUFF_createDCtx(void); ZBUFF_DEPRECATED("use ZSTD_freeDStream") size_t ZBUFF_freeDCtx(ZBUFF_DCtx* dctx); ZBUFF_DEPRECATED("use ZSTD_initDStream") size_t ZBUFF_decompressInit(ZBUFF_DCtx* dctx); ZBUFF_DEPRECATED("use ZSTD_initDStream_usingDict") size_t ZBUFF_decompressInitDictionary(ZBUFF_DCtx* dctx, const void* dict, size_t dictSize); ZBUFF_DEPRECATED("use ZSTD_decompressStream") size_t ZBUFF_decompressContinue(ZBUFF_DCtx* dctx, void* dst, size_t* dstCapacityPtr, const void* src, size_t* srcSizePtr); /*-*************************************************************************** * Streaming decompression howto * * A ZBUFF_DCtx object is required to track streaming operations. * Use ZBUFF_createDCtx() and ZBUFF_freeDCtx() to create/release resources. * Use ZBUFF_decompressInit() to start a new decompression operation, * or ZBUFF_decompressInitDictionary() if decompression requires a dictionary. * Note that ZBUFF_DCtx objects can be re-init multiple times. * * Use ZBUFF_decompressContinue() repetitively to consume your input. * *srcSizePtr and *dstCapacityPtr can be any size. * The function will report how many bytes were read or written by modifying *srcSizePtr and *dstCapacityPtr. * Note that it may not consume the entire input, in which case it's up to the caller to present remaining input again. * The content of `dst` will be overwritten (up to *dstCapacityPtr) at each function call, so save its content if it matters, or change `dst`. * @return : 0 when a frame is completely decoded and fully flushed, * 1 when there is still some data left within internal buffer to flush, * >1 when more data is expected, with value being a suggested next input size (it's just a hint, which helps latency), * or an error code, which can be tested using ZBUFF_isError(). * * Hint : recommended buffer sizes (not compulsory) : ZBUFF_recommendedDInSize() and ZBUFF_recommendedDOutSize() * output : ZBUFF_recommendedDOutSize== 128 KB block size is the internal unit, it ensures it's always possible to write a full block when decoded. * input : ZBUFF_recommendedDInSize == 128KB + 3; * just follow indications from ZBUFF_decompressContinue() to minimize latency. It should always be <= 128 KB + 3 . * *******************************************************************************/ /* ************************************* * Tool functions ***************************************/ ZBUFF_DEPRECATED("use ZSTD_isError") unsigned ZBUFF_isError(size_t errorCode); ZBUFF_DEPRECATED("use ZSTD_getErrorName") const char* ZBUFF_getErrorName(size_t errorCode); /** Functions below provide recommended buffer sizes for Compression or Decompression operations. * These sizes are just hints, they tend to offer better latency */ ZBUFF_DEPRECATED("use ZSTD_CStreamInSize") size_t ZBUFF_recommendedCInSize(void); ZBUFF_DEPRECATED("use ZSTD_CStreamOutSize") size_t ZBUFF_recommendedCOutSize(void); ZBUFF_DEPRECATED("use ZSTD_DStreamInSize") size_t ZBUFF_recommendedDInSize(void); ZBUFF_DEPRECATED("use ZSTD_DStreamOutSize") size_t ZBUFF_recommendedDOutSize(void); #endif /* ZSTD_BUFFERED_H_23987 */ #ifdef ZBUFF_STATIC_LINKING_ONLY #ifndef ZBUFF_STATIC_H_30298098432 #define ZBUFF_STATIC_H_30298098432 /* ==================================================================================== * The definitions in this section are considered experimental. * They should never be used in association with a dynamic library, as they may change in the future. * They are provided for advanced usages. * Use them only in association with static linking. * ==================================================================================== */ /*--- Dependency ---*/ #define ZSTD_STATIC_LINKING_ONLY /* ZSTD_parameters, ZSTD_customMem */ #include "../zstd.h" /*--- Custom memory allocator ---*/ /*! ZBUFF_createCCtx_advanced() : * Create a ZBUFF compression context using external alloc and free functions */ ZBUFF_DEPRECATED("use ZSTD_createCStream_advanced") ZBUFF_CCtx* ZBUFF_createCCtx_advanced(ZSTD_customMem customMem); /*! ZBUFF_createDCtx_advanced() : * Create a ZBUFF decompression context using external alloc and free functions */ ZBUFF_DEPRECATED("use ZSTD_createDStream_advanced") ZBUFF_DCtx* ZBUFF_createDCtx_advanced(ZSTD_customMem customMem); /*--- Advanced Streaming Initialization ---*/ ZBUFF_DEPRECATED("use ZSTD_initDStream_usingDict") size_t ZBUFF_compressInit_advanced(ZBUFF_CCtx* zbc, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); #endif /* ZBUFF_STATIC_H_30298098432 */ #endif /* ZBUFF_STATIC_LINKING_ONLY */ #if defined (__cplusplus) } #endif zmat-0.9.9/src/blosc2/internal-complibs/zstd/BUCK0000644000175200007730000001060014515254731021764 0ustar rlaboissrlaboisscxx_library( name='zstd', header_namespace='', exported_headers=['zstd.h'], visibility=['PUBLIC'], deps=[ ':common', ':compress', ':decompress', ':deprecated', ], ) cxx_library( name='compress', header_namespace='', visibility=['PUBLIC'], exported_headers=subdir_glob([ ('compress', 'zstd*.h'), ]), srcs=glob(['compress/zstd*.c', 'compress/hist.c']), deps=[':common'], ) cxx_library( name='decompress', header_namespace='', visibility=['PUBLIC'], headers=subdir_glob([ ('decompress', '*_impl.h'), ]), srcs=glob(['decompress/zstd*.c']), deps=[ ':common', ':legacy', ], ) cxx_library( name='deprecated', header_namespace='', visibility=['PUBLIC'], exported_headers=subdir_glob([ ('deprecated', '*.h'), ]), srcs=glob(['deprecated/*.c']), deps=[':common'], ) cxx_library( name='legacy', header_namespace='', visibility=['PUBLIC'], exported_headers=subdir_glob([ ('legacy', '*.h'), ]), srcs=glob(['legacy/*.c']), deps=[':common'], exported_preprocessor_flags=[ '-DZSTD_LEGACY_SUPPORT=4', ], ) cxx_library( name='zdict', header_namespace='', visibility=['PUBLIC'], exported_headers=['zdict.h'], headers=subdir_glob([ ('dictBuilder', 'divsufsort.h'), ('dictBuilder', 'cover.h'), ]), srcs=glob(['dictBuilder/*.c']), deps=[':common'], ) cxx_library( name='compiler', header_namespace='', visibility=['PUBLIC'], exported_headers=subdir_glob([ ('common', 'compiler.h'), ]), ) cxx_library( name='cpu', header_namespace='', visibility=['PUBLIC'], exported_headers=subdir_glob([ ('common', 'cpu.h'), ]), ) cxx_library( name='bitstream', header_namespace='', visibility=['PUBLIC'], exported_headers=subdir_glob([ ('common', 'bitstream.h'), ]), ) cxx_library( name='entropy', header_namespace='', visibility=['PUBLIC'], exported_headers=subdir_glob([ ('common', 'fse.h'), ('common', 'huf.h'), ]), srcs=[ 'common/entropy_common.c', 'common/fse_decompress.c', 'compress/fse_compress.c', 'compress/huf_compress.c', 'decompress/huf_decompress.c', ], deps=[ ':debug', ':bitstream', ':compiler', ':errors', ':mem', ], ) cxx_library( name='errors', header_namespace='', visibility=['PUBLIC'], exported_headers=[ 'zstd_errors.h', 'common/error_private.h', ] srcs=['common/error_private.c'], ) cxx_library( name='mem', header_namespace='', visibility=['PUBLIC'], exported_headers=subdir_glob([ ('common', 'mem.h'), ]), ) cxx_library( name='pool', header_namespace='', visibility=['PUBLIC'], exported_headers=subdir_glob([ ('common', 'pool.h'), ]), srcs=['common/pool.c'], deps=[ ':threading', ':zstd_common', ], ) cxx_library( name='threading', header_namespace='', visibility=['PUBLIC'], exported_headers=subdir_glob([ ('common', 'threading.h'), ]), srcs=['common/threading.c'], exported_preprocessor_flags=[ '-DZSTD_MULTITHREAD', ], exported_linker_flags=[ '-pthread', ], ) cxx_library( name='xxhash', header_namespace='', visibility=['PUBLIC'], exported_headers=subdir_glob([ ('common', 'xxhash.h'), ]), srcs=['common/xxhash.c'], exported_preprocessor_flags=[ '-DXXH_NAMESPACE=ZSTD_', ], ) cxx_library( name='zstd_common', header_namespace='', visibility=['PUBLIC'], exported_headers=subdir_glob([ ('', 'zstd.h'), ('common', 'zstd_internal.h'), ]), srcs=['common/zstd_common.c'], deps=[ ':compiler', ':errors', ':mem', ], ) cxx_library( name='debug', header_namespace='', visibility=['PUBLIC'], exported_headers=subdir_glob([ ('common', 'debug.h'), ]), srcs=['common/debug.c'], ) cxx_library( name='common', deps=[ ':debug', ':bitstream', ':compiler', ':cpu', ':entropy', ':errors', ':mem', ':pool', ':threading', ':xxhash', ':zstd_common', ] ) zmat-0.9.9/src/blosc2/internal-complibs/zstd/zstd_errors.h0000644000175200007730000001066414515254731024060 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_ERRORS_H_398273423 #define ZSTD_ERRORS_H_398273423 #if defined (__cplusplus) extern "C" { #endif /*===== dependency =====*/ #include /* size_t */ /* ===== ZSTDERRORLIB_API : control library symbols visibility ===== */ #ifndef ZSTDERRORLIB_VISIBLE /* Backwards compatibility with old macro name */ # ifdef ZSTDERRORLIB_VISIBILITY # define ZSTDERRORLIB_VISIBLE ZSTDERRORLIB_VISIBILITY # elif defined(__GNUC__) && (__GNUC__ >= 4) && !defined(__MINGW32__) # define ZSTDERRORLIB_VISIBLE __attribute__ ((visibility ("default"))) # else # define ZSTDERRORLIB_VISIBLE # endif #endif #ifndef ZSTDERRORLIB_HIDDEN # if defined(__GNUC__) && (__GNUC__ >= 4) && !defined(__MINGW32__) # define ZSTDERRORLIB_HIDDEN __attribute__ ((visibility ("hidden"))) # else # define ZSTDERRORLIB_HIDDEN # endif #endif #if defined(ZSTD_DLL_EXPORT) && (ZSTD_DLL_EXPORT==1) # define ZSTDERRORLIB_API __declspec(dllexport) ZSTDERRORLIB_VISIBLE #elif defined(ZSTD_DLL_IMPORT) && (ZSTD_DLL_IMPORT==1) # define ZSTDERRORLIB_API __declspec(dllimport) ZSTDERRORLIB_VISIBLE /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/ #else # define ZSTDERRORLIB_API ZSTDERRORLIB_VISIBLE #endif /*-********************************************* * Error codes list *-********************************************* * Error codes _values_ are pinned down since v1.3.1 only. * Therefore, don't rely on values if you may link to any version < v1.3.1. * * Only values < 100 are considered stable. * * note 1 : this API shall be used with static linking only. * dynamic linking is not yet officially supported. * note 2 : Prefer relying on the enum than on its value whenever possible * This is the only supported way to use the error list < v1.3.1 * note 3 : ZSTD_isError() is always correct, whatever the library version. **********************************************/ typedef enum { ZSTD_error_no_error = 0, ZSTD_error_GENERIC = 1, ZSTD_error_prefix_unknown = 10, ZSTD_error_version_unsupported = 12, ZSTD_error_frameParameter_unsupported = 14, ZSTD_error_frameParameter_windowTooLarge = 16, ZSTD_error_corruption_detected = 20, ZSTD_error_checksum_wrong = 22, ZSTD_error_literals_headerWrong = 24, ZSTD_error_dictionary_corrupted = 30, ZSTD_error_dictionary_wrong = 32, ZSTD_error_dictionaryCreation_failed = 34, ZSTD_error_parameter_unsupported = 40, ZSTD_error_parameter_combination_unsupported = 41, ZSTD_error_parameter_outOfBound = 42, ZSTD_error_tableLog_tooLarge = 44, ZSTD_error_maxSymbolValue_tooLarge = 46, ZSTD_error_maxSymbolValue_tooSmall = 48, ZSTD_error_stabilityCondition_notRespected = 50, ZSTD_error_stage_wrong = 60, ZSTD_error_init_missing = 62, ZSTD_error_memory_allocation = 64, ZSTD_error_workSpace_tooSmall= 66, ZSTD_error_dstSize_tooSmall = 70, ZSTD_error_srcSize_wrong = 72, ZSTD_error_dstBuffer_null = 74, ZSTD_error_noForwardProgress_destFull = 80, ZSTD_error_noForwardProgress_inputEmpty = 82, /* following error codes are __NOT STABLE__, they can be removed or changed in future versions */ ZSTD_error_frameIndex_tooLarge = 100, ZSTD_error_seekableIO = 102, ZSTD_error_dstBuffer_wrong = 104, ZSTD_error_srcBuffer_wrong = 105, ZSTD_error_sequenceProducer_failed = 106, ZSTD_error_externalSequences_invalid = 107, ZSTD_error_maxCode = 120 /* never EVER use this value directly, it can change in future versions! Use ZSTD_isError() instead */ } ZSTD_ErrorCode; /*! ZSTD_getErrorCode() : convert a `size_t` function result into a `ZSTD_ErrorCode` enum type, which can be used to compare with enum list published above */ ZSTDERRORLIB_API ZSTD_ErrorCode ZSTD_getErrorCode(size_t functionResult); ZSTDERRORLIB_API const char* ZSTD_getErrorString(ZSTD_ErrorCode code); /**< Same as ZSTD_getErrorName, but using a `ZSTD_ErrorCode` enum argument */ #if defined (__cplusplus) } #endif #endif /* ZSTD_ERRORS_H_398273423 */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/README.md0000644000175200007730000002624014515254731022603 0ustar rlaboissrlaboissZstandard library files ================================ The __lib__ directory is split into several sub-directories, in order to make it easier to select or exclude features. #### Building `Makefile` script is provided, supporting [Makefile conventions](https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html#Makefile-Conventions), including commands variables, staged install, directory variables and standard targets. - `make` : generates both static and dynamic libraries - `make install` : install libraries and headers in target system directories `libzstd` default scope is pretty large, including compression, decompression, dictionary builder, and support for decoding legacy formats >= v0.5.0. The scope can be reduced on demand (see paragraph _modular build_). #### Multithreading support When building with `make`, by default the dynamic library is multithreaded and static library is single-threaded (for compatibility reasons). Enabling multithreading requires 2 conditions : - set build macro `ZSTD_MULTITHREAD` (`-DZSTD_MULTITHREAD` for `gcc`) - for POSIX systems : compile with pthread (`-pthread` compilation flag for `gcc`) For convenience, we provide a build target to generate multi and single threaded libraries: - Force enable multithreading on both dynamic and static libraries by appending `-mt` to the target, e.g. `make lib-mt`. - Force disable multithreading on both dynamic and static libraries by appending `-nomt` to the target, e.g. `make lib-nomt`. - By default, as mentioned before, dynamic library is multithreaded, and static library is single-threaded, e.g. `make lib`. When linking a POSIX program with a multithreaded version of `libzstd`, note that it's necessary to invoke the `-pthread` flag during link stage. Multithreading capabilities are exposed via the [advanced API defined in `lib/zstd.h`](https://github.com/facebook/zstd/blob/v1.4.3/lib/zstd.h#L351). #### API Zstandard's stable API is exposed within [lib/zstd.h](zstd.h). #### Advanced API Optional advanced features are exposed via : - `lib/zstd_errors.h` : translates `size_t` function results into a `ZSTD_ErrorCode`, for accurate error handling. - `ZSTD_STATIC_LINKING_ONLY` : if this macro is defined _before_ including `zstd.h`, it unlocks access to the experimental API, exposed in the second part of `zstd.h`. All definitions in the experimental APIs are unstable, they may still change in the future, or even be removed. As a consequence, experimental definitions shall ___never be used with dynamic library___ ! Only static linking is allowed. #### Modular build It's possible to compile only a limited set of features within `libzstd`. The file structure is designed to make this selection manually achievable for any build system : - Directory `lib/common` is always required, for all variants. - Compression source code lies in `lib/compress` - Decompression source code lies in `lib/decompress` - It's possible to include only `compress` or only `decompress`, they don't depend on each other. - `lib/dictBuilder` : makes it possible to generate dictionaries from a set of samples. The API is exposed in `lib/dictBuilder/zdict.h`. This module depends on both `lib/common` and `lib/compress` . - `lib/legacy` : makes it possible to decompress legacy zstd formats, starting from `v0.1.0`. This module depends on `lib/common` and `lib/decompress`. To enable this feature, define `ZSTD_LEGACY_SUPPORT` during compilation. Specifying a number limits versions supported to that version onward. For example, `ZSTD_LEGACY_SUPPORT=2` means : "support legacy formats >= v0.2.0". Conversely, `ZSTD_LEGACY_SUPPORT=0` means "do __not__ support legacy formats". By default, this build macro is set as `ZSTD_LEGACY_SUPPORT=5`. Decoding supported legacy format is a transparent capability triggered within decompression functions. It's also allowed to invoke legacy API directly, exposed in `lib/legacy/zstd_legacy.h`. Each version does also provide its own set of advanced API. For example, advanced API for version `v0.4` is exposed in `lib/legacy/zstd_v04.h` . - While invoking `make libzstd`, it's possible to define build macros `ZSTD_LIB_COMPRESSION, ZSTD_LIB_DECOMPRESSION`, `ZSTD_LIB_DICTBUILDER`, and `ZSTD_LIB_DEPRECATED` as `0` to forgo compilation of the corresponding features. This will also disable compilation of all dependencies (e.g. `ZSTD_LIB_COMPRESSION=0` will also disable dictBuilder). - There are a number of options that can help minimize the binary size of `libzstd`. The first step is to select the components needed (using the above-described `ZSTD_LIB_COMPRESSION` etc.). The next step is to set `ZSTD_LIB_MINIFY` to `1` when invoking `make`. This disables various optional components and changes the compilation flags to prioritize space-saving. Detailed options: Zstandard's code and build environment is set up by default to optimize above all else for performance. In pursuit of this goal, Zstandard makes significant trade-offs in code size. For example, Zstandard often has more than one implementation of a particular component, with each implementation optimized for different scenarios. For example, the Huffman decoder has complementary implementations that decode the stream one symbol at a time or two symbols at a time. Zstd normally includes both (and dispatches between them at runtime), but by defining `HUF_FORCE_DECOMPRESS_X1` or `HUF_FORCE_DECOMPRESS_X2`, you can force the use of one or the other, avoiding compilation of the other. Similarly, `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT` and `ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG` force the compilation and use of only one or the other of two decompression implementations. The smallest binary is achieved by using `HUF_FORCE_DECOMPRESS_X1` and `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT` (implied by `ZSTD_LIB_MINIFY`). For squeezing the last ounce of size out, you can also define `ZSTD_NO_INLINE`, which disables inlining, and `ZSTD_STRIP_ERROR_STRINGS`, which removes the error messages that are otherwise returned by `ZSTD_getErrorName` (implied by `ZSTD_LIB_MINIFY`). Finally, when integrating into your application, make sure you're doing link- time optimization and unused symbol garbage collection (via some combination of, e.g., `-flto`, `-ffat-lto-objects`, `-fuse-linker-plugin`, `-ffunction-sections`, `-fdata-sections`, `-fmerge-all-constants`, `-Wl,--gc-sections`, `-Wl,-z,norelro`, and an archiver that understands the compiler's intermediate representation, e.g., `AR=gcc-ar`). Consult your compiler's documentation. - While invoking `make libzstd`, the build macro `ZSTD_LEGACY_MULTITHREADED_API=1` will expose the deprecated `ZSTDMT` API exposed by `zstdmt_compress.h` in the shared library, which is now hidden by default. - The build macro `DYNAMIC_BMI2` can be set to 1 or 0 in order to generate binaries which can detect at runtime the presence of BMI2 instructions, and use them only if present. These instructions contribute to better performance, notably on the decoder side. By default, this feature is automatically enabled on detecting the right instruction set (x64) and compiler (clang or gcc >= 5). It's obviously disabled for different cpus, or when BMI2 instruction set is _required_ by the compiler command line (in this case, only the BMI2 code path is generated). Setting this macro will either force to generate the BMI2 dispatcher (1) or prevent it (0). It overrides automatic detection. - The build macro `ZSTD_NO_UNUSED_FUNCTIONS` can be defined to hide the definitions of functions that zstd does not use. Not all unused functions are hidden, but they can be if needed. Currently, this macro will hide function definitions in FSE and HUF that use an excessive amount of stack space. - The build macro `ZSTD_NO_INTRINSICS` can be defined to disable all explicit intrinsics. Compiler builtins are still used. - The build macro `ZSTD_DECODER_INTERNAL_BUFFER` can be set to control the amount of extra memory used during decompression to store literals. This defaults to 64kB. Reducing this value reduces the memory footprint of `ZSTD_DCtx` decompression contexts, but might also result in a small decompression speed cost. - The C compiler macros `ZSTDLIB_VISIBLE`, `ZSTDERRORLIB_VISIBLE` and `ZDICTLIB_VISIBLE` can be overridden to control the visibility of zstd's API. Additionally, `ZSTDLIB_STATIC_API` and `ZDICTLIB_STATIC_API` can be overridden to control the visibility of zstd's static API. Specifically, it can be set to `ZSTDLIB_HIDDEN` to hide the symbols from the shared library. These macros default to `ZSTDLIB_VISIBILITY`, `ZSTDERRORLIB_VSIBILITY`, and `ZDICTLIB_VISIBILITY` if unset, for backwards compatibility with the old macro names. #### Windows : using MinGW+MSYS to create DLL DLL can be created using MinGW+MSYS with the `make libzstd` command. This command creates `dll\libzstd.dll` and the import library `dll\libzstd.lib`. The import library is only required with Visual C++. The header file `zstd.h` and the dynamic library `dll\libzstd.dll` are required to compile a project using gcc/MinGW. The dynamic library has to be added to linking options. It means that if a project that uses ZSTD consists of a single `test-dll.c` file it should be linked with `dll\libzstd.dll`. For example: ``` gcc $(CFLAGS) -Iinclude/ test-dll.c -o test-dll dll\libzstd.dll ``` The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`. #### Advanced Build options The build system requires a hash function in order to separate object files created with different compilation flags. By default, it tries to use `md5sum` or equivalent. The hash function can be manually switched by setting the `HASH` variable. For example : `make HASH=xxhsum` The hash function needs to generate at least 64-bit using hexadecimal format. When no hash function is found, the Makefile just generates all object files into the same default directory, irrespective of compilation flags. This functionality only matters if `libzstd` is compiled multiple times with different build flags. The build directory, where object files are stored can also be manually controlled using variable `BUILD_DIR`, for example `make BUILD_DIR=objectDir/v1`. In which case, the hash function doesn't matter. #### Deprecated API Obsolete API on their way out are stored in directory `lib/deprecated`. At this stage, it contains older streaming prototypes, in `lib/deprecated/zbuff.h`. These prototypes will be removed in some future version. Consider migrating code towards supported streaming API exposed in `zstd.h`. #### Miscellaneous The other files are not source code. There are : - `BUCK` : support for `buck` build system (https://buckbuild.com/) - `Makefile` : `make` script to build and install zstd library (static and dynamic) - `README.md` : this file - `dll/` : resources directory for Windows compilation - `libzstd.pc.in` : script for `pkg-config` (used in `make install`) zmat-0.9.9/src/blosc2/internal-complibs/zstd/module.modulemap0000644000175200007730000000156714515254731024523 0ustar rlaboissrlaboissmodule libzstd [extern_c] { header "zstd.h" export * config_macros [exhaustive] \ /* zstd.h */ \ ZSTD_STATIC_LINKING_ONLY, \ ZSTDLIB_VISIBILITY, \ ZSTDLIB_VISIBLE, \ ZSTDLIB_HIDDEN, \ ZSTD_DLL_EXPORT, \ ZSTDLIB_STATIC_API, \ ZSTD_DISABLE_DEPRECATE_WARNINGS, \ ZSTD_CLEVEL_DEFAULT, \ /* zdict.h */ \ ZDICT_STATIC_LINKING_ONLY, \ ZDICTLIB_VISIBLE, \ ZDICTLIB_HIDDEN, \ ZDICTLIB_VISIBILITY, \ ZDICTLIB_STATIC_API, \ ZDICT_DISABLE_DEPRECATE_WARNINGS, \ /* zstd_errors.h */ \ ZSTDERRORLIB_VISIBLE, \ ZSTDERRORLIB_HIDDEN, \ ZSTDERRORLIB_VISIBILITY module dictbuilder [extern_c] { header "zdict.h" export * } module errors [extern_c] { header "zstd_errors.h" export * } } zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/0000755000175200007730000000000014515254731022610 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/internal-complibs/zstd/common/debug.c0000644000175200007730000000151314515254731024042 0ustar rlaboissrlaboiss/* ****************************************************************** * debug * Part of FSE library * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ /* * This module only hosts one global variable * which can be used to dynamically influence the verbosity of traces, * such as DEBUGLOG and RAWLOG */ #include "debug.h" int g_debuglevel = DEBUGLEVEL; zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/threading.c0000644000175200007730000001120414515254731024717 0ustar rlaboissrlaboiss/** * Copyright (c) 2016 Tino Reichardt * All rights reserved. * * You can contact the author at: * - zstdmt source repository: https://github.com/mcmilk/zstdmt * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /** * This file will hold wrapper for systems, which do not support pthreads */ #include "threading.h" /* create fake symbol to avoid empty translation unit warning */ int g_ZSTD_threading_useless_symbol; #if defined(ZSTD_MULTITHREAD) && defined(_WIN32) /** * Windows minimalist Pthread Wrapper */ /* === Dependencies === */ #include #include /* === Implementation === */ typedef struct { void* (*start_routine)(void*); void* arg; int initialized; ZSTD_pthread_cond_t initialized_cond; ZSTD_pthread_mutex_t initialized_mutex; } ZSTD_thread_params_t; static unsigned __stdcall worker(void *arg) { void* (*start_routine)(void*); void* thread_arg; /* Initialized thread_arg and start_routine and signal main thread that we don't need it * to wait any longer. */ { ZSTD_thread_params_t* thread_param = (ZSTD_thread_params_t*)arg; thread_arg = thread_param->arg; start_routine = thread_param->start_routine; /* Signal main thread that we are running and do not depend on its memory anymore */ ZSTD_pthread_mutex_lock(&thread_param->initialized_mutex); thread_param->initialized = 1; ZSTD_pthread_cond_signal(&thread_param->initialized_cond); ZSTD_pthread_mutex_unlock(&thread_param->initialized_mutex); } start_routine(thread_arg); return 0; } int ZSTD_pthread_create(ZSTD_pthread_t* thread, const void* unused, void* (*start_routine) (void*), void* arg) { ZSTD_thread_params_t thread_param; (void)unused; thread_param.start_routine = start_routine; thread_param.arg = arg; thread_param.initialized = 0; *thread = NULL; /* Setup thread initialization synchronization */ if(ZSTD_pthread_cond_init(&thread_param.initialized_cond, NULL)) { /* Should never happen on Windows */ return -1; } if(ZSTD_pthread_mutex_init(&thread_param.initialized_mutex, NULL)) { /* Should never happen on Windows */ ZSTD_pthread_cond_destroy(&thread_param.initialized_cond); return -1; } /* Spawn thread */ *thread = (HANDLE)_beginthreadex(NULL, 0, worker, &thread_param, 0, NULL); if (!thread) { ZSTD_pthread_mutex_destroy(&thread_param.initialized_mutex); ZSTD_pthread_cond_destroy(&thread_param.initialized_cond); return errno; } /* Wait for thread to be initialized */ ZSTD_pthread_mutex_lock(&thread_param.initialized_mutex); while(!thread_param.initialized) { ZSTD_pthread_cond_wait(&thread_param.initialized_cond, &thread_param.initialized_mutex); } ZSTD_pthread_mutex_unlock(&thread_param.initialized_mutex); ZSTD_pthread_mutex_destroy(&thread_param.initialized_mutex); ZSTD_pthread_cond_destroy(&thread_param.initialized_cond); return 0; } int ZSTD_pthread_join(ZSTD_pthread_t thread) { DWORD result; if (!thread) return 0; result = WaitForSingleObject(thread, INFINITE); CloseHandle(thread); switch (result) { case WAIT_OBJECT_0: return 0; case WAIT_ABANDONED: return EINVAL; default: return GetLastError(); } } #endif /* ZSTD_MULTITHREAD */ #if defined(ZSTD_MULTITHREAD) && DEBUGLEVEL >= 1 && !defined(_WIN32) #define ZSTD_DEPS_NEED_MALLOC #include "zstd_deps.h" int ZSTD_pthread_mutex_init(ZSTD_pthread_mutex_t* mutex, pthread_mutexattr_t const* attr) { *mutex = (pthread_mutex_t*)ZSTD_malloc(sizeof(pthread_mutex_t)); if (!*mutex) return 1; return pthread_mutex_init(*mutex, attr); } int ZSTD_pthread_mutex_destroy(ZSTD_pthread_mutex_t* mutex) { if (!*mutex) return 0; { int const ret = pthread_mutex_destroy(*mutex); ZSTD_free(*mutex); return ret; } } int ZSTD_pthread_cond_init(ZSTD_pthread_cond_t* cond, pthread_condattr_t const* attr) { *cond = (pthread_cond_t*)ZSTD_malloc(sizeof(pthread_cond_t)); if (!*cond) return 1; return pthread_cond_init(*cond, attr); } int ZSTD_pthread_cond_destroy(ZSTD_pthread_cond_t* cond) { if (!*cond) return 0; { int const ret = pthread_cond_destroy(*cond); ZSTD_free(*cond); return ret; } } #endif zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/mem.h0000644000175200007730000003173614515254731023551 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef MEM_H_MODULE #define MEM_H_MODULE #if defined (__cplusplus) extern "C" { #endif /*-**************************************** * Dependencies ******************************************/ #include /* size_t, ptrdiff_t */ #include "compiler.h" /* __has_builtin */ #include "debug.h" /* DEBUG_STATIC_ASSERT */ #include "zstd_deps.h" /* ZSTD_memcpy */ /*-**************************************** * Compiler specifics ******************************************/ #if defined(_MSC_VER) /* Visual Studio */ # include /* _byteswap_ulong */ # include /* _byteswap_* */ #endif #if defined(__GNUC__) # define MEM_STATIC static __inline __attribute__((unused)) #elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) # define MEM_STATIC static inline #elif defined(_MSC_VER) # define MEM_STATIC static __inline #else # define MEM_STATIC static /* this version may generate warnings for unused static functions; disable the relevant warning */ #endif /*-************************************************************** * Basic Types *****************************************************************/ #if !defined (__VMS) && (defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) # if defined(_AIX) # include # else # include /* intptr_t */ # endif typedef uint8_t BYTE; typedef uint8_t U8; typedef int8_t S8; typedef uint16_t U16; typedef int16_t S16; typedef uint32_t U32; typedef int32_t S32; typedef uint64_t U64; typedef int64_t S64; #else # include #if CHAR_BIT != 8 # error "this implementation requires char to be exactly 8-bit type" #endif typedef unsigned char BYTE; typedef unsigned char U8; typedef signed char S8; #if USHRT_MAX != 65535 # error "this implementation requires short to be exactly 16-bit type" #endif typedef unsigned short U16; typedef signed short S16; #if UINT_MAX != 4294967295 # error "this implementation requires int to be exactly 32-bit type" #endif typedef unsigned int U32; typedef signed int S32; /* note : there are no limits defined for long long type in C90. * limits exist in C99, however, in such case, is preferred */ typedef unsigned long long U64; typedef signed long long S64; #endif /*-************************************************************** * Memory I/O API *****************************************************************/ /*=== Static platform detection ===*/ MEM_STATIC unsigned MEM_32bits(void); MEM_STATIC unsigned MEM_64bits(void); MEM_STATIC unsigned MEM_isLittleEndian(void); /*=== Native unaligned read/write ===*/ MEM_STATIC U16 MEM_read16(const void* memPtr); MEM_STATIC U32 MEM_read32(const void* memPtr); MEM_STATIC U64 MEM_read64(const void* memPtr); MEM_STATIC size_t MEM_readST(const void* memPtr); MEM_STATIC void MEM_write16(void* memPtr, U16 value); MEM_STATIC void MEM_write32(void* memPtr, U32 value); MEM_STATIC void MEM_write64(void* memPtr, U64 value); /*=== Little endian unaligned read/write ===*/ MEM_STATIC U16 MEM_readLE16(const void* memPtr); MEM_STATIC U32 MEM_readLE24(const void* memPtr); MEM_STATIC U32 MEM_readLE32(const void* memPtr); MEM_STATIC U64 MEM_readLE64(const void* memPtr); MEM_STATIC size_t MEM_readLEST(const void* memPtr); MEM_STATIC void MEM_writeLE16(void* memPtr, U16 val); MEM_STATIC void MEM_writeLE24(void* memPtr, U32 val); MEM_STATIC void MEM_writeLE32(void* memPtr, U32 val32); MEM_STATIC void MEM_writeLE64(void* memPtr, U64 val64); MEM_STATIC void MEM_writeLEST(void* memPtr, size_t val); /*=== Big endian unaligned read/write ===*/ MEM_STATIC U32 MEM_readBE32(const void* memPtr); MEM_STATIC U64 MEM_readBE64(const void* memPtr); MEM_STATIC size_t MEM_readBEST(const void* memPtr); MEM_STATIC void MEM_writeBE32(void* memPtr, U32 val32); MEM_STATIC void MEM_writeBE64(void* memPtr, U64 val64); MEM_STATIC void MEM_writeBEST(void* memPtr, size_t val); /*=== Byteswap ===*/ MEM_STATIC U32 MEM_swap32(U32 in); MEM_STATIC U64 MEM_swap64(U64 in); MEM_STATIC size_t MEM_swapST(size_t in); /*-************************************************************** * Memory I/O Implementation *****************************************************************/ /* MEM_FORCE_MEMORY_ACCESS : For accessing unaligned memory: * Method 0 : always use `memcpy()`. Safe and portable. * Method 1 : Use compiler extension to set unaligned access. * Method 2 : direct access. This method is portable but violate C standard. * It can generate buggy code on targets depending on alignment. * Default : method 1 if supported, else method 0 */ #ifndef MEM_FORCE_MEMORY_ACCESS /* can be defined externally, on command line for example */ # ifdef __GNUC__ # define MEM_FORCE_MEMORY_ACCESS 1 # endif #endif MEM_STATIC unsigned MEM_32bits(void) { return sizeof(size_t)==4; } MEM_STATIC unsigned MEM_64bits(void) { return sizeof(size_t)==8; } MEM_STATIC unsigned MEM_isLittleEndian(void) { #if defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && (__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) return 1; #elif defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) return 0; #elif defined(__clang__) && __LITTLE_ENDIAN__ return 1; #elif defined(__clang__) && __BIG_ENDIAN__ return 0; #elif defined(_MSC_VER) && (_M_AMD64 || _M_IX86) return 1; #elif defined(__DMC__) && defined(_M_IX86) return 1; #else const union { U32 u; BYTE c[4]; } one = { 1 }; /* don't use static : performance detrimental */ return one.c[0]; #endif } #if defined(MEM_FORCE_MEMORY_ACCESS) && (MEM_FORCE_MEMORY_ACCESS==2) /* violates C standard, by lying on structure alignment. Only use if no other choice to achieve best performance on target platform */ MEM_STATIC U16 MEM_read16(const void* memPtr) { return *(const U16*) memPtr; } MEM_STATIC U32 MEM_read32(const void* memPtr) { return *(const U32*) memPtr; } MEM_STATIC U64 MEM_read64(const void* memPtr) { return *(const U64*) memPtr; } MEM_STATIC size_t MEM_readST(const void* memPtr) { return *(const size_t*) memPtr; } MEM_STATIC void MEM_write16(void* memPtr, U16 value) { *(U16*)memPtr = value; } MEM_STATIC void MEM_write32(void* memPtr, U32 value) { *(U32*)memPtr = value; } MEM_STATIC void MEM_write64(void* memPtr, U64 value) { *(U64*)memPtr = value; } #elif defined(MEM_FORCE_MEMORY_ACCESS) && (MEM_FORCE_MEMORY_ACCESS==1) typedef __attribute__((aligned(1))) U16 unalign16; typedef __attribute__((aligned(1))) U32 unalign32; typedef __attribute__((aligned(1))) U64 unalign64; typedef __attribute__((aligned(1))) size_t unalignArch; MEM_STATIC U16 MEM_read16(const void* ptr) { return *(const unalign16*)ptr; } MEM_STATIC U32 MEM_read32(const void* ptr) { return *(const unalign32*)ptr; } MEM_STATIC U64 MEM_read64(const void* ptr) { return *(const unalign64*)ptr; } MEM_STATIC size_t MEM_readST(const void* ptr) { return *(const unalignArch*)ptr; } MEM_STATIC void MEM_write16(void* memPtr, U16 value) { *(unalign16*)memPtr = value; } MEM_STATIC void MEM_write32(void* memPtr, U32 value) { *(unalign32*)memPtr = value; } MEM_STATIC void MEM_write64(void* memPtr, U64 value) { *(unalign64*)memPtr = value; } #else /* default method, safe and standard. can sometimes prove slower */ MEM_STATIC U16 MEM_read16(const void* memPtr) { U16 val; ZSTD_memcpy(&val, memPtr, sizeof(val)); return val; } MEM_STATIC U32 MEM_read32(const void* memPtr) { U32 val; ZSTD_memcpy(&val, memPtr, sizeof(val)); return val; } MEM_STATIC U64 MEM_read64(const void* memPtr) { U64 val; ZSTD_memcpy(&val, memPtr, sizeof(val)); return val; } MEM_STATIC size_t MEM_readST(const void* memPtr) { size_t val; ZSTD_memcpy(&val, memPtr, sizeof(val)); return val; } MEM_STATIC void MEM_write16(void* memPtr, U16 value) { ZSTD_memcpy(memPtr, &value, sizeof(value)); } MEM_STATIC void MEM_write32(void* memPtr, U32 value) { ZSTD_memcpy(memPtr, &value, sizeof(value)); } MEM_STATIC void MEM_write64(void* memPtr, U64 value) { ZSTD_memcpy(memPtr, &value, sizeof(value)); } #endif /* MEM_FORCE_MEMORY_ACCESS */ MEM_STATIC U32 MEM_swap32_fallback(U32 in) { return ((in << 24) & 0xff000000 ) | ((in << 8) & 0x00ff0000 ) | ((in >> 8) & 0x0000ff00 ) | ((in >> 24) & 0x000000ff ); } MEM_STATIC U32 MEM_swap32(U32 in) { #if defined(_MSC_VER) /* Visual Studio */ return _byteswap_ulong(in); #elif (defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)) \ || (defined(__clang__) && __has_builtin(__builtin_bswap32)) return __builtin_bswap32(in); #else return MEM_swap32_fallback(in); #endif } MEM_STATIC U64 MEM_swap64_fallback(U64 in) { return ((in << 56) & 0xff00000000000000ULL) | ((in << 40) & 0x00ff000000000000ULL) | ((in << 24) & 0x0000ff0000000000ULL) | ((in << 8) & 0x000000ff00000000ULL) | ((in >> 8) & 0x00000000ff000000ULL) | ((in >> 24) & 0x0000000000ff0000ULL) | ((in >> 40) & 0x000000000000ff00ULL) | ((in >> 56) & 0x00000000000000ffULL); } MEM_STATIC U64 MEM_swap64(U64 in) { #if defined(_MSC_VER) /* Visual Studio */ return _byteswap_uint64(in); #elif (defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)) \ || (defined(__clang__) && __has_builtin(__builtin_bswap64)) return __builtin_bswap64(in); #else return MEM_swap64_fallback(in); #endif } MEM_STATIC size_t MEM_swapST(size_t in) { if (MEM_32bits()) return (size_t)MEM_swap32((U32)in); else return (size_t)MEM_swap64((U64)in); } /*=== Little endian r/w ===*/ MEM_STATIC U16 MEM_readLE16(const void* memPtr) { if (MEM_isLittleEndian()) return MEM_read16(memPtr); else { const BYTE* p = (const BYTE*)memPtr; return (U16)(p[0] + (p[1]<<8)); } } MEM_STATIC void MEM_writeLE16(void* memPtr, U16 val) { if (MEM_isLittleEndian()) { MEM_write16(memPtr, val); } else { BYTE* p = (BYTE*)memPtr; p[0] = (BYTE)val; p[1] = (BYTE)(val>>8); } } MEM_STATIC U32 MEM_readLE24(const void* memPtr) { return (U32)MEM_readLE16(memPtr) + ((U32)(((const BYTE*)memPtr)[2]) << 16); } MEM_STATIC void MEM_writeLE24(void* memPtr, U32 val) { MEM_writeLE16(memPtr, (U16)val); ((BYTE*)memPtr)[2] = (BYTE)(val>>16); } MEM_STATIC U32 MEM_readLE32(const void* memPtr) { if (MEM_isLittleEndian()) return MEM_read32(memPtr); else return MEM_swap32(MEM_read32(memPtr)); } MEM_STATIC void MEM_writeLE32(void* memPtr, U32 val32) { if (MEM_isLittleEndian()) MEM_write32(memPtr, val32); else MEM_write32(memPtr, MEM_swap32(val32)); } MEM_STATIC U64 MEM_readLE64(const void* memPtr) { if (MEM_isLittleEndian()) return MEM_read64(memPtr); else return MEM_swap64(MEM_read64(memPtr)); } MEM_STATIC void MEM_writeLE64(void* memPtr, U64 val64) { if (MEM_isLittleEndian()) MEM_write64(memPtr, val64); else MEM_write64(memPtr, MEM_swap64(val64)); } MEM_STATIC size_t MEM_readLEST(const void* memPtr) { if (MEM_32bits()) return (size_t)MEM_readLE32(memPtr); else return (size_t)MEM_readLE64(memPtr); } MEM_STATIC void MEM_writeLEST(void* memPtr, size_t val) { if (MEM_32bits()) MEM_writeLE32(memPtr, (U32)val); else MEM_writeLE64(memPtr, (U64)val); } /*=== Big endian r/w ===*/ MEM_STATIC U32 MEM_readBE32(const void* memPtr) { if (MEM_isLittleEndian()) return MEM_swap32(MEM_read32(memPtr)); else return MEM_read32(memPtr); } MEM_STATIC void MEM_writeBE32(void* memPtr, U32 val32) { if (MEM_isLittleEndian()) MEM_write32(memPtr, MEM_swap32(val32)); else MEM_write32(memPtr, val32); } MEM_STATIC U64 MEM_readBE64(const void* memPtr) { if (MEM_isLittleEndian()) return MEM_swap64(MEM_read64(memPtr)); else return MEM_read64(memPtr); } MEM_STATIC void MEM_writeBE64(void* memPtr, U64 val64) { if (MEM_isLittleEndian()) MEM_write64(memPtr, MEM_swap64(val64)); else MEM_write64(memPtr, val64); } MEM_STATIC size_t MEM_readBEST(const void* memPtr) { if (MEM_32bits()) return (size_t)MEM_readBE32(memPtr); else return (size_t)MEM_readBE64(memPtr); } MEM_STATIC void MEM_writeBEST(void* memPtr, size_t val) { if (MEM_32bits()) MEM_writeBE32(memPtr, (U32)val); else MEM_writeBE64(memPtr, (U64)val); } /* code only tested on 32 and 64 bits systems */ MEM_STATIC void MEM_check(void) { DEBUG_STATIC_ASSERT((sizeof(size_t)==4) || (sizeof(size_t)==8)); } #if defined (__cplusplus) } #endif #endif /* MEM_H_MODULE */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/huf.h0000644000175200007730000003276014515254731023553 0ustar rlaboissrlaboiss/* ****************************************************************** * huff0 huffman codec, * part of Finite State Entropy library * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ #if defined (__cplusplus) extern "C" { #endif #ifndef HUF_H_298734234 #define HUF_H_298734234 /* *** Dependencies *** */ #include "zstd_deps.h" /* size_t */ #include "mem.h" /* U32 */ #define FSE_STATIC_LINKING_ONLY #include "fse.h" /* *** Tool functions *** */ #define HUF_BLOCKSIZE_MAX (128 * 1024) /**< maximum input size for a single block compressed with HUF_compress */ size_t HUF_compressBound(size_t size); /**< maximum compressed size (worst case) */ /* Error Management */ unsigned HUF_isError(size_t code); /**< tells if a return value is an error code */ const char* HUF_getErrorName(size_t code); /**< provides error code string (useful for debugging) */ #define HUF_WORKSPACE_SIZE ((8 << 10) + 512 /* sorting scratch space */) #define HUF_WORKSPACE_SIZE_U64 (HUF_WORKSPACE_SIZE / sizeof(U64)) /* *** Constants *** */ #define HUF_TABLELOG_MAX 12 /* max runtime value of tableLog (due to static allocation); can be modified up to HUF_TABLELOG_ABSOLUTEMAX */ #define HUF_TABLELOG_DEFAULT 11 /* default tableLog value when none specified */ #define HUF_SYMBOLVALUE_MAX 255 #define HUF_TABLELOG_ABSOLUTEMAX 12 /* absolute limit of HUF_MAX_TABLELOG. Beyond that value, code does not work */ #if (HUF_TABLELOG_MAX > HUF_TABLELOG_ABSOLUTEMAX) # error "HUF_TABLELOG_MAX is too large !" #endif /* **************************************** * Static allocation ******************************************/ /* HUF buffer bounds */ #define HUF_CTABLEBOUND 129 #define HUF_BLOCKBOUND(size) (size + (size>>8) + 8) /* only true when incompressible is pre-filtered with fast heuristic */ #define HUF_COMPRESSBOUND(size) (HUF_CTABLEBOUND + HUF_BLOCKBOUND(size)) /* Macro version, useful for static allocation */ /* static allocation of HUF's Compression Table */ /* this is a private definition, just exposed for allocation and strict aliasing purpose. never EVER access its members directly */ typedef size_t HUF_CElt; /* consider it an incomplete type */ #define HUF_CTABLE_SIZE_ST(maxSymbolValue) ((maxSymbolValue)+2) /* Use tables of size_t, for proper alignment */ #define HUF_CTABLE_SIZE(maxSymbolValue) (HUF_CTABLE_SIZE_ST(maxSymbolValue) * sizeof(size_t)) #define HUF_CREATE_STATIC_CTABLE(name, maxSymbolValue) \ HUF_CElt name[HUF_CTABLE_SIZE_ST(maxSymbolValue)] /* no final ; */ /* static allocation of HUF's DTable */ typedef U32 HUF_DTable; #define HUF_DTABLE_SIZE(maxTableLog) (1 + (1<<(maxTableLog))) #define HUF_CREATE_STATIC_DTABLEX1(DTable, maxTableLog) \ HUF_DTable DTable[HUF_DTABLE_SIZE((maxTableLog)-1)] = { ((U32)((maxTableLog)-1) * 0x01000001) } #define HUF_CREATE_STATIC_DTABLEX2(DTable, maxTableLog) \ HUF_DTable DTable[HUF_DTABLE_SIZE(maxTableLog)] = { ((U32)(maxTableLog) * 0x01000001) } /* **************************************** * Advanced decompression functions ******************************************/ /** * Huffman flags bitset. * For all flags, 0 is the default value. */ typedef enum { /** * If compiled with DYNAMIC_BMI2: Set flag only if the CPU supports BMI2 at runtime. * Otherwise: Ignored. */ HUF_flags_bmi2 = (1 << 0), /** * If set: Test possible table depths to find the one that produces the smallest header + encoded size. * If unset: Use heuristic to find the table depth. */ HUF_flags_optimalDepth = (1 << 1), /** * If set: If the previous table can encode the input, always reuse the previous table. * If unset: If the previous table can encode the input, reuse the previous table if it results in a smaller output. */ HUF_flags_preferRepeat = (1 << 2), /** * If set: Sample the input and check if the sample is uncompressible, if it is then don't attempt to compress. * If unset: Always histogram the entire input. */ HUF_flags_suspectUncompressible = (1 << 3), /** * If set: Don't use assembly implementations * If unset: Allow using assembly implementations */ HUF_flags_disableAsm = (1 << 4), /** * If set: Don't use the fast decoding loop, always use the fallback decoding loop. * If unset: Use the fast decoding loop when possible. */ HUF_flags_disableFast = (1 << 5) } HUF_flags_e; /* **************************************** * HUF detailed API * ****************************************/ #define HUF_OPTIMAL_DEPTH_THRESHOLD ZSTD_btultra /*! HUF_compress() does the following: * 1. count symbol occurrence from source[] into table count[] using FSE_count() (exposed within "fse.h") * 2. (optional) refine tableLog using HUF_optimalTableLog() * 3. build Huffman table from count using HUF_buildCTable() * 4. save Huffman table to memory buffer using HUF_writeCTable() * 5. encode the data stream using HUF_compress4X_usingCTable() * * The following API allows targeting specific sub-functions for advanced tasks. * For example, it's possible to compress several blocks using the same 'CTable', * or to save and regenerate 'CTable' using external methods. */ unsigned HUF_minTableLog(unsigned symbolCardinality); unsigned HUF_cardinality(const unsigned* count, unsigned maxSymbolValue); unsigned HUF_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, void* workSpace, size_t wkspSize, HUF_CElt* table, const unsigned* count, int flags); /* table is used as scratch space for building and testing tables, not a return value */ size_t HUF_writeCTable_wksp(void* dst, size_t maxDstSize, const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog, void* workspace, size_t workspaceSize); size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, int flags); size_t HUF_estimateCompressedSize(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue); int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue); typedef enum { HUF_repeat_none, /**< Cannot use the previous table */ HUF_repeat_check, /**< Can use the previous table but it must be checked. Note : The previous table must have been constructed by HUF_compress{1, 4}X_repeat */ HUF_repeat_valid /**< Can use the previous table and it is assumed to be valid */ } HUF_repeat; /** HUF_compress4X_repeat() : * Same as HUF_compress4X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none. * If it uses hufTable it does not modify hufTable or repeat. * If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used. * If preferRepeat then the old table will always be used if valid. * If suspectUncompressible then some sampling checks will be run to potentially skip huffman coding */ size_t HUF_compress4X_repeat(void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize, /**< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE */ HUF_CElt* hufTable, HUF_repeat* repeat, int flags); /** HUF_buildCTable_wksp() : * Same as HUF_buildCTable(), but using externally allocated scratch buffer. * `workSpace` must be aligned on 4-bytes boundaries, and its size must be >= HUF_CTABLE_WORKSPACE_SIZE. */ #define HUF_CTABLE_WORKSPACE_SIZE_U32 ((4 * (HUF_SYMBOLVALUE_MAX + 1)) + 192) #define HUF_CTABLE_WORKSPACE_SIZE (HUF_CTABLE_WORKSPACE_SIZE_U32 * sizeof(unsigned)) size_t HUF_buildCTable_wksp (HUF_CElt* tree, const unsigned* count, U32 maxSymbolValue, U32 maxNbBits, void* workSpace, size_t wkspSize); /*! HUF_readStats() : * Read compact Huffman tree, saved by HUF_writeCTable(). * `huffWeight` is destination buffer. * @return : size read from `src` , or an error Code . * Note : Needed by HUF_readCTable() and HUF_readDTableXn() . */ size_t HUF_readStats(BYTE* huffWeight, size_t hwSize, U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr, const void* src, size_t srcSize); /*! HUF_readStats_wksp() : * Same as HUF_readStats() but takes an external workspace which must be * 4-byte aligned and its size must be >= HUF_READ_STATS_WORKSPACE_SIZE. * If the CPU has BMI2 support, pass bmi2=1, otherwise pass bmi2=0. */ #define HUF_READ_STATS_WORKSPACE_SIZE_U32 FSE_DECOMPRESS_WKSP_SIZE_U32(6, HUF_TABLELOG_MAX-1) #define HUF_READ_STATS_WORKSPACE_SIZE (HUF_READ_STATS_WORKSPACE_SIZE_U32 * sizeof(unsigned)) size_t HUF_readStats_wksp(BYTE* huffWeight, size_t hwSize, U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr, const void* src, size_t srcSize, void* workspace, size_t wkspSize, int flags); /** HUF_readCTable() : * Loading a CTable saved with HUF_writeCTable() */ size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, unsigned *hasZeroWeights); /** HUF_getNbBitsFromCTable() : * Read nbBits from CTable symbolTable, for symbol `symbolValue` presumed <= HUF_SYMBOLVALUE_MAX * Note 1 : is not inlined, as HUF_CElt definition is private */ U32 HUF_getNbBitsFromCTable(const HUF_CElt* symbolTable, U32 symbolValue); /* * HUF_decompress() does the following: * 1. select the decompression algorithm (X1, X2) based on pre-computed heuristics * 2. build Huffman table from save, using HUF_readDTableX?() * 3. decode 1 or 4 segments in parallel using HUF_decompress?X?_usingDTable() */ /** HUF_selectDecoder() : * Tells which decoder is likely to decode faster, * based on a set of pre-computed metrics. * @return : 0==HUF_decompress4X1, 1==HUF_decompress4X2 . * Assumption : 0 < dstSize <= 128 KB */ U32 HUF_selectDecoder (size_t dstSize, size_t cSrcSize); /** * The minimum workspace size for the `workSpace` used in * HUF_readDTableX1_wksp() and HUF_readDTableX2_wksp(). * * The space used depends on HUF_TABLELOG_MAX, ranging from ~1500 bytes when * HUF_TABLE_LOG_MAX=12 to ~1850 bytes when HUF_TABLE_LOG_MAX=15. * Buffer overflow errors may potentially occur if code modifications result in * a required workspace size greater than that specified in the following * macro. */ #define HUF_DECOMPRESS_WORKSPACE_SIZE ((2 << 10) + (1 << 9)) #define HUF_DECOMPRESS_WORKSPACE_SIZE_U32 (HUF_DECOMPRESS_WORKSPACE_SIZE / sizeof(U32)) /* ====================== */ /* single stream variants */ /* ====================== */ size_t HUF_compress1X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, int flags); /** HUF_compress1X_repeat() : * Same as HUF_compress1X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none. * If it uses hufTable it does not modify hufTable or repeat. * If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used. * If preferRepeat then the old table will always be used if valid. * If suspectUncompressible then some sampling checks will be run to potentially skip huffman coding */ size_t HUF_compress1X_repeat(void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize, /**< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE */ HUF_CElt* hufTable, HUF_repeat* repeat, int flags); size_t HUF_decompress1X_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags); #ifndef HUF_FORCE_DECOMPRESS_X1 size_t HUF_decompress1X2_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags); /**< double-symbols decoder */ #endif /* BMI2 variants. * If the CPU has BMI2 support, pass bmi2=1, otherwise pass bmi2=0. */ size_t HUF_decompress1X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int flags); #ifndef HUF_FORCE_DECOMPRESS_X2 size_t HUF_decompress1X1_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags); #endif size_t HUF_decompress4X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int flags); size_t HUF_decompress4X_hufOnly_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags); #ifndef HUF_FORCE_DECOMPRESS_X2 size_t HUF_readDTableX1_wksp(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int flags); #endif #ifndef HUF_FORCE_DECOMPRESS_X1 size_t HUF_readDTableX2_wksp(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int flags); #endif #endif /* HUF_H_298734234 */ #if defined (__cplusplus) } #endif zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/xxhash.c0000644000175200007730000000137714515254731024267 0ustar rlaboissrlaboiss/* * xxHash - Fast Hash algorithm * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - xxHash homepage: https://cyan4973.github.io/xxHash/ * - xxHash source repository : https://github.com/Cyan4973/xxHash * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* * xxhash.c instantiates functions defined in xxhash.h */ #define XXH_STATIC_LINKING_ONLY /* access advanced declarations */ #define XXH_IMPLEMENTATION /* access definitions */ #include "xxhash.h" zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/allocations.h0000644000175200007730000000314014515254731025267 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* This file provides custom allocation primitives */ #define ZSTD_DEPS_NEED_MALLOC #include "zstd_deps.h" /* ZSTD_malloc, ZSTD_calloc, ZSTD_free, ZSTD_memset */ #include "mem.h" /* MEM_STATIC */ #define ZSTD_STATIC_LINKING_ONLY #include "../zstd.h" /* ZSTD_customMem */ #ifndef ZSTD_ALLOCATIONS_H #define ZSTD_ALLOCATIONS_H /* custom memory allocation functions */ MEM_STATIC void* ZSTD_customMalloc(size_t size, ZSTD_customMem customMem) { if (customMem.customAlloc) return customMem.customAlloc(customMem.opaque, size); return ZSTD_malloc(size); } MEM_STATIC void* ZSTD_customCalloc(size_t size, ZSTD_customMem customMem) { if (customMem.customAlloc) { /* calloc implemented as malloc+memset; * not as efficient as calloc, but next best guess for custom malloc */ void* const ptr = customMem.customAlloc(customMem.opaque, size); ZSTD_memset(ptr, 0, size); return ptr; } return ZSTD_calloc(1, size); } MEM_STATIC void ZSTD_customFree(void* ptr, ZSTD_customMem customMem) { if (ptr!=NULL) { if (customMem.customFree) customMem.customFree(customMem.opaque, ptr); else ZSTD_free(ptr); } } #endif /* ZSTD_ALLOCATIONS_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/bits.h0000644000175200007730000001412214515254731023722 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_BITS_H #define ZSTD_BITS_H #include "mem.h" MEM_STATIC unsigned ZSTD_countTrailingZeros32_fallback(U32 val) { assert(val != 0); { static const U32 DeBruijnBytePos[32] = {0, 1, 28, 2, 29, 14, 24, 3, 30, 22, 20, 15, 25, 17, 4, 8, 31, 27, 13, 23, 21, 19, 16, 7, 26, 12, 18, 6, 11, 5, 10, 9}; return DeBruijnBytePos[((U32) ((val & -(S32) val) * 0x077CB531U)) >> 27]; } } MEM_STATIC unsigned ZSTD_countTrailingZeros32(U32 val) { assert(val != 0); # if defined(_MSC_VER) # if STATIC_BMI2 == 1 return (unsigned)_tzcnt_u32(val); # else if (val != 0) { unsigned long r; _BitScanForward(&r, val); return (unsigned)r; } else { /* Should not reach this code path */ __assume(0); } # endif # elif defined(__GNUC__) && (__GNUC__ >= 4) return (unsigned)__builtin_ctz(val); # else return ZSTD_countTrailingZeros32_fallback(val); # endif } MEM_STATIC unsigned ZSTD_countLeadingZeros32_fallback(U32 val) { assert(val != 0); { static const U32 DeBruijnClz[32] = {0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30, 8, 12, 20, 28, 15, 17, 24, 7, 19, 27, 23, 6, 26, 5, 4, 31}; val |= val >> 1; val |= val >> 2; val |= val >> 4; val |= val >> 8; val |= val >> 16; return 31 - DeBruijnClz[(val * 0x07C4ACDDU) >> 27]; } } MEM_STATIC unsigned ZSTD_countLeadingZeros32(U32 val) { assert(val != 0); # if defined(_MSC_VER) # if STATIC_BMI2 == 1 return (unsigned)_lzcnt_u32(val); # else if (val != 0) { unsigned long r; _BitScanReverse(&r, val); return (unsigned)(31 - r); } else { /* Should not reach this code path */ __assume(0); } # endif # elif defined(__GNUC__) && (__GNUC__ >= 4) return (unsigned)__builtin_clz(val); # else return ZSTD_countLeadingZeros32_fallback(val); # endif } MEM_STATIC unsigned ZSTD_countTrailingZeros64(U64 val) { assert(val != 0); # if defined(_MSC_VER) && defined(_WIN64) # if STATIC_BMI2 == 1 return (unsigned)_tzcnt_u64(val); # else if (val != 0) { unsigned long r; _BitScanForward64(&r, val); return (unsigned)r; } else { /* Should not reach this code path */ __assume(0); } # endif # elif defined(__GNUC__) && (__GNUC__ >= 4) && defined(__LP64__) return (unsigned)__builtin_ctzll(val); # else { U32 mostSignificantWord = (U32)(val >> 32); U32 leastSignificantWord = (U32)val; if (leastSignificantWord == 0) { return 32 + ZSTD_countTrailingZeros32(mostSignificantWord); } else { return ZSTD_countTrailingZeros32(leastSignificantWord); } } # endif } MEM_STATIC unsigned ZSTD_countLeadingZeros64(U64 val) { assert(val != 0); # if defined(_MSC_VER) && defined(_WIN64) # if STATIC_BMI2 == 1 return (unsigned)_lzcnt_u64(val); # else if (val != 0) { unsigned long r; _BitScanReverse64(&r, val); return (unsigned)(63 - r); } else { /* Should not reach this code path */ __assume(0); } # endif # elif defined(__GNUC__) && (__GNUC__ >= 4) return (unsigned)(__builtin_clzll(val)); # else { U32 mostSignificantWord = (U32)(val >> 32); U32 leastSignificantWord = (U32)val; if (mostSignificantWord == 0) { return 32 + ZSTD_countLeadingZeros32(leastSignificantWord); } else { return ZSTD_countLeadingZeros32(mostSignificantWord); } } # endif } MEM_STATIC unsigned ZSTD_NbCommonBytes(size_t val) { if (MEM_isLittleEndian()) { if (MEM_64bits()) { return ZSTD_countTrailingZeros64((U64)val) >> 3; } else { return ZSTD_countTrailingZeros32((U32)val) >> 3; } } else { /* Big Endian CPU */ if (MEM_64bits()) { return ZSTD_countLeadingZeros64((U64)val) >> 3; } else { return ZSTD_countLeadingZeros32((U32)val) >> 3; } } } MEM_STATIC unsigned ZSTD_highbit32(U32 val) /* compress, dictBuilder, decodeCorpus */ { assert(val != 0); return 31 - ZSTD_countLeadingZeros32(val); } /* ZSTD_rotateRight_*(): * Rotates a bitfield to the right by "count" bits. * https://en.wikipedia.org/w/index.php?title=Circular_shift&oldid=991635599#Implementing_circular_shifts */ MEM_STATIC U64 ZSTD_rotateRight_U64(U64 const value, U32 count) { assert(count < 64); count &= 0x3F; /* for fickle pattern recognition */ return (value >> count) | (U64)(value << ((0U - count) & 0x3F)); } MEM_STATIC U32 ZSTD_rotateRight_U32(U32 const value, U32 count) { assert(count < 32); count &= 0x1F; /* for fickle pattern recognition */ return (value >> count) | (U32)(value << ((0U - count) & 0x1F)); } MEM_STATIC U16 ZSTD_rotateRight_U16(U16 const value, U32 count) { assert(count < 16); count &= 0x0F; /* for fickle pattern recognition */ return (value >> count) | (U16)(value << ((0U - count) & 0x0F)); } #endif /* ZSTD_BITS_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/compiler.h0000644000175200007730000003132414515254731024576 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_COMPILER_H #define ZSTD_COMPILER_H #include "portability_macros.h" /*-******************************************************* * Compiler specifics *********************************************************/ /* force inlining */ #if !defined(ZSTD_NO_INLINE) #if (defined(__GNUC__) && !defined(__STRICT_ANSI__)) || defined(__cplusplus) || defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */ # define INLINE_KEYWORD inline #else # define INLINE_KEYWORD #endif #if defined(__GNUC__) || defined(__ICCARM__) # define FORCE_INLINE_ATTR __attribute__((always_inline)) #elif defined(_MSC_VER) # define FORCE_INLINE_ATTR __forceinline #else # define FORCE_INLINE_ATTR #endif #else #define INLINE_KEYWORD #define FORCE_INLINE_ATTR #endif /** On MSVC qsort requires that functions passed into it use the __cdecl calling conversion(CC). This explicitly marks such functions as __cdecl so that the code will still compile if a CC other than __cdecl has been made the default. */ #if defined(_MSC_VER) # define WIN_CDECL __cdecl #else # define WIN_CDECL #endif /** * FORCE_INLINE_TEMPLATE is used to define C "templates", which take constant * parameters. They must be inlined for the compiler to eliminate the constant * branches. */ #define FORCE_INLINE_TEMPLATE static INLINE_KEYWORD FORCE_INLINE_ATTR /** * HINT_INLINE is used to help the compiler generate better code. It is *not* * used for "templates", so it can be tweaked based on the compilers * performance. * * gcc-4.8 and gcc-4.9 have been shown to benefit from leaving off the * always_inline attribute. * * clang up to 5.0.0 (trunk) benefit tremendously from the always_inline * attribute. */ #if !defined(__clang__) && defined(__GNUC__) && __GNUC__ >= 4 && __GNUC_MINOR__ >= 8 && __GNUC__ < 5 # define HINT_INLINE static INLINE_KEYWORD #else # define HINT_INLINE static INLINE_KEYWORD FORCE_INLINE_ATTR #endif /* UNUSED_ATTR tells the compiler it is okay if the function is unused. */ #if defined(__GNUC__) # define UNUSED_ATTR __attribute__((unused)) #else # define UNUSED_ATTR #endif /* force no inlining */ #ifdef _MSC_VER # define FORCE_NOINLINE static __declspec(noinline) #else # if defined(__GNUC__) || defined(__ICCARM__) # define FORCE_NOINLINE static __attribute__((__noinline__)) # else # define FORCE_NOINLINE static # endif #endif /* target attribute */ #if defined(__GNUC__) || defined(__ICCARM__) # define TARGET_ATTRIBUTE(target) __attribute__((__target__(target))) #else # define TARGET_ATTRIBUTE(target) #endif /* Target attribute for BMI2 dynamic dispatch. * Enable lzcnt, bmi, and bmi2. * We test for bmi1 & bmi2. lzcnt is included in bmi1. */ #define BMI2_TARGET_ATTRIBUTE TARGET_ATTRIBUTE("lzcnt,bmi,bmi2") /* prefetch * can be disabled, by declaring NO_PREFETCH build macro */ #if defined(NO_PREFETCH) # define PREFETCH_L1(ptr) (void)(ptr) /* disabled */ # define PREFETCH_L2(ptr) (void)(ptr) /* disabled */ #else # if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_I86)) /* _mm_prefetch() is not defined outside of x86/x64 */ # include /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */ # define PREFETCH_L1(ptr) _mm_prefetch((const char*)(ptr), _MM_HINT_T0) # define PREFETCH_L2(ptr) _mm_prefetch((const char*)(ptr), _MM_HINT_T1) # elif defined(__GNUC__) && ( (__GNUC__ >= 4) || ( (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) ) ) # define PREFETCH_L1(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 3 /* locality */) # define PREFETCH_L2(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 2 /* locality */) # elif defined(__aarch64__) # define PREFETCH_L1(ptr) __asm__ __volatile__("prfm pldl1keep, %0" ::"Q"(*(ptr))) # define PREFETCH_L2(ptr) __asm__ __volatile__("prfm pldl2keep, %0" ::"Q"(*(ptr))) # else # define PREFETCH_L1(ptr) (void)(ptr) /* disabled */ # define PREFETCH_L2(ptr) (void)(ptr) /* disabled */ # endif #endif /* NO_PREFETCH */ #define CACHELINE_SIZE 64 #define PREFETCH_AREA(p, s) { \ const char* const _ptr = (const char*)(p); \ size_t const _size = (size_t)(s); \ size_t _pos; \ for (_pos=0; _pos<_size; _pos+=CACHELINE_SIZE) { \ PREFETCH_L2(_ptr + _pos); \ } \ } /* vectorization * older GCC (pre gcc-4.3 picked as the cutoff) uses a different syntax, * and some compilers, like Intel ICC and MCST LCC, do not support it at all. */ #if !defined(__INTEL_COMPILER) && !defined(__clang__) && defined(__GNUC__) && !defined(__LCC__) # if (__GNUC__ == 4 && __GNUC_MINOR__ > 3) || (__GNUC__ >= 5) # define DONT_VECTORIZE __attribute__((optimize("no-tree-vectorize"))) # else # define DONT_VECTORIZE _Pragma("GCC optimize(\"no-tree-vectorize\")") # endif #else # define DONT_VECTORIZE #endif /* Tell the compiler that a branch is likely or unlikely. * Only use these macros if it causes the compiler to generate better code. * If you can remove a LIKELY/UNLIKELY annotation without speed changes in gcc * and clang, please do. */ #if defined(__GNUC__) #define LIKELY(x) (__builtin_expect((x), 1)) #define UNLIKELY(x) (__builtin_expect((x), 0)) #else #define LIKELY(x) (x) #define UNLIKELY(x) (x) #endif #if __has_builtin(__builtin_unreachable) || (defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 5))) # define ZSTD_UNREACHABLE { assert(0), __builtin_unreachable(); } #else # define ZSTD_UNREACHABLE { assert(0); } #endif /* disable warnings */ #ifdef _MSC_VER /* Visual Studio */ # include /* For Visual 2005 */ # pragma warning(disable : 4100) /* disable: C4100: unreferenced formal parameter */ # pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ # pragma warning(disable : 4204) /* disable: C4204: non-constant aggregate initializer */ # pragma warning(disable : 4214) /* disable: C4214: non-int bitfields */ # pragma warning(disable : 4324) /* disable: C4324: padded structure */ #endif /*Like DYNAMIC_BMI2 but for compile time determination of BMI2 support*/ #ifndef STATIC_BMI2 # if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_I86)) # ifdef __AVX2__ //MSVC does not have a BMI2 specific flag, but every CPU that supports AVX2 also supports BMI2 # define STATIC_BMI2 1 # endif # elif defined(__BMI2__) && defined(__x86_64__) && defined(__GNUC__) # define STATIC_BMI2 1 # endif #endif #ifndef STATIC_BMI2 #define STATIC_BMI2 0 #endif /* compile time determination of SIMD support */ #if !defined(ZSTD_NO_INTRINSICS) # if defined(__SSE2__) || defined(_M_AMD64) || (defined (_M_IX86) && defined(_M_IX86_FP) && (_M_IX86_FP >= 2)) # define ZSTD_ARCH_X86_SSE2 # endif # if defined(__ARM_NEON) || defined(_M_ARM64) # define ZSTD_ARCH_ARM_NEON # endif # # if defined(ZSTD_ARCH_X86_SSE2) # include # elif defined(ZSTD_ARCH_ARM_NEON) # include # endif #endif /* C-language Attributes are added in C23. */ #if defined(__STDC_VERSION__) && (__STDC_VERSION__ > 201710L) && defined(__has_c_attribute) # define ZSTD_HAS_C_ATTRIBUTE(x) __has_c_attribute(x) #else # define ZSTD_HAS_C_ATTRIBUTE(x) 0 #endif /* Only use C++ attributes in C++. Some compilers report support for C++ * attributes when compiling with C. */ #if defined(__cplusplus) && defined(__has_cpp_attribute) # define ZSTD_HAS_CPP_ATTRIBUTE(x) __has_cpp_attribute(x) #else # define ZSTD_HAS_CPP_ATTRIBUTE(x) 0 #endif /* Define ZSTD_FALLTHROUGH macro for annotating switch case with the 'fallthrough' attribute. * - C23: https://en.cppreference.com/w/c/language/attributes/fallthrough * - CPP17: https://en.cppreference.com/w/cpp/language/attributes/fallthrough * - Else: __attribute__((__fallthrough__)) */ #ifndef ZSTD_FALLTHROUGH # if ZSTD_HAS_C_ATTRIBUTE(fallthrough) # define ZSTD_FALLTHROUGH [[fallthrough]] # elif ZSTD_HAS_CPP_ATTRIBUTE(fallthrough) # define ZSTD_FALLTHROUGH [[fallthrough]] # elif __has_attribute(__fallthrough__) /* Leading semicolon is to satisfy gcc-11 with -pedantic. Without the semicolon * gcc complains about: a label can only be part of a statement and a declaration is not a statement. */ # define ZSTD_FALLTHROUGH ; __attribute__((__fallthrough__)) # else # define ZSTD_FALLTHROUGH # endif #endif /*-************************************************************** * Alignment check *****************************************************************/ /* this test was initially positioned in mem.h, * but this file is removed (or replaced) for linux kernel * so it's now hosted in compiler.h, * which remains valid for both user & kernel spaces. */ #ifndef ZSTD_ALIGNOF # if defined(__GNUC__) || defined(_MSC_VER) /* covers gcc, clang & MSVC */ /* note : this section must come first, before C11, * due to a limitation in the kernel source generator */ # define ZSTD_ALIGNOF(T) __alignof(T) # elif defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L) /* C11 support */ # include # define ZSTD_ALIGNOF(T) alignof(T) # else /* No known support for alignof() - imperfect backup */ # define ZSTD_ALIGNOF(T) (sizeof(void*) < sizeof(T) ? sizeof(void*) : sizeof(T)) # endif #endif /* ZSTD_ALIGNOF */ /*-************************************************************** * Sanitizer *****************************************************************/ /* Issue #3240 reports an ASAN failure on an llvm-mingw build. Out of an * abundance of caution, disable our custom poisoning on mingw. */ #ifdef __MINGW32__ #ifndef ZSTD_ASAN_DONT_POISON_WORKSPACE #define ZSTD_ASAN_DONT_POISON_WORKSPACE 1 #endif #ifndef ZSTD_MSAN_DONT_POISON_WORKSPACE #define ZSTD_MSAN_DONT_POISON_WORKSPACE 1 #endif #endif #if ZSTD_MEMORY_SANITIZER && !defined(ZSTD_MSAN_DONT_POISON_WORKSPACE) /* Not all platforms that support msan provide sanitizers/msan_interface.h. * We therefore declare the functions we need ourselves, rather than trying to * include the header file... */ #include /* size_t */ #define ZSTD_DEPS_NEED_STDINT #include "zstd_deps.h" /* intptr_t */ /* Make memory region fully initialized (without changing its contents). */ void __msan_unpoison(const volatile void *a, size_t size); /* Make memory region fully uninitialized (without changing its contents). This is a legacy interface that does not update origin information. Use __msan_allocated_memory() instead. */ void __msan_poison(const volatile void *a, size_t size); /* Returns the offset of the first (at least partially) poisoned byte in the memory range, or -1 if the whole range is good. */ intptr_t __msan_test_shadow(const volatile void *x, size_t size); /* Print shadow and origin for the memory range to stderr in a human-readable format. */ void __msan_print_shadow(const volatile void *x, size_t size); #endif #if ZSTD_ADDRESS_SANITIZER && !defined(ZSTD_ASAN_DONT_POISON_WORKSPACE) /* Not all platforms that support asan provide sanitizers/asan_interface.h. * We therefore declare the functions we need ourselves, rather than trying to * include the header file... */ #include /* size_t */ /** * Marks a memory region ([addr, addr+size)) as unaddressable. * * This memory must be previously allocated by your program. Instrumented * code is forbidden from accessing addresses in this region until it is * unpoisoned. This function is not guaranteed to poison the entire region - * it could poison only a subregion of [addr, addr+size) due to ASan * alignment restrictions. * * \note This function is not thread-safe because no two threads can poison or * unpoison memory in the same memory region simultaneously. * * \param addr Start of memory region. * \param size Size of memory region. */ void __asan_poison_memory_region(void const volatile *addr, size_t size); /** * Marks a memory region ([addr, addr+size)) as addressable. * * This memory must be previously allocated by your program. Accessing * addresses in this region is allowed until this region is poisoned again. * This function could unpoison a super-region of [addr, addr+size) due * to ASan alignment restrictions. * * \note This function is not thread-safe because no two threads can * poison or unpoison memory in the same memory region simultaneously. * * \param addr Start of memory region. * \param size Size of memory region. */ void __asan_unpoison_memory_region(void const volatile *addr, size_t size); #endif #endif /* ZSTD_COMPILER_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/zstd_internal.h0000644000175200007730000003203614515254731025645 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_CCOMMON_H_MODULE #define ZSTD_CCOMMON_H_MODULE /* this module contains definitions which must be identical * across compression, decompression and dictBuilder. * It also contains a few functions useful to at least 2 of them * and which benefit from being inlined */ /*-************************************* * Dependencies ***************************************/ #include "compiler.h" #include "cpu.h" #include "mem.h" #include "debug.h" /* assert, DEBUGLOG, RAWLOG, g_debuglevel */ #include "error_private.h" #define ZSTD_STATIC_LINKING_ONLY #include "../zstd.h" #define FSE_STATIC_LINKING_ONLY #include "fse.h" #include "huf.h" #ifndef XXH_STATIC_LINKING_ONLY # define XXH_STATIC_LINKING_ONLY /* XXH64_state_t */ #endif #include "xxhash.h" /* XXH_reset, update, digest */ #ifndef ZSTD_NO_TRACE # include "zstd_trace.h" #else # define ZSTD_TRACE 0 #endif #if defined (__cplusplus) extern "C" { #endif /* ---- static assert (debug) --- */ #define ZSTD_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c) #define ZSTD_isError ERR_isError /* for inlining */ #define FSE_isError ERR_isError #define HUF_isError ERR_isError /*-************************************* * shared macros ***************************************/ #undef MIN #undef MAX #define MIN(a,b) ((a)<(b) ? (a) : (b)) #define MAX(a,b) ((a)>(b) ? (a) : (b)) #define BOUNDED(min,val,max) (MAX(min,MIN(val,max))) /*-************************************* * Common constants ***************************************/ #define ZSTD_OPT_NUM (1<<12) #define ZSTD_REP_NUM 3 /* number of repcodes */ static UNUSED_ATTR const U32 repStartValue[ZSTD_REP_NUM] = { 1, 4, 8 }; #define KB *(1 <<10) #define MB *(1 <<20) #define GB *(1U<<30) #define BIT7 128 #define BIT6 64 #define BIT5 32 #define BIT4 16 #define BIT1 2 #define BIT0 1 #define ZSTD_WINDOWLOG_ABSOLUTEMIN 10 static UNUSED_ATTR const size_t ZSTD_fcs_fieldSize[4] = { 0, 2, 4, 8 }; static UNUSED_ATTR const size_t ZSTD_did_fieldSize[4] = { 0, 1, 2, 4 }; #define ZSTD_FRAMEIDSIZE 4 /* magic number size */ #define ZSTD_BLOCKHEADERSIZE 3 /* C standard doesn't allow `static const` variable to be init using another `static const` variable */ static UNUSED_ATTR const size_t ZSTD_blockHeaderSize = ZSTD_BLOCKHEADERSIZE; typedef enum { bt_raw, bt_rle, bt_compressed, bt_reserved } blockType_e; #define ZSTD_FRAMECHECKSUMSIZE 4 #define MIN_SEQUENCES_SIZE 1 /* nbSeq==0 */ #define MIN_CBLOCK_SIZE (1 /*litCSize*/ + 1 /* RLE or RAW */) /* for a non-null block */ #define MIN_LITERALS_FOR_4_STREAMS 6 typedef enum { set_basic, set_rle, set_compressed, set_repeat } symbolEncodingType_e; #define LONGNBSEQ 0x7F00 #define MINMATCH 3 #define Litbits 8 #define LitHufLog 11 #define MaxLit ((1<= WILDCOPY_VECLEN || diff <= -WILDCOPY_VECLEN); /* Separate out the first COPY16() call because the copy length is * almost certain to be short, so the branches have different * probabilities. Since it is almost certain to be short, only do * one COPY16() in the first call. Then, do two calls per loop since * at that point it is more likely to have a high trip count. */ ZSTD_copy16(op, ip); if (16 >= length) return; op += 16; ip += 16; do { COPY16(op, ip); COPY16(op, ip); } while (op < oend); } } MEM_STATIC size_t ZSTD_limitCopy(void* dst, size_t dstCapacity, const void* src, size_t srcSize) { size_t const length = MIN(dstCapacity, srcSize); if (length > 0) { ZSTD_memcpy(dst, src, length); } return length; } /* define "workspace is too large" as this number of times larger than needed */ #define ZSTD_WORKSPACETOOLARGE_FACTOR 3 /* when workspace is continuously too large * during at least this number of times, * context's memory usage is considered wasteful, * because it's sized to handle a worst case scenario which rarely happens. * In which case, resize it down to free some memory */ #define ZSTD_WORKSPACETOOLARGE_MAXDURATION 128 /* Controls whether the input/output buffer is buffered or stable. */ typedef enum { ZSTD_bm_buffered = 0, /* Buffer the input/output */ ZSTD_bm_stable = 1 /* ZSTD_inBuffer/ZSTD_outBuffer is stable */ } ZSTD_bufferMode_e; /*-******************************************* * Private declarations *********************************************/ typedef struct seqDef_s { U32 offBase; /* offBase == Offset + ZSTD_REP_NUM, or repcode 1,2,3 */ U16 litLength; U16 mlBase; /* mlBase == matchLength - MINMATCH */ } seqDef; /* Controls whether seqStore has a single "long" litLength or matchLength. See seqStore_t. */ typedef enum { ZSTD_llt_none = 0, /* no longLengthType */ ZSTD_llt_literalLength = 1, /* represents a long literal */ ZSTD_llt_matchLength = 2 /* represents a long match */ } ZSTD_longLengthType_e; typedef struct { seqDef* sequencesStart; seqDef* sequences; /* ptr to end of sequences */ BYTE* litStart; BYTE* lit; /* ptr to end of literals */ BYTE* llCode; BYTE* mlCode; BYTE* ofCode; size_t maxNbSeq; size_t maxNbLit; /* longLengthPos and longLengthType to allow us to represent either a single litLength or matchLength * in the seqStore that has a value larger than U16 (if it exists). To do so, we increment * the existing value of the litLength or matchLength by 0x10000. */ ZSTD_longLengthType_e longLengthType; U32 longLengthPos; /* Index of the sequence to apply long length modification to */ } seqStore_t; typedef struct { U32 litLength; U32 matchLength; } ZSTD_sequenceLength; /** * Returns the ZSTD_sequenceLength for the given sequences. It handles the decoding of long sequences * indicated by longLengthPos and longLengthType, and adds MINMATCH back to matchLength. */ MEM_STATIC ZSTD_sequenceLength ZSTD_getSequenceLength(seqStore_t const* seqStore, seqDef const* seq) { ZSTD_sequenceLength seqLen; seqLen.litLength = seq->litLength; seqLen.matchLength = seq->mlBase + MINMATCH; if (seqStore->longLengthPos == (U32)(seq - seqStore->sequencesStart)) { if (seqStore->longLengthType == ZSTD_llt_literalLength) { seqLen.litLength += 0x10000; } if (seqStore->longLengthType == ZSTD_llt_matchLength) { seqLen.matchLength += 0x10000; } } return seqLen; } /** * Contains the compressed frame size and an upper-bound for the decompressed frame size. * Note: before using `compressedSize`, check for errors using ZSTD_isError(). * similarly, before using `decompressedBound`, check for errors using: * `decompressedBound != ZSTD_CONTENTSIZE_ERROR` */ typedef struct { size_t nbBlocks; size_t compressedSize; unsigned long long decompressedBound; } ZSTD_frameSizeInfo; /* decompress & legacy */ const seqStore_t* ZSTD_getSeqStore(const ZSTD_CCtx* ctx); /* compress & dictBuilder */ int ZSTD_seqToCodes(const seqStore_t* seqStorePtr); /* compress, dictBuilder, decodeCorpus (shouldn't get its definition from here) */ /* ZSTD_invalidateRepCodes() : * ensures next compression will not use repcodes from previous block. * Note : only works with regular variant; * do not use with extDict variant ! */ void ZSTD_invalidateRepCodes(ZSTD_CCtx* cctx); /* zstdmt, adaptive_compression (shouldn't get this definition from here) */ typedef struct { blockType_e blockType; U32 lastBlock; U32 origSize; } blockProperties_t; /* declared here for decompress and fullbench */ /*! ZSTD_getcBlockSize() : * Provides the size of compressed block from block header `src` */ /* Used by: decompress, fullbench (does not get its definition from here) */ size_t ZSTD_getcBlockSize(const void* src, size_t srcSize, blockProperties_t* bpPtr); /*! ZSTD_decodeSeqHeaders() : * decode sequence header from src */ /* Used by: decompress, fullbench (does not get its definition from here) */ size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr, const void* src, size_t srcSize); /** * @returns true iff the CPU supports dynamic BMI2 dispatch. */ MEM_STATIC int ZSTD_cpuSupportsBmi2(void) { ZSTD_cpuid_t cpuid = ZSTD_cpuid(); return ZSTD_cpuid_bmi1(cpuid) && ZSTD_cpuid_bmi2(cpuid); } #if defined (__cplusplus) } #endif #endif /* ZSTD_CCOMMON_H_MODULE */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/pool.c0000644000175200007730000002700314515254731023727 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* ====== Dependencies ======= */ #include "../common/allocations.h" /* ZSTD_customCalloc, ZSTD_customFree */ #include "zstd_deps.h" /* size_t */ #include "debug.h" /* assert */ #include "pool.h" /* ====== Compiler specifics ====== */ #if defined(_MSC_VER) # pragma warning(disable : 4204) /* disable: C4204: non-constant aggregate initializer */ #endif #ifdef ZSTD_MULTITHREAD #include "threading.h" /* pthread adaptation */ /* A job is a function and an opaque argument */ typedef struct POOL_job_s { POOL_function function; void *opaque; } POOL_job; struct POOL_ctx_s { ZSTD_customMem customMem; /* Keep track of the threads */ ZSTD_pthread_t* threads; size_t threadCapacity; size_t threadLimit; /* The queue is a circular buffer */ POOL_job *queue; size_t queueHead; size_t queueTail; size_t queueSize; /* The number of threads working on jobs */ size_t numThreadsBusy; /* Indicates if the queue is empty */ int queueEmpty; /* The mutex protects the queue */ ZSTD_pthread_mutex_t queueMutex; /* Condition variable for pushers to wait on when the queue is full */ ZSTD_pthread_cond_t queuePushCond; /* Condition variables for poppers to wait on when the queue is empty */ ZSTD_pthread_cond_t queuePopCond; /* Indicates if the queue is shutting down */ int shutdown; }; /* POOL_thread() : * Work thread for the thread pool. * Waits for jobs and executes them. * @returns : NULL on failure else non-null. */ static void* POOL_thread(void* opaque) { POOL_ctx* const ctx = (POOL_ctx*)opaque; if (!ctx) { return NULL; } for (;;) { /* Lock the mutex and wait for a non-empty queue or until shutdown */ ZSTD_pthread_mutex_lock(&ctx->queueMutex); while ( ctx->queueEmpty || (ctx->numThreadsBusy >= ctx->threadLimit) ) { if (ctx->shutdown) { /* even if !queueEmpty, (possible if numThreadsBusy >= threadLimit), * a few threads will be shutdown while !queueEmpty, * but enough threads will remain active to finish the queue */ ZSTD_pthread_mutex_unlock(&ctx->queueMutex); return opaque; } ZSTD_pthread_cond_wait(&ctx->queuePopCond, &ctx->queueMutex); } /* Pop a job off the queue */ { POOL_job const job = ctx->queue[ctx->queueHead]; ctx->queueHead = (ctx->queueHead + 1) % ctx->queueSize; ctx->numThreadsBusy++; ctx->queueEmpty = (ctx->queueHead == ctx->queueTail); /* Unlock the mutex, signal a pusher, and run the job */ ZSTD_pthread_cond_signal(&ctx->queuePushCond); ZSTD_pthread_mutex_unlock(&ctx->queueMutex); job.function(job.opaque); /* If the intended queue size was 0, signal after finishing job */ ZSTD_pthread_mutex_lock(&ctx->queueMutex); ctx->numThreadsBusy--; ZSTD_pthread_cond_signal(&ctx->queuePushCond); ZSTD_pthread_mutex_unlock(&ctx->queueMutex); } } /* for (;;) */ assert(0); /* Unreachable */ } /* ZSTD_createThreadPool() : public access point */ POOL_ctx* ZSTD_createThreadPool(size_t numThreads) { return POOL_create (numThreads, 0); } POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) { return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem); } POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem) { POOL_ctx* ctx; /* Check parameters */ if (!numThreads) { return NULL; } /* Allocate the context and zero initialize */ ctx = (POOL_ctx*)ZSTD_customCalloc(sizeof(POOL_ctx), customMem); if (!ctx) { return NULL; } /* Initialize the job queue. * It needs one extra space since one space is wasted to differentiate * empty and full queues. */ ctx->queueSize = queueSize + 1; ctx->queue = (POOL_job*)ZSTD_customCalloc(ctx->queueSize * sizeof(POOL_job), customMem); ctx->queueHead = 0; ctx->queueTail = 0; ctx->numThreadsBusy = 0; ctx->queueEmpty = 1; { int error = 0; error |= ZSTD_pthread_mutex_init(&ctx->queueMutex, NULL); error |= ZSTD_pthread_cond_init(&ctx->queuePushCond, NULL); error |= ZSTD_pthread_cond_init(&ctx->queuePopCond, NULL); if (error) { POOL_free(ctx); return NULL; } } ctx->shutdown = 0; /* Allocate space for the thread handles */ ctx->threads = (ZSTD_pthread_t*)ZSTD_customCalloc(numThreads * sizeof(ZSTD_pthread_t), customMem); ctx->threadCapacity = 0; ctx->customMem = customMem; /* Check for errors */ if (!ctx->threads || !ctx->queue) { POOL_free(ctx); return NULL; } /* Initialize the threads */ { size_t i; for (i = 0; i < numThreads; ++i) { if (ZSTD_pthread_create(&ctx->threads[i], NULL, &POOL_thread, ctx)) { ctx->threadCapacity = i; POOL_free(ctx); return NULL; } } ctx->threadCapacity = numThreads; ctx->threadLimit = numThreads; } return ctx; } /*! POOL_join() : Shutdown the queue, wake any sleeping threads, and join all of the threads. */ static void POOL_join(POOL_ctx* ctx) { /* Shut down the queue */ ZSTD_pthread_mutex_lock(&ctx->queueMutex); ctx->shutdown = 1; ZSTD_pthread_mutex_unlock(&ctx->queueMutex); /* Wake up sleeping threads */ ZSTD_pthread_cond_broadcast(&ctx->queuePushCond); ZSTD_pthread_cond_broadcast(&ctx->queuePopCond); /* Join all of the threads */ { size_t i; for (i = 0; i < ctx->threadCapacity; ++i) { ZSTD_pthread_join(ctx->threads[i]); /* note : could fail */ } } } void POOL_free(POOL_ctx *ctx) { if (!ctx) { return; } POOL_join(ctx); ZSTD_pthread_mutex_destroy(&ctx->queueMutex); ZSTD_pthread_cond_destroy(&ctx->queuePushCond); ZSTD_pthread_cond_destroy(&ctx->queuePopCond); ZSTD_customFree(ctx->queue, ctx->customMem); ZSTD_customFree(ctx->threads, ctx->customMem); ZSTD_customFree(ctx, ctx->customMem); } /*! POOL_joinJobs() : * Waits for all queued jobs to finish executing. */ void POOL_joinJobs(POOL_ctx* ctx) { ZSTD_pthread_mutex_lock(&ctx->queueMutex); while(!ctx->queueEmpty || ctx->numThreadsBusy > 0) { ZSTD_pthread_cond_wait(&ctx->queuePushCond, &ctx->queueMutex); } ZSTD_pthread_mutex_unlock(&ctx->queueMutex); } void ZSTD_freeThreadPool (ZSTD_threadPool* pool) { POOL_free (pool); } size_t POOL_sizeof(const POOL_ctx* ctx) { if (ctx==NULL) return 0; /* supports sizeof NULL */ return sizeof(*ctx) + ctx->queueSize * sizeof(POOL_job) + ctx->threadCapacity * sizeof(ZSTD_pthread_t); } /* @return : 0 on success, 1 on error */ static int POOL_resize_internal(POOL_ctx* ctx, size_t numThreads) { if (numThreads <= ctx->threadCapacity) { if (!numThreads) return 1; ctx->threadLimit = numThreads; return 0; } /* numThreads > threadCapacity */ { ZSTD_pthread_t* const threadPool = (ZSTD_pthread_t*)ZSTD_customCalloc(numThreads * sizeof(ZSTD_pthread_t), ctx->customMem); if (!threadPool) return 1; /* replace existing thread pool */ ZSTD_memcpy(threadPool, ctx->threads, ctx->threadCapacity * sizeof(*threadPool)); ZSTD_customFree(ctx->threads, ctx->customMem); ctx->threads = threadPool; /* Initialize additional threads */ { size_t threadId; for (threadId = ctx->threadCapacity; threadId < numThreads; ++threadId) { if (ZSTD_pthread_create(&threadPool[threadId], NULL, &POOL_thread, ctx)) { ctx->threadCapacity = threadId; return 1; } } } } /* successfully expanded */ ctx->threadCapacity = numThreads; ctx->threadLimit = numThreads; return 0; } /* @return : 0 on success, 1 on error */ int POOL_resize(POOL_ctx* ctx, size_t numThreads) { int result; if (ctx==NULL) return 1; ZSTD_pthread_mutex_lock(&ctx->queueMutex); result = POOL_resize_internal(ctx, numThreads); ZSTD_pthread_cond_broadcast(&ctx->queuePopCond); ZSTD_pthread_mutex_unlock(&ctx->queueMutex); return result; } /** * Returns 1 if the queue is full and 0 otherwise. * * When queueSize is 1 (pool was created with an intended queueSize of 0), * then a queue is empty if there is a thread free _and_ no job is waiting. */ static int isQueueFull(POOL_ctx const* ctx) { if (ctx->queueSize > 1) { return ctx->queueHead == ((ctx->queueTail + 1) % ctx->queueSize); } else { return (ctx->numThreadsBusy == ctx->threadLimit) || !ctx->queueEmpty; } } static void POOL_add_internal(POOL_ctx* ctx, POOL_function function, void *opaque) { POOL_job job; job.function = function; job.opaque = opaque; assert(ctx != NULL); if (ctx->shutdown) return; ctx->queueEmpty = 0; ctx->queue[ctx->queueTail] = job; ctx->queueTail = (ctx->queueTail + 1) % ctx->queueSize; ZSTD_pthread_cond_signal(&ctx->queuePopCond); } void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque) { assert(ctx != NULL); ZSTD_pthread_mutex_lock(&ctx->queueMutex); /* Wait until there is space in the queue for the new job */ while (isQueueFull(ctx) && (!ctx->shutdown)) { ZSTD_pthread_cond_wait(&ctx->queuePushCond, &ctx->queueMutex); } POOL_add_internal(ctx, function, opaque); ZSTD_pthread_mutex_unlock(&ctx->queueMutex); } int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque) { assert(ctx != NULL); ZSTD_pthread_mutex_lock(&ctx->queueMutex); if (isQueueFull(ctx)) { ZSTD_pthread_mutex_unlock(&ctx->queueMutex); return 0; } POOL_add_internal(ctx, function, opaque); ZSTD_pthread_mutex_unlock(&ctx->queueMutex); return 1; } #else /* ZSTD_MULTITHREAD not defined */ /* ========================== */ /* No multi-threading support */ /* ========================== */ /* We don't need any data, but if it is empty, malloc() might return NULL. */ struct POOL_ctx_s { int dummy; }; static POOL_ctx g_poolCtx; POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) { return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem); } POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem) { (void)numThreads; (void)queueSize; (void)customMem; return &g_poolCtx; } void POOL_free(POOL_ctx* ctx) { assert(!ctx || ctx == &g_poolCtx); (void)ctx; } void POOL_joinJobs(POOL_ctx* ctx){ assert(!ctx || ctx == &g_poolCtx); (void)ctx; } int POOL_resize(POOL_ctx* ctx, size_t numThreads) { (void)ctx; (void)numThreads; return 0; } void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque) { (void)ctx; function(opaque); } int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque) { (void)ctx; function(opaque); return 1; } size_t POOL_sizeof(const POOL_ctx* ctx) { if (ctx==NULL) return 0; /* supports sizeof NULL */ assert(ctx == &g_poolCtx); return sizeof(*ctx); } #endif /* ZSTD_MULTITHREAD */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/debug.h0000644000175200007730000000726114515254731024055 0ustar rlaboissrlaboiss/* ****************************************************************** * debug * Part of FSE library * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ /* * The purpose of this header is to enable debug functions. * They regroup assert(), DEBUGLOG() and RAWLOG() for run-time, * and DEBUG_STATIC_ASSERT() for compile-time. * * By default, DEBUGLEVEL==0, which means run-time debug is disabled. * * Level 1 enables assert() only. * Starting level 2, traces can be generated and pushed to stderr. * The higher the level, the more verbose the traces. * * It's possible to dynamically adjust level using variable g_debug_level, * which is only declared if DEBUGLEVEL>=2, * and is a global variable, not multi-thread protected (use with care) */ #ifndef DEBUG_H_12987983217 #define DEBUG_H_12987983217 #if defined (__cplusplus) extern "C" { #endif /* static assert is triggered at compile time, leaving no runtime artefact. * static assert only works with compile-time constants. * Also, this variant can only be used inside a function. */ #define DEBUG_STATIC_ASSERT(c) (void)sizeof(char[(c) ? 1 : -1]) /* DEBUGLEVEL is expected to be defined externally, * typically through compiler command line. * Value must be a number. */ #ifndef DEBUGLEVEL # define DEBUGLEVEL 0 #endif /* recommended values for DEBUGLEVEL : * 0 : release mode, no debug, all run-time checks disabled * 1 : enables assert() only, no display * 2 : reserved, for currently active debug path * 3 : events once per object lifetime (CCtx, CDict, etc.) * 4 : events once per frame * 5 : events once per block * 6 : events once per sequence (verbose) * 7+: events at every position (*very* verbose) * * It's generally inconvenient to output traces > 5. * In which case, it's possible to selectively trigger high verbosity levels * by modifying g_debug_level. */ #if (DEBUGLEVEL>=1) # define ZSTD_DEPS_NEED_ASSERT # include "zstd_deps.h" #else # ifndef assert /* assert may be already defined, due to prior #include */ # define assert(condition) ((void)0) /* disable assert (default) */ # endif #endif #if (DEBUGLEVEL>=2) # define ZSTD_DEPS_NEED_IO # include "zstd_deps.h" extern int g_debuglevel; /* the variable is only declared, it actually lives in debug.c, and is shared by the whole process. It's not thread-safe. It's useful when enabling very verbose levels on selective conditions (such as position in src) */ # define RAWLOG(l, ...) { \ if (l<=g_debuglevel) { \ ZSTD_DEBUG_PRINT(__VA_ARGS__); \ } } # define DEBUGLOG(l, ...) { \ if (l<=g_debuglevel) { \ ZSTD_DEBUG_PRINT(__FILE__ ": " __VA_ARGS__); \ ZSTD_DEBUG_PRINT(" \n"); \ } } #else # define RAWLOG(l, ...) {} /* disabled */ # define DEBUGLOG(l, ...) {} /* disabled */ #endif #if defined (__cplusplus) } #endif #endif /* DEBUG_H_12987983217 */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/error_private.c0000644000175200007730000000732514515254731025646 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* The purpose of this file is to have a single list of error strings embedded in binary */ #include "error_private.h" const char* ERR_getErrorString(ERR_enum code) { #ifdef ZSTD_STRIP_ERROR_STRINGS (void)code; return "Error strings stripped"; #else static const char* const notErrorCode = "Unspecified error code"; switch( code ) { case PREFIX(no_error): return "No error detected"; case PREFIX(GENERIC): return "Error (generic)"; case PREFIX(prefix_unknown): return "Unknown frame descriptor"; case PREFIX(version_unsupported): return "Version not supported"; case PREFIX(frameParameter_unsupported): return "Unsupported frame parameter"; case PREFIX(frameParameter_windowTooLarge): return "Frame requires too much memory for decoding"; case PREFIX(corruption_detected): return "Data corruption detected"; case PREFIX(checksum_wrong): return "Restored data doesn't match checksum"; case PREFIX(literals_headerWrong): return "Header of Literals' block doesn't respect format specification"; case PREFIX(parameter_unsupported): return "Unsupported parameter"; case PREFIX(parameter_combination_unsupported): return "Unsupported combination of parameters"; case PREFIX(parameter_outOfBound): return "Parameter is out of bound"; case PREFIX(init_missing): return "Context should be init first"; case PREFIX(memory_allocation): return "Allocation error : not enough memory"; case PREFIX(workSpace_tooSmall): return "workSpace buffer is not large enough"; case PREFIX(stage_wrong): return "Operation not authorized at current processing stage"; case PREFIX(tableLog_tooLarge): return "tableLog requires too much memory : unsupported"; case PREFIX(maxSymbolValue_tooLarge): return "Unsupported max Symbol Value : too large"; case PREFIX(maxSymbolValue_tooSmall): return "Specified maxSymbolValue is too small"; case PREFIX(stabilityCondition_notRespected): return "pledged buffer stability condition is not respected"; case PREFIX(dictionary_corrupted): return "Dictionary is corrupted"; case PREFIX(dictionary_wrong): return "Dictionary mismatch"; case PREFIX(dictionaryCreation_failed): return "Cannot create Dictionary from provided samples"; case PREFIX(dstSize_tooSmall): return "Destination buffer is too small"; case PREFIX(srcSize_wrong): return "Src size is incorrect"; case PREFIX(dstBuffer_null): return "Operation on NULL destination buffer"; case PREFIX(noForwardProgress_destFull): return "Operation made no progress over multiple calls, due to output buffer being full"; case PREFIX(noForwardProgress_inputEmpty): return "Operation made no progress over multiple calls, due to input being empty"; /* following error codes are not stable and may be removed or changed in a future version */ case PREFIX(frameIndex_tooLarge): return "Frame index is too large"; case PREFIX(seekableIO): return "An I/O error occurred when reading/seeking"; case PREFIX(dstBuffer_wrong): return "Destination buffer is wrong"; case PREFIX(srcBuffer_wrong): return "Source buffer is wrong"; case PREFIX(sequenceProducer_failed): return "Block-level external sequence producer returned an error code"; case PREFIX(externalSequences_invalid): return "External sequences are not valid"; case PREFIX(maxCode): default: return notErrorCode; } #endif } zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/zstd_common.c0000644000175200007730000000322314515254731025310 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /*-************************************* * Dependencies ***************************************/ #define ZSTD_DEPS_NEED_MALLOC #include "error_private.h" #include "zstd_internal.h" /*-**************************************** * Version ******************************************/ unsigned ZSTD_versionNumber(void) { return ZSTD_VERSION_NUMBER; } const char* ZSTD_versionString(void) { return ZSTD_VERSION_STRING; } /*-**************************************** * ZSTD Error Management ******************************************/ #undef ZSTD_isError /* defined within zstd_internal.h */ /*! ZSTD_isError() : * tells if a return value is an error code * symbol is required for external callers */ unsigned ZSTD_isError(size_t code) { return ERR_isError(code); } /*! ZSTD_getErrorName() : * provides error code string from function result (useful for debugging) */ const char* ZSTD_getErrorName(size_t code) { return ERR_getErrorName(code); } /*! ZSTD_getError() : * convert a `size_t` function result into a proper ZSTD_errorCode enum */ ZSTD_ErrorCode ZSTD_getErrorCode(size_t code) { return ERR_getErrorCode(code); } /*! ZSTD_getErrorString() : * provides error code string from enum */ const char* ZSTD_getErrorString(ZSTD_ErrorCode code) { return ERR_getErrorString(code); } zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/zstd_trace.h0000644000175200007730000001101314515254731025117 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_TRACE_H #define ZSTD_TRACE_H #if defined (__cplusplus) extern "C" { #endif #include /* weak symbol support * For now, enable conservatively: * - Only GNUC * - Only ELF * - Only x86-64, i386 and aarch64 * Also, explicitly disable on platforms known not to work so they aren't * forgotten in the future. */ #if !defined(ZSTD_HAVE_WEAK_SYMBOLS) && \ defined(__GNUC__) && defined(__ELF__) && \ (defined(__x86_64__) || defined(_M_X64) || defined(__i386__) || defined(_M_IX86) || defined(__aarch64__)) && \ !defined(__APPLE__) && !defined(_WIN32) && !defined(__MINGW32__) && \ !defined(__CYGWIN__) && !defined(_AIX) # define ZSTD_HAVE_WEAK_SYMBOLS 1 #else # define ZSTD_HAVE_WEAK_SYMBOLS 0 #endif #if ZSTD_HAVE_WEAK_SYMBOLS # define ZSTD_WEAK_ATTR __attribute__((__weak__)) #else # define ZSTD_WEAK_ATTR #endif /* Only enable tracing when weak symbols are available. */ #ifndef ZSTD_TRACE # define ZSTD_TRACE ZSTD_HAVE_WEAK_SYMBOLS #endif #if ZSTD_TRACE struct ZSTD_CCtx_s; struct ZSTD_DCtx_s; struct ZSTD_CCtx_params_s; typedef struct { /** * ZSTD_VERSION_NUMBER * * This is guaranteed to be the first member of ZSTD_trace. * Otherwise, this struct is not stable between versions. If * the version number does not match your expectation, you * should not interpret the rest of the struct. */ unsigned version; /** * Non-zero if streaming (de)compression is used. */ unsigned streaming; /** * The dictionary ID. */ unsigned dictionaryID; /** * Is the dictionary cold? * Only set on decompression. */ unsigned dictionaryIsCold; /** * The dictionary size or zero if no dictionary. */ size_t dictionarySize; /** * The uncompressed size of the data. */ size_t uncompressedSize; /** * The compressed size of the data. */ size_t compressedSize; /** * The fully resolved CCtx parameters (NULL on decompression). */ struct ZSTD_CCtx_params_s const* params; /** * The ZSTD_CCtx pointer (NULL on decompression). */ struct ZSTD_CCtx_s const* cctx; /** * The ZSTD_DCtx pointer (NULL on compression). */ struct ZSTD_DCtx_s const* dctx; } ZSTD_Trace; /** * A tracing context. It must be 0 when tracing is disabled. * Otherwise, any non-zero value returned by a tracing begin() * function is presented to any subsequent calls to end(). * * Any non-zero value is treated as tracing is enabled and not * interpreted by the library. * * Two possible uses are: * * A timestamp for when the begin() function was called. * * A unique key identifying the (de)compression, like the * address of the [dc]ctx pointer if you need to track * more information than just a timestamp. */ typedef unsigned long long ZSTD_TraceCtx; /** * Trace the beginning of a compression call. * @param cctx The dctx pointer for the compression. * It can be used as a key to map begin() to end(). * @returns Non-zero if tracing is enabled. The return value is * passed to ZSTD_trace_compress_end(). */ ZSTD_WEAK_ATTR ZSTD_TraceCtx ZSTD_trace_compress_begin( struct ZSTD_CCtx_s const* cctx); /** * Trace the end of a compression call. * @param ctx The return value of ZSTD_trace_compress_begin(). * @param trace The zstd tracing info. */ ZSTD_WEAK_ATTR void ZSTD_trace_compress_end( ZSTD_TraceCtx ctx, ZSTD_Trace const* trace); /** * Trace the beginning of a decompression call. * @param dctx The dctx pointer for the decompression. * It can be used as a key to map begin() to end(). * @returns Non-zero if tracing is enabled. The return value is * passed to ZSTD_trace_compress_end(). */ ZSTD_WEAK_ATTR ZSTD_TraceCtx ZSTD_trace_decompress_begin( struct ZSTD_DCtx_s const* dctx); /** * Trace the end of a decompression call. * @param ctx The return value of ZSTD_trace_decompress_begin(). * @param trace The zstd tracing info. */ ZSTD_WEAK_ATTR void ZSTD_trace_decompress_end( ZSTD_TraceCtx ctx, ZSTD_Trace const* trace); #endif /* ZSTD_TRACE */ #if defined (__cplusplus) } #endif #endif /* ZSTD_TRACE_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/threading.h0000644000175200007730000001211314515254731024724 0ustar rlaboissrlaboiss/** * Copyright (c) 2016 Tino Reichardt * All rights reserved. * * You can contact the author at: * - zstdmt source repository: https://github.com/mcmilk/zstdmt * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef THREADING_H_938743 #define THREADING_H_938743 #include "debug.h" #if defined (__cplusplus) extern "C" { #endif #if defined(ZSTD_MULTITHREAD) && defined(_WIN32) /** * Windows minimalist Pthread Wrapper */ #ifdef WINVER # undef WINVER #endif #define WINVER 0x0600 #ifdef _WIN32_WINNT # undef _WIN32_WINNT #endif #define _WIN32_WINNT 0x0600 #ifndef WIN32_LEAN_AND_MEAN # define WIN32_LEAN_AND_MEAN #endif #undef ERROR /* reported already defined on VS 2015 (Rich Geldreich) */ #include #undef ERROR #define ERROR(name) ZSTD_ERROR(name) /* mutex */ #define ZSTD_pthread_mutex_t CRITICAL_SECTION #define ZSTD_pthread_mutex_init(a, b) ((void)(b), InitializeCriticalSection((a)), 0) #define ZSTD_pthread_mutex_destroy(a) DeleteCriticalSection((a)) #define ZSTD_pthread_mutex_lock(a) EnterCriticalSection((a)) #define ZSTD_pthread_mutex_unlock(a) LeaveCriticalSection((a)) /* condition variable */ #define ZSTD_pthread_cond_t CONDITION_VARIABLE #define ZSTD_pthread_cond_init(a, b) ((void)(b), InitializeConditionVariable((a)), 0) #define ZSTD_pthread_cond_destroy(a) ((void)(a)) #define ZSTD_pthread_cond_wait(a, b) SleepConditionVariableCS((a), (b), INFINITE) #define ZSTD_pthread_cond_signal(a) WakeConditionVariable((a)) #define ZSTD_pthread_cond_broadcast(a) WakeAllConditionVariable((a)) /* ZSTD_pthread_create() and ZSTD_pthread_join() */ typedef HANDLE ZSTD_pthread_t; int ZSTD_pthread_create(ZSTD_pthread_t* thread, const void* unused, void* (*start_routine) (void*), void* arg); int ZSTD_pthread_join(ZSTD_pthread_t thread); /** * add here more wrappers as required */ #elif defined(ZSTD_MULTITHREAD) /* posix assumed ; need a better detection method */ /* === POSIX Systems === */ # include #if DEBUGLEVEL < 1 #define ZSTD_pthread_mutex_t pthread_mutex_t #define ZSTD_pthread_mutex_init(a, b) pthread_mutex_init((a), (b)) #define ZSTD_pthread_mutex_destroy(a) pthread_mutex_destroy((a)) #define ZSTD_pthread_mutex_lock(a) pthread_mutex_lock((a)) #define ZSTD_pthread_mutex_unlock(a) pthread_mutex_unlock((a)) #define ZSTD_pthread_cond_t pthread_cond_t #define ZSTD_pthread_cond_init(a, b) pthread_cond_init((a), (b)) #define ZSTD_pthread_cond_destroy(a) pthread_cond_destroy((a)) #define ZSTD_pthread_cond_wait(a, b) pthread_cond_wait((a), (b)) #define ZSTD_pthread_cond_signal(a) pthread_cond_signal((a)) #define ZSTD_pthread_cond_broadcast(a) pthread_cond_broadcast((a)) #define ZSTD_pthread_t pthread_t #define ZSTD_pthread_create(a, b, c, d) pthread_create((a), (b), (c), (d)) #define ZSTD_pthread_join(a) pthread_join((a),NULL) #else /* DEBUGLEVEL >= 1 */ /* Debug implementation of threading. * In this implementation we use pointers for mutexes and condition variables. * This way, if we forget to init/destroy them the program will crash or ASAN * will report leaks. */ #define ZSTD_pthread_mutex_t pthread_mutex_t* int ZSTD_pthread_mutex_init(ZSTD_pthread_mutex_t* mutex, pthread_mutexattr_t const* attr); int ZSTD_pthread_mutex_destroy(ZSTD_pthread_mutex_t* mutex); #define ZSTD_pthread_mutex_lock(a) pthread_mutex_lock(*(a)) #define ZSTD_pthread_mutex_unlock(a) pthread_mutex_unlock(*(a)) #define ZSTD_pthread_cond_t pthread_cond_t* int ZSTD_pthread_cond_init(ZSTD_pthread_cond_t* cond, pthread_condattr_t const* attr); int ZSTD_pthread_cond_destroy(ZSTD_pthread_cond_t* cond); #define ZSTD_pthread_cond_wait(a, b) pthread_cond_wait(*(a), *(b)) #define ZSTD_pthread_cond_signal(a) pthread_cond_signal(*(a)) #define ZSTD_pthread_cond_broadcast(a) pthread_cond_broadcast(*(a)) #define ZSTD_pthread_t pthread_t #define ZSTD_pthread_create(a, b, c, d) pthread_create((a), (b), (c), (d)) #define ZSTD_pthread_join(a) pthread_join((a),NULL) #endif #else /* ZSTD_MULTITHREAD not defined */ /* No multithreading support */ typedef int ZSTD_pthread_mutex_t; #define ZSTD_pthread_mutex_init(a, b) ((void)(a), (void)(b), 0) #define ZSTD_pthread_mutex_destroy(a) ((void)(a)) #define ZSTD_pthread_mutex_lock(a) ((void)(a)) #define ZSTD_pthread_mutex_unlock(a) ((void)(a)) typedef int ZSTD_pthread_cond_t; #define ZSTD_pthread_cond_init(a, b) ((void)(a), (void)(b), 0) #define ZSTD_pthread_cond_destroy(a) ((void)(a)) #define ZSTD_pthread_cond_wait(a, b) ((void)(a), (void)(b)) #define ZSTD_pthread_cond_signal(a) ((void)(a)) #define ZSTD_pthread_cond_broadcast(a) ((void)(a)) /* do not use ZSTD_pthread_t */ #endif /* ZSTD_MULTITHREAD */ #if defined (__cplusplus) } #endif #endif /* THREADING_H_938743 */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/fse_decompress.c0000644000175200007730000002757014515254731025770 0ustar rlaboissrlaboiss/* ****************************************************************** * FSE : Finite State Entropy decoder * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy * - Public forum : https://groups.google.com/forum/#!forum/lz4c * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ /* ************************************************************** * Includes ****************************************************************/ #include "debug.h" /* assert */ #include "bitstream.h" #include "compiler.h" #define FSE_STATIC_LINKING_ONLY #include "fse.h" #include "error_private.h" #define ZSTD_DEPS_NEED_MALLOC #include "zstd_deps.h" #include "bits.h" /* ZSTD_highbit32 */ /* ************************************************************** * Error Management ****************************************************************/ #define FSE_isError ERR_isError #define FSE_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c) /* use only *after* variable declarations */ /* ************************************************************** * Templates ****************************************************************/ /* designed to be included for type-specific functions (template emulation in C) Objective is to write these functions only once, for improved maintenance */ /* safety checks */ #ifndef FSE_FUNCTION_EXTENSION # error "FSE_FUNCTION_EXTENSION must be defined" #endif #ifndef FSE_FUNCTION_TYPE # error "FSE_FUNCTION_TYPE must be defined" #endif /* Function names */ #define FSE_CAT(X,Y) X##Y #define FSE_FUNCTION_NAME(X,Y) FSE_CAT(X,Y) #define FSE_TYPE_NAME(X,Y) FSE_CAT(X,Y) static size_t FSE_buildDTable_internal(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize) { void* const tdPtr = dt+1; /* because *dt is unsigned, 32-bits aligned on 32-bits */ FSE_DECODE_TYPE* const tableDecode = (FSE_DECODE_TYPE*) (tdPtr); U16* symbolNext = (U16*)workSpace; BYTE* spread = (BYTE*)(symbolNext + maxSymbolValue + 1); U32 const maxSV1 = maxSymbolValue + 1; U32 const tableSize = 1 << tableLog; U32 highThreshold = tableSize-1; /* Sanity Checks */ if (FSE_BUILD_DTABLE_WKSP_SIZE(tableLog, maxSymbolValue) > wkspSize) return ERROR(maxSymbolValue_tooLarge); if (maxSymbolValue > FSE_MAX_SYMBOL_VALUE) return ERROR(maxSymbolValue_tooLarge); if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge); /* Init, lay down lowprob symbols */ { FSE_DTableHeader DTableH; DTableH.tableLog = (U16)tableLog; DTableH.fastMode = 1; { S16 const largeLimit= (S16)(1 << (tableLog-1)); U32 s; for (s=0; s= largeLimit) DTableH.fastMode=0; symbolNext[s] = normalizedCounter[s]; } } } ZSTD_memcpy(dt, &DTableH, sizeof(DTableH)); } /* Spread symbols */ if (highThreshold == tableSize - 1) { size_t const tableMask = tableSize-1; size_t const step = FSE_TABLESTEP(tableSize); /* First lay down the symbols in order. * We use a uint64_t to lay down 8 bytes at a time. This reduces branch * misses since small blocks generally have small table logs, so nearly * all symbols have counts <= 8. We ensure we have 8 bytes at the end of * our buffer to handle the over-write. */ { U64 const add = 0x0101010101010101ull; size_t pos = 0; U64 sv = 0; U32 s; for (s=0; s highThreshold) position = (position + step) & tableMask; /* lowprob area */ } } if (position!=0) return ERROR(GENERIC); /* position must reach all cells once, otherwise normalizedCounter is incorrect */ } /* Build Decoding table */ { U32 u; for (u=0; u sizeof(bitD.bitContainer)*8) /* This test must be static */ BIT_reloadDStream(&bitD); op[1] = FSE_GETSYMBOL(&state2); if (FSE_MAX_TABLELOG*4+7 > sizeof(bitD.bitContainer)*8) /* This test must be static */ { if (BIT_reloadDStream(&bitD) > BIT_DStream_unfinished) { op+=2; break; } } op[2] = FSE_GETSYMBOL(&state1); if (FSE_MAX_TABLELOG*2+7 > sizeof(bitD.bitContainer)*8) /* This test must be static */ BIT_reloadDStream(&bitD); op[3] = FSE_GETSYMBOL(&state2); } /* tail */ /* note : BIT_reloadDStream(&bitD) >= FSE_DStream_partiallyFilled; Ends at exactly BIT_DStream_completed */ while (1) { if (op>(omax-2)) return ERROR(dstSize_tooSmall); *op++ = FSE_GETSYMBOL(&state1); if (BIT_reloadDStream(&bitD)==BIT_DStream_overflow) { *op++ = FSE_GETSYMBOL(&state2); break; } if (op>(omax-2)) return ERROR(dstSize_tooSmall); *op++ = FSE_GETSYMBOL(&state2); if (BIT_reloadDStream(&bitD)==BIT_DStream_overflow) { *op++ = FSE_GETSYMBOL(&state1); break; } } return op-ostart; } typedef struct { short ncount[FSE_MAX_SYMBOL_VALUE + 1]; FSE_DTable dtable[1]; /* Dynamically sized */ } FSE_DecompressWksp; FORCE_INLINE_TEMPLATE size_t FSE_decompress_wksp_body( void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize, int bmi2) { const BYTE* const istart = (const BYTE*)cSrc; const BYTE* ip = istart; unsigned tableLog; unsigned maxSymbolValue = FSE_MAX_SYMBOL_VALUE; FSE_DecompressWksp* const wksp = (FSE_DecompressWksp*)workSpace; DEBUG_STATIC_ASSERT((FSE_MAX_SYMBOL_VALUE + 1) % 2 == 0); if (wkspSize < sizeof(*wksp)) return ERROR(GENERIC); /* normal FSE decoding mode */ { size_t const NCountLength = FSE_readNCount_bmi2(wksp->ncount, &maxSymbolValue, &tableLog, istart, cSrcSize, bmi2); if (FSE_isError(NCountLength)) return NCountLength; if (tableLog > maxLog) return ERROR(tableLog_tooLarge); assert(NCountLength <= cSrcSize); ip += NCountLength; cSrcSize -= NCountLength; } if (FSE_DECOMPRESS_WKSP_SIZE(tableLog, maxSymbolValue) > wkspSize) return ERROR(tableLog_tooLarge); assert(sizeof(*wksp) + FSE_DTABLE_SIZE(tableLog) <= wkspSize); workSpace = (BYTE*)workSpace + sizeof(*wksp) + FSE_DTABLE_SIZE(tableLog); wkspSize -= sizeof(*wksp) + FSE_DTABLE_SIZE(tableLog); CHECK_F( FSE_buildDTable_internal(wksp->dtable, wksp->ncount, maxSymbolValue, tableLog, workSpace, wkspSize) ); { const void* ptr = wksp->dtable; const FSE_DTableHeader* DTableH = (const FSE_DTableHeader*)ptr; const U32 fastMode = DTableH->fastMode; /* select fast mode (static) */ if (fastMode) return FSE_decompress_usingDTable_generic(dst, dstCapacity, ip, cSrcSize, wksp->dtable, 1); return FSE_decompress_usingDTable_generic(dst, dstCapacity, ip, cSrcSize, wksp->dtable, 0); } } /* Avoids the FORCE_INLINE of the _body() function. */ static size_t FSE_decompress_wksp_body_default(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize) { return FSE_decompress_wksp_body(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize, 0); } #if DYNAMIC_BMI2 BMI2_TARGET_ATTRIBUTE static size_t FSE_decompress_wksp_body_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize) { return FSE_decompress_wksp_body(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize, 1); } #endif size_t FSE_decompress_wksp_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize, int bmi2) { #if DYNAMIC_BMI2 if (bmi2) { return FSE_decompress_wksp_body_bmi2(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize); } #endif (void)bmi2; return FSE_decompress_wksp_body_default(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize); } #endif /* FSE_COMMONDEFS_ONLY */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/bitstream.h0000644000175200007730000004216214515254731024760 0ustar rlaboissrlaboiss/* ****************************************************************** * bitstream * Part of FSE library * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ #ifndef BITSTREAM_H_MODULE #define BITSTREAM_H_MODULE #if defined (__cplusplus) extern "C" { #endif /* * This API consists of small unitary functions, which must be inlined for best performance. * Since link-time-optimization is not available for all compilers, * these functions are defined into a .h to be included. */ /*-**************************************** * Dependencies ******************************************/ #include "mem.h" /* unaligned access routines */ #include "compiler.h" /* UNLIKELY() */ #include "debug.h" /* assert(), DEBUGLOG(), RAWLOG() */ #include "error_private.h" /* error codes and messages */ #include "bits.h" /* ZSTD_highbit32 */ /*========================================= * Target specific =========================================*/ #ifndef ZSTD_NO_INTRINSICS # if (defined(__BMI__) || defined(__BMI2__)) && defined(__GNUC__) # include /* support for bextr (experimental)/bzhi */ # elif defined(__ICCARM__) # include # endif #endif #define STREAM_ACCUMULATOR_MIN_32 25 #define STREAM_ACCUMULATOR_MIN_64 57 #define STREAM_ACCUMULATOR_MIN ((U32)(MEM_32bits() ? STREAM_ACCUMULATOR_MIN_32 : STREAM_ACCUMULATOR_MIN_64)) /*-****************************************** * bitStream encoding API (write forward) ********************************************/ /* bitStream can mix input from multiple sources. * A critical property of these streams is that they encode and decode in **reverse** direction. * So the first bit sequence you add will be the last to be read, like a LIFO stack. */ typedef struct { size_t bitContainer; unsigned bitPos; char* startPtr; char* ptr; char* endPtr; } BIT_CStream_t; MEM_STATIC size_t BIT_initCStream(BIT_CStream_t* bitC, void* dstBuffer, size_t dstCapacity); MEM_STATIC void BIT_addBits(BIT_CStream_t* bitC, size_t value, unsigned nbBits); MEM_STATIC void BIT_flushBits(BIT_CStream_t* bitC); MEM_STATIC size_t BIT_closeCStream(BIT_CStream_t* bitC); /* Start with initCStream, providing the size of buffer to write into. * bitStream will never write outside of this buffer. * `dstCapacity` must be >= sizeof(bitD->bitContainer), otherwise @return will be an error code. * * bits are first added to a local register. * Local register is size_t, hence 64-bits on 64-bits systems, or 32-bits on 32-bits systems. * Writing data into memory is an explicit operation, performed by the flushBits function. * Hence keep track how many bits are potentially stored into local register to avoid register overflow. * After a flushBits, a maximum of 7 bits might still be stored into local register. * * Avoid storing elements of more than 24 bits if you want compatibility with 32-bits bitstream readers. * * Last operation is to close the bitStream. * The function returns the final size of CStream in bytes. * If data couldn't fit into `dstBuffer`, it will return a 0 ( == not storable) */ /*-******************************************** * bitStream decoding API (read backward) **********************************************/ typedef struct { size_t bitContainer; unsigned bitsConsumed; const char* ptr; const char* start; const char* limitPtr; } BIT_DStream_t; typedef enum { BIT_DStream_unfinished = 0, BIT_DStream_endOfBuffer = 1, BIT_DStream_completed = 2, BIT_DStream_overflow = 3 } BIT_DStream_status; /* result of BIT_reloadDStream() */ /* 1,2,4,8 would be better for bitmap combinations, but slows down performance a bit ... :( */ MEM_STATIC size_t BIT_initDStream(BIT_DStream_t* bitD, const void* srcBuffer, size_t srcSize); MEM_STATIC size_t BIT_readBits(BIT_DStream_t* bitD, unsigned nbBits); MEM_STATIC BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD); MEM_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t* bitD); /* Start by invoking BIT_initDStream(). * A chunk of the bitStream is then stored into a local register. * Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (size_t). * You can then retrieve bitFields stored into the local register, **in reverse order**. * Local register is explicitly reloaded from memory by the BIT_reloadDStream() method. * A reload guarantee a minimum of ((8*sizeof(bitD->bitContainer))-7) bits when its result is BIT_DStream_unfinished. * Otherwise, it can be less than that, so proceed accordingly. * Checking if DStream has reached its end can be performed with BIT_endOfDStream(). */ /*-**************************************** * unsafe API ******************************************/ MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC, size_t value, unsigned nbBits); /* faster, but works only if value is "clean", meaning all high bits above nbBits are 0 */ MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC); /* unsafe version; does not check buffer overflow */ MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits); /* faster, but works only if nbBits >= 1 */ /*===== Local Constants =====*/ static const unsigned BIT_mask[] = { 0, 1, 3, 7, 0xF, 0x1F, 0x3F, 0x7F, 0xFF, 0x1FF, 0x3FF, 0x7FF, 0xFFF, 0x1FFF, 0x3FFF, 0x7FFF, 0xFFFF, 0x1FFFF, 0x3FFFF, 0x7FFFF, 0xFFFFF, 0x1FFFFF, 0x3FFFFF, 0x7FFFFF, 0xFFFFFF, 0x1FFFFFF, 0x3FFFFFF, 0x7FFFFFF, 0xFFFFFFF, 0x1FFFFFFF, 0x3FFFFFFF, 0x7FFFFFFF}; /* up to 31 bits */ #define BIT_MASK_SIZE (sizeof(BIT_mask) / sizeof(BIT_mask[0])) /*-************************************************************** * bitStream encoding ****************************************************************/ /*! BIT_initCStream() : * `dstCapacity` must be > sizeof(size_t) * @return : 0 if success, * otherwise an error code (can be tested using ERR_isError()) */ MEM_STATIC size_t BIT_initCStream(BIT_CStream_t* bitC, void* startPtr, size_t dstCapacity) { bitC->bitContainer = 0; bitC->bitPos = 0; bitC->startPtr = (char*)startPtr; bitC->ptr = bitC->startPtr; bitC->endPtr = bitC->startPtr + dstCapacity - sizeof(bitC->bitContainer); if (dstCapacity <= sizeof(bitC->bitContainer)) return ERROR(dstSize_tooSmall); return 0; } MEM_STATIC FORCE_INLINE_ATTR size_t BIT_getLowerBits(size_t bitContainer, U32 const nbBits) { #if defined(STATIC_BMI2) && STATIC_BMI2 == 1 && !defined(ZSTD_NO_INTRINSICS) return _bzhi_u64(bitContainer, nbBits); #else assert(nbBits < BIT_MASK_SIZE); return bitContainer & BIT_mask[nbBits]; #endif } /*! BIT_addBits() : * can add up to 31 bits into `bitC`. * Note : does not check for register overflow ! */ MEM_STATIC void BIT_addBits(BIT_CStream_t* bitC, size_t value, unsigned nbBits) { DEBUG_STATIC_ASSERT(BIT_MASK_SIZE == 32); assert(nbBits < BIT_MASK_SIZE); assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8); bitC->bitContainer |= BIT_getLowerBits(value, nbBits) << bitC->bitPos; bitC->bitPos += nbBits; } /*! BIT_addBitsFast() : * works only if `value` is _clean_, * meaning all high bits above nbBits are 0 */ MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC, size_t value, unsigned nbBits) { assert((value>>nbBits) == 0); assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8); bitC->bitContainer |= value << bitC->bitPos; bitC->bitPos += nbBits; } /*! BIT_flushBitsFast() : * assumption : bitContainer has not overflowed * unsafe version; does not check buffer overflow */ MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC) { size_t const nbBytes = bitC->bitPos >> 3; assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8); assert(bitC->ptr <= bitC->endPtr); MEM_writeLEST(bitC->ptr, bitC->bitContainer); bitC->ptr += nbBytes; bitC->bitPos &= 7; bitC->bitContainer >>= nbBytes*8; } /*! BIT_flushBits() : * assumption : bitContainer has not overflowed * safe version; check for buffer overflow, and prevents it. * note : does not signal buffer overflow. * overflow will be revealed later on using BIT_closeCStream() */ MEM_STATIC void BIT_flushBits(BIT_CStream_t* bitC) { size_t const nbBytes = bitC->bitPos >> 3; assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8); assert(bitC->ptr <= bitC->endPtr); MEM_writeLEST(bitC->ptr, bitC->bitContainer); bitC->ptr += nbBytes; if (bitC->ptr > bitC->endPtr) bitC->ptr = bitC->endPtr; bitC->bitPos &= 7; bitC->bitContainer >>= nbBytes*8; } /*! BIT_closeCStream() : * @return : size of CStream, in bytes, * or 0 if it could not fit into dstBuffer */ MEM_STATIC size_t BIT_closeCStream(BIT_CStream_t* bitC) { BIT_addBitsFast(bitC, 1, 1); /* endMark */ BIT_flushBits(bitC); if (bitC->ptr >= bitC->endPtr) return 0; /* overflow detected */ return (bitC->ptr - bitC->startPtr) + (bitC->bitPos > 0); } /*-******************************************************** * bitStream decoding **********************************************************/ /*! BIT_initDStream() : * Initialize a BIT_DStream_t. * `bitD` : a pointer to an already allocated BIT_DStream_t structure. * `srcSize` must be the *exact* size of the bitStream, in bytes. * @return : size of stream (== srcSize), or an errorCode if a problem is detected */ MEM_STATIC size_t BIT_initDStream(BIT_DStream_t* bitD, const void* srcBuffer, size_t srcSize) { if (srcSize < 1) { ZSTD_memset(bitD, 0, sizeof(*bitD)); return ERROR(srcSize_wrong); } bitD->start = (const char*)srcBuffer; bitD->limitPtr = bitD->start + sizeof(bitD->bitContainer); if (srcSize >= sizeof(bitD->bitContainer)) { /* normal case */ bitD->ptr = (const char*)srcBuffer + srcSize - sizeof(bitD->bitContainer); bitD->bitContainer = MEM_readLEST(bitD->ptr); { BYTE const lastByte = ((const BYTE*)srcBuffer)[srcSize-1]; bitD->bitsConsumed = lastByte ? 8 - ZSTD_highbit32(lastByte) : 0; /* ensures bitsConsumed is always set */ if (lastByte == 0) return ERROR(GENERIC); /* endMark not present */ } } else { bitD->ptr = bitD->start; bitD->bitContainer = *(const BYTE*)(bitD->start); switch(srcSize) { case 7: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[6]) << (sizeof(bitD->bitContainer)*8 - 16); ZSTD_FALLTHROUGH; case 6: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[5]) << (sizeof(bitD->bitContainer)*8 - 24); ZSTD_FALLTHROUGH; case 5: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[4]) << (sizeof(bitD->bitContainer)*8 - 32); ZSTD_FALLTHROUGH; case 4: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[3]) << 24; ZSTD_FALLTHROUGH; case 3: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[2]) << 16; ZSTD_FALLTHROUGH; case 2: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[1]) << 8; ZSTD_FALLTHROUGH; default: break; } { BYTE const lastByte = ((const BYTE*)srcBuffer)[srcSize-1]; bitD->bitsConsumed = lastByte ? 8 - ZSTD_highbit32(lastByte) : 0; if (lastByte == 0) return ERROR(corruption_detected); /* endMark not present */ } bitD->bitsConsumed += (U32)(sizeof(bitD->bitContainer) - srcSize)*8; } return srcSize; } MEM_STATIC FORCE_INLINE_ATTR size_t BIT_getUpperBits(size_t bitContainer, U32 const start) { return bitContainer >> start; } MEM_STATIC FORCE_INLINE_ATTR size_t BIT_getMiddleBits(size_t bitContainer, U32 const start, U32 const nbBits) { U32 const regMask = sizeof(bitContainer)*8 - 1; /* if start > regMask, bitstream is corrupted, and result is undefined */ assert(nbBits < BIT_MASK_SIZE); /* x86 transform & ((1 << nbBits) - 1) to bzhi instruction, it is better * than accessing memory. When bmi2 instruction is not present, we consider * such cpus old (pre-Haswell, 2013) and their performance is not of that * importance. */ #if defined(__x86_64__) || defined(_M_X86) return (bitContainer >> (start & regMask)) & ((((U64)1) << nbBits) - 1); #else return (bitContainer >> (start & regMask)) & BIT_mask[nbBits]; #endif } /*! BIT_lookBits() : * Provides next n bits from local register. * local register is not modified. * On 32-bits, maxNbBits==24. * On 64-bits, maxNbBits==56. * @return : value extracted */ MEM_STATIC FORCE_INLINE_ATTR size_t BIT_lookBits(const BIT_DStream_t* bitD, U32 nbBits) { /* arbitrate between double-shift and shift+mask */ #if 1 /* if bitD->bitsConsumed + nbBits > sizeof(bitD->bitContainer)*8, * bitstream is likely corrupted, and result is undefined */ return BIT_getMiddleBits(bitD->bitContainer, (sizeof(bitD->bitContainer)*8) - bitD->bitsConsumed - nbBits, nbBits); #else /* this code path is slower on my os-x laptop */ U32 const regMask = sizeof(bitD->bitContainer)*8 - 1; return ((bitD->bitContainer << (bitD->bitsConsumed & regMask)) >> 1) >> ((regMask-nbBits) & regMask); #endif } /*! BIT_lookBitsFast() : * unsafe version; only works if nbBits >= 1 */ MEM_STATIC size_t BIT_lookBitsFast(const BIT_DStream_t* bitD, U32 nbBits) { U32 const regMask = sizeof(bitD->bitContainer)*8 - 1; assert(nbBits >= 1); return (bitD->bitContainer << (bitD->bitsConsumed & regMask)) >> (((regMask+1)-nbBits) & regMask); } MEM_STATIC FORCE_INLINE_ATTR void BIT_skipBits(BIT_DStream_t* bitD, U32 nbBits) { bitD->bitsConsumed += nbBits; } /*! BIT_readBits() : * Read (consume) next n bits from local register and update. * Pay attention to not read more than nbBits contained into local register. * @return : extracted value. */ MEM_STATIC FORCE_INLINE_ATTR size_t BIT_readBits(BIT_DStream_t* bitD, unsigned nbBits) { size_t const value = BIT_lookBits(bitD, nbBits); BIT_skipBits(bitD, nbBits); return value; } /*! BIT_readBitsFast() : * unsafe version; only works if nbBits >= 1 */ MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits) { size_t const value = BIT_lookBitsFast(bitD, nbBits); assert(nbBits >= 1); BIT_skipBits(bitD, nbBits); return value; } /*! BIT_reloadDStreamFast() : * Similar to BIT_reloadDStream(), but with two differences: * 1. bitsConsumed <= sizeof(bitD->bitContainer)*8 must hold! * 2. Returns BIT_DStream_overflow when bitD->ptr < bitD->limitPtr, at this * point you must use BIT_reloadDStream() to reload. */ MEM_STATIC BIT_DStream_status BIT_reloadDStreamFast(BIT_DStream_t* bitD) { if (UNLIKELY(bitD->ptr < bitD->limitPtr)) return BIT_DStream_overflow; assert(bitD->bitsConsumed <= sizeof(bitD->bitContainer)*8); bitD->ptr -= bitD->bitsConsumed >> 3; bitD->bitsConsumed &= 7; bitD->bitContainer = MEM_readLEST(bitD->ptr); return BIT_DStream_unfinished; } /*! BIT_reloadDStream() : * Refill `bitD` from buffer previously set in BIT_initDStream() . * This function is safe, it guarantees it will not read beyond src buffer. * @return : status of `BIT_DStream_t` internal register. * when status == BIT_DStream_unfinished, internal register is filled with at least 25 or 57 bits */ MEM_STATIC FORCE_INLINE_ATTR BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD) { if (bitD->bitsConsumed > (sizeof(bitD->bitContainer)*8)) /* overflow detected, like end of stream */ return BIT_DStream_overflow; if (bitD->ptr >= bitD->limitPtr) { return BIT_reloadDStreamFast(bitD); } if (bitD->ptr == bitD->start) { if (bitD->bitsConsumed < sizeof(bitD->bitContainer)*8) return BIT_DStream_endOfBuffer; return BIT_DStream_completed; } /* start < ptr < limitPtr */ { U32 nbBytes = bitD->bitsConsumed >> 3; BIT_DStream_status result = BIT_DStream_unfinished; if (bitD->ptr - nbBytes < bitD->start) { nbBytes = (U32)(bitD->ptr - bitD->start); /* ptr > start */ result = BIT_DStream_endOfBuffer; } bitD->ptr -= nbBytes; bitD->bitsConsumed -= nbBytes*8; bitD->bitContainer = MEM_readLEST(bitD->ptr); /* reminder : srcSize > sizeof(bitD->bitContainer), otherwise bitD->ptr == bitD->start */ return result; } } /*! BIT_endOfDStream() : * @return : 1 if DStream has _exactly_ reached its end (all bits consumed). */ MEM_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t* DStream) { return ((DStream->ptr == DStream->start) && (DStream->bitsConsumed == sizeof(DStream->bitContainer)*8)); } #if defined (__cplusplus) } #endif #endif /* BITSTREAM_H_MODULE */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/xxhash.h0000644000175200007730000063764014515254731024304 0ustar rlaboissrlaboiss/* * xxHash - Fast Hash algorithm * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - xxHash homepage: https://cyan4973.github.io/xxHash/ * - xxHash source repository : https://github.com/Cyan4973/xxHash * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef XXH_NO_XXH3 # define XXH_NO_XXH3 #endif #ifndef XXH_NAMESPACE # define XXH_NAMESPACE ZSTD_ #endif /*! * @mainpage xxHash * * @file xxhash.h * xxHash prototypes and implementation */ /* TODO: update */ /* Notice extracted from xxHash homepage: xxHash is an extremely fast hash algorithm, running at RAM speed limits. It also successfully passes all tests from the SMHasher suite. Comparison (single thread, Windows Seven 32 bits, using SMHasher on a Core 2 Duo @3GHz) Name Speed Q.Score Author xxHash 5.4 GB/s 10 CrapWow 3.2 GB/s 2 Andrew MurmurHash 3a 2.7 GB/s 10 Austin Appleby SpookyHash 2.0 GB/s 10 Bob Jenkins SBox 1.4 GB/s 9 Bret Mulvey Lookup3 1.2 GB/s 9 Bob Jenkins SuperFastHash 1.2 GB/s 1 Paul Hsieh CityHash64 1.05 GB/s 10 Pike & Alakuijala FNV 0.55 GB/s 5 Fowler, Noll, Vo CRC32 0.43 GB/s 9 MD5-32 0.33 GB/s 10 Ronald L. Rivest SHA1-32 0.28 GB/s 10 Q.Score is a measure of quality of the hash function. It depends on successfully passing SMHasher test set. 10 is a perfect score. Note: SMHasher's CRC32 implementation is not the fastest one. Other speed-oriented implementations can be faster, especially in combination with PCLMUL instruction: https://fastcompression.blogspot.com/2019/03/presenting-xxh3.html?showComment=1552696407071#c3490092340461170735 A 64-bit version, named XXH64, is available since r35. It offers much better speed, but for 64-bit applications only. Name Speed on 64 bits Speed on 32 bits XXH64 13.8 GB/s 1.9 GB/s XXH32 6.8 GB/s 6.0 GB/s */ #if defined (__cplusplus) extern "C" { #endif /* **************************** * INLINE mode ******************************/ /*! * XXH_INLINE_ALL (and XXH_PRIVATE_API) * Use these build macros to inline xxhash into the target unit. * Inlining improves performance on small inputs, especially when the length is * expressed as a compile-time constant: * * https://fastcompression.blogspot.com/2018/03/xxhash-for-small-keys-impressive-power.html * * It also keeps xxHash symbols private to the unit, so they are not exported. * * Usage: * #define XXH_INLINE_ALL * #include "xxhash.h" * * Do not compile and link xxhash.o as a separate object, as it is not useful. */ #if (defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)) \ && !defined(XXH_INLINE_ALL_31684351384) /* this section should be traversed only once */ # define XXH_INLINE_ALL_31684351384 /* give access to the advanced API, required to compile implementations */ # undef XXH_STATIC_LINKING_ONLY /* avoid macro redef */ # define XXH_STATIC_LINKING_ONLY /* make all functions private */ # undef XXH_PUBLIC_API # if defined(__GNUC__) # define XXH_PUBLIC_API static __inline __attribute__((unused)) # elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) # define XXH_PUBLIC_API static inline # elif defined(_MSC_VER) # define XXH_PUBLIC_API static __inline # else /* note: this version may generate warnings for unused static functions */ # define XXH_PUBLIC_API static # endif /* * This part deals with the special case where a unit wants to inline xxHash, * but "xxhash.h" has previously been included without XXH_INLINE_ALL, * such as part of some previously included *.h header file. * Without further action, the new include would just be ignored, * and functions would effectively _not_ be inlined (silent failure). * The following macros solve this situation by prefixing all inlined names, * avoiding naming collision with previous inclusions. */ /* Before that, we unconditionally #undef all symbols, * in case they were already defined with XXH_NAMESPACE. * They will then be redefined for XXH_INLINE_ALL */ # undef XXH_versionNumber /* XXH32 */ # undef XXH32 # undef XXH32_createState # undef XXH32_freeState # undef XXH32_reset # undef XXH32_update # undef XXH32_digest # undef XXH32_copyState # undef XXH32_canonicalFromHash # undef XXH32_hashFromCanonical /* XXH64 */ # undef XXH64 # undef XXH64_createState # undef XXH64_freeState # undef XXH64_reset # undef XXH64_update # undef XXH64_digest # undef XXH64_copyState # undef XXH64_canonicalFromHash # undef XXH64_hashFromCanonical /* XXH3_64bits */ # undef XXH3_64bits # undef XXH3_64bits_withSecret # undef XXH3_64bits_withSeed # undef XXH3_64bits_withSecretandSeed # undef XXH3_createState # undef XXH3_freeState # undef XXH3_copyState # undef XXH3_64bits_reset # undef XXH3_64bits_reset_withSeed # undef XXH3_64bits_reset_withSecret # undef XXH3_64bits_update # undef XXH3_64bits_digest # undef XXH3_generateSecret /* XXH3_128bits */ # undef XXH128 # undef XXH3_128bits # undef XXH3_128bits_withSeed # undef XXH3_128bits_withSecret # undef XXH3_128bits_reset # undef XXH3_128bits_reset_withSeed # undef XXH3_128bits_reset_withSecret # undef XXH3_128bits_reset_withSecretandSeed # undef XXH3_128bits_update # undef XXH3_128bits_digest # undef XXH128_isEqual # undef XXH128_cmp # undef XXH128_canonicalFromHash # undef XXH128_hashFromCanonical /* Finally, free the namespace itself */ # undef XXH_NAMESPACE /* employ the namespace for XXH_INLINE_ALL */ # define XXH_NAMESPACE XXH_INLINE_ /* * Some identifiers (enums, type names) are not symbols, * but they must nonetheless be renamed to avoid redeclaration. * Alternative solution: do not redeclare them. * However, this requires some #ifdefs, and has a more dispersed impact. * Meanwhile, renaming can be achieved in a single place. */ # define XXH_IPREF(Id) XXH_NAMESPACE ## Id # define XXH_OK XXH_IPREF(XXH_OK) # define XXH_ERROR XXH_IPREF(XXH_ERROR) # define XXH_errorcode XXH_IPREF(XXH_errorcode) # define XXH32_canonical_t XXH_IPREF(XXH32_canonical_t) # define XXH64_canonical_t XXH_IPREF(XXH64_canonical_t) # define XXH128_canonical_t XXH_IPREF(XXH128_canonical_t) # define XXH32_state_s XXH_IPREF(XXH32_state_s) # define XXH32_state_t XXH_IPREF(XXH32_state_t) # define XXH64_state_s XXH_IPREF(XXH64_state_s) # define XXH64_state_t XXH_IPREF(XXH64_state_t) # define XXH3_state_s XXH_IPREF(XXH3_state_s) # define XXH3_state_t XXH_IPREF(XXH3_state_t) # define XXH128_hash_t XXH_IPREF(XXH128_hash_t) /* Ensure the header is parsed again, even if it was previously included */ # undef XXHASH_H_5627135585666179 # undef XXHASH_H_STATIC_13879238742 #endif /* XXH_INLINE_ALL || XXH_PRIVATE_API */ /* **************************************************************** * Stable API *****************************************************************/ #ifndef XXHASH_H_5627135585666179 #define XXHASH_H_5627135585666179 1 /*! * @defgroup public Public API * Contains details on the public xxHash functions. * @{ */ /* specific declaration modes for Windows */ #if !defined(XXH_INLINE_ALL) && !defined(XXH_PRIVATE_API) # if defined(WIN32) && defined(_MSC_VER) && (defined(XXH_IMPORT) || defined(XXH_EXPORT)) # ifdef XXH_EXPORT # define XXH_PUBLIC_API __declspec(dllexport) # elif XXH_IMPORT # define XXH_PUBLIC_API __declspec(dllimport) # endif # else # define XXH_PUBLIC_API /* do nothing */ # endif #endif #ifdef XXH_DOXYGEN /*! * @brief Emulate a namespace by transparently prefixing all symbols. * * If you want to include _and expose_ xxHash functions from within your own * library, but also want to avoid symbol collisions with other libraries which * may also include xxHash, you can use XXH_NAMESPACE to automatically prefix * any public symbol from xxhash library with the value of XXH_NAMESPACE * (therefore, avoid empty or numeric values). * * Note that no change is required within the calling program as long as it * includes `xxhash.h`: Regular symbol names will be automatically translated * by this header. */ # define XXH_NAMESPACE /* YOUR NAME HERE */ # undef XXH_NAMESPACE #endif #ifdef XXH_NAMESPACE # define XXH_CAT(A,B) A##B # define XXH_NAME2(A,B) XXH_CAT(A,B) # define XXH_versionNumber XXH_NAME2(XXH_NAMESPACE, XXH_versionNumber) /* XXH32 */ # define XXH32 XXH_NAME2(XXH_NAMESPACE, XXH32) # define XXH32_createState XXH_NAME2(XXH_NAMESPACE, XXH32_createState) # define XXH32_freeState XXH_NAME2(XXH_NAMESPACE, XXH32_freeState) # define XXH32_reset XXH_NAME2(XXH_NAMESPACE, XXH32_reset) # define XXH32_update XXH_NAME2(XXH_NAMESPACE, XXH32_update) # define XXH32_digest XXH_NAME2(XXH_NAMESPACE, XXH32_digest) # define XXH32_copyState XXH_NAME2(XXH_NAMESPACE, XXH32_copyState) # define XXH32_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH32_canonicalFromHash) # define XXH32_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH32_hashFromCanonical) /* XXH64 */ # define XXH64 XXH_NAME2(XXH_NAMESPACE, XXH64) # define XXH64_createState XXH_NAME2(XXH_NAMESPACE, XXH64_createState) # define XXH64_freeState XXH_NAME2(XXH_NAMESPACE, XXH64_freeState) # define XXH64_reset XXH_NAME2(XXH_NAMESPACE, XXH64_reset) # define XXH64_update XXH_NAME2(XXH_NAMESPACE, XXH64_update) # define XXH64_digest XXH_NAME2(XXH_NAMESPACE, XXH64_digest) # define XXH64_copyState XXH_NAME2(XXH_NAMESPACE, XXH64_copyState) # define XXH64_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH64_canonicalFromHash) # define XXH64_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH64_hashFromCanonical) /* XXH3_64bits */ # define XXH3_64bits XXH_NAME2(XXH_NAMESPACE, XXH3_64bits) # define XXH3_64bits_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSecret) # define XXH3_64bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSeed) # define XXH3_64bits_withSecretandSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSecretandSeed) # define XXH3_createState XXH_NAME2(XXH_NAMESPACE, XXH3_createState) # define XXH3_freeState XXH_NAME2(XXH_NAMESPACE, XXH3_freeState) # define XXH3_copyState XXH_NAME2(XXH_NAMESPACE, XXH3_copyState) # define XXH3_64bits_reset XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset) # define XXH3_64bits_reset_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset_withSeed) # define XXH3_64bits_reset_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset_withSecret) # define XXH3_64bits_reset_withSecretandSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset_withSecretandSeed) # define XXH3_64bits_update XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_update) # define XXH3_64bits_digest XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_digest) # define XXH3_generateSecret XXH_NAME2(XXH_NAMESPACE, XXH3_generateSecret) # define XXH3_generateSecret_fromSeed XXH_NAME2(XXH_NAMESPACE, XXH3_generateSecret_fromSeed) /* XXH3_128bits */ # define XXH128 XXH_NAME2(XXH_NAMESPACE, XXH128) # define XXH3_128bits XXH_NAME2(XXH_NAMESPACE, XXH3_128bits) # define XXH3_128bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSeed) # define XXH3_128bits_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSecret) # define XXH3_128bits_withSecretandSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSecretandSeed) # define XXH3_128bits_reset XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset) # define XXH3_128bits_reset_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset_withSeed) # define XXH3_128bits_reset_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset_withSecret) # define XXH3_128bits_reset_withSecretandSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset_withSecretandSeed) # define XXH3_128bits_update XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_update) # define XXH3_128bits_digest XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_digest) # define XXH128_isEqual XXH_NAME2(XXH_NAMESPACE, XXH128_isEqual) # define XXH128_cmp XXH_NAME2(XXH_NAMESPACE, XXH128_cmp) # define XXH128_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH128_canonicalFromHash) # define XXH128_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH128_hashFromCanonical) #endif /* ************************************* * Version ***************************************/ #define XXH_VERSION_MAJOR 0 #define XXH_VERSION_MINOR 8 #define XXH_VERSION_RELEASE 1 #define XXH_VERSION_NUMBER (XXH_VERSION_MAJOR *100*100 + XXH_VERSION_MINOR *100 + XXH_VERSION_RELEASE) /*! * @brief Obtains the xxHash version. * * This is mostly useful when xxHash is compiled as a shared library, * since the returned value comes from the library, as opposed to header file. * * @return `XXH_VERSION_NUMBER` of the invoked library. */ XXH_PUBLIC_API unsigned XXH_versionNumber (void); /* **************************** * Common basic types ******************************/ #include /* size_t */ typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode; /*-********************************************************************** * 32-bit hash ************************************************************************/ #if defined(XXH_DOXYGEN) /* Don't show include */ /*! * @brief An unsigned 32-bit integer. * * Not necessarily defined to `uint32_t` but functionally equivalent. */ typedef uint32_t XXH32_hash_t; #elif !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) # include typedef uint32_t XXH32_hash_t; #else # include # if UINT_MAX == 0xFFFFFFFFUL typedef unsigned int XXH32_hash_t; # else # if ULONG_MAX == 0xFFFFFFFFUL typedef unsigned long XXH32_hash_t; # else # error "unsupported platform: need a 32-bit type" # endif # endif #endif /*! * @} * * @defgroup xxh32_family XXH32 family * @ingroup public * Contains functions used in the classic 32-bit xxHash algorithm. * * @note * XXH32 is useful for older platforms, with no or poor 64-bit performance. * Note that @ref xxh3_family provides competitive speed * for both 32-bit and 64-bit systems, and offers true 64/128 bit hash results. * * @see @ref xxh64_family, @ref xxh3_family : Other xxHash families * @see @ref xxh32_impl for implementation details * @{ */ /*! * @brief Calculates the 32-bit hash of @p input using xxHash32. * * Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark): 5.4 GB/s * * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * @param seed The 32-bit seed to alter the hash's output predictably. * * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return The calculated 32-bit hash value. * * @see * XXH64(), XXH3_64bits_withSeed(), XXH3_128bits_withSeed(), XXH128(): * Direct equivalents for the other variants of xxHash. * @see * XXH32_createState(), XXH32_update(), XXH32_digest(): Streaming version. */ XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t length, XXH32_hash_t seed); /*! * Streaming functions generate the xxHash value from an incremental input. * This method is slower than single-call functions, due to state management. * For small inputs, prefer `XXH32()` and `XXH64()`, which are better optimized. * * An XXH state must first be allocated using `XXH*_createState()`. * * Start a new hash by initializing the state with a seed using `XXH*_reset()`. * * Then, feed the hash state by calling `XXH*_update()` as many times as necessary. * * The function returns an error code, with 0 meaning OK, and any other value * meaning there is an error. * * Finally, a hash value can be produced anytime, by using `XXH*_digest()`. * This function returns the nn-bits hash as an int or long long. * * It's still possible to continue inserting input into the hash state after a * digest, and generate new hash values later on by invoking `XXH*_digest()`. * * When done, release the state using `XXH*_freeState()`. * * Example code for incrementally hashing a file: * @code{.c} * #include * #include * #define BUFFER_SIZE 256 * * // Note: XXH64 and XXH3 use the same interface. * XXH32_hash_t * hashFile(FILE* stream) * { * XXH32_state_t* state; * unsigned char buf[BUFFER_SIZE]; * size_t amt; * XXH32_hash_t hash; * * state = XXH32_createState(); // Create a state * assert(state != NULL); // Error check here * XXH32_reset(state, 0xbaad5eed); // Reset state with our seed * while ((amt = fread(buf, 1, sizeof(buf), stream)) != 0) { * XXH32_update(state, buf, amt); // Hash the file in chunks * } * hash = XXH32_digest(state); // Finalize the hash * XXH32_freeState(state); // Clean up * return hash; * } * @endcode */ /*! * @typedef struct XXH32_state_s XXH32_state_t * @brief The opaque state struct for the XXH32 streaming API. * * @see XXH32_state_s for details. */ typedef struct XXH32_state_s XXH32_state_t; /*! * @brief Allocates an @ref XXH32_state_t. * * Must be freed with XXH32_freeState(). * @return An allocated XXH32_state_t on success, `NULL` on failure. */ XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void); /*! * @brief Frees an @ref XXH32_state_t. * * Must be allocated with XXH32_createState(). * @param statePtr A pointer to an @ref XXH32_state_t allocated with @ref XXH32_createState(). * @return XXH_OK. */ XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr); /*! * @brief Copies one @ref XXH32_state_t to another. * * @param dst_state The state to copy to. * @param src_state The state to copy from. * @pre * @p dst_state and @p src_state must not be `NULL` and must not overlap. */ XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dst_state, const XXH32_state_t* src_state); /*! * @brief Resets an @ref XXH32_state_t to begin a new hash. * * This function resets and seeds a state. Call it before @ref XXH32_update(). * * @param statePtr The state struct to reset. * @param seed The 32-bit seed to alter the hash result predictably. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH32_reset (XXH32_state_t* statePtr, XXH32_hash_t seed); /*! * @brief Consumes a block of @p input to an @ref XXH32_state_t. * * Call this to incrementally consume blocks of data. * * @param statePtr The state struct to update. * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * * @pre * @p statePtr must not be `NULL`. * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length); /*! * @brief Returns the calculated hash value from an @ref XXH32_state_t. * * @note * Calling XXH32_digest() will not affect @p statePtr, so you can update, * digest, and update again. * * @param statePtr The state struct to calculate the hash from. * * @pre * @p statePtr must not be `NULL`. * * @return The calculated xxHash32 value from that state. */ XXH_PUBLIC_API XXH32_hash_t XXH32_digest (const XXH32_state_t* statePtr); /******* Canonical representation *******/ /* * The default return values from XXH functions are unsigned 32 and 64 bit * integers. * This the simplest and fastest format for further post-processing. * * However, this leaves open the question of what is the order on the byte level, * since little and big endian conventions will store the same number differently. * * The canonical representation settles this issue by mandating big-endian * convention, the same convention as human-readable numbers (large digits first). * * When writing hash values to storage, sending them over a network, or printing * them, it's highly recommended to use the canonical representation to ensure * portability across a wider range of systems, present and future. * * The following functions allow transformation of hash values to and from * canonical format. */ /*! * @brief Canonical (big endian) representation of @ref XXH32_hash_t. */ typedef struct { unsigned char digest[4]; /*!< Hash bytes, big endian */ } XXH32_canonical_t; /*! * @brief Converts an @ref XXH32_hash_t to a big endian @ref XXH32_canonical_t. * * @param dst The @ref XXH32_canonical_t pointer to be stored to. * @param hash The @ref XXH32_hash_t to be converted. * * @pre * @p dst must not be `NULL`. */ XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash); /*! * @brief Converts an @ref XXH32_canonical_t to a native @ref XXH32_hash_t. * * @param src The @ref XXH32_canonical_t to convert. * * @pre * @p src must not be `NULL`. * * @return The converted hash. */ XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src); #ifdef __has_attribute # define XXH_HAS_ATTRIBUTE(x) __has_attribute(x) #else # define XXH_HAS_ATTRIBUTE(x) 0 #endif /* C-language Attributes are added in C23. */ #if defined(__STDC_VERSION__) && (__STDC_VERSION__ > 201710L) && defined(__has_c_attribute) # define XXH_HAS_C_ATTRIBUTE(x) __has_c_attribute(x) #else # define XXH_HAS_C_ATTRIBUTE(x) 0 #endif #if defined(__cplusplus) && defined(__has_cpp_attribute) # define XXH_HAS_CPP_ATTRIBUTE(x) __has_cpp_attribute(x) #else # define XXH_HAS_CPP_ATTRIBUTE(x) 0 #endif /* Define XXH_FALLTHROUGH macro for annotating switch case with the 'fallthrough' attribute introduced in CPP17 and C23. CPP17 : https://en.cppreference.com/w/cpp/language/attributes/fallthrough C23 : https://en.cppreference.com/w/c/language/attributes/fallthrough */ #if XXH_HAS_C_ATTRIBUTE(x) # define XXH_FALLTHROUGH [[fallthrough]] #elif XXH_HAS_CPP_ATTRIBUTE(x) # define XXH_FALLTHROUGH [[fallthrough]] #elif XXH_HAS_ATTRIBUTE(__fallthrough__) # define XXH_FALLTHROUGH __attribute__ ((fallthrough)) #else # define XXH_FALLTHROUGH #endif /*! * @} * @ingroup public * @{ */ #ifndef XXH_NO_LONG_LONG /*-********************************************************************** * 64-bit hash ************************************************************************/ #if defined(XXH_DOXYGEN) /* don't include */ /*! * @brief An unsigned 64-bit integer. * * Not necessarily defined to `uint64_t` but functionally equivalent. */ typedef uint64_t XXH64_hash_t; #elif !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) # include typedef uint64_t XXH64_hash_t; #else # include # if defined(__LP64__) && ULONG_MAX == 0xFFFFFFFFFFFFFFFFULL /* LP64 ABI says uint64_t is unsigned long */ typedef unsigned long XXH64_hash_t; # else /* the following type must have a width of 64-bit */ typedef unsigned long long XXH64_hash_t; # endif #endif /*! * @} * * @defgroup xxh64_family XXH64 family * @ingroup public * @{ * Contains functions used in the classic 64-bit xxHash algorithm. * * @note * XXH3 provides competitive speed for both 32-bit and 64-bit systems, * and offers true 64/128 bit hash results. * It provides better speed for systems with vector processing capabilities. */ /*! * @brief Calculates the 64-bit hash of @p input using xxHash64. * * This function usually runs faster on 64-bit systems, but slower on 32-bit * systems (see benchmark). * * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * @param seed The 64-bit seed to alter the hash's output predictably. * * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return The calculated 64-bit hash. * * @see * XXH32(), XXH3_64bits_withSeed(), XXH3_128bits_withSeed(), XXH128(): * Direct equivalents for the other variants of xxHash. * @see * XXH64_createState(), XXH64_update(), XXH64_digest(): Streaming version. */ XXH_PUBLIC_API XXH64_hash_t XXH64(const void* input, size_t length, XXH64_hash_t seed); /******* Streaming *******/ /*! * @brief The opaque state struct for the XXH64 streaming API. * * @see XXH64_state_s for details. */ typedef struct XXH64_state_s XXH64_state_t; /* incomplete type */ XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void); XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr); XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dst_state, const XXH64_state_t* src_state); XXH_PUBLIC_API XXH_errorcode XXH64_reset (XXH64_state_t* statePtr, XXH64_hash_t seed); XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* statePtr, const void* input, size_t length); XXH_PUBLIC_API XXH64_hash_t XXH64_digest (const XXH64_state_t* statePtr); /******* Canonical representation *******/ typedef struct { unsigned char digest[sizeof(XXH64_hash_t)]; } XXH64_canonical_t; XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash); XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src); #ifndef XXH_NO_XXH3 /*! * @} * ************************************************************************ * @defgroup xxh3_family XXH3 family * @ingroup public * @{ * * XXH3 is a more recent hash algorithm featuring: * - Improved speed for both small and large inputs * - True 64-bit and 128-bit outputs * - SIMD acceleration * - Improved 32-bit viability * * Speed analysis methodology is explained here: * * https://fastcompression.blogspot.com/2019/03/presenting-xxh3.html * * Compared to XXH64, expect XXH3 to run approximately * ~2x faster on large inputs and >3x faster on small ones, * exact differences vary depending on platform. * * XXH3's speed benefits greatly from SIMD and 64-bit arithmetic, * but does not require it. * Any 32-bit and 64-bit targets that can run XXH32 smoothly * can run XXH3 at competitive speeds, even without vector support. * Further details are explained in the implementation. * * Optimized implementations are provided for AVX512, AVX2, SSE2, NEON, POWER8, * ZVector and scalar targets. This can be controlled via the XXH_VECTOR macro. * * XXH3 implementation is portable: * it has a generic C90 formulation that can be compiled on any platform, * all implementations generage exactly the same hash value on all platforms. * Starting from v0.8.0, it's also labelled "stable", meaning that * any future version will also generate the same hash value. * * XXH3 offers 2 variants, _64bits and _128bits. * * When only 64 bits are needed, prefer invoking the _64bits variant, as it * reduces the amount of mixing, resulting in faster speed on small inputs. * It's also generally simpler to manipulate a scalar return type than a struct. * * The API supports one-shot hashing, streaming mode, and custom secrets. */ /*-********************************************************************** * XXH3 64-bit variant ************************************************************************/ /* XXH3_64bits(): * default 64-bit variant, using default secret and default seed of 0. * It's the fastest variant. */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits(const void* data, size_t len); /* * XXH3_64bits_withSeed(): * This variant generates a custom secret on the fly * based on default secret altered using the `seed` value. * While this operation is decently fast, note that it's not completely free. * Note: seed==0 produces the same results as XXH3_64bits(). */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSeed(const void* data, size_t len, XXH64_hash_t seed); /*! * The bare minimum size for a custom secret. * * @see * XXH3_64bits_withSecret(), XXH3_64bits_reset_withSecret(), * XXH3_128bits_withSecret(), XXH3_128bits_reset_withSecret(). */ #define XXH3_SECRET_SIZE_MIN 136 /* * XXH3_64bits_withSecret(): * It's possible to provide any blob of bytes as a "secret" to generate the hash. * This makes it more difficult for an external actor to prepare an intentional collision. * The main condition is that secretSize *must* be large enough (>= XXH3_SECRET_SIZE_MIN). * However, the quality of the secret impacts the dispersion of the hash algorithm. * Therefore, the secret _must_ look like a bunch of random bytes. * Avoid "trivial" or structured data such as repeated sequences or a text document. * Whenever in doubt about the "randomness" of the blob of bytes, * consider employing "XXH3_generateSecret()" instead (see below). * It will generate a proper high entropy secret derived from the blob of bytes. * Another advantage of using XXH3_generateSecret() is that * it guarantees that all bits within the initial blob of bytes * will impact every bit of the output. * This is not necessarily the case when using the blob of bytes directly * because, when hashing _small_ inputs, only a portion of the secret is employed. */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSecret(const void* data, size_t len, const void* secret, size_t secretSize); /******* Streaming *******/ /* * Streaming requires state maintenance. * This operation costs memory and CPU. * As a consequence, streaming is slower than one-shot hashing. * For better performance, prefer one-shot functions whenever applicable. */ /*! * @brief The state struct for the XXH3 streaming API. * * @see XXH3_state_s for details. */ typedef struct XXH3_state_s XXH3_state_t; XXH_PUBLIC_API XXH3_state_t* XXH3_createState(void); XXH_PUBLIC_API XXH_errorcode XXH3_freeState(XXH3_state_t* statePtr); XXH_PUBLIC_API void XXH3_copyState(XXH3_state_t* dst_state, const XXH3_state_t* src_state); /* * XXH3_64bits_reset(): * Initialize with default parameters. * digest will be equivalent to `XXH3_64bits()`. */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset(XXH3_state_t* statePtr); /* * XXH3_64bits_reset_withSeed(): * Generate a custom secret from `seed`, and store it into `statePtr`. * digest will be equivalent to `XXH3_64bits_withSeed()`. */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSeed(XXH3_state_t* statePtr, XXH64_hash_t seed); /* * XXH3_64bits_reset_withSecret(): * `secret` is referenced, it _must outlive_ the hash streaming session. * Similar to one-shot API, `secretSize` must be >= `XXH3_SECRET_SIZE_MIN`, * and the quality of produced hash values depends on secret's entropy * (secret's content should look like a bunch of random bytes). * When in doubt about the randomness of a candidate `secret`, * consider employing `XXH3_generateSecret()` instead (see below). */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecret(XXH3_state_t* statePtr, const void* secret, size_t secretSize); XXH_PUBLIC_API XXH_errorcode XXH3_64bits_update (XXH3_state_t* statePtr, const void* input, size_t length); XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_digest (const XXH3_state_t* statePtr); /* note : canonical representation of XXH3 is the same as XXH64 * since they both produce XXH64_hash_t values */ /*-********************************************************************** * XXH3 128-bit variant ************************************************************************/ /*! * @brief The return value from 128-bit hashes. * * Stored in little endian order, although the fields themselves are in native * endianness. */ typedef struct { XXH64_hash_t low64; /*!< `value & 0xFFFFFFFFFFFFFFFF` */ XXH64_hash_t high64; /*!< `value >> 64` */ } XXH128_hash_t; XXH_PUBLIC_API XXH128_hash_t XXH3_128bits(const void* data, size_t len); XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSeed(const void* data, size_t len, XXH64_hash_t seed); XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSecret(const void* data, size_t len, const void* secret, size_t secretSize); /******* Streaming *******/ /* * Streaming requires state maintenance. * This operation costs memory and CPU. * As a consequence, streaming is slower than one-shot hashing. * For better performance, prefer one-shot functions whenever applicable. * * XXH3_128bits uses the same XXH3_state_t as XXH3_64bits(). * Use already declared XXH3_createState() and XXH3_freeState(). * * All reset and streaming functions have same meaning as their 64-bit counterpart. */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset(XXH3_state_t* statePtr); XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSeed(XXH3_state_t* statePtr, XXH64_hash_t seed); XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecret(XXH3_state_t* statePtr, const void* secret, size_t secretSize); XXH_PUBLIC_API XXH_errorcode XXH3_128bits_update (XXH3_state_t* statePtr, const void* input, size_t length); XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_digest (const XXH3_state_t* statePtr); /* Following helper functions make it possible to compare XXH128_hast_t values. * Since XXH128_hash_t is a structure, this capability is not offered by the language. * Note: For better performance, these functions can be inlined using XXH_INLINE_ALL */ /*! * XXH128_isEqual(): * Return: 1 if `h1` and `h2` are equal, 0 if they are not. */ XXH_PUBLIC_API int XXH128_isEqual(XXH128_hash_t h1, XXH128_hash_t h2); /*! * XXH128_cmp(): * * This comparator is compatible with stdlib's `qsort()`/`bsearch()`. * * return: >0 if *h128_1 > *h128_2 * =0 if *h128_1 == *h128_2 * <0 if *h128_1 < *h128_2 */ XXH_PUBLIC_API int XXH128_cmp(const void* h128_1, const void* h128_2); /******* Canonical representation *******/ typedef struct { unsigned char digest[sizeof(XXH128_hash_t)]; } XXH128_canonical_t; XXH_PUBLIC_API void XXH128_canonicalFromHash(XXH128_canonical_t* dst, XXH128_hash_t hash); XXH_PUBLIC_API XXH128_hash_t XXH128_hashFromCanonical(const XXH128_canonical_t* src); #endif /* !XXH_NO_XXH3 */ #endif /* XXH_NO_LONG_LONG */ /*! * @} */ #endif /* XXHASH_H_5627135585666179 */ #if defined(XXH_STATIC_LINKING_ONLY) && !defined(XXHASH_H_STATIC_13879238742) #define XXHASH_H_STATIC_13879238742 /* **************************************************************************** * This section contains declarations which are not guaranteed to remain stable. * They may change in future versions, becoming incompatible with a different * version of the library. * These declarations should only be used with static linking. * Never use them in association with dynamic linking! ***************************************************************************** */ /* * These definitions are only present to allow static allocation * of XXH states, on stack or in a struct, for example. * Never **ever** access their members directly. */ /*! * @internal * @brief Structure for XXH32 streaming API. * * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. Otherwise it is * an opaque type. This allows fields to safely be changed. * * Typedef'd to @ref XXH32_state_t. * Do not access the members of this struct directly. * @see XXH64_state_s, XXH3_state_s */ struct XXH32_state_s { XXH32_hash_t total_len_32; /*!< Total length hashed, modulo 2^32 */ XXH32_hash_t large_len; /*!< Whether the hash is >= 16 (handles @ref total_len_32 overflow) */ XXH32_hash_t v[4]; /*!< Accumulator lanes */ XXH32_hash_t mem32[4]; /*!< Internal buffer for partial reads. Treated as unsigned char[16]. */ XXH32_hash_t memsize; /*!< Amount of data in @ref mem32 */ XXH32_hash_t reserved; /*!< Reserved field. Do not read nor write to it. */ }; /* typedef'd to XXH32_state_t */ #ifndef XXH_NO_LONG_LONG /* defined when there is no 64-bit support */ /*! * @internal * @brief Structure for XXH64 streaming API. * * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. Otherwise it is * an opaque type. This allows fields to safely be changed. * * Typedef'd to @ref XXH64_state_t. * Do not access the members of this struct directly. * @see XXH32_state_s, XXH3_state_s */ struct XXH64_state_s { XXH64_hash_t total_len; /*!< Total length hashed. This is always 64-bit. */ XXH64_hash_t v[4]; /*!< Accumulator lanes */ XXH64_hash_t mem64[4]; /*!< Internal buffer for partial reads. Treated as unsigned char[32]. */ XXH32_hash_t memsize; /*!< Amount of data in @ref mem64 */ XXH32_hash_t reserved32; /*!< Reserved field, needed for padding anyways*/ XXH64_hash_t reserved64; /*!< Reserved field. Do not read or write to it. */ }; /* typedef'd to XXH64_state_t */ #ifndef XXH_NO_XXH3 #if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L) /* >= C11 */ # include # define XXH_ALIGN(n) alignas(n) #elif defined(__cplusplus) && (__cplusplus >= 201103L) /* >= C++11 */ /* In C++ alignas() is a keyword */ # define XXH_ALIGN(n) alignas(n) #elif defined(__GNUC__) # define XXH_ALIGN(n) __attribute__ ((aligned(n))) #elif defined(_MSC_VER) # define XXH_ALIGN(n) __declspec(align(n)) #else # define XXH_ALIGN(n) /* disabled */ #endif /* Old GCC versions only accept the attribute after the type in structures. */ #if !(defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L)) /* C11+ */ \ && ! (defined(__cplusplus) && (__cplusplus >= 201103L)) /* >= C++11 */ \ && defined(__GNUC__) # define XXH_ALIGN_MEMBER(align, type) type XXH_ALIGN(align) #else # define XXH_ALIGN_MEMBER(align, type) XXH_ALIGN(align) type #endif /*! * @brief The size of the internal XXH3 buffer. * * This is the optimal update size for incremental hashing. * * @see XXH3_64b_update(), XXH3_128b_update(). */ #define XXH3_INTERNALBUFFER_SIZE 256 /*! * @brief Default size of the secret buffer (and @ref XXH3_kSecret). * * This is the size used in @ref XXH3_kSecret and the seeded functions. * * Not to be confused with @ref XXH3_SECRET_SIZE_MIN. */ #define XXH3_SECRET_DEFAULT_SIZE 192 /*! * @internal * @brief Structure for XXH3 streaming API. * * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. * Otherwise it is an opaque type. * Never use this definition in combination with dynamic library. * This allows fields to safely be changed in the future. * * @note ** This structure has a strict alignment requirement of 64 bytes!! ** * Do not allocate this with `malloc()` or `new`, * it will not be sufficiently aligned. * Use @ref XXH3_createState() and @ref XXH3_freeState(), or stack allocation. * * Typedef'd to @ref XXH3_state_t. * Do never access the members of this struct directly. * * @see XXH3_INITSTATE() for stack initialization. * @see XXH3_createState(), XXH3_freeState(). * @see XXH32_state_s, XXH64_state_s */ struct XXH3_state_s { XXH_ALIGN_MEMBER(64, XXH64_hash_t acc[8]); /*!< The 8 accumulators. Similar to `vN` in @ref XXH32_state_s::v1 and @ref XXH64_state_s */ XXH_ALIGN_MEMBER(64, unsigned char customSecret[XXH3_SECRET_DEFAULT_SIZE]); /*!< Used to store a custom secret generated from a seed. */ XXH_ALIGN_MEMBER(64, unsigned char buffer[XXH3_INTERNALBUFFER_SIZE]); /*!< The internal buffer. @see XXH32_state_s::mem32 */ XXH32_hash_t bufferedSize; /*!< The amount of memory in @ref buffer, @see XXH32_state_s::memsize */ XXH32_hash_t useSeed; /*!< Reserved field. Needed for padding on 64-bit. */ size_t nbStripesSoFar; /*!< Number or stripes processed. */ XXH64_hash_t totalLen; /*!< Total length hashed. 64-bit even on 32-bit targets. */ size_t nbStripesPerBlock; /*!< Number of stripes per block. */ size_t secretLimit; /*!< Size of @ref customSecret or @ref extSecret */ XXH64_hash_t seed; /*!< Seed for _withSeed variants. Must be zero otherwise, @see XXH3_INITSTATE() */ XXH64_hash_t reserved64; /*!< Reserved field. */ const unsigned char* extSecret; /*!< Reference to an external secret for the _withSecret variants, NULL * for other variants. */ /* note: there may be some padding at the end due to alignment on 64 bytes */ }; /* typedef'd to XXH3_state_t */ #undef XXH_ALIGN_MEMBER /*! * @brief Initializes a stack-allocated `XXH3_state_s`. * * When the @ref XXH3_state_t structure is merely emplaced on stack, * it should be initialized with XXH3_INITSTATE() or a memset() * in case its first reset uses XXH3_NNbits_reset_withSeed(). * This init can be omitted if the first reset uses default or _withSecret mode. * This operation isn't necessary when the state is created with XXH3_createState(). * Note that this doesn't prepare the state for a streaming operation, * it's still necessary to use XXH3_NNbits_reset*() afterwards. */ #define XXH3_INITSTATE(XXH3_state_ptr) { (XXH3_state_ptr)->seed = 0; } /* XXH128() : * simple alias to pre-selected XXH3_128bits variant */ XXH_PUBLIC_API XXH128_hash_t XXH128(const void* data, size_t len, XXH64_hash_t seed); /* === Experimental API === */ /* Symbols defined below must be considered tied to a specific library version. */ /* * XXH3_generateSecret(): * * Derive a high-entropy secret from any user-defined content, named customSeed. * The generated secret can be used in combination with `*_withSecret()` functions. * The `_withSecret()` variants are useful to provide a higher level of protection than 64-bit seed, * as it becomes much more difficult for an external actor to guess how to impact the calculation logic. * * The function accepts as input a custom seed of any length and any content, * and derives from it a high-entropy secret of length @secretSize * into an already allocated buffer @secretBuffer. * @secretSize must be >= XXH3_SECRET_SIZE_MIN * * The generated secret can then be used with any `*_withSecret()` variant. * Functions `XXH3_128bits_withSecret()`, `XXH3_64bits_withSecret()`, * `XXH3_128bits_reset_withSecret()` and `XXH3_64bits_reset_withSecret()` * are part of this list. They all accept a `secret` parameter * which must be large enough for implementation reasons (>= XXH3_SECRET_SIZE_MIN) * _and_ feature very high entropy (consist of random-looking bytes). * These conditions can be a high bar to meet, so * XXH3_generateSecret() can be employed to ensure proper quality. * * customSeed can be anything. It can have any size, even small ones, * and its content can be anything, even "poor entropy" sources such as a bunch of zeroes. * The resulting `secret` will nonetheless provide all required qualities. * * When customSeedSize > 0, supplying NULL as customSeed is undefined behavior. */ XXH_PUBLIC_API XXH_errorcode XXH3_generateSecret(void* secretBuffer, size_t secretSize, const void* customSeed, size_t customSeedSize); /* * XXH3_generateSecret_fromSeed(): * * Generate the same secret as the _withSeed() variants. * * The resulting secret has a length of XXH3_SECRET_DEFAULT_SIZE (necessarily). * @secretBuffer must be already allocated, of size at least XXH3_SECRET_DEFAULT_SIZE bytes. * * The generated secret can be used in combination with *`*_withSecret()` and `_withSecretandSeed()` variants. * This generator is notably useful in combination with `_withSecretandSeed()`, * as a way to emulate a faster `_withSeed()` variant. */ XXH_PUBLIC_API void XXH3_generateSecret_fromSeed(void* secretBuffer, XXH64_hash_t seed); /* * *_withSecretandSeed() : * These variants generate hash values using either * @seed for "short" keys (< XXH3_MIDSIZE_MAX = 240 bytes) * or @secret for "large" keys (>= XXH3_MIDSIZE_MAX). * * This generally benefits speed, compared to `_withSeed()` or `_withSecret()`. * `_withSeed()` has to generate the secret on the fly for "large" keys. * It's fast, but can be perceptible for "not so large" keys (< 1 KB). * `_withSecret()` has to generate the masks on the fly for "small" keys, * which requires more instructions than _withSeed() variants. * Therefore, _withSecretandSeed variant combines the best of both worlds. * * When @secret has been generated by XXH3_generateSecret_fromSeed(), * this variant produces *exactly* the same results as `_withSeed()` variant, * hence offering only a pure speed benefit on "large" input, * by skipping the need to regenerate the secret for every large input. * * Another usage scenario is to hash the secret to a 64-bit hash value, * for example with XXH3_64bits(), which then becomes the seed, * and then employ both the seed and the secret in _withSecretandSeed(). * On top of speed, an added benefit is that each bit in the secret * has a 50% chance to swap each bit in the output, * via its impact to the seed. * This is not guaranteed when using the secret directly in "small data" scenarios, * because only portions of the secret are employed for small data. */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSecretandSeed(const void* data, size_t len, const void* secret, size_t secretSize, XXH64_hash_t seed); XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSecretandSeed(const void* data, size_t len, const void* secret, size_t secretSize, XXH64_hash_t seed64); XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecretandSeed(XXH3_state_t* statePtr, const void* secret, size_t secretSize, XXH64_hash_t seed64); XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecretandSeed(XXH3_state_t* statePtr, const void* secret, size_t secretSize, XXH64_hash_t seed64); #endif /* XXH_NO_XXH3 */ #endif /* XXH_NO_LONG_LONG */ #if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API) # define XXH_IMPLEMENTATION #endif #endif /* defined(XXH_STATIC_LINKING_ONLY) && !defined(XXHASH_H_STATIC_13879238742) */ /* ======================================================================== */ /* ======================================================================== */ /* ======================================================================== */ /*-********************************************************************** * xxHash implementation *-********************************************************************** * xxHash's implementation used to be hosted inside xxhash.c. * * However, inlining requires implementation to be visible to the compiler, * hence be included alongside the header. * Previously, implementation was hosted inside xxhash.c, * which was then #included when inlining was activated. * This construction created issues with a few build and install systems, * as it required xxhash.c to be stored in /include directory. * * xxHash implementation is now directly integrated within xxhash.h. * As a consequence, xxhash.c is no longer needed in /include. * * xxhash.c is still available and is still useful. * In a "normal" setup, when xxhash is not inlined, * xxhash.h only exposes the prototypes and public symbols, * while xxhash.c can be built into an object file xxhash.o * which can then be linked into the final binary. ************************************************************************/ #if ( defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API) \ || defined(XXH_IMPLEMENTATION) ) && !defined(XXH_IMPLEM_13a8737387) # define XXH_IMPLEM_13a8737387 /* ************************************* * Tuning parameters ***************************************/ /*! * @defgroup tuning Tuning parameters * @{ * * Various macros to control xxHash's behavior. */ #ifdef XXH_DOXYGEN /*! * @brief Define this to disable 64-bit code. * * Useful if only using the @ref xxh32_family and you have a strict C90 compiler. */ # define XXH_NO_LONG_LONG # undef XXH_NO_LONG_LONG /* don't actually */ /*! * @brief Controls how unaligned memory is accessed. * * By default, access to unaligned memory is controlled by `memcpy()`, which is * safe and portable. * * Unfortunately, on some target/compiler combinations, the generated assembly * is sub-optimal. * * The below switch allow selection of a different access method * in the search for improved performance. * * @par Possible options: * * - `XXH_FORCE_MEMORY_ACCESS=0` (default): `memcpy` * @par * Use `memcpy()`. Safe and portable. Note that most modern compilers will * eliminate the function call and treat it as an unaligned access. * * - `XXH_FORCE_MEMORY_ACCESS=1`: `__attribute__((packed))` * @par * Depends on compiler extensions and is therefore not portable. * This method is safe _if_ your compiler supports it, * and *generally* as fast or faster than `memcpy`. * * - `XXH_FORCE_MEMORY_ACCESS=2`: Direct cast * @par * Casts directly and dereferences. This method doesn't depend on the * compiler, but it violates the C standard as it directly dereferences an * unaligned pointer. It can generate buggy code on targets which do not * support unaligned memory accesses, but in some circumstances, it's the * only known way to get the most performance. * * - `XXH_FORCE_MEMORY_ACCESS=3`: Byteshift * @par * Also portable. This can generate the best code on old compilers which don't * inline small `memcpy()` calls, and it might also be faster on big-endian * systems which lack a native byteswap instruction. However, some compilers * will emit literal byteshifts even if the target supports unaligned access. * . * * @warning * Methods 1 and 2 rely on implementation-defined behavior. Use these with * care, as what works on one compiler/platform/optimization level may cause * another to read garbage data or even crash. * * See https://fastcompression.blogspot.com/2015/08/accessing-unaligned-memory.html for details. * * Prefer these methods in priority order (0 > 3 > 1 > 2) */ # define XXH_FORCE_MEMORY_ACCESS 0 /*! * @def XXH_FORCE_ALIGN_CHECK * @brief If defined to non-zero, adds a special path for aligned inputs (XXH32() * and XXH64() only). * * This is an important performance trick for architectures without decent * unaligned memory access performance. * * It checks for input alignment, and when conditions are met, uses a "fast * path" employing direct 32-bit/64-bit reads, resulting in _dramatically * faster_ read speed. * * The check costs one initial branch per hash, which is generally negligible, * but not zero. * * Moreover, it's not useful to generate an additional code path if memory * access uses the same instruction for both aligned and unaligned * addresses (e.g. x86 and aarch64). * * In these cases, the alignment check can be removed by setting this macro to 0. * Then the code will always use unaligned memory access. * Align check is automatically disabled on x86, x64 & arm64, * which are platforms known to offer good unaligned memory accesses performance. * * This option does not affect XXH3 (only XXH32 and XXH64). */ # define XXH_FORCE_ALIGN_CHECK 0 /*! * @def XXH_NO_INLINE_HINTS * @brief When non-zero, sets all functions to `static`. * * By default, xxHash tries to force the compiler to inline almost all internal * functions. * * This can usually improve performance due to reduced jumping and improved * constant folding, but significantly increases the size of the binary which * might not be favorable. * * Additionally, sometimes the forced inlining can be detrimental to performance, * depending on the architecture. * * XXH_NO_INLINE_HINTS marks all internal functions as static, giving the * compiler full control on whether to inline or not. * * When not optimizing (-O0), optimizing for size (-Os, -Oz), or using * -fno-inline with GCC or Clang, this will automatically be defined. */ # define XXH_NO_INLINE_HINTS 0 /*! * @def XXH32_ENDJMP * @brief Whether to use a jump for `XXH32_finalize`. * * For performance, `XXH32_finalize` uses multiple branches in the finalizer. * This is generally preferable for performance, * but depending on exact architecture, a jmp may be preferable. * * This setting is only possibly making a difference for very small inputs. */ # define XXH32_ENDJMP 0 /*! * @internal * @brief Redefines old internal names. * * For compatibility with code that uses xxHash's internals before the names * were changed to improve namespacing. There is no other reason to use this. */ # define XXH_OLD_NAMES # undef XXH_OLD_NAMES /* don't actually use, it is ugly. */ #endif /* XXH_DOXYGEN */ /*! * @} */ #ifndef XXH_FORCE_MEMORY_ACCESS /* can be defined externally, on command line for example */ /* prefer __packed__ structures (method 1) for gcc on armv7+ and mips */ # if !defined(__clang__) && \ ( \ (defined(__INTEL_COMPILER) && !defined(_WIN32)) || \ ( \ defined(__GNUC__) && ( \ (defined(__ARM_ARCH) && __ARM_ARCH >= 7) || \ ( \ defined(__mips__) && \ (__mips <= 5 || __mips_isa_rev < 6) && \ (!defined(__mips16) || defined(__mips_mips16e2)) \ ) \ ) \ ) \ ) # define XXH_FORCE_MEMORY_ACCESS 1 # endif #endif #ifndef XXH_FORCE_ALIGN_CHECK /* can be defined externally */ # if defined(__i386) || defined(__x86_64__) || defined(__aarch64__) \ || defined(_M_IX86) || defined(_M_X64) || defined(_M_ARM64) /* visual */ # define XXH_FORCE_ALIGN_CHECK 0 # else # define XXH_FORCE_ALIGN_CHECK 1 # endif #endif #ifndef XXH_NO_INLINE_HINTS # if defined(__OPTIMIZE_SIZE__) /* -Os, -Oz */ \ || defined(__NO_INLINE__) /* -O0, -fno-inline */ # define XXH_NO_INLINE_HINTS 1 # else # define XXH_NO_INLINE_HINTS 0 # endif #endif #ifndef XXH32_ENDJMP /* generally preferable for performance */ # define XXH32_ENDJMP 0 #endif /*! * @defgroup impl Implementation * @{ */ /* ************************************* * Includes & Memory related functions ***************************************/ /* Modify the local functions below should you wish to use some other memory routines */ /* for ZSTD_malloc(), ZSTD_free() */ #define ZSTD_DEPS_NEED_MALLOC #include "zstd_deps.h" /* size_t, ZSTD_malloc, ZSTD_free, ZSTD_memcpy */ static void* XXH_malloc(size_t s) { return ZSTD_malloc(s); } static void XXH_free (void* p) { ZSTD_free(p); } static void* XXH_memcpy(void* dest, const void* src, size_t size) { return ZSTD_memcpy(dest,src,size); } /* ************************************* * Compiler Specific Options ***************************************/ #ifdef _MSC_VER /* Visual Studio warning fix */ # pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ #endif #if XXH_NO_INLINE_HINTS /* disable inlining hints */ # if defined(__GNUC__) || defined(__clang__) # define XXH_FORCE_INLINE static __attribute__((unused)) # else # define XXH_FORCE_INLINE static # endif # define XXH_NO_INLINE static /* enable inlining hints */ #elif defined(__GNUC__) || defined(__clang__) # define XXH_FORCE_INLINE static __inline__ __attribute__((always_inline, unused)) # define XXH_NO_INLINE static __attribute__((noinline)) #elif defined(_MSC_VER) /* Visual Studio */ # define XXH_FORCE_INLINE static __forceinline # define XXH_NO_INLINE static __declspec(noinline) #elif defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L)) /* C99 */ # define XXH_FORCE_INLINE static inline # define XXH_NO_INLINE static #else # define XXH_FORCE_INLINE static # define XXH_NO_INLINE static #endif /* ************************************* * Debug ***************************************/ /*! * @ingroup tuning * @def XXH_DEBUGLEVEL * @brief Sets the debugging level. * * XXH_DEBUGLEVEL is expected to be defined externally, typically via the * compiler's command line options. The value must be a number. */ #ifndef XXH_DEBUGLEVEL # ifdef DEBUGLEVEL /* backwards compat */ # define XXH_DEBUGLEVEL DEBUGLEVEL # else # define XXH_DEBUGLEVEL 0 # endif #endif #if (XXH_DEBUGLEVEL>=1) # include /* note: can still be disabled with NDEBUG */ # define XXH_ASSERT(c) assert(c) #else # define XXH_ASSERT(c) ((void)0) #endif /* note: use after variable declarations */ #ifndef XXH_STATIC_ASSERT # if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L) /* C11 */ # include # define XXH_STATIC_ASSERT_WITH_MESSAGE(c,m) do { static_assert((c),m); } while(0) # elif defined(__cplusplus) && (__cplusplus >= 201103L) /* C++11 */ # define XXH_STATIC_ASSERT_WITH_MESSAGE(c,m) do { static_assert((c),m); } while(0) # else # define XXH_STATIC_ASSERT_WITH_MESSAGE(c,m) do { struct xxh_sa { char x[(c) ? 1 : -1]; }; } while(0) # endif # define XXH_STATIC_ASSERT(c) XXH_STATIC_ASSERT_WITH_MESSAGE((c),#c) #endif /*! * @internal * @def XXH_COMPILER_GUARD(var) * @brief Used to prevent unwanted optimizations for @p var. * * It uses an empty GCC inline assembly statement with a register constraint * which forces @p var into a general purpose register (e.g. eax, ebx, ecx * on x86) and marks it as modified. * * This is used in a few places to avoid unwanted autovectorization (e.g. * XXH32_round()). All vectorization we want is explicit via intrinsics, * and _usually_ isn't wanted elsewhere. * * We also use it to prevent unwanted constant folding for AArch64 in * XXH3_initCustomSecret_scalar(). */ #if defined(__GNUC__) || defined(__clang__) # define XXH_COMPILER_GUARD(var) __asm__ __volatile__("" : "+r" (var)) #else # define XXH_COMPILER_GUARD(var) ((void)0) #endif /* ************************************* * Basic Types ***************************************/ #if !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) # include typedef uint8_t xxh_u8; #else typedef unsigned char xxh_u8; #endif typedef XXH32_hash_t xxh_u32; #ifdef XXH_OLD_NAMES # define BYTE xxh_u8 # define U8 xxh_u8 # define U32 xxh_u32 #endif /* *** Memory access *** */ /*! * @internal * @fn xxh_u32 XXH_read32(const void* ptr) * @brief Reads an unaligned 32-bit integer from @p ptr in native endianness. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * * @param ptr The pointer to read from. * @return The 32-bit native endian integer from the bytes at @p ptr. */ /*! * @internal * @fn xxh_u32 XXH_readLE32(const void* ptr) * @brief Reads an unaligned 32-bit little endian integer from @p ptr. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * * @param ptr The pointer to read from. * @return The 32-bit little endian integer from the bytes at @p ptr. */ /*! * @internal * @fn xxh_u32 XXH_readBE32(const void* ptr) * @brief Reads an unaligned 32-bit big endian integer from @p ptr. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * * @param ptr The pointer to read from. * @return The 32-bit big endian integer from the bytes at @p ptr. */ /*! * @internal * @fn xxh_u32 XXH_readLE32_align(const void* ptr, XXH_alignment align) * @brief Like @ref XXH_readLE32(), but has an option for aligned reads. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * Note that when @ref XXH_FORCE_ALIGN_CHECK == 0, the @p align parameter is * always @ref XXH_alignment::XXH_unaligned. * * @param ptr The pointer to read from. * @param align Whether @p ptr is aligned. * @pre * If @p align == @ref XXH_alignment::XXH_aligned, @p ptr must be 4 byte * aligned. * @return The 32-bit little endian integer from the bytes at @p ptr. */ #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) /* * Manual byteshift. Best for old compilers which don't inline memcpy. * We actually directly use XXH_readLE32 and XXH_readBE32. */ #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2)) /* * Force direct memory access. Only works on CPU which support unaligned memory * access in hardware. */ static xxh_u32 XXH_read32(const void* memPtr) { return *(const xxh_u32*) memPtr; } #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1)) /* * __pack instructions are safer but compiler specific, hence potentially * problematic for some compilers. * * Currently only defined for GCC and ICC. */ #ifdef XXH_OLD_NAMES typedef union { xxh_u32 u32; } __attribute__((packed)) unalign; #endif static xxh_u32 XXH_read32(const void* ptr) { typedef union { xxh_u32 u32; } __attribute__((packed)) xxh_unalign; return ((const xxh_unalign*)ptr)->u32; } #else /* * Portable and safe solution. Generally efficient. * see: https://fastcompression.blogspot.com/2015/08/accessing-unaligned-memory.html */ static xxh_u32 XXH_read32(const void* memPtr) { xxh_u32 val; XXH_memcpy(&val, memPtr, sizeof(val)); return val; } #endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */ /* *** Endianness *** */ /*! * @ingroup tuning * @def XXH_CPU_LITTLE_ENDIAN * @brief Whether the target is little endian. * * Defined to 1 if the target is little endian, or 0 if it is big endian. * It can be defined externally, for example on the compiler command line. * * If it is not defined, * a runtime check (which is usually constant folded) is used instead. * * @note * This is not necessarily defined to an integer constant. * * @see XXH_isLittleEndian() for the runtime check. */ #ifndef XXH_CPU_LITTLE_ENDIAN /* * Try to detect endianness automatically, to avoid the nonstandard behavior * in `XXH_isLittleEndian()` */ # if defined(_WIN32) /* Windows is always little endian */ \ || defined(__LITTLE_ENDIAN__) \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) # define XXH_CPU_LITTLE_ENDIAN 1 # elif defined(__BIG_ENDIAN__) \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) # define XXH_CPU_LITTLE_ENDIAN 0 # else /*! * @internal * @brief Runtime check for @ref XXH_CPU_LITTLE_ENDIAN. * * Most compilers will constant fold this. */ static int XXH_isLittleEndian(void) { /* * Portable and well-defined behavior. * Don't use static: it is detrimental to performance. */ const union { xxh_u32 u; xxh_u8 c[4]; } one = { 1 }; return one.c[0]; } # define XXH_CPU_LITTLE_ENDIAN XXH_isLittleEndian() # endif #endif /* **************************************** * Compiler-specific Functions and Macros ******************************************/ #define XXH_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) #ifdef __has_builtin # define XXH_HAS_BUILTIN(x) __has_builtin(x) #else # define XXH_HAS_BUILTIN(x) 0 #endif /*! * @internal * @def XXH_rotl32(x,r) * @brief 32-bit rotate left. * * @param x The 32-bit integer to be rotated. * @param r The number of bits to rotate. * @pre * @p r > 0 && @p r < 32 * @note * @p x and @p r may be evaluated multiple times. * @return The rotated result. */ #if !defined(NO_CLANG_BUILTIN) && XXH_HAS_BUILTIN(__builtin_rotateleft32) \ && XXH_HAS_BUILTIN(__builtin_rotateleft64) # define XXH_rotl32 __builtin_rotateleft32 # define XXH_rotl64 __builtin_rotateleft64 /* Note: although _rotl exists for minGW (GCC under windows), performance seems poor */ #elif defined(_MSC_VER) # define XXH_rotl32(x,r) _rotl(x,r) # define XXH_rotl64(x,r) _rotl64(x,r) #else # define XXH_rotl32(x,r) (((x) << (r)) | ((x) >> (32 - (r)))) # define XXH_rotl64(x,r) (((x) << (r)) | ((x) >> (64 - (r)))) #endif /*! * @internal * @fn xxh_u32 XXH_swap32(xxh_u32 x) * @brief A 32-bit byteswap. * * @param x The 32-bit integer to byteswap. * @return @p x, byteswapped. */ #if defined(_MSC_VER) /* Visual Studio */ # define XXH_swap32 _byteswap_ulong #elif XXH_GCC_VERSION >= 403 # define XXH_swap32 __builtin_bswap32 #else static xxh_u32 XXH_swap32 (xxh_u32 x) { return ((x << 24) & 0xff000000 ) | ((x << 8) & 0x00ff0000 ) | ((x >> 8) & 0x0000ff00 ) | ((x >> 24) & 0x000000ff ); } #endif /* *************************** * Memory reads *****************************/ /*! * @internal * @brief Enum to indicate whether a pointer is aligned. */ typedef enum { XXH_aligned, /*!< Aligned */ XXH_unaligned /*!< Possibly unaligned */ } XXH_alignment; /* * XXH_FORCE_MEMORY_ACCESS==3 is an endian-independent byteshift load. * * This is ideal for older compilers which don't inline memcpy. */ #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) XXH_FORCE_INLINE xxh_u32 XXH_readLE32(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[0] | ((xxh_u32)bytePtr[1] << 8) | ((xxh_u32)bytePtr[2] << 16) | ((xxh_u32)bytePtr[3] << 24); } XXH_FORCE_INLINE xxh_u32 XXH_readBE32(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[3] | ((xxh_u32)bytePtr[2] << 8) | ((xxh_u32)bytePtr[1] << 16) | ((xxh_u32)bytePtr[0] << 24); } #else XXH_FORCE_INLINE xxh_u32 XXH_readLE32(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr)); } static xxh_u32 XXH_readBE32(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_swap32(XXH_read32(ptr)) : XXH_read32(ptr); } #endif XXH_FORCE_INLINE xxh_u32 XXH_readLE32_align(const void* ptr, XXH_alignment align) { if (align==XXH_unaligned) { return XXH_readLE32(ptr); } else { return XXH_CPU_LITTLE_ENDIAN ? *(const xxh_u32*)ptr : XXH_swap32(*(const xxh_u32*)ptr); } } /* ************************************* * Misc ***************************************/ /*! @ingroup public */ XXH_PUBLIC_API unsigned XXH_versionNumber (void) { return XXH_VERSION_NUMBER; } /* ******************************************************************* * 32-bit hash functions *********************************************************************/ /*! * @} * @defgroup xxh32_impl XXH32 implementation * @ingroup impl * @{ */ /* #define instead of static const, to be used as initializers */ #define XXH_PRIME32_1 0x9E3779B1U /*!< 0b10011110001101110111100110110001 */ #define XXH_PRIME32_2 0x85EBCA77U /*!< 0b10000101111010111100101001110111 */ #define XXH_PRIME32_3 0xC2B2AE3DU /*!< 0b11000010101100101010111000111101 */ #define XXH_PRIME32_4 0x27D4EB2FU /*!< 0b00100111110101001110101100101111 */ #define XXH_PRIME32_5 0x165667B1U /*!< 0b00010110010101100110011110110001 */ #ifdef XXH_OLD_NAMES # define PRIME32_1 XXH_PRIME32_1 # define PRIME32_2 XXH_PRIME32_2 # define PRIME32_3 XXH_PRIME32_3 # define PRIME32_4 XXH_PRIME32_4 # define PRIME32_5 XXH_PRIME32_5 #endif /*! * @internal * @brief Normal stripe processing routine. * * This shuffles the bits so that any bit from @p input impacts several bits in * @p acc. * * @param acc The accumulator lane. * @param input The stripe of input to mix. * @return The mixed accumulator lane. */ static xxh_u32 XXH32_round(xxh_u32 acc, xxh_u32 input) { acc += input * XXH_PRIME32_2; acc = XXH_rotl32(acc, 13); acc *= XXH_PRIME32_1; #if (defined(__SSE4_1__) || defined(__aarch64__)) && !defined(XXH_ENABLE_AUTOVECTORIZE) /* * UGLY HACK: * A compiler fence is the only thing that prevents GCC and Clang from * autovectorizing the XXH32 loop (pragmas and attributes don't work for some * reason) without globally disabling SSE4.1. * * The reason we want to avoid vectorization is because despite working on * 4 integers at a time, there are multiple factors slowing XXH32 down on * SSE4: * - There's a ridiculous amount of lag from pmulld (10 cycles of latency on * newer chips!) making it slightly slower to multiply four integers at * once compared to four integers independently. Even when pmulld was * fastest, Sandy/Ivy Bridge, it is still not worth it to go into SSE * just to multiply unless doing a long operation. * * - Four instructions are required to rotate, * movqda tmp, v // not required with VEX encoding * pslld tmp, 13 // tmp <<= 13 * psrld v, 19 // x >>= 19 * por v, tmp // x |= tmp * compared to one for scalar: * roll v, 13 // reliably fast across the board * shldl v, v, 13 // Sandy Bridge and later prefer this for some reason * * - Instruction level parallelism is actually more beneficial here because * the SIMD actually serializes this operation: While v1 is rotating, v2 * can load data, while v3 can multiply. SSE forces them to operate * together. * * This is also enabled on AArch64, as Clang autovectorizes it incorrectly * and it is pointless writing a NEON implementation that is basically the * same speed as scalar for XXH32. */ XXH_COMPILER_GUARD(acc); #endif return acc; } /*! * @internal * @brief Mixes all bits to finalize the hash. * * The final mix ensures that all input bits have a chance to impact any bit in * the output digest, resulting in an unbiased distribution. * * @param h32 The hash to avalanche. * @return The avalanched hash. */ static xxh_u32 XXH32_avalanche(xxh_u32 h32) { h32 ^= h32 >> 15; h32 *= XXH_PRIME32_2; h32 ^= h32 >> 13; h32 *= XXH_PRIME32_3; h32 ^= h32 >> 16; return(h32); } #define XXH_get32bits(p) XXH_readLE32_align(p, align) /*! * @internal * @brief Processes the last 0-15 bytes of @p ptr. * * There may be up to 15 bytes remaining to consume from the input. * This final stage will digest them to ensure that all input bytes are present * in the final mix. * * @param h32 The hash to finalize. * @param ptr The pointer to the remaining input. * @param len The remaining length, modulo 16. * @param align Whether @p ptr is aligned. * @return The finalized hash. */ static xxh_u32 XXH32_finalize(xxh_u32 h32, const xxh_u8* ptr, size_t len, XXH_alignment align) { #define XXH_PROCESS1 do { \ h32 += (*ptr++) * XXH_PRIME32_5; \ h32 = XXH_rotl32(h32, 11) * XXH_PRIME32_1; \ } while (0) #define XXH_PROCESS4 do { \ h32 += XXH_get32bits(ptr) * XXH_PRIME32_3; \ ptr += 4; \ h32 = XXH_rotl32(h32, 17) * XXH_PRIME32_4; \ } while (0) if (ptr==NULL) XXH_ASSERT(len == 0); /* Compact rerolled version; generally faster */ if (!XXH32_ENDJMP) { len &= 15; while (len >= 4) { XXH_PROCESS4; len -= 4; } while (len > 0) { XXH_PROCESS1; --len; } return XXH32_avalanche(h32); } else { switch(len&15) /* or switch(bEnd - p) */ { case 12: XXH_PROCESS4; XXH_FALLTHROUGH; case 8: XXH_PROCESS4; XXH_FALLTHROUGH; case 4: XXH_PROCESS4; return XXH32_avalanche(h32); case 13: XXH_PROCESS4; XXH_FALLTHROUGH; case 9: XXH_PROCESS4; XXH_FALLTHROUGH; case 5: XXH_PROCESS4; XXH_PROCESS1; return XXH32_avalanche(h32); case 14: XXH_PROCESS4; XXH_FALLTHROUGH; case 10: XXH_PROCESS4; XXH_FALLTHROUGH; case 6: XXH_PROCESS4; XXH_PROCESS1; XXH_PROCESS1; return XXH32_avalanche(h32); case 15: XXH_PROCESS4; XXH_FALLTHROUGH; case 11: XXH_PROCESS4; XXH_FALLTHROUGH; case 7: XXH_PROCESS4; XXH_FALLTHROUGH; case 3: XXH_PROCESS1; XXH_FALLTHROUGH; case 2: XXH_PROCESS1; XXH_FALLTHROUGH; case 1: XXH_PROCESS1; XXH_FALLTHROUGH; case 0: return XXH32_avalanche(h32); } XXH_ASSERT(0); return h32; /* reaching this point is deemed impossible */ } } #ifdef XXH_OLD_NAMES # define PROCESS1 XXH_PROCESS1 # define PROCESS4 XXH_PROCESS4 #else # undef XXH_PROCESS1 # undef XXH_PROCESS4 #endif /*! * @internal * @brief The implementation for @ref XXH32(). * * @param input , len , seed Directly passed from @ref XXH32(). * @param align Whether @p input is aligned. * @return The calculated hash. */ XXH_FORCE_INLINE xxh_u32 XXH32_endian_align(const xxh_u8* input, size_t len, xxh_u32 seed, XXH_alignment align) { xxh_u32 h32; if (input==NULL) XXH_ASSERT(len == 0); if (len>=16) { const xxh_u8* const bEnd = input + len; const xxh_u8* const limit = bEnd - 15; xxh_u32 v1 = seed + XXH_PRIME32_1 + XXH_PRIME32_2; xxh_u32 v2 = seed + XXH_PRIME32_2; xxh_u32 v3 = seed + 0; xxh_u32 v4 = seed - XXH_PRIME32_1; do { v1 = XXH32_round(v1, XXH_get32bits(input)); input += 4; v2 = XXH32_round(v2, XXH_get32bits(input)); input += 4; v3 = XXH32_round(v3, XXH_get32bits(input)); input += 4; v4 = XXH32_round(v4, XXH_get32bits(input)); input += 4; } while (input < limit); h32 = XXH_rotl32(v1, 1) + XXH_rotl32(v2, 7) + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18); } else { h32 = seed + XXH_PRIME32_5; } h32 += (xxh_u32)len; return XXH32_finalize(h32, input, len&15, align); } /*! @ingroup xxh32_family */ XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t len, XXH32_hash_t seed) { #if 0 /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ XXH32_state_t state; XXH32_reset(&state, seed); XXH32_update(&state, (const xxh_u8*)input, len); return XXH32_digest(&state); #else if (XXH_FORCE_ALIGN_CHECK) { if ((((size_t)input) & 3) == 0) { /* Input is 4-bytes aligned, leverage the speed benefit */ return XXH32_endian_align((const xxh_u8*)input, len, seed, XXH_aligned); } } return XXH32_endian_align((const xxh_u8*)input, len, seed, XXH_unaligned); #endif } /******* Hash streaming *******/ /*! * @ingroup xxh32_family */ XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void) { return (XXH32_state_t*)XXH_malloc(sizeof(XXH32_state_t)); } /*! @ingroup xxh32_family */ XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr) { XXH_free(statePtr); return XXH_OK; } /*! @ingroup xxh32_family */ XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dstState, const XXH32_state_t* srcState) { XXH_memcpy(dstState, srcState, sizeof(*dstState)); } /*! @ingroup xxh32_family */ XXH_PUBLIC_API XXH_errorcode XXH32_reset(XXH32_state_t* statePtr, XXH32_hash_t seed) { XXH_ASSERT(statePtr != NULL); memset(statePtr, 0, sizeof(*statePtr)); statePtr->v[0] = seed + XXH_PRIME32_1 + XXH_PRIME32_2; statePtr->v[1] = seed + XXH_PRIME32_2; statePtr->v[2] = seed + 0; statePtr->v[3] = seed - XXH_PRIME32_1; return XXH_OK; } /*! @ingroup xxh32_family */ XXH_PUBLIC_API XXH_errorcode XXH32_update(XXH32_state_t* state, const void* input, size_t len) { if (input==NULL) { XXH_ASSERT(len == 0); return XXH_OK; } { const xxh_u8* p = (const xxh_u8*)input; const xxh_u8* const bEnd = p + len; state->total_len_32 += (XXH32_hash_t)len; state->large_len |= (XXH32_hash_t)((len>=16) | (state->total_len_32>=16)); if (state->memsize + len < 16) { /* fill in tmp buffer */ XXH_memcpy((xxh_u8*)(state->mem32) + state->memsize, input, len); state->memsize += (XXH32_hash_t)len; return XXH_OK; } if (state->memsize) { /* some data left from previous update */ XXH_memcpy((xxh_u8*)(state->mem32) + state->memsize, input, 16-state->memsize); { const xxh_u32* p32 = state->mem32; state->v[0] = XXH32_round(state->v[0], XXH_readLE32(p32)); p32++; state->v[1] = XXH32_round(state->v[1], XXH_readLE32(p32)); p32++; state->v[2] = XXH32_round(state->v[2], XXH_readLE32(p32)); p32++; state->v[3] = XXH32_round(state->v[3], XXH_readLE32(p32)); } p += 16-state->memsize; state->memsize = 0; } if (p <= bEnd-16) { const xxh_u8* const limit = bEnd - 16; do { state->v[0] = XXH32_round(state->v[0], XXH_readLE32(p)); p+=4; state->v[1] = XXH32_round(state->v[1], XXH_readLE32(p)); p+=4; state->v[2] = XXH32_round(state->v[2], XXH_readLE32(p)); p+=4; state->v[3] = XXH32_round(state->v[3], XXH_readLE32(p)); p+=4; } while (p<=limit); } if (p < bEnd) { XXH_memcpy(state->mem32, p, (size_t)(bEnd-p)); state->memsize = (unsigned)(bEnd-p); } } return XXH_OK; } /*! @ingroup xxh32_family */ XXH_PUBLIC_API XXH32_hash_t XXH32_digest(const XXH32_state_t* state) { xxh_u32 h32; if (state->large_len) { h32 = XXH_rotl32(state->v[0], 1) + XXH_rotl32(state->v[1], 7) + XXH_rotl32(state->v[2], 12) + XXH_rotl32(state->v[3], 18); } else { h32 = state->v[2] /* == seed */ + XXH_PRIME32_5; } h32 += state->total_len_32; return XXH32_finalize(h32, (const xxh_u8*)state->mem32, state->memsize, XXH_aligned); } /******* Canonical representation *******/ /*! * @ingroup xxh32_family * The default return values from XXH functions are unsigned 32 and 64 bit * integers. * * The canonical representation uses big endian convention, the same convention * as human-readable numbers (large digits first). * * This way, hash values can be written into a file or buffer, remaining * comparable across different systems. * * The following functions allow transformation of hash values to and from their * canonical format. */ XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash) { /* XXH_STATIC_ASSERT(sizeof(XXH32_canonical_t) == sizeof(XXH32_hash_t)); */ if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap32(hash); XXH_memcpy(dst, &hash, sizeof(*dst)); } /*! @ingroup xxh32_family */ XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src) { return XXH_readBE32(src); } #ifndef XXH_NO_LONG_LONG /* ******************************************************************* * 64-bit hash functions *********************************************************************/ /*! * @} * @ingroup impl * @{ */ /******* Memory access *******/ typedef XXH64_hash_t xxh_u64; #ifdef XXH_OLD_NAMES # define U64 xxh_u64 #endif #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) /* * Manual byteshift. Best for old compilers which don't inline memcpy. * We actually directly use XXH_readLE64 and XXH_readBE64. */ #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2)) /* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */ static xxh_u64 XXH_read64(const void* memPtr) { return *(const xxh_u64*) memPtr; } #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1)) /* * __pack instructions are safer, but compiler specific, hence potentially * problematic for some compilers. * * Currently only defined for GCC and ICC. */ #ifdef XXH_OLD_NAMES typedef union { xxh_u32 u32; xxh_u64 u64; } __attribute__((packed)) unalign64; #endif static xxh_u64 XXH_read64(const void* ptr) { typedef union { xxh_u32 u32; xxh_u64 u64; } __attribute__((packed)) xxh_unalign64; return ((const xxh_unalign64*)ptr)->u64; } #else /* * Portable and safe solution. Generally efficient. * see: https://fastcompression.blogspot.com/2015/08/accessing-unaligned-memory.html */ static xxh_u64 XXH_read64(const void* memPtr) { xxh_u64 val; XXH_memcpy(&val, memPtr, sizeof(val)); return val; } #endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */ #if defined(_MSC_VER) /* Visual Studio */ # define XXH_swap64 _byteswap_uint64 #elif XXH_GCC_VERSION >= 403 # define XXH_swap64 __builtin_bswap64 #else static xxh_u64 XXH_swap64(xxh_u64 x) { return ((x << 56) & 0xff00000000000000ULL) | ((x << 40) & 0x00ff000000000000ULL) | ((x << 24) & 0x0000ff0000000000ULL) | ((x << 8) & 0x000000ff00000000ULL) | ((x >> 8) & 0x00000000ff000000ULL) | ((x >> 24) & 0x0000000000ff0000ULL) | ((x >> 40) & 0x000000000000ff00ULL) | ((x >> 56) & 0x00000000000000ffULL); } #endif /* XXH_FORCE_MEMORY_ACCESS==3 is an endian-independent byteshift load. */ #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) XXH_FORCE_INLINE xxh_u64 XXH_readLE64(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[0] | ((xxh_u64)bytePtr[1] << 8) | ((xxh_u64)bytePtr[2] << 16) | ((xxh_u64)bytePtr[3] << 24) | ((xxh_u64)bytePtr[4] << 32) | ((xxh_u64)bytePtr[5] << 40) | ((xxh_u64)bytePtr[6] << 48) | ((xxh_u64)bytePtr[7] << 56); } XXH_FORCE_INLINE xxh_u64 XXH_readBE64(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[7] | ((xxh_u64)bytePtr[6] << 8) | ((xxh_u64)bytePtr[5] << 16) | ((xxh_u64)bytePtr[4] << 24) | ((xxh_u64)bytePtr[3] << 32) | ((xxh_u64)bytePtr[2] << 40) | ((xxh_u64)bytePtr[1] << 48) | ((xxh_u64)bytePtr[0] << 56); } #else XXH_FORCE_INLINE xxh_u64 XXH_readLE64(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr)); } static xxh_u64 XXH_readBE64(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_swap64(XXH_read64(ptr)) : XXH_read64(ptr); } #endif XXH_FORCE_INLINE xxh_u64 XXH_readLE64_align(const void* ptr, XXH_alignment align) { if (align==XXH_unaligned) return XXH_readLE64(ptr); else return XXH_CPU_LITTLE_ENDIAN ? *(const xxh_u64*)ptr : XXH_swap64(*(const xxh_u64*)ptr); } /******* xxh64 *******/ /*! * @} * @defgroup xxh64_impl XXH64 implementation * @ingroup impl * @{ */ /* #define rather that static const, to be used as initializers */ #define XXH_PRIME64_1 0x9E3779B185EBCA87ULL /*!< 0b1001111000110111011110011011000110000101111010111100101010000111 */ #define XXH_PRIME64_2 0xC2B2AE3D27D4EB4FULL /*!< 0b1100001010110010101011100011110100100111110101001110101101001111 */ #define XXH_PRIME64_3 0x165667B19E3779F9ULL /*!< 0b0001011001010110011001111011000110011110001101110111100111111001 */ #define XXH_PRIME64_4 0x85EBCA77C2B2AE63ULL /*!< 0b1000010111101011110010100111011111000010101100101010111001100011 */ #define XXH_PRIME64_5 0x27D4EB2F165667C5ULL /*!< 0b0010011111010100111010110010111100010110010101100110011111000101 */ #ifdef XXH_OLD_NAMES # define PRIME64_1 XXH_PRIME64_1 # define PRIME64_2 XXH_PRIME64_2 # define PRIME64_3 XXH_PRIME64_3 # define PRIME64_4 XXH_PRIME64_4 # define PRIME64_5 XXH_PRIME64_5 #endif static xxh_u64 XXH64_round(xxh_u64 acc, xxh_u64 input) { acc += input * XXH_PRIME64_2; acc = XXH_rotl64(acc, 31); acc *= XXH_PRIME64_1; return acc; } static xxh_u64 XXH64_mergeRound(xxh_u64 acc, xxh_u64 val) { val = XXH64_round(0, val); acc ^= val; acc = acc * XXH_PRIME64_1 + XXH_PRIME64_4; return acc; } static xxh_u64 XXH64_avalanche(xxh_u64 h64) { h64 ^= h64 >> 33; h64 *= XXH_PRIME64_2; h64 ^= h64 >> 29; h64 *= XXH_PRIME64_3; h64 ^= h64 >> 32; return h64; } #define XXH_get64bits(p) XXH_readLE64_align(p, align) static xxh_u64 XXH64_finalize(xxh_u64 h64, const xxh_u8* ptr, size_t len, XXH_alignment align) { if (ptr==NULL) XXH_ASSERT(len == 0); len &= 31; while (len >= 8) { xxh_u64 const k1 = XXH64_round(0, XXH_get64bits(ptr)); ptr += 8; h64 ^= k1; h64 = XXH_rotl64(h64,27) * XXH_PRIME64_1 + XXH_PRIME64_4; len -= 8; } if (len >= 4) { h64 ^= (xxh_u64)(XXH_get32bits(ptr)) * XXH_PRIME64_1; ptr += 4; h64 = XXH_rotl64(h64, 23) * XXH_PRIME64_2 + XXH_PRIME64_3; len -= 4; } while (len > 0) { h64 ^= (*ptr++) * XXH_PRIME64_5; h64 = XXH_rotl64(h64, 11) * XXH_PRIME64_1; --len; } return XXH64_avalanche(h64); } #ifdef XXH_OLD_NAMES # define PROCESS1_64 XXH_PROCESS1_64 # define PROCESS4_64 XXH_PROCESS4_64 # define PROCESS8_64 XXH_PROCESS8_64 #else # undef XXH_PROCESS1_64 # undef XXH_PROCESS4_64 # undef XXH_PROCESS8_64 #endif XXH_FORCE_INLINE xxh_u64 XXH64_endian_align(const xxh_u8* input, size_t len, xxh_u64 seed, XXH_alignment align) { xxh_u64 h64; if (input==NULL) XXH_ASSERT(len == 0); if (len>=32) { const xxh_u8* const bEnd = input + len; const xxh_u8* const limit = bEnd - 31; xxh_u64 v1 = seed + XXH_PRIME64_1 + XXH_PRIME64_2; xxh_u64 v2 = seed + XXH_PRIME64_2; xxh_u64 v3 = seed + 0; xxh_u64 v4 = seed - XXH_PRIME64_1; do { v1 = XXH64_round(v1, XXH_get64bits(input)); input+=8; v2 = XXH64_round(v2, XXH_get64bits(input)); input+=8; v3 = XXH64_round(v3, XXH_get64bits(input)); input+=8; v4 = XXH64_round(v4, XXH_get64bits(input)); input+=8; } while (inputv[0] = seed + XXH_PRIME64_1 + XXH_PRIME64_2; statePtr->v[1] = seed + XXH_PRIME64_2; statePtr->v[2] = seed + 0; statePtr->v[3] = seed - XXH_PRIME64_1; return XXH_OK; } /*! @ingroup xxh64_family */ XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* state, const void* input, size_t len) { if (input==NULL) { XXH_ASSERT(len == 0); return XXH_OK; } { const xxh_u8* p = (const xxh_u8*)input; const xxh_u8* const bEnd = p + len; state->total_len += len; if (state->memsize + len < 32) { /* fill in tmp buffer */ XXH_memcpy(((xxh_u8*)state->mem64) + state->memsize, input, len); state->memsize += (xxh_u32)len; return XXH_OK; } if (state->memsize) { /* tmp buffer is full */ XXH_memcpy(((xxh_u8*)state->mem64) + state->memsize, input, 32-state->memsize); state->v[0] = XXH64_round(state->v[0], XXH_readLE64(state->mem64+0)); state->v[1] = XXH64_round(state->v[1], XXH_readLE64(state->mem64+1)); state->v[2] = XXH64_round(state->v[2], XXH_readLE64(state->mem64+2)); state->v[3] = XXH64_round(state->v[3], XXH_readLE64(state->mem64+3)); p += 32 - state->memsize; state->memsize = 0; } if (p+32 <= bEnd) { const xxh_u8* const limit = bEnd - 32; do { state->v[0] = XXH64_round(state->v[0], XXH_readLE64(p)); p+=8; state->v[1] = XXH64_round(state->v[1], XXH_readLE64(p)); p+=8; state->v[2] = XXH64_round(state->v[2], XXH_readLE64(p)); p+=8; state->v[3] = XXH64_round(state->v[3], XXH_readLE64(p)); p+=8; } while (p<=limit); } if (p < bEnd) { XXH_memcpy(state->mem64, p, (size_t)(bEnd-p)); state->memsize = (unsigned)(bEnd-p); } } return XXH_OK; } /*! @ingroup xxh64_family */ XXH_PUBLIC_API XXH64_hash_t XXH64_digest(const XXH64_state_t* state) { xxh_u64 h64; if (state->total_len >= 32) { h64 = XXH_rotl64(state->v[0], 1) + XXH_rotl64(state->v[1], 7) + XXH_rotl64(state->v[2], 12) + XXH_rotl64(state->v[3], 18); h64 = XXH64_mergeRound(h64, state->v[0]); h64 = XXH64_mergeRound(h64, state->v[1]); h64 = XXH64_mergeRound(h64, state->v[2]); h64 = XXH64_mergeRound(h64, state->v[3]); } else { h64 = state->v[2] /*seed*/ + XXH_PRIME64_5; } h64 += (xxh_u64) state->total_len; return XXH64_finalize(h64, (const xxh_u8*)state->mem64, (size_t)state->total_len, XXH_aligned); } /******* Canonical representation *******/ /*! @ingroup xxh64_family */ XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash) { /* XXH_STATIC_ASSERT(sizeof(XXH64_canonical_t) == sizeof(XXH64_hash_t)); */ if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap64(hash); XXH_memcpy(dst, &hash, sizeof(*dst)); } /*! @ingroup xxh64_family */ XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src) { return XXH_readBE64(src); } #ifndef XXH_NO_XXH3 /* ********************************************************************* * XXH3 * New generation hash designed for speed on small keys and vectorization ************************************************************************ */ /*! * @} * @defgroup xxh3_impl XXH3 implementation * @ingroup impl * @{ */ /* === Compiler specifics === */ #if ((defined(sun) || defined(__sun)) && __cplusplus) /* Solaris includes __STDC_VERSION__ with C++. Tested with GCC 5.5 */ # define XXH_RESTRICT /* disable */ #elif defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* >= C99 */ # define XXH_RESTRICT restrict #else /* Note: it might be useful to define __restrict or __restrict__ for some C++ compilers */ # define XXH_RESTRICT /* disable */ #endif #if (defined(__GNUC__) && (__GNUC__ >= 3)) \ || (defined(__INTEL_COMPILER) && (__INTEL_COMPILER >= 800)) \ || defined(__clang__) # define XXH_likely(x) __builtin_expect(x, 1) # define XXH_unlikely(x) __builtin_expect(x, 0) #else # define XXH_likely(x) (x) # define XXH_unlikely(x) (x) #endif #if defined(__GNUC__) || defined(__clang__) # if defined(__ARM_NEON__) || defined(__ARM_NEON) \ || defined(__aarch64__) || defined(_M_ARM) \ || defined(_M_ARM64) || defined(_M_ARM64EC) # define inline __inline__ /* circumvent a clang bug */ # include # undef inline # elif defined(__AVX2__) # include # elif defined(__SSE2__) # include # endif #endif #if defined(_MSC_VER) # include #endif /* * One goal of XXH3 is to make it fast on both 32-bit and 64-bit, while * remaining a true 64-bit/128-bit hash function. * * This is done by prioritizing a subset of 64-bit operations that can be * emulated without too many steps on the average 32-bit machine. * * For example, these two lines seem similar, and run equally fast on 64-bit: * * xxh_u64 x; * x ^= (x >> 47); // good * x ^= (x >> 13); // bad * * However, to a 32-bit machine, there is a major difference. * * x ^= (x >> 47) looks like this: * * x.lo ^= (x.hi >> (47 - 32)); * * while x ^= (x >> 13) looks like this: * * // note: funnel shifts are not usually cheap. * x.lo ^= (x.lo >> 13) | (x.hi << (32 - 13)); * x.hi ^= (x.hi >> 13); * * The first one is significantly faster than the second, simply because the * shift is larger than 32. This means: * - All the bits we need are in the upper 32 bits, so we can ignore the lower * 32 bits in the shift. * - The shift result will always fit in the lower 32 bits, and therefore, * we can ignore the upper 32 bits in the xor. * * Thanks to this optimization, XXH3 only requires these features to be efficient: * * - Usable unaligned access * - A 32-bit or 64-bit ALU * - If 32-bit, a decent ADC instruction * - A 32 or 64-bit multiply with a 64-bit result * - For the 128-bit variant, a decent byteswap helps short inputs. * * The first two are already required by XXH32, and almost all 32-bit and 64-bit * platforms which can run XXH32 can run XXH3 efficiently. * * Thumb-1, the classic 16-bit only subset of ARM's instruction set, is one * notable exception. * * First of all, Thumb-1 lacks support for the UMULL instruction which * performs the important long multiply. This means numerous __aeabi_lmul * calls. * * Second of all, the 8 functional registers are just not enough. * Setup for __aeabi_lmul, byteshift loads, pointers, and all arithmetic need * Lo registers, and this shuffling results in thousands more MOVs than A32. * * A32 and T32 don't have this limitation. They can access all 14 registers, * do a 32->64 multiply with UMULL, and the flexible operand allowing free * shifts is helpful, too. * * Therefore, we do a quick sanity check. * * If compiling Thumb-1 for a target which supports ARM instructions, we will * emit a warning, as it is not a "sane" platform to compile for. * * Usually, if this happens, it is because of an accident and you probably need * to specify -march, as you likely meant to compile for a newer architecture. * * Credit: large sections of the vectorial and asm source code paths * have been contributed by @easyaspi314 */ #if defined(__thumb__) && !defined(__thumb2__) && defined(__ARM_ARCH_ISA_ARM) # warning "XXH3 is highly inefficient without ARM or Thumb-2." #endif /* ========================================== * Vectorization detection * ========================================== */ #ifdef XXH_DOXYGEN /*! * @ingroup tuning * @brief Overrides the vectorization implementation chosen for XXH3. * * Can be defined to 0 to disable SIMD or any of the values mentioned in * @ref XXH_VECTOR_TYPE. * * If this is not defined, it uses predefined macros to determine the best * implementation. */ # define XXH_VECTOR XXH_SCALAR /*! * @ingroup tuning * @brief Possible values for @ref XXH_VECTOR. * * Note that these are actually implemented as macros. * * If this is not defined, it is detected automatically. * @ref XXH_X86DISPATCH overrides this. */ enum XXH_VECTOR_TYPE /* fake enum */ { XXH_SCALAR = 0, /*!< Portable scalar version */ XXH_SSE2 = 1, /*!< * SSE2 for Pentium 4, Opteron, all x86_64. * * @note SSE2 is also guaranteed on Windows 10, macOS, and * Android x86. */ XXH_AVX2 = 2, /*!< AVX2 for Haswell and Bulldozer */ XXH_AVX512 = 3, /*!< AVX512 for Skylake and Icelake */ XXH_NEON = 4, /*!< NEON for most ARMv7-A and all AArch64 */ XXH_VSX = 5, /*!< VSX and ZVector for POWER8/z13 (64-bit) */ }; /*! * @ingroup tuning * @brief Selects the minimum alignment for XXH3's accumulators. * * When using SIMD, this should match the alignment required for said vector * type, so, for example, 32 for AVX2. * * Default: Auto detected. */ # define XXH_ACC_ALIGN 8 #endif /* Actual definition */ #ifndef XXH_DOXYGEN # define XXH_SCALAR 0 # define XXH_SSE2 1 # define XXH_AVX2 2 # define XXH_AVX512 3 # define XXH_NEON 4 # define XXH_VSX 5 #endif #ifndef XXH_VECTOR /* can be defined on command line */ # if ( \ defined(__ARM_NEON__) || defined(__ARM_NEON) /* gcc */ \ || defined(_M_ARM) || defined(_M_ARM64) || defined(_M_ARM64EC) /* msvc */ \ ) && ( \ defined(_WIN32) || defined(__LITTLE_ENDIAN__) /* little endian only */ \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) \ ) # define XXH_VECTOR XXH_NEON # elif defined(__AVX512F__) # define XXH_VECTOR XXH_AVX512 # elif defined(__AVX2__) # define XXH_VECTOR XXH_AVX2 # elif defined(__SSE2__) || defined(_M_AMD64) || defined(_M_X64) || (defined(_M_IX86_FP) && (_M_IX86_FP == 2)) # define XXH_VECTOR XXH_SSE2 # elif (defined(__PPC64__) && defined(__POWER8_VECTOR__)) \ || (defined(__s390x__) && defined(__VEC__)) \ && defined(__GNUC__) /* TODO: IBM XL */ # define XXH_VECTOR XXH_VSX # else # define XXH_VECTOR XXH_SCALAR # endif #endif /* * Controls the alignment of the accumulator, * for compatibility with aligned vector loads, which are usually faster. */ #ifndef XXH_ACC_ALIGN # if defined(XXH_X86DISPATCH) # define XXH_ACC_ALIGN 64 /* for compatibility with avx512 */ # elif XXH_VECTOR == XXH_SCALAR /* scalar */ # define XXH_ACC_ALIGN 8 # elif XXH_VECTOR == XXH_SSE2 /* sse2 */ # define XXH_ACC_ALIGN 16 # elif XXH_VECTOR == XXH_AVX2 /* avx2 */ # define XXH_ACC_ALIGN 32 # elif XXH_VECTOR == XXH_NEON /* neon */ # define XXH_ACC_ALIGN 16 # elif XXH_VECTOR == XXH_VSX /* vsx */ # define XXH_ACC_ALIGN 16 # elif XXH_VECTOR == XXH_AVX512 /* avx512 */ # define XXH_ACC_ALIGN 64 # endif #endif #if defined(XXH_X86DISPATCH) || XXH_VECTOR == XXH_SSE2 \ || XXH_VECTOR == XXH_AVX2 || XXH_VECTOR == XXH_AVX512 # define XXH_SEC_ALIGN XXH_ACC_ALIGN #else # define XXH_SEC_ALIGN 8 #endif /* * UGLY HACK: * GCC usually generates the best code with -O3 for xxHash. * * However, when targeting AVX2, it is overzealous in its unrolling resulting * in code roughly 3/4 the speed of Clang. * * There are other issues, such as GCC splitting _mm256_loadu_si256 into * _mm_loadu_si128 + _mm256_inserti128_si256. This is an optimization which * only applies to Sandy and Ivy Bridge... which don't even support AVX2. * * That is why when compiling the AVX2 version, it is recommended to use either * -O2 -mavx2 -march=haswell * or * -O2 -mavx2 -mno-avx256-split-unaligned-load * for decent performance, or to use Clang instead. * * Fortunately, we can control the first one with a pragma that forces GCC into * -O2, but the other one we can't control without "failed to inline always * inline function due to target mismatch" warnings. */ #if XXH_VECTOR == XXH_AVX2 /* AVX2 */ \ && defined(__GNUC__) && !defined(__clang__) /* GCC, not Clang */ \ && defined(__OPTIMIZE__) && !defined(__OPTIMIZE_SIZE__) /* respect -O0 and -Os */ # pragma GCC push_options # pragma GCC optimize("-O2") #endif #if XXH_VECTOR == XXH_NEON /* * NEON's setup for vmlal_u32 is a little more complicated than it is on * SSE2, AVX2, and VSX. * * While PMULUDQ and VMULEUW both perform a mask, VMLAL.U32 performs an upcast. * * To do the same operation, the 128-bit 'Q' register needs to be split into * two 64-bit 'D' registers, performing this operation:: * * [ a | b ] * | '---------. .--------' | * | x | * | .---------' '--------. | * [ a & 0xFFFFFFFF | b & 0xFFFFFFFF ],[ a >> 32 | b >> 32 ] * * Due to significant changes in aarch64, the fastest method for aarch64 is * completely different than the fastest method for ARMv7-A. * * ARMv7-A treats D registers as unions overlaying Q registers, so modifying * D11 will modify the high half of Q5. This is similar to how modifying AH * will only affect bits 8-15 of AX on x86. * * VZIP takes two registers, and puts even lanes in one register and odd lanes * in the other. * * On ARMv7-A, this strangely modifies both parameters in place instead of * taking the usual 3-operand form. * * Therefore, if we want to do this, we can simply use a D-form VZIP.32 on the * lower and upper halves of the Q register to end up with the high and low * halves where we want - all in one instruction. * * vzip.32 d10, d11 @ d10 = { d10[0], d11[0] }; d11 = { d10[1], d11[1] } * * Unfortunately we need inline assembly for this: Instructions modifying two * registers at once is not possible in GCC or Clang's IR, and they have to * create a copy. * * aarch64 requires a different approach. * * In order to make it easier to write a decent compiler for aarch64, many * quirks were removed, such as conditional execution. * * NEON was also affected by this. * * aarch64 cannot access the high bits of a Q-form register, and writes to a * D-form register zero the high bits, similar to how writes to W-form scalar * registers (or DWORD registers on x86_64) work. * * The formerly free vget_high intrinsics now require a vext (with a few * exceptions) * * Additionally, VZIP was replaced by ZIP1 and ZIP2, which are the equivalent * of PUNPCKL* and PUNPCKH* in SSE, respectively, in order to only modify one * operand. * * The equivalent of the VZIP.32 on the lower and upper halves would be this * mess: * * ext v2.4s, v0.4s, v0.4s, #2 // v2 = { v0[2], v0[3], v0[0], v0[1] } * zip1 v1.2s, v0.2s, v2.2s // v1 = { v0[0], v2[0] } * zip2 v0.2s, v0.2s, v1.2s // v0 = { v0[1], v2[1] } * * Instead, we use a literal downcast, vmovn_u64 (XTN), and vshrn_n_u64 (SHRN): * * shrn v1.2s, v0.2d, #32 // v1 = (uint32x2_t)(v0 >> 32); * xtn v0.2s, v0.2d // v0 = (uint32x2_t)(v0 & 0xFFFFFFFF); * * This is available on ARMv7-A, but is less efficient than a single VZIP.32. */ /*! * Function-like macro: * void XXH_SPLIT_IN_PLACE(uint64x2_t &in, uint32x2_t &outLo, uint32x2_t &outHi) * { * outLo = (uint32x2_t)(in & 0xFFFFFFFF); * outHi = (uint32x2_t)(in >> 32); * in = UNDEFINED; * } */ # if !defined(XXH_NO_VZIP_HACK) /* define to disable */ \ && (defined(__GNUC__) || defined(__clang__)) \ && (defined(__arm__) || defined(__thumb__) || defined(_M_ARM)) # define XXH_SPLIT_IN_PLACE(in, outLo, outHi) \ do { \ /* Undocumented GCC/Clang operand modifier: %e0 = lower D half, %f0 = upper D half */ \ /* https://github.com/gcc-mirror/gcc/blob/38cf91e5/gcc/config/arm/arm.c#L22486 */ \ /* https://github.com/llvm-mirror/llvm/blob/2c4ca683/lib/Target/ARM/ARMAsmPrinter.cpp#L399 */ \ __asm__("vzip.32 %e0, %f0" : "+w" (in)); \ (outLo) = vget_low_u32 (vreinterpretq_u32_u64(in)); \ (outHi) = vget_high_u32(vreinterpretq_u32_u64(in)); \ } while (0) # else # define XXH_SPLIT_IN_PLACE(in, outLo, outHi) \ do { \ (outLo) = vmovn_u64 (in); \ (outHi) = vshrn_n_u64 ((in), 32); \ } while (0) # endif /*! * @ingroup tuning * @brief Controls the NEON to scalar ratio for XXH3 * * On AArch64 when not optimizing for size, XXH3 will run 6 lanes using NEON and * 2 lanes on scalar by default. * * This can be set to 2, 4, 6, or 8. ARMv7 will default to all 8 NEON lanes, as the * emulated 64-bit arithmetic is too slow. * * Modern ARM CPUs are _very_ sensitive to how their pipelines are used. * * For example, the Cortex-A73 can dispatch 3 micro-ops per cycle, but it can't * have more than 2 NEON (F0/F1) micro-ops. If you are only using NEON instructions, * you are only using 2/3 of the CPU bandwidth. * * This is even more noticeable on the more advanced cores like the A76 which * can dispatch 8 micro-ops per cycle, but still only 2 NEON micro-ops at once. * * Therefore, @ref XXH3_NEON_LANES lanes will be processed using NEON, and the * remaining lanes will use scalar instructions. This improves the bandwidth * and also gives the integer pipelines something to do besides twiddling loop * counters and pointers. * * This change benefits CPUs with large micro-op buffers without negatively affecting * other CPUs: * * | Chipset | Dispatch type | NEON only | 6:2 hybrid | Diff. | * |:----------------------|:--------------------|----------:|-----------:|------:| * | Snapdragon 730 (A76) | 2 NEON/8 micro-ops | 8.8 GB/s | 10.1 GB/s | ~16% | * | Snapdragon 835 (A73) | 2 NEON/3 micro-ops | 5.1 GB/s | 5.3 GB/s | ~5% | * | Marvell PXA1928 (A53) | In-order dual-issue | 1.9 GB/s | 1.9 GB/s | 0% | * * It also seems to fix some bad codegen on GCC, making it almost as fast as clang. * * @see XXH3_accumulate_512_neon() */ # ifndef XXH3_NEON_LANES # if (defined(__aarch64__) || defined(__arm64__) || defined(_M_ARM64) || defined(_M_ARM64EC)) \ && !defined(__OPTIMIZE_SIZE__) # define XXH3_NEON_LANES 6 # else # define XXH3_NEON_LANES XXH_ACC_NB # endif # endif #endif /* XXH_VECTOR == XXH_NEON */ /* * VSX and Z Vector helpers. * * This is very messy, and any pull requests to clean this up are welcome. * * There are a lot of problems with supporting VSX and s390x, due to * inconsistent intrinsics, spotty coverage, and multiple endiannesses. */ #if XXH_VECTOR == XXH_VSX # if defined(__s390x__) # include # else /* gcc's altivec.h can have the unwanted consequence to unconditionally * #define bool, vector, and pixel keywords, * with bad consequences for programs already using these keywords for other purposes. * The paragraph defining these macros is skipped when __APPLE_ALTIVEC__ is defined. * __APPLE_ALTIVEC__ is _generally_ defined automatically by the compiler, * but it seems that, in some cases, it isn't. * Force the build macro to be defined, so that keywords are not altered. */ # if defined(__GNUC__) && !defined(__APPLE_ALTIVEC__) # define __APPLE_ALTIVEC__ # endif # include # endif typedef __vector unsigned long long xxh_u64x2; typedef __vector unsigned char xxh_u8x16; typedef __vector unsigned xxh_u32x4; # ifndef XXH_VSX_BE # if defined(__BIG_ENDIAN__) \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) # define XXH_VSX_BE 1 # elif defined(__VEC_ELEMENT_REG_ORDER__) && __VEC_ELEMENT_REG_ORDER__ == __ORDER_BIG_ENDIAN__ # warning "-maltivec=be is not recommended. Please use native endianness." # define XXH_VSX_BE 1 # else # define XXH_VSX_BE 0 # endif # endif /* !defined(XXH_VSX_BE) */ # if XXH_VSX_BE # if defined(__POWER9_VECTOR__) || (defined(__clang__) && defined(__s390x__)) # define XXH_vec_revb vec_revb # else /*! * A polyfill for POWER9's vec_revb(). */ XXH_FORCE_INLINE xxh_u64x2 XXH_vec_revb(xxh_u64x2 val) { xxh_u8x16 const vByteSwap = { 0x07, 0x06, 0x05, 0x04, 0x03, 0x02, 0x01, 0x00, 0x0F, 0x0E, 0x0D, 0x0C, 0x0B, 0x0A, 0x09, 0x08 }; return vec_perm(val, val, vByteSwap); } # endif # endif /* XXH_VSX_BE */ /*! * Performs an unaligned vector load and byte swaps it on big endian. */ XXH_FORCE_INLINE xxh_u64x2 XXH_vec_loadu(const void *ptr) { xxh_u64x2 ret; XXH_memcpy(&ret, ptr, sizeof(xxh_u64x2)); # if XXH_VSX_BE ret = XXH_vec_revb(ret); # endif return ret; } /* * vec_mulo and vec_mule are very problematic intrinsics on PowerPC * * These intrinsics weren't added until GCC 8, despite existing for a while, * and they are endian dependent. Also, their meaning swap depending on version. * */ # if defined(__s390x__) /* s390x is always big endian, no issue on this platform */ # define XXH_vec_mulo vec_mulo # define XXH_vec_mule vec_mule # elif defined(__clang__) && XXH_HAS_BUILTIN(__builtin_altivec_vmuleuw) /* Clang has a better way to control this, we can just use the builtin which doesn't swap. */ # define XXH_vec_mulo __builtin_altivec_vmulouw # define XXH_vec_mule __builtin_altivec_vmuleuw # else /* gcc needs inline assembly */ /* Adapted from https://github.com/google/highwayhash/blob/master/highwayhash/hh_vsx.h. */ XXH_FORCE_INLINE xxh_u64x2 XXH_vec_mulo(xxh_u32x4 a, xxh_u32x4 b) { xxh_u64x2 result; __asm__("vmulouw %0, %1, %2" : "=v" (result) : "v" (a), "v" (b)); return result; } XXH_FORCE_INLINE xxh_u64x2 XXH_vec_mule(xxh_u32x4 a, xxh_u32x4 b) { xxh_u64x2 result; __asm__("vmuleuw %0, %1, %2" : "=v" (result) : "v" (a), "v" (b)); return result; } # endif /* XXH_vec_mulo, XXH_vec_mule */ #endif /* XXH_VECTOR == XXH_VSX */ /* prefetch * can be disabled, by declaring XXH_NO_PREFETCH build macro */ #if defined(XXH_NO_PREFETCH) # define XXH_PREFETCH(ptr) (void)(ptr) /* disabled */ #else # if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_IX86)) /* _mm_prefetch() not defined outside of x86/x64 */ # include /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */ # define XXH_PREFETCH(ptr) _mm_prefetch((const char*)(ptr), _MM_HINT_T0) # elif defined(__GNUC__) && ( (__GNUC__ >= 4) || ( (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) ) ) # define XXH_PREFETCH(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 3 /* locality */) # else # define XXH_PREFETCH(ptr) (void)(ptr) /* disabled */ # endif #endif /* XXH_NO_PREFETCH */ /* ========================================== * XXH3 default settings * ========================================== */ #define XXH_SECRET_DEFAULT_SIZE 192 /* minimum XXH3_SECRET_SIZE_MIN */ #if (XXH_SECRET_DEFAULT_SIZE < XXH3_SECRET_SIZE_MIN) # error "default keyset is not large enough" #endif /*! Pseudorandom secret taken directly from FARSH. */ XXH_ALIGN(64) static const xxh_u8 XXH3_kSecret[XXH_SECRET_DEFAULT_SIZE] = { 0xb8, 0xfe, 0x6c, 0x39, 0x23, 0xa4, 0x4b, 0xbe, 0x7c, 0x01, 0x81, 0x2c, 0xf7, 0x21, 0xad, 0x1c, 0xde, 0xd4, 0x6d, 0xe9, 0x83, 0x90, 0x97, 0xdb, 0x72, 0x40, 0xa4, 0xa4, 0xb7, 0xb3, 0x67, 0x1f, 0xcb, 0x79, 0xe6, 0x4e, 0xcc, 0xc0, 0xe5, 0x78, 0x82, 0x5a, 0xd0, 0x7d, 0xcc, 0xff, 0x72, 0x21, 0xb8, 0x08, 0x46, 0x74, 0xf7, 0x43, 0x24, 0x8e, 0xe0, 0x35, 0x90, 0xe6, 0x81, 0x3a, 0x26, 0x4c, 0x3c, 0x28, 0x52, 0xbb, 0x91, 0xc3, 0x00, 0xcb, 0x88, 0xd0, 0x65, 0x8b, 0x1b, 0x53, 0x2e, 0xa3, 0x71, 0x64, 0x48, 0x97, 0xa2, 0x0d, 0xf9, 0x4e, 0x38, 0x19, 0xef, 0x46, 0xa9, 0xde, 0xac, 0xd8, 0xa8, 0xfa, 0x76, 0x3f, 0xe3, 0x9c, 0x34, 0x3f, 0xf9, 0xdc, 0xbb, 0xc7, 0xc7, 0x0b, 0x4f, 0x1d, 0x8a, 0x51, 0xe0, 0x4b, 0xcd, 0xb4, 0x59, 0x31, 0xc8, 0x9f, 0x7e, 0xc9, 0xd9, 0x78, 0x73, 0x64, 0xea, 0xc5, 0xac, 0x83, 0x34, 0xd3, 0xeb, 0xc3, 0xc5, 0x81, 0xa0, 0xff, 0xfa, 0x13, 0x63, 0xeb, 0x17, 0x0d, 0xdd, 0x51, 0xb7, 0xf0, 0xda, 0x49, 0xd3, 0x16, 0x55, 0x26, 0x29, 0xd4, 0x68, 0x9e, 0x2b, 0x16, 0xbe, 0x58, 0x7d, 0x47, 0xa1, 0xfc, 0x8f, 0xf8, 0xb8, 0xd1, 0x7a, 0xd0, 0x31, 0xce, 0x45, 0xcb, 0x3a, 0x8f, 0x95, 0x16, 0x04, 0x28, 0xaf, 0xd7, 0xfb, 0xca, 0xbb, 0x4b, 0x40, 0x7e, }; #ifdef XXH_OLD_NAMES # define kSecret XXH3_kSecret #endif #ifdef XXH_DOXYGEN /*! * @brief Calculates a 32-bit to 64-bit long multiply. * * Implemented as a macro. * * Wraps `__emulu` on MSVC x86 because it tends to call `__allmul` when it doesn't * need to (but it shouldn't need to anyways, it is about 7 instructions to do * a 64x64 multiply...). Since we know that this will _always_ emit `MULL`, we * use that instead of the normal method. * * If you are compiling for platforms like Thumb-1 and don't have a better option, * you may also want to write your own long multiply routine here. * * @param x, y Numbers to be multiplied * @return 64-bit product of the low 32 bits of @p x and @p y. */ XXH_FORCE_INLINE xxh_u64 XXH_mult32to64(xxh_u64 x, xxh_u64 y) { return (x & 0xFFFFFFFF) * (y & 0xFFFFFFFF); } #elif defined(_MSC_VER) && defined(_M_IX86) # define XXH_mult32to64(x, y) __emulu((unsigned)(x), (unsigned)(y)) #else /* * Downcast + upcast is usually better than masking on older compilers like * GCC 4.2 (especially 32-bit ones), all without affecting newer compilers. * * The other method, (x & 0xFFFFFFFF) * (y & 0xFFFFFFFF), will AND both operands * and perform a full 64x64 multiply -- entirely redundant on 32-bit. */ # define XXH_mult32to64(x, y) ((xxh_u64)(xxh_u32)(x) * (xxh_u64)(xxh_u32)(y)) #endif /*! * @brief Calculates a 64->128-bit long multiply. * * Uses `__uint128_t` and `_umul128` if available, otherwise uses a scalar * version. * * @param lhs , rhs The 64-bit integers to be multiplied * @return The 128-bit result represented in an @ref XXH128_hash_t. */ static XXH128_hash_t XXH_mult64to128(xxh_u64 lhs, xxh_u64 rhs) { /* * GCC/Clang __uint128_t method. * * On most 64-bit targets, GCC and Clang define a __uint128_t type. * This is usually the best way as it usually uses a native long 64-bit * multiply, such as MULQ on x86_64 or MUL + UMULH on aarch64. * * Usually. * * Despite being a 32-bit platform, Clang (and emscripten) define this type * despite not having the arithmetic for it. This results in a laggy * compiler builtin call which calculates a full 128-bit multiply. * In that case it is best to use the portable one. * https://github.com/Cyan4973/xxHash/issues/211#issuecomment-515575677 */ #if (defined(__GNUC__) || defined(__clang__)) && !defined(__wasm__) \ && defined(__SIZEOF_INT128__) \ || (defined(_INTEGRAL_MAX_BITS) && _INTEGRAL_MAX_BITS >= 128) __uint128_t const product = (__uint128_t)lhs * (__uint128_t)rhs; XXH128_hash_t r128; r128.low64 = (xxh_u64)(product); r128.high64 = (xxh_u64)(product >> 64); return r128; /* * MSVC for x64's _umul128 method. * * xxh_u64 _umul128(xxh_u64 Multiplier, xxh_u64 Multiplicand, xxh_u64 *HighProduct); * * This compiles to single operand MUL on x64. */ #elif (defined(_M_X64) || defined(_M_IA64)) && !defined(_M_ARM64EC) #ifndef _MSC_VER # pragma intrinsic(_umul128) #endif xxh_u64 product_high; xxh_u64 const product_low = _umul128(lhs, rhs, &product_high); XXH128_hash_t r128; r128.low64 = product_low; r128.high64 = product_high; return r128; /* * MSVC for ARM64's __umulh method. * * This compiles to the same MUL + UMULH as GCC/Clang's __uint128_t method. */ #elif defined(_M_ARM64) || defined(_M_ARM64EC) #ifndef _MSC_VER # pragma intrinsic(__umulh) #endif XXH128_hash_t r128; r128.low64 = lhs * rhs; r128.high64 = __umulh(lhs, rhs); return r128; #else /* * Portable scalar method. Optimized for 32-bit and 64-bit ALUs. * * This is a fast and simple grade school multiply, which is shown below * with base 10 arithmetic instead of base 0x100000000. * * 9 3 // D2 lhs = 93 * x 7 5 // D2 rhs = 75 * ---------- * 1 5 // D2 lo_lo = (93 % 10) * (75 % 10) = 15 * 4 5 | // D2 hi_lo = (93 / 10) * (75 % 10) = 45 * 2 1 | // D2 lo_hi = (93 % 10) * (75 / 10) = 21 * + 6 3 | | // D2 hi_hi = (93 / 10) * (75 / 10) = 63 * --------- * 2 7 | // D2 cross = (15 / 10) + (45 % 10) + 21 = 27 * + 6 7 | | // D2 upper = (27 / 10) + (45 / 10) + 63 = 67 * --------- * 6 9 7 5 // D4 res = (27 * 10) + (15 % 10) + (67 * 100) = 6975 * * The reasons for adding the products like this are: * 1. It avoids manual carry tracking. Just like how * (9 * 9) + 9 + 9 = 99, the same applies with this for UINT64_MAX. * This avoids a lot of complexity. * * 2. It hints for, and on Clang, compiles to, the powerful UMAAL * instruction available in ARM's Digital Signal Processing extension * in 32-bit ARMv6 and later, which is shown below: * * void UMAAL(xxh_u32 *RdLo, xxh_u32 *RdHi, xxh_u32 Rn, xxh_u32 Rm) * { * xxh_u64 product = (xxh_u64)*RdLo * (xxh_u64)*RdHi + Rn + Rm; * *RdLo = (xxh_u32)(product & 0xFFFFFFFF); * *RdHi = (xxh_u32)(product >> 32); * } * * This instruction was designed for efficient long multiplication, and * allows this to be calculated in only 4 instructions at speeds * comparable to some 64-bit ALUs. * * 3. It isn't terrible on other platforms. Usually this will be a couple * of 32-bit ADD/ADCs. */ /* First calculate all of the cross products. */ xxh_u64 const lo_lo = XXH_mult32to64(lhs & 0xFFFFFFFF, rhs & 0xFFFFFFFF); xxh_u64 const hi_lo = XXH_mult32to64(lhs >> 32, rhs & 0xFFFFFFFF); xxh_u64 const lo_hi = XXH_mult32to64(lhs & 0xFFFFFFFF, rhs >> 32); xxh_u64 const hi_hi = XXH_mult32to64(lhs >> 32, rhs >> 32); /* Now add the products together. These will never overflow. */ xxh_u64 const cross = (lo_lo >> 32) + (hi_lo & 0xFFFFFFFF) + lo_hi; xxh_u64 const upper = (hi_lo >> 32) + (cross >> 32) + hi_hi; xxh_u64 const lower = (cross << 32) | (lo_lo & 0xFFFFFFFF); XXH128_hash_t r128; r128.low64 = lower; r128.high64 = upper; return r128; #endif } /*! * @brief Calculates a 64-bit to 128-bit multiply, then XOR folds it. * * The reason for the separate function is to prevent passing too many structs * around by value. This will hopefully inline the multiply, but we don't force it. * * @param lhs , rhs The 64-bit integers to multiply * @return The low 64 bits of the product XOR'd by the high 64 bits. * @see XXH_mult64to128() */ static xxh_u64 XXH3_mul128_fold64(xxh_u64 lhs, xxh_u64 rhs) { XXH128_hash_t product = XXH_mult64to128(lhs, rhs); return product.low64 ^ product.high64; } /*! Seems to produce slightly better code on GCC for some reason. */ XXH_FORCE_INLINE xxh_u64 XXH_xorshift64(xxh_u64 v64, int shift) { XXH_ASSERT(0 <= shift && shift < 64); return v64 ^ (v64 >> shift); } /* * This is a fast avalanche stage, * suitable when input bits are already partially mixed */ static XXH64_hash_t XXH3_avalanche(xxh_u64 h64) { h64 = XXH_xorshift64(h64, 37); h64 *= 0x165667919E3779F9ULL; h64 = XXH_xorshift64(h64, 32); return h64; } /* * This is a stronger avalanche, * inspired by Pelle Evensen's rrmxmx * preferable when input has not been previously mixed */ static XXH64_hash_t XXH3_rrmxmx(xxh_u64 h64, xxh_u64 len) { /* this mix is inspired by Pelle Evensen's rrmxmx */ h64 ^= XXH_rotl64(h64, 49) ^ XXH_rotl64(h64, 24); h64 *= 0x9FB21C651E98DF25ULL; h64 ^= (h64 >> 35) + len ; h64 *= 0x9FB21C651E98DF25ULL; return XXH_xorshift64(h64, 28); } /* ========================================== * Short keys * ========================================== * One of the shortcomings of XXH32 and XXH64 was that their performance was * sub-optimal on short lengths. It used an iterative algorithm which strongly * favored lengths that were a multiple of 4 or 8. * * Instead of iterating over individual inputs, we use a set of single shot * functions which piece together a range of lengths and operate in constant time. * * Additionally, the number of multiplies has been significantly reduced. This * reduces latency, especially when emulating 64-bit multiplies on 32-bit. * * Depending on the platform, this may or may not be faster than XXH32, but it * is almost guaranteed to be faster than XXH64. */ /* * At very short lengths, there isn't enough input to fully hide secrets, or use * the entire secret. * * There is also only a limited amount of mixing we can do before significantly * impacting performance. * * Therefore, we use different sections of the secret and always mix two secret * samples with an XOR. This should have no effect on performance on the * seedless or withSeed variants because everything _should_ be constant folded * by modern compilers. * * The XOR mixing hides individual parts of the secret and increases entropy. * * This adds an extra layer of strength for custom secrets. */ XXH_FORCE_INLINE XXH64_hash_t XXH3_len_1to3_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(1 <= len && len <= 3); XXH_ASSERT(secret != NULL); /* * len = 1: combined = { input[0], 0x01, input[0], input[0] } * len = 2: combined = { input[1], 0x02, input[0], input[1] } * len = 3: combined = { input[2], 0x03, input[0], input[1] } */ { xxh_u8 const c1 = input[0]; xxh_u8 const c2 = input[len >> 1]; xxh_u8 const c3 = input[len - 1]; xxh_u32 const combined = ((xxh_u32)c1 << 16) | ((xxh_u32)c2 << 24) | ((xxh_u32)c3 << 0) | ((xxh_u32)len << 8); xxh_u64 const bitflip = (XXH_readLE32(secret) ^ XXH_readLE32(secret+4)) + seed; xxh_u64 const keyed = (xxh_u64)combined ^ bitflip; return XXH64_avalanche(keyed); } } XXH_FORCE_INLINE XXH64_hash_t XXH3_len_4to8_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(4 <= len && len <= 8); seed ^= (xxh_u64)XXH_swap32((xxh_u32)seed) << 32; { xxh_u32 const input1 = XXH_readLE32(input); xxh_u32 const input2 = XXH_readLE32(input + len - 4); xxh_u64 const bitflip = (XXH_readLE64(secret+8) ^ XXH_readLE64(secret+16)) - seed; xxh_u64 const input64 = input2 + (((xxh_u64)input1) << 32); xxh_u64 const keyed = input64 ^ bitflip; return XXH3_rrmxmx(keyed, len); } } XXH_FORCE_INLINE XXH64_hash_t XXH3_len_9to16_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(9 <= len && len <= 16); { xxh_u64 const bitflip1 = (XXH_readLE64(secret+24) ^ XXH_readLE64(secret+32)) + seed; xxh_u64 const bitflip2 = (XXH_readLE64(secret+40) ^ XXH_readLE64(secret+48)) - seed; xxh_u64 const input_lo = XXH_readLE64(input) ^ bitflip1; xxh_u64 const input_hi = XXH_readLE64(input + len - 8) ^ bitflip2; xxh_u64 const acc = len + XXH_swap64(input_lo) + input_hi + XXH3_mul128_fold64(input_lo, input_hi); return XXH3_avalanche(acc); } } XXH_FORCE_INLINE XXH64_hash_t XXH3_len_0to16_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(len <= 16); { if (XXH_likely(len > 8)) return XXH3_len_9to16_64b(input, len, secret, seed); if (XXH_likely(len >= 4)) return XXH3_len_4to8_64b(input, len, secret, seed); if (len) return XXH3_len_1to3_64b(input, len, secret, seed); return XXH64_avalanche(seed ^ (XXH_readLE64(secret+56) ^ XXH_readLE64(secret+64))); } } /* * DISCLAIMER: There are known *seed-dependent* multicollisions here due to * multiplication by zero, affecting hashes of lengths 17 to 240. * * However, they are very unlikely. * * Keep this in mind when using the unseeded XXH3_64bits() variant: As with all * unseeded non-cryptographic hashes, it does not attempt to defend itself * against specially crafted inputs, only random inputs. * * Compared to classic UMAC where a 1 in 2^31 chance of 4 consecutive bytes * cancelling out the secret is taken an arbitrary number of times (addressed * in XXH3_accumulate_512), this collision is very unlikely with random inputs * and/or proper seeding: * * This only has a 1 in 2^63 chance of 8 consecutive bytes cancelling out, in a * function that is only called up to 16 times per hash with up to 240 bytes of * input. * * This is not too bad for a non-cryptographic hash function, especially with * only 64 bit outputs. * * The 128-bit variant (which trades some speed for strength) is NOT affected * by this, although it is always a good idea to use a proper seed if you care * about strength. */ XXH_FORCE_INLINE xxh_u64 XXH3_mix16B(const xxh_u8* XXH_RESTRICT input, const xxh_u8* XXH_RESTRICT secret, xxh_u64 seed64) { #if defined(__GNUC__) && !defined(__clang__) /* GCC, not Clang */ \ && defined(__i386__) && defined(__SSE2__) /* x86 + SSE2 */ \ && !defined(XXH_ENABLE_AUTOVECTORIZE) /* Define to disable like XXH32 hack */ /* * UGLY HACK: * GCC for x86 tends to autovectorize the 128-bit multiply, resulting in * slower code. * * By forcing seed64 into a register, we disrupt the cost model and * cause it to scalarize. See `XXH32_round()` * * FIXME: Clang's output is still _much_ faster -- On an AMD Ryzen 3600, * XXH3_64bits @ len=240 runs at 4.6 GB/s with Clang 9, but 3.3 GB/s on * GCC 9.2, despite both emitting scalar code. * * GCC generates much better scalar code than Clang for the rest of XXH3, * which is why finding a more optimal codepath is an interest. */ XXH_COMPILER_GUARD(seed64); #endif { xxh_u64 const input_lo = XXH_readLE64(input); xxh_u64 const input_hi = XXH_readLE64(input+8); return XXH3_mul128_fold64( input_lo ^ (XXH_readLE64(secret) + seed64), input_hi ^ (XXH_readLE64(secret+8) - seed64) ); } } /* For mid range keys, XXH3 uses a Mum-hash variant. */ XXH_FORCE_INLINE XXH64_hash_t XXH3_len_17to128_64b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(16 < len && len <= 128); { xxh_u64 acc = len * XXH_PRIME64_1; if (len > 32) { if (len > 64) { if (len > 96) { acc += XXH3_mix16B(input+48, secret+96, seed); acc += XXH3_mix16B(input+len-64, secret+112, seed); } acc += XXH3_mix16B(input+32, secret+64, seed); acc += XXH3_mix16B(input+len-48, secret+80, seed); } acc += XXH3_mix16B(input+16, secret+32, seed); acc += XXH3_mix16B(input+len-32, secret+48, seed); } acc += XXH3_mix16B(input+0, secret+0, seed); acc += XXH3_mix16B(input+len-16, secret+16, seed); return XXH3_avalanche(acc); } } #define XXH3_MIDSIZE_MAX 240 XXH_NO_INLINE XXH64_hash_t XXH3_len_129to240_64b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(128 < len && len <= XXH3_MIDSIZE_MAX); #define XXH3_MIDSIZE_STARTOFFSET 3 #define XXH3_MIDSIZE_LASTOFFSET 17 { xxh_u64 acc = len * XXH_PRIME64_1; int const nbRounds = (int)len / 16; int i; for (i=0; i<8; i++) { acc += XXH3_mix16B(input+(16*i), secret+(16*i), seed); } acc = XXH3_avalanche(acc); XXH_ASSERT(nbRounds >= 8); #if defined(__clang__) /* Clang */ \ && (defined(__ARM_NEON) || defined(__ARM_NEON__)) /* NEON */ \ && !defined(XXH_ENABLE_AUTOVECTORIZE) /* Define to disable */ /* * UGLY HACK: * Clang for ARMv7-A tries to vectorize this loop, similar to GCC x86. * In everywhere else, it uses scalar code. * * For 64->128-bit multiplies, even if the NEON was 100% optimal, it * would still be slower than UMAAL (see XXH_mult64to128). * * Unfortunately, Clang doesn't handle the long multiplies properly and * converts them to the nonexistent "vmulq_u64" intrinsic, which is then * scalarized into an ugly mess of VMOV.32 instructions. * * This mess is difficult to avoid without turning autovectorization * off completely, but they are usually relatively minor and/or not * worth it to fix. * * This loop is the easiest to fix, as unlike XXH32, this pragma * _actually works_ because it is a loop vectorization instead of an * SLP vectorization. */ #pragma clang loop vectorize(disable) #endif for (i=8 ; i < nbRounds; i++) { acc += XXH3_mix16B(input+(16*i), secret+(16*(i-8)) + XXH3_MIDSIZE_STARTOFFSET, seed); } /* last bytes */ acc += XXH3_mix16B(input + len - 16, secret + XXH3_SECRET_SIZE_MIN - XXH3_MIDSIZE_LASTOFFSET, seed); return XXH3_avalanche(acc); } } /* ======= Long Keys ======= */ #define XXH_STRIPE_LEN 64 #define XXH_SECRET_CONSUME_RATE 8 /* nb of secret bytes consumed at each accumulation */ #define XXH_ACC_NB (XXH_STRIPE_LEN / sizeof(xxh_u64)) #ifdef XXH_OLD_NAMES # define STRIPE_LEN XXH_STRIPE_LEN # define ACC_NB XXH_ACC_NB #endif XXH_FORCE_INLINE void XXH_writeLE64(void* dst, xxh_u64 v64) { if (!XXH_CPU_LITTLE_ENDIAN) v64 = XXH_swap64(v64); XXH_memcpy(dst, &v64, sizeof(v64)); } /* Several intrinsic functions below are supposed to accept __int64 as argument, * as documented in https://software.intel.com/sites/landingpage/IntrinsicsGuide/ . * However, several environments do not define __int64 type, * requiring a workaround. */ #if !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) typedef int64_t xxh_i64; #else /* the following type must have a width of 64-bit */ typedef long long xxh_i64; #endif /* * XXH3_accumulate_512 is the tightest loop for long inputs, and it is the most optimized. * * It is a hardened version of UMAC, based off of FARSH's implementation. * * This was chosen because it adapts quite well to 32-bit, 64-bit, and SIMD * implementations, and it is ridiculously fast. * * We harden it by mixing the original input to the accumulators as well as the product. * * This means that in the (relatively likely) case of a multiply by zero, the * original input is preserved. * * On 128-bit inputs, we swap 64-bit pairs when we add the input to improve * cross-pollination, as otherwise the upper and lower halves would be * essentially independent. * * This doesn't matter on 64-bit hashes since they all get merged together in * the end, so we skip the extra step. * * Both XXH3_64bits and XXH3_128bits use this subroutine. */ #if (XXH_VECTOR == XXH_AVX512) \ || (defined(XXH_DISPATCH_AVX512) && XXH_DISPATCH_AVX512 != 0) #ifndef XXH_TARGET_AVX512 # define XXH_TARGET_AVX512 /* disable attribute target */ #endif XXH_FORCE_INLINE XXH_TARGET_AVX512 void XXH3_accumulate_512_avx512(void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { __m512i* const xacc = (__m512i *) acc; XXH_ASSERT((((size_t)acc) & 63) == 0); XXH_STATIC_ASSERT(XXH_STRIPE_LEN == sizeof(__m512i)); { /* data_vec = input[0]; */ __m512i const data_vec = _mm512_loadu_si512 (input); /* key_vec = secret[0]; */ __m512i const key_vec = _mm512_loadu_si512 (secret); /* data_key = data_vec ^ key_vec; */ __m512i const data_key = _mm512_xor_si512 (data_vec, key_vec); /* data_key_lo = data_key >> 32; */ __m512i const data_key_lo = _mm512_shuffle_epi32 (data_key, (_MM_PERM_ENUM)_MM_SHUFFLE(0, 3, 0, 1)); /* product = (data_key & 0xffffffff) * (data_key_lo & 0xffffffff); */ __m512i const product = _mm512_mul_epu32 (data_key, data_key_lo); /* xacc[0] += swap(data_vec); */ __m512i const data_swap = _mm512_shuffle_epi32(data_vec, (_MM_PERM_ENUM)_MM_SHUFFLE(1, 0, 3, 2)); __m512i const sum = _mm512_add_epi64(*xacc, data_swap); /* xacc[0] += product; */ *xacc = _mm512_add_epi64(product, sum); } } /* * XXH3_scrambleAcc: Scrambles the accumulators to improve mixing. * * Multiplication isn't perfect, as explained by Google in HighwayHash: * * // Multiplication mixes/scrambles bytes 0-7 of the 64-bit result to * // varying degrees. In descending order of goodness, bytes * // 3 4 2 5 1 6 0 7 have quality 228 224 164 160 100 96 36 32. * // As expected, the upper and lower bytes are much worse. * * Source: https://github.com/google/highwayhash/blob/0aaf66b/highwayhash/hh_avx2.h#L291 * * Since our algorithm uses a pseudorandom secret to add some variance into the * mix, we don't need to (or want to) mix as often or as much as HighwayHash does. * * This isn't as tight as XXH3_accumulate, but still written in SIMD to avoid * extraction. * * Both XXH3_64bits and XXH3_128bits use this subroutine. */ XXH_FORCE_INLINE XXH_TARGET_AVX512 void XXH3_scrambleAcc_avx512(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 63) == 0); XXH_STATIC_ASSERT(XXH_STRIPE_LEN == sizeof(__m512i)); { __m512i* const xacc = (__m512i*) acc; const __m512i prime32 = _mm512_set1_epi32((int)XXH_PRIME32_1); /* xacc[0] ^= (xacc[0] >> 47) */ __m512i const acc_vec = *xacc; __m512i const shifted = _mm512_srli_epi64 (acc_vec, 47); __m512i const data_vec = _mm512_xor_si512 (acc_vec, shifted); /* xacc[0] ^= secret; */ __m512i const key_vec = _mm512_loadu_si512 (secret); __m512i const data_key = _mm512_xor_si512 (data_vec, key_vec); /* xacc[0] *= XXH_PRIME32_1; */ __m512i const data_key_hi = _mm512_shuffle_epi32 (data_key, (_MM_PERM_ENUM)_MM_SHUFFLE(0, 3, 0, 1)); __m512i const prod_lo = _mm512_mul_epu32 (data_key, prime32); __m512i const prod_hi = _mm512_mul_epu32 (data_key_hi, prime32); *xacc = _mm512_add_epi64(prod_lo, _mm512_slli_epi64(prod_hi, 32)); } } XXH_FORCE_INLINE XXH_TARGET_AVX512 void XXH3_initCustomSecret_avx512(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 63) == 0); XXH_STATIC_ASSERT(XXH_SEC_ALIGN == 64); XXH_ASSERT(((size_t)customSecret & 63) == 0); (void)(&XXH_writeLE64); { int const nbRounds = XXH_SECRET_DEFAULT_SIZE / sizeof(__m512i); __m512i const seed = _mm512_mask_set1_epi64(_mm512_set1_epi64((xxh_i64)seed64), 0xAA, (xxh_i64)(0U - seed64)); const __m512i* const src = (const __m512i*) ((const void*) XXH3_kSecret); __m512i* const dest = ( __m512i*) customSecret; int i; XXH_ASSERT(((size_t)src & 63) == 0); /* control alignment */ XXH_ASSERT(((size_t)dest & 63) == 0); for (i=0; i < nbRounds; ++i) { /* GCC has a bug, _mm512_stream_load_si512 accepts 'void*', not 'void const*', * this will warn "discards 'const' qualifier". */ union { const __m512i* cp; void* p; } remote_const_void; remote_const_void.cp = src + i; dest[i] = _mm512_add_epi64(_mm512_stream_load_si512(remote_const_void.p), seed); } } } #endif #if (XXH_VECTOR == XXH_AVX2) \ || (defined(XXH_DISPATCH_AVX2) && XXH_DISPATCH_AVX2 != 0) #ifndef XXH_TARGET_AVX2 # define XXH_TARGET_AVX2 /* disable attribute target */ #endif XXH_FORCE_INLINE XXH_TARGET_AVX2 void XXH3_accumulate_512_avx2( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 31) == 0); { __m256i* const xacc = (__m256i *) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm256_loadu_si256 requires a const __m256i * pointer for some reason. */ const __m256i* const xinput = (const __m256i *) input; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm256_loadu_si256 requires a const __m256i * pointer for some reason. */ const __m256i* const xsecret = (const __m256i *) secret; size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m256i); i++) { /* data_vec = xinput[i]; */ __m256i const data_vec = _mm256_loadu_si256 (xinput+i); /* key_vec = xsecret[i]; */ __m256i const key_vec = _mm256_loadu_si256 (xsecret+i); /* data_key = data_vec ^ key_vec; */ __m256i const data_key = _mm256_xor_si256 (data_vec, key_vec); /* data_key_lo = data_key >> 32; */ __m256i const data_key_lo = _mm256_shuffle_epi32 (data_key, _MM_SHUFFLE(0, 3, 0, 1)); /* product = (data_key & 0xffffffff) * (data_key_lo & 0xffffffff); */ __m256i const product = _mm256_mul_epu32 (data_key, data_key_lo); /* xacc[i] += swap(data_vec); */ __m256i const data_swap = _mm256_shuffle_epi32(data_vec, _MM_SHUFFLE(1, 0, 3, 2)); __m256i const sum = _mm256_add_epi64(xacc[i], data_swap); /* xacc[i] += product; */ xacc[i] = _mm256_add_epi64(product, sum); } } } XXH_FORCE_INLINE XXH_TARGET_AVX2 void XXH3_scrambleAcc_avx2(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 31) == 0); { __m256i* const xacc = (__m256i*) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm256_loadu_si256 requires a const __m256i * pointer for some reason. */ const __m256i* const xsecret = (const __m256i *) secret; const __m256i prime32 = _mm256_set1_epi32((int)XXH_PRIME32_1); size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m256i); i++) { /* xacc[i] ^= (xacc[i] >> 47) */ __m256i const acc_vec = xacc[i]; __m256i const shifted = _mm256_srli_epi64 (acc_vec, 47); __m256i const data_vec = _mm256_xor_si256 (acc_vec, shifted); /* xacc[i] ^= xsecret; */ __m256i const key_vec = _mm256_loadu_si256 (xsecret+i); __m256i const data_key = _mm256_xor_si256 (data_vec, key_vec); /* xacc[i] *= XXH_PRIME32_1; */ __m256i const data_key_hi = _mm256_shuffle_epi32 (data_key, _MM_SHUFFLE(0, 3, 0, 1)); __m256i const prod_lo = _mm256_mul_epu32 (data_key, prime32); __m256i const prod_hi = _mm256_mul_epu32 (data_key_hi, prime32); xacc[i] = _mm256_add_epi64(prod_lo, _mm256_slli_epi64(prod_hi, 32)); } } } XXH_FORCE_INLINE XXH_TARGET_AVX2 void XXH3_initCustomSecret_avx2(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 31) == 0); XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE / sizeof(__m256i)) == 6); XXH_STATIC_ASSERT(XXH_SEC_ALIGN <= 64); (void)(&XXH_writeLE64); XXH_PREFETCH(customSecret); { __m256i const seed = _mm256_set_epi64x((xxh_i64)(0U - seed64), (xxh_i64)seed64, (xxh_i64)(0U - seed64), (xxh_i64)seed64); const __m256i* const src = (const __m256i*) ((const void*) XXH3_kSecret); __m256i* dest = ( __m256i*) customSecret; # if defined(__GNUC__) || defined(__clang__) /* * On GCC & Clang, marking 'dest' as modified will cause the compiler: * - do not extract the secret from sse registers in the internal loop * - use less common registers, and avoid pushing these reg into stack */ XXH_COMPILER_GUARD(dest); # endif XXH_ASSERT(((size_t)src & 31) == 0); /* control alignment */ XXH_ASSERT(((size_t)dest & 31) == 0); /* GCC -O2 need unroll loop manually */ dest[0] = _mm256_add_epi64(_mm256_stream_load_si256(src+0), seed); dest[1] = _mm256_add_epi64(_mm256_stream_load_si256(src+1), seed); dest[2] = _mm256_add_epi64(_mm256_stream_load_si256(src+2), seed); dest[3] = _mm256_add_epi64(_mm256_stream_load_si256(src+3), seed); dest[4] = _mm256_add_epi64(_mm256_stream_load_si256(src+4), seed); dest[5] = _mm256_add_epi64(_mm256_stream_load_si256(src+5), seed); } } #endif /* x86dispatch always generates SSE2 */ #if (XXH_VECTOR == XXH_SSE2) || defined(XXH_X86DISPATCH) #ifndef XXH_TARGET_SSE2 # define XXH_TARGET_SSE2 /* disable attribute target */ #endif XXH_FORCE_INLINE XXH_TARGET_SSE2 void XXH3_accumulate_512_sse2( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { /* SSE2 is just a half-scale version of the AVX2 version. */ XXH_ASSERT((((size_t)acc) & 15) == 0); { __m128i* const xacc = (__m128i *) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm_loadu_si128 requires a const __m128i * pointer for some reason. */ const __m128i* const xinput = (const __m128i *) input; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm_loadu_si128 requires a const __m128i * pointer for some reason. */ const __m128i* const xsecret = (const __m128i *) secret; size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m128i); i++) { /* data_vec = xinput[i]; */ __m128i const data_vec = _mm_loadu_si128 (xinput+i); /* key_vec = xsecret[i]; */ __m128i const key_vec = _mm_loadu_si128 (xsecret+i); /* data_key = data_vec ^ key_vec; */ __m128i const data_key = _mm_xor_si128 (data_vec, key_vec); /* data_key_lo = data_key >> 32; */ __m128i const data_key_lo = _mm_shuffle_epi32 (data_key, _MM_SHUFFLE(0, 3, 0, 1)); /* product = (data_key & 0xffffffff) * (data_key_lo & 0xffffffff); */ __m128i const product = _mm_mul_epu32 (data_key, data_key_lo); /* xacc[i] += swap(data_vec); */ __m128i const data_swap = _mm_shuffle_epi32(data_vec, _MM_SHUFFLE(1,0,3,2)); __m128i const sum = _mm_add_epi64(xacc[i], data_swap); /* xacc[i] += product; */ xacc[i] = _mm_add_epi64(product, sum); } } } XXH_FORCE_INLINE XXH_TARGET_SSE2 void XXH3_scrambleAcc_sse2(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); { __m128i* const xacc = (__m128i*) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm_loadu_si128 requires a const __m128i * pointer for some reason. */ const __m128i* const xsecret = (const __m128i *) secret; const __m128i prime32 = _mm_set1_epi32((int)XXH_PRIME32_1); size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m128i); i++) { /* xacc[i] ^= (xacc[i] >> 47) */ __m128i const acc_vec = xacc[i]; __m128i const shifted = _mm_srli_epi64 (acc_vec, 47); __m128i const data_vec = _mm_xor_si128 (acc_vec, shifted); /* xacc[i] ^= xsecret[i]; */ __m128i const key_vec = _mm_loadu_si128 (xsecret+i); __m128i const data_key = _mm_xor_si128 (data_vec, key_vec); /* xacc[i] *= XXH_PRIME32_1; */ __m128i const data_key_hi = _mm_shuffle_epi32 (data_key, _MM_SHUFFLE(0, 3, 0, 1)); __m128i const prod_lo = _mm_mul_epu32 (data_key, prime32); __m128i const prod_hi = _mm_mul_epu32 (data_key_hi, prime32); xacc[i] = _mm_add_epi64(prod_lo, _mm_slli_epi64(prod_hi, 32)); } } } XXH_FORCE_INLINE XXH_TARGET_SSE2 void XXH3_initCustomSecret_sse2(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 15) == 0); (void)(&XXH_writeLE64); { int const nbRounds = XXH_SECRET_DEFAULT_SIZE / sizeof(__m128i); # if defined(_MSC_VER) && defined(_M_IX86) && _MSC_VER < 1900 /* MSVC 32bit mode does not support _mm_set_epi64x before 2015 */ XXH_ALIGN(16) const xxh_i64 seed64x2[2] = { (xxh_i64)seed64, (xxh_i64)(0U - seed64) }; __m128i const seed = _mm_load_si128((__m128i const*)seed64x2); # else __m128i const seed = _mm_set_epi64x((xxh_i64)(0U - seed64), (xxh_i64)seed64); # endif int i; const void* const src16 = XXH3_kSecret; __m128i* dst16 = (__m128i*) customSecret; # if defined(__GNUC__) || defined(__clang__) /* * On GCC & Clang, marking 'dest' as modified will cause the compiler: * - do not extract the secret from sse registers in the internal loop * - use less common registers, and avoid pushing these reg into stack */ XXH_COMPILER_GUARD(dst16); # endif XXH_ASSERT(((size_t)src16 & 15) == 0); /* control alignment */ XXH_ASSERT(((size_t)dst16 & 15) == 0); for (i=0; i < nbRounds; ++i) { dst16[i] = _mm_add_epi64(_mm_load_si128((const __m128i *)src16+i), seed); } } } #endif #if (XXH_VECTOR == XXH_NEON) /* forward declarations for the scalar routines */ XXH_FORCE_INLINE void XXH3_scalarRound(void* XXH_RESTRICT acc, void const* XXH_RESTRICT input, void const* XXH_RESTRICT secret, size_t lane); XXH_FORCE_INLINE void XXH3_scalarScrambleRound(void* XXH_RESTRICT acc, void const* XXH_RESTRICT secret, size_t lane); /*! * @internal * @brief The bulk processing loop for NEON. * * The NEON code path is actually partially scalar when running on AArch64. This * is to optimize the pipelining and can have up to 15% speedup depending on the * CPU, and it also mitigates some GCC codegen issues. * * @see XXH3_NEON_LANES for configuring this and details about this optimization. */ XXH_FORCE_INLINE void XXH3_accumulate_512_neon( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); XXH_STATIC_ASSERT(XXH3_NEON_LANES > 0 && XXH3_NEON_LANES <= XXH_ACC_NB && XXH3_NEON_LANES % 2 == 0); { uint64x2_t* const xacc = (uint64x2_t *) acc; /* We don't use a uint32x4_t pointer because it causes bus errors on ARMv7. */ uint8_t const* const xinput = (const uint8_t *) input; uint8_t const* const xsecret = (const uint8_t *) secret; size_t i; /* NEON for the first few lanes (these loops are normally interleaved) */ for (i=0; i < XXH3_NEON_LANES / 2; i++) { /* data_vec = xinput[i]; */ uint8x16_t data_vec = vld1q_u8(xinput + (i * 16)); /* key_vec = xsecret[i]; */ uint8x16_t key_vec = vld1q_u8(xsecret + (i * 16)); uint64x2_t data_key; uint32x2_t data_key_lo, data_key_hi; /* xacc[i] += swap(data_vec); */ uint64x2_t const data64 = vreinterpretq_u64_u8(data_vec); uint64x2_t const swapped = vextq_u64(data64, data64, 1); xacc[i] = vaddq_u64 (xacc[i], swapped); /* data_key = data_vec ^ key_vec; */ data_key = vreinterpretq_u64_u8(veorq_u8(data_vec, key_vec)); /* data_key_lo = (uint32x2_t) (data_key & 0xFFFFFFFF); * data_key_hi = (uint32x2_t) (data_key >> 32); * data_key = UNDEFINED; */ XXH_SPLIT_IN_PLACE(data_key, data_key_lo, data_key_hi); /* xacc[i] += (uint64x2_t) data_key_lo * (uint64x2_t) data_key_hi; */ xacc[i] = vmlal_u32 (xacc[i], data_key_lo, data_key_hi); } /* Scalar for the remainder. This may be a zero iteration loop. */ for (i = XXH3_NEON_LANES; i < XXH_ACC_NB; i++) { XXH3_scalarRound(acc, input, secret, i); } } } XXH_FORCE_INLINE void XXH3_scrambleAcc_neon(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); { uint64x2_t* xacc = (uint64x2_t*) acc; uint8_t const* xsecret = (uint8_t const*) secret; uint32x2_t prime = vdup_n_u32 (XXH_PRIME32_1); size_t i; /* NEON for the first few lanes (these loops are normally interleaved) */ for (i=0; i < XXH3_NEON_LANES / 2; i++) { /* xacc[i] ^= (xacc[i] >> 47); */ uint64x2_t acc_vec = xacc[i]; uint64x2_t shifted = vshrq_n_u64 (acc_vec, 47); uint64x2_t data_vec = veorq_u64 (acc_vec, shifted); /* xacc[i] ^= xsecret[i]; */ uint8x16_t key_vec = vld1q_u8 (xsecret + (i * 16)); uint64x2_t data_key = veorq_u64 (data_vec, vreinterpretq_u64_u8(key_vec)); /* xacc[i] *= XXH_PRIME32_1 */ uint32x2_t data_key_lo, data_key_hi; /* data_key_lo = (uint32x2_t) (xacc[i] & 0xFFFFFFFF); * data_key_hi = (uint32x2_t) (xacc[i] >> 32); * xacc[i] = UNDEFINED; */ XXH_SPLIT_IN_PLACE(data_key, data_key_lo, data_key_hi); { /* * prod_hi = (data_key >> 32) * XXH_PRIME32_1; * * Avoid vmul_u32 + vshll_n_u32 since Clang 6 and 7 will * incorrectly "optimize" this: * tmp = vmul_u32(vmovn_u64(a), vmovn_u64(b)); * shifted = vshll_n_u32(tmp, 32); * to this: * tmp = "vmulq_u64"(a, b); // no such thing! * shifted = vshlq_n_u64(tmp, 32); * * However, unlike SSE, Clang lacks a 64-bit multiply routine * for NEON, and it scalarizes two 64-bit multiplies instead. * * vmull_u32 has the same timing as vmul_u32, and it avoids * this bug completely. * See https://bugs.llvm.org/show_bug.cgi?id=39967 */ uint64x2_t prod_hi = vmull_u32 (data_key_hi, prime); /* xacc[i] = prod_hi << 32; */ xacc[i] = vshlq_n_u64(prod_hi, 32); /* xacc[i] += (prod_hi & 0xFFFFFFFF) * XXH_PRIME32_1; */ xacc[i] = vmlal_u32(xacc[i], data_key_lo, prime); } } /* Scalar for the remainder. This may be a zero iteration loop. */ for (i = XXH3_NEON_LANES; i < XXH_ACC_NB; i++) { XXH3_scalarScrambleRound(acc, secret, i); } } } #endif #if (XXH_VECTOR == XXH_VSX) XXH_FORCE_INLINE void XXH3_accumulate_512_vsx( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { /* presumed aligned */ unsigned int* const xacc = (unsigned int*) acc; xxh_u64x2 const* const xinput = (xxh_u64x2 const*) input; /* no alignment restriction */ xxh_u64x2 const* const xsecret = (xxh_u64x2 const*) secret; /* no alignment restriction */ xxh_u64x2 const v32 = { 32, 32 }; size_t i; for (i = 0; i < XXH_STRIPE_LEN / sizeof(xxh_u64x2); i++) { /* data_vec = xinput[i]; */ xxh_u64x2 const data_vec = XXH_vec_loadu(xinput + i); /* key_vec = xsecret[i]; */ xxh_u64x2 const key_vec = XXH_vec_loadu(xsecret + i); xxh_u64x2 const data_key = data_vec ^ key_vec; /* shuffled = (data_key << 32) | (data_key >> 32); */ xxh_u32x4 const shuffled = (xxh_u32x4)vec_rl(data_key, v32); /* product = ((xxh_u64x2)data_key & 0xFFFFFFFF) * ((xxh_u64x2)shuffled & 0xFFFFFFFF); */ xxh_u64x2 const product = XXH_vec_mulo((xxh_u32x4)data_key, shuffled); /* acc_vec = xacc[i]; */ xxh_u64x2 acc_vec = (xxh_u64x2)vec_xl(0, xacc + 4 * i); acc_vec += product; /* swap high and low halves */ #ifdef __s390x__ acc_vec += vec_permi(data_vec, data_vec, 2); #else acc_vec += vec_xxpermdi(data_vec, data_vec, 2); #endif /* xacc[i] = acc_vec; */ vec_xst((xxh_u32x4)acc_vec, 0, xacc + 4 * i); } } XXH_FORCE_INLINE void XXH3_scrambleAcc_vsx(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); { xxh_u64x2* const xacc = (xxh_u64x2*) acc; const xxh_u64x2* const xsecret = (const xxh_u64x2*) secret; /* constants */ xxh_u64x2 const v32 = { 32, 32 }; xxh_u64x2 const v47 = { 47, 47 }; xxh_u32x4 const prime = { XXH_PRIME32_1, XXH_PRIME32_1, XXH_PRIME32_1, XXH_PRIME32_1 }; size_t i; for (i = 0; i < XXH_STRIPE_LEN / sizeof(xxh_u64x2); i++) { /* xacc[i] ^= (xacc[i] >> 47); */ xxh_u64x2 const acc_vec = xacc[i]; xxh_u64x2 const data_vec = acc_vec ^ (acc_vec >> v47); /* xacc[i] ^= xsecret[i]; */ xxh_u64x2 const key_vec = XXH_vec_loadu(xsecret + i); xxh_u64x2 const data_key = data_vec ^ key_vec; /* xacc[i] *= XXH_PRIME32_1 */ /* prod_lo = ((xxh_u64x2)data_key & 0xFFFFFFFF) * ((xxh_u64x2)prime & 0xFFFFFFFF); */ xxh_u64x2 const prod_even = XXH_vec_mule((xxh_u32x4)data_key, prime); /* prod_hi = ((xxh_u64x2)data_key >> 32) * ((xxh_u64x2)prime >> 32); */ xxh_u64x2 const prod_odd = XXH_vec_mulo((xxh_u32x4)data_key, prime); xacc[i] = prod_odd + (prod_even << v32); } } } #endif /* scalar variants - universal */ /*! * @internal * @brief Scalar round for @ref XXH3_accumulate_512_scalar(). * * This is extracted to its own function because the NEON path uses a combination * of NEON and scalar. */ XXH_FORCE_INLINE void XXH3_scalarRound(void* XXH_RESTRICT acc, void const* XXH_RESTRICT input, void const* XXH_RESTRICT secret, size_t lane) { xxh_u64* xacc = (xxh_u64*) acc; xxh_u8 const* xinput = (xxh_u8 const*) input; xxh_u8 const* xsecret = (xxh_u8 const*) secret; XXH_ASSERT(lane < XXH_ACC_NB); XXH_ASSERT(((size_t)acc & (XXH_ACC_ALIGN-1)) == 0); { xxh_u64 const data_val = XXH_readLE64(xinput + lane * 8); xxh_u64 const data_key = data_val ^ XXH_readLE64(xsecret + lane * 8); xacc[lane ^ 1] += data_val; /* swap adjacent lanes */ xacc[lane] += XXH_mult32to64(data_key & 0xFFFFFFFF, data_key >> 32); } } /*! * @internal * @brief Processes a 64 byte block of data using the scalar path. */ XXH_FORCE_INLINE void XXH3_accumulate_512_scalar(void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { size_t i; for (i=0; i < XXH_ACC_NB; i++) { XXH3_scalarRound(acc, input, secret, i); } } /*! * @internal * @brief Scalar scramble step for @ref XXH3_scrambleAcc_scalar(). * * This is extracted to its own function because the NEON path uses a combination * of NEON and scalar. */ XXH_FORCE_INLINE void XXH3_scalarScrambleRound(void* XXH_RESTRICT acc, void const* XXH_RESTRICT secret, size_t lane) { xxh_u64* const xacc = (xxh_u64*) acc; /* presumed aligned */ const xxh_u8* const xsecret = (const xxh_u8*) secret; /* no alignment restriction */ XXH_ASSERT((((size_t)acc) & (XXH_ACC_ALIGN-1)) == 0); XXH_ASSERT(lane < XXH_ACC_NB); { xxh_u64 const key64 = XXH_readLE64(xsecret + lane * 8); xxh_u64 acc64 = xacc[lane]; acc64 = XXH_xorshift64(acc64, 47); acc64 ^= key64; acc64 *= XXH_PRIME32_1; xacc[lane] = acc64; } } /*! * @internal * @brief Scrambles the accumulators after a large chunk has been read */ XXH_FORCE_INLINE void XXH3_scrambleAcc_scalar(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { size_t i; for (i=0; i < XXH_ACC_NB; i++) { XXH3_scalarScrambleRound(acc, secret, i); } } XXH_FORCE_INLINE void XXH3_initCustomSecret_scalar(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { /* * We need a separate pointer for the hack below, * which requires a non-const pointer. * Any decent compiler will optimize this out otherwise. */ const xxh_u8* kSecretPtr = XXH3_kSecret; XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 15) == 0); #if defined(__clang__) && defined(__aarch64__) /* * UGLY HACK: * Clang generates a bunch of MOV/MOVK pairs for aarch64, and they are * placed sequentially, in order, at the top of the unrolled loop. * * While MOVK is great for generating constants (2 cycles for a 64-bit * constant compared to 4 cycles for LDR), it fights for bandwidth with * the arithmetic instructions. * * I L S * MOVK * MOVK * MOVK * MOVK * ADD * SUB STR * STR * By forcing loads from memory (as the asm line causes Clang to assume * that XXH3_kSecretPtr has been changed), the pipelines are used more * efficiently: * I L S * LDR * ADD LDR * SUB STR * STR * * See XXH3_NEON_LANES for details on the pipsline. * * XXH3_64bits_withSeed, len == 256, Snapdragon 835 * without hack: 2654.4 MB/s * with hack: 3202.9 MB/s */ XXH_COMPILER_GUARD(kSecretPtr); #endif /* * Note: in debug mode, this overrides the asm optimization * and Clang will emit MOVK chains again. */ XXH_ASSERT(kSecretPtr == XXH3_kSecret); { int const nbRounds = XXH_SECRET_DEFAULT_SIZE / 16; int i; for (i=0; i < nbRounds; i++) { /* * The asm hack causes Clang to assume that kSecretPtr aliases with * customSecret, and on aarch64, this prevented LDP from merging two * loads together for free. Putting the loads together before the stores * properly generates LDP. */ xxh_u64 lo = XXH_readLE64(kSecretPtr + 16*i) + seed64; xxh_u64 hi = XXH_readLE64(kSecretPtr + 16*i + 8) - seed64; XXH_writeLE64((xxh_u8*)customSecret + 16*i, lo); XXH_writeLE64((xxh_u8*)customSecret + 16*i + 8, hi); } } } typedef void (*XXH3_f_accumulate_512)(void* XXH_RESTRICT, const void*, const void*); typedef void (*XXH3_f_scrambleAcc)(void* XXH_RESTRICT, const void*); typedef void (*XXH3_f_initCustomSecret)(void* XXH_RESTRICT, xxh_u64); #if (XXH_VECTOR == XXH_AVX512) #define XXH3_accumulate_512 XXH3_accumulate_512_avx512 #define XXH3_scrambleAcc XXH3_scrambleAcc_avx512 #define XXH3_initCustomSecret XXH3_initCustomSecret_avx512 #elif (XXH_VECTOR == XXH_AVX2) #define XXH3_accumulate_512 XXH3_accumulate_512_avx2 #define XXH3_scrambleAcc XXH3_scrambleAcc_avx2 #define XXH3_initCustomSecret XXH3_initCustomSecret_avx2 #elif (XXH_VECTOR == XXH_SSE2) #define XXH3_accumulate_512 XXH3_accumulate_512_sse2 #define XXH3_scrambleAcc XXH3_scrambleAcc_sse2 #define XXH3_initCustomSecret XXH3_initCustomSecret_sse2 #elif (XXH_VECTOR == XXH_NEON) #define XXH3_accumulate_512 XXH3_accumulate_512_neon #define XXH3_scrambleAcc XXH3_scrambleAcc_neon #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #elif (XXH_VECTOR == XXH_VSX) #define XXH3_accumulate_512 XXH3_accumulate_512_vsx #define XXH3_scrambleAcc XXH3_scrambleAcc_vsx #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #else /* scalar */ #define XXH3_accumulate_512 XXH3_accumulate_512_scalar #define XXH3_scrambleAcc XXH3_scrambleAcc_scalar #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #endif #ifndef XXH_PREFETCH_DIST # ifdef __clang__ # define XXH_PREFETCH_DIST 320 # else # if (XXH_VECTOR == XXH_AVX512) # define XXH_PREFETCH_DIST 512 # else # define XXH_PREFETCH_DIST 384 # endif # endif /* __clang__ */ #endif /* XXH_PREFETCH_DIST */ /* * XXH3_accumulate() * Loops over XXH3_accumulate_512(). * Assumption: nbStripes will not overflow the secret size */ XXH_FORCE_INLINE void XXH3_accumulate( xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT input, const xxh_u8* XXH_RESTRICT secret, size_t nbStripes, XXH3_f_accumulate_512 f_acc512) { size_t n; for (n = 0; n < nbStripes; n++ ) { const xxh_u8* const in = input + n*XXH_STRIPE_LEN; XXH_PREFETCH(in + XXH_PREFETCH_DIST); f_acc512(acc, in, secret + n*XXH_SECRET_CONSUME_RATE); } } XXH_FORCE_INLINE void XXH3_hashLong_internal_loop(xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble) { size_t const nbStripesPerBlock = (secretSize - XXH_STRIPE_LEN) / XXH_SECRET_CONSUME_RATE; size_t const block_len = XXH_STRIPE_LEN * nbStripesPerBlock; size_t const nb_blocks = (len - 1) / block_len; size_t n; XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); for (n = 0; n < nb_blocks; n++) { XXH3_accumulate(acc, input + n*block_len, secret, nbStripesPerBlock, f_acc512); f_scramble(acc, secret + secretSize - XXH_STRIPE_LEN); } /* last partial block */ XXH_ASSERT(len > XXH_STRIPE_LEN); { size_t const nbStripes = ((len - 1) - (block_len * nb_blocks)) / XXH_STRIPE_LEN; XXH_ASSERT(nbStripes <= (secretSize / XXH_SECRET_CONSUME_RATE)); XXH3_accumulate(acc, input + nb_blocks*block_len, secret, nbStripes, f_acc512); /* last stripe */ { const xxh_u8* const p = input + len - XXH_STRIPE_LEN; #define XXH_SECRET_LASTACC_START 7 /* not aligned on 8, last secret is different from acc & scrambler */ f_acc512(acc, p, secret + secretSize - XXH_STRIPE_LEN - XXH_SECRET_LASTACC_START); } } } XXH_FORCE_INLINE xxh_u64 XXH3_mix2Accs(const xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT secret) { return XXH3_mul128_fold64( acc[0] ^ XXH_readLE64(secret), acc[1] ^ XXH_readLE64(secret+8) ); } static XXH64_hash_t XXH3_mergeAccs(const xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT secret, xxh_u64 start) { xxh_u64 result64 = start; size_t i = 0; for (i = 0; i < 4; i++) { result64 += XXH3_mix2Accs(acc+2*i, secret + 16*i); #if defined(__clang__) /* Clang */ \ && (defined(__arm__) || defined(__thumb__)) /* ARMv7 */ \ && (defined(__ARM_NEON) || defined(__ARM_NEON__)) /* NEON */ \ && !defined(XXH_ENABLE_AUTOVECTORIZE) /* Define to disable */ /* * UGLY HACK: * Prevent autovectorization on Clang ARMv7-a. Exact same problem as * the one in XXH3_len_129to240_64b. Speeds up shorter keys > 240b. * XXH3_64bits, len == 256, Snapdragon 835: * without hack: 2063.7 MB/s * with hack: 2560.7 MB/s */ XXH_COMPILER_GUARD(result64); #endif } return XXH3_avalanche(result64); } #define XXH3_INIT_ACC { XXH_PRIME32_3, XXH_PRIME64_1, XXH_PRIME64_2, XXH_PRIME64_3, \ XXH_PRIME64_4, XXH_PRIME32_2, XXH_PRIME64_5, XXH_PRIME32_1 } XXH_FORCE_INLINE XXH64_hash_t XXH3_hashLong_64b_internal(const void* XXH_RESTRICT input, size_t len, const void* XXH_RESTRICT secret, size_t secretSize, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble) { XXH_ALIGN(XXH_ACC_ALIGN) xxh_u64 acc[XXH_ACC_NB] = XXH3_INIT_ACC; XXH3_hashLong_internal_loop(acc, (const xxh_u8*)input, len, (const xxh_u8*)secret, secretSize, f_acc512, f_scramble); /* converge into final hash */ XXH_STATIC_ASSERT(sizeof(acc) == 64); /* do not align on 8, so that the secret is different from the accumulator */ #define XXH_SECRET_MERGEACCS_START 11 XXH_ASSERT(secretSize >= sizeof(acc) + XXH_SECRET_MERGEACCS_START); return XXH3_mergeAccs(acc, (const xxh_u8*)secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)len * XXH_PRIME64_1); } /* * It's important for performance to transmit secret's size (when it's static) * so that the compiler can properly optimize the vectorized loop. * This makes a big performance difference for "medium" keys (<1 KB) when using AVX instruction set. */ XXH_FORCE_INLINE XXH64_hash_t XXH3_hashLong_64b_withSecret(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const xxh_u8* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; return XXH3_hashLong_64b_internal(input, len, secret, secretLen, XXH3_accumulate_512, XXH3_scrambleAcc); } /* * It's preferable for performance that XXH3_hashLong is not inlined, * as it results in a smaller function for small data, easier to the instruction cache. * Note that inside this no_inline function, we do inline the internal loop, * and provide a statically defined secret size to allow optimization of vector loop. */ XXH_NO_INLINE XXH64_hash_t XXH3_hashLong_64b_default(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const xxh_u8* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; (void)secret; (void)secretLen; return XXH3_hashLong_64b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_accumulate_512, XXH3_scrambleAcc); } /* * XXH3_hashLong_64b_withSeed(): * Generate a custom key based on alteration of default XXH3_kSecret with the seed, * and then use this key for long mode hashing. * * This operation is decently fast but nonetheless costs a little bit of time. * Try to avoid it whenever possible (typically when seed==0). * * It's important for performance that XXH3_hashLong is not inlined. Not sure * why (uop cache maybe?), but the difference is large and easily measurable. */ XXH_FORCE_INLINE XXH64_hash_t XXH3_hashLong_64b_withSeed_internal(const void* input, size_t len, XXH64_hash_t seed, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble, XXH3_f_initCustomSecret f_initSec) { if (seed == 0) return XXH3_hashLong_64b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), f_acc512, f_scramble); { XXH_ALIGN(XXH_SEC_ALIGN) xxh_u8 secret[XXH_SECRET_DEFAULT_SIZE]; f_initSec(secret, seed); return XXH3_hashLong_64b_internal(input, len, secret, sizeof(secret), f_acc512, f_scramble); } } /* * It's important for performance that XXH3_hashLong is not inlined. */ XXH_NO_INLINE XXH64_hash_t XXH3_hashLong_64b_withSeed(const void* input, size_t len, XXH64_hash_t seed, const xxh_u8* secret, size_t secretLen) { (void)secret; (void)secretLen; return XXH3_hashLong_64b_withSeed_internal(input, len, seed, XXH3_accumulate_512, XXH3_scrambleAcc, XXH3_initCustomSecret); } typedef XXH64_hash_t (*XXH3_hashLong64_f)(const void* XXH_RESTRICT, size_t, XXH64_hash_t, const xxh_u8* XXH_RESTRICT, size_t); XXH_FORCE_INLINE XXH64_hash_t XXH3_64bits_internal(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen, XXH3_hashLong64_f f_hashLong) { XXH_ASSERT(secretLen >= XXH3_SECRET_SIZE_MIN); /* * If an action is to be taken if `secretLen` condition is not respected, * it should be done here. * For now, it's a contract pre-condition. * Adding a check and a branch here would cost performance at every hash. * Also, note that function signature doesn't offer room to return an error. */ if (len <= 16) return XXH3_len_0to16_64b((const xxh_u8*)input, len, (const xxh_u8*)secret, seed64); if (len <= 128) return XXH3_len_17to128_64b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); if (len <= XXH3_MIDSIZE_MAX) return XXH3_len_129to240_64b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); return f_hashLong(input, len, seed64, (const xxh_u8*)secret, secretLen); } /* === Public entry point === */ /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits(const void* input, size_t len) { return XXH3_64bits_internal(input, len, 0, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_64b_default); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSecret(const void* input, size_t len, const void* secret, size_t secretSize) { return XXH3_64bits_internal(input, len, 0, secret, secretSize, XXH3_hashLong_64b_withSecret); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSeed(const void* input, size_t len, XXH64_hash_t seed) { return XXH3_64bits_internal(input, len, seed, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_64b_withSeed); } XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSecretandSeed(const void* input, size_t len, const void* secret, size_t secretSize, XXH64_hash_t seed) { if (len <= XXH3_MIDSIZE_MAX) return XXH3_64bits_internal(input, len, seed, XXH3_kSecret, sizeof(XXH3_kSecret), NULL); return XXH3_hashLong_64b_withSecret(input, len, seed, (const xxh_u8*)secret, secretSize); } /* === XXH3 streaming === */ /* * Malloc's a pointer that is always aligned to align. * * This must be freed with `XXH_alignedFree()`. * * malloc typically guarantees 16 byte alignment on 64-bit systems and 8 byte * alignment on 32-bit. This isn't enough for the 32 byte aligned loads in AVX2 * or on 32-bit, the 16 byte aligned loads in SSE2 and NEON. * * This underalignment previously caused a rather obvious crash which went * completely unnoticed due to XXH3_createState() not actually being tested. * Credit to RedSpah for noticing this bug. * * The alignment is done manually: Functions like posix_memalign or _mm_malloc * are avoided: To maintain portability, we would have to write a fallback * like this anyways, and besides, testing for the existence of library * functions without relying on external build tools is impossible. * * The method is simple: Overallocate, manually align, and store the offset * to the original behind the returned pointer. * * Align must be a power of 2 and 8 <= align <= 128. */ static void* XXH_alignedMalloc(size_t s, size_t align) { XXH_ASSERT(align <= 128 && align >= 8); /* range check */ XXH_ASSERT((align & (align-1)) == 0); /* power of 2 */ XXH_ASSERT(s != 0 && s < (s + align)); /* empty/overflow */ { /* Overallocate to make room for manual realignment and an offset byte */ xxh_u8* base = (xxh_u8*)XXH_malloc(s + align); if (base != NULL) { /* * Get the offset needed to align this pointer. * * Even if the returned pointer is aligned, there will always be * at least one byte to store the offset to the original pointer. */ size_t offset = align - ((size_t)base & (align - 1)); /* base % align */ /* Add the offset for the now-aligned pointer */ xxh_u8* ptr = base + offset; XXH_ASSERT((size_t)ptr % align == 0); /* Store the offset immediately before the returned pointer. */ ptr[-1] = (xxh_u8)offset; return ptr; } return NULL; } } /* * Frees an aligned pointer allocated by XXH_alignedMalloc(). Don't pass * normal malloc'd pointers, XXH_alignedMalloc has a specific data layout. */ static void XXH_alignedFree(void* p) { if (p != NULL) { xxh_u8* ptr = (xxh_u8*)p; /* Get the offset byte we added in XXH_malloc. */ xxh_u8 offset = ptr[-1]; /* Free the original malloc'd pointer */ xxh_u8* base = ptr - offset; XXH_free(base); } } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH3_state_t* XXH3_createState(void) { XXH3_state_t* const state = (XXH3_state_t*)XXH_alignedMalloc(sizeof(XXH3_state_t), 64); if (state==NULL) return NULL; XXH3_INITSTATE(state); return state; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_freeState(XXH3_state_t* statePtr) { XXH_alignedFree(statePtr); return XXH_OK; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API void XXH3_copyState(XXH3_state_t* dst_state, const XXH3_state_t* src_state) { XXH_memcpy(dst_state, src_state, sizeof(*dst_state)); } static void XXH3_reset_internal(XXH3_state_t* statePtr, XXH64_hash_t seed, const void* secret, size_t secretSize) { size_t const initStart = offsetof(XXH3_state_t, bufferedSize); size_t const initLength = offsetof(XXH3_state_t, nbStripesPerBlock) - initStart; XXH_ASSERT(offsetof(XXH3_state_t, nbStripesPerBlock) > initStart); XXH_ASSERT(statePtr != NULL); /* set members from bufferedSize to nbStripesPerBlock (excluded) to 0 */ memset((char*)statePtr + initStart, 0, initLength); statePtr->acc[0] = XXH_PRIME32_3; statePtr->acc[1] = XXH_PRIME64_1; statePtr->acc[2] = XXH_PRIME64_2; statePtr->acc[3] = XXH_PRIME64_3; statePtr->acc[4] = XXH_PRIME64_4; statePtr->acc[5] = XXH_PRIME32_2; statePtr->acc[6] = XXH_PRIME64_5; statePtr->acc[7] = XXH_PRIME32_1; statePtr->seed = seed; statePtr->useSeed = (seed != 0); statePtr->extSecret = (const unsigned char*)secret; XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); statePtr->secretLimit = secretSize - XXH_STRIPE_LEN; statePtr->nbStripesPerBlock = statePtr->secretLimit / XXH_SECRET_CONSUME_RATE; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset(XXH3_state_t* statePtr) { if (statePtr == NULL) return XXH_ERROR; XXH3_reset_internal(statePtr, 0, XXH3_kSecret, XXH_SECRET_DEFAULT_SIZE); return XXH_OK; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecret(XXH3_state_t* statePtr, const void* secret, size_t secretSize) { if (statePtr == NULL) return XXH_ERROR; XXH3_reset_internal(statePtr, 0, secret, secretSize); if (secret == NULL) return XXH_ERROR; if (secretSize < XXH3_SECRET_SIZE_MIN) return XXH_ERROR; return XXH_OK; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSeed(XXH3_state_t* statePtr, XXH64_hash_t seed) { if (statePtr == NULL) return XXH_ERROR; if (seed==0) return XXH3_64bits_reset(statePtr); if ((seed != statePtr->seed) || (statePtr->extSecret != NULL)) XXH3_initCustomSecret(statePtr->customSecret, seed); XXH3_reset_internal(statePtr, seed, NULL, XXH_SECRET_DEFAULT_SIZE); return XXH_OK; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecretandSeed(XXH3_state_t* statePtr, const void* secret, size_t secretSize, XXH64_hash_t seed64) { if (statePtr == NULL) return XXH_ERROR; if (secret == NULL) return XXH_ERROR; if (secretSize < XXH3_SECRET_SIZE_MIN) return XXH_ERROR; XXH3_reset_internal(statePtr, seed64, secret, secretSize); statePtr->useSeed = 1; /* always, even if seed64==0 */ return XXH_OK; } /* Note : when XXH3_consumeStripes() is invoked, * there must be a guarantee that at least one more byte must be consumed from input * so that the function can blindly consume all stripes using the "normal" secret segment */ XXH_FORCE_INLINE void XXH3_consumeStripes(xxh_u64* XXH_RESTRICT acc, size_t* XXH_RESTRICT nbStripesSoFarPtr, size_t nbStripesPerBlock, const xxh_u8* XXH_RESTRICT input, size_t nbStripes, const xxh_u8* XXH_RESTRICT secret, size_t secretLimit, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble) { XXH_ASSERT(nbStripes <= nbStripesPerBlock); /* can handle max 1 scramble per invocation */ XXH_ASSERT(*nbStripesSoFarPtr < nbStripesPerBlock); if (nbStripesPerBlock - *nbStripesSoFarPtr <= nbStripes) { /* need a scrambling operation */ size_t const nbStripesToEndofBlock = nbStripesPerBlock - *nbStripesSoFarPtr; size_t const nbStripesAfterBlock = nbStripes - nbStripesToEndofBlock; XXH3_accumulate(acc, input, secret + nbStripesSoFarPtr[0] * XXH_SECRET_CONSUME_RATE, nbStripesToEndofBlock, f_acc512); f_scramble(acc, secret + secretLimit); XXH3_accumulate(acc, input + nbStripesToEndofBlock * XXH_STRIPE_LEN, secret, nbStripesAfterBlock, f_acc512); *nbStripesSoFarPtr = nbStripesAfterBlock; } else { XXH3_accumulate(acc, input, secret + nbStripesSoFarPtr[0] * XXH_SECRET_CONSUME_RATE, nbStripes, f_acc512); *nbStripesSoFarPtr += nbStripes; } } #ifndef XXH3_STREAM_USE_STACK # ifndef __clang__ /* clang doesn't need additional stack space */ # define XXH3_STREAM_USE_STACK 1 # endif #endif /* * Both XXH3_64bits_update and XXH3_128bits_update use this routine. */ XXH_FORCE_INLINE XXH_errorcode XXH3_update(XXH3_state_t* XXH_RESTRICT const state, const xxh_u8* XXH_RESTRICT input, size_t len, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble) { if (input==NULL) { XXH_ASSERT(len == 0); return XXH_OK; } XXH_ASSERT(state != NULL); { const xxh_u8* const bEnd = input + len; const unsigned char* const secret = (state->extSecret == NULL) ? state->customSecret : state->extSecret; #if defined(XXH3_STREAM_USE_STACK) && XXH3_STREAM_USE_STACK >= 1 /* For some reason, gcc and MSVC seem to suffer greatly * when operating accumulators directly into state. * Operating into stack space seems to enable proper optimization. * clang, on the other hand, doesn't seem to need this trick */ XXH_ALIGN(XXH_ACC_ALIGN) xxh_u64 acc[8]; memcpy(acc, state->acc, sizeof(acc)); #else xxh_u64* XXH_RESTRICT const acc = state->acc; #endif state->totalLen += len; XXH_ASSERT(state->bufferedSize <= XXH3_INTERNALBUFFER_SIZE); /* small input : just fill in tmp buffer */ if (state->bufferedSize + len <= XXH3_INTERNALBUFFER_SIZE) { XXH_memcpy(state->buffer + state->bufferedSize, input, len); state->bufferedSize += (XXH32_hash_t)len; return XXH_OK; } /* total input is now > XXH3_INTERNALBUFFER_SIZE */ #define XXH3_INTERNALBUFFER_STRIPES (XXH3_INTERNALBUFFER_SIZE / XXH_STRIPE_LEN) XXH_STATIC_ASSERT(XXH3_INTERNALBUFFER_SIZE % XXH_STRIPE_LEN == 0); /* clean multiple */ /* * Internal buffer is partially filled (always, except at beginning) * Complete it, then consume it. */ if (state->bufferedSize) { size_t const loadSize = XXH3_INTERNALBUFFER_SIZE - state->bufferedSize; XXH_memcpy(state->buffer + state->bufferedSize, input, loadSize); input += loadSize; XXH3_consumeStripes(acc, &state->nbStripesSoFar, state->nbStripesPerBlock, state->buffer, XXH3_INTERNALBUFFER_STRIPES, secret, state->secretLimit, f_acc512, f_scramble); state->bufferedSize = 0; } XXH_ASSERT(input < bEnd); /* large input to consume : ingest per full block */ if ((size_t)(bEnd - input) > state->nbStripesPerBlock * XXH_STRIPE_LEN) { size_t nbStripes = (size_t)(bEnd - 1 - input) / XXH_STRIPE_LEN; XXH_ASSERT(state->nbStripesPerBlock >= state->nbStripesSoFar); /* join to current block's end */ { size_t const nbStripesToEnd = state->nbStripesPerBlock - state->nbStripesSoFar; XXH_ASSERT(nbStripesToEnd <= nbStripes); XXH3_accumulate(acc, input, secret + state->nbStripesSoFar * XXH_SECRET_CONSUME_RATE, nbStripesToEnd, f_acc512); f_scramble(acc, secret + state->secretLimit); state->nbStripesSoFar = 0; input += nbStripesToEnd * XXH_STRIPE_LEN; nbStripes -= nbStripesToEnd; } /* consume per entire blocks */ while(nbStripes >= state->nbStripesPerBlock) { XXH3_accumulate(acc, input, secret, state->nbStripesPerBlock, f_acc512); f_scramble(acc, secret + state->secretLimit); input += state->nbStripesPerBlock * XXH_STRIPE_LEN; nbStripes -= state->nbStripesPerBlock; } /* consume last partial block */ XXH3_accumulate(acc, input, secret, nbStripes, f_acc512); input += nbStripes * XXH_STRIPE_LEN; XXH_ASSERT(input < bEnd); /* at least some bytes left */ state->nbStripesSoFar = nbStripes; /* buffer predecessor of last partial stripe */ XXH_memcpy(state->buffer + sizeof(state->buffer) - XXH_STRIPE_LEN, input - XXH_STRIPE_LEN, XXH_STRIPE_LEN); XXH_ASSERT(bEnd - input <= XXH_STRIPE_LEN); } else { /* content to consume <= block size */ /* Consume input by a multiple of internal buffer size */ if (bEnd - input > XXH3_INTERNALBUFFER_SIZE) { const xxh_u8* const limit = bEnd - XXH3_INTERNALBUFFER_SIZE; do { XXH3_consumeStripes(acc, &state->nbStripesSoFar, state->nbStripesPerBlock, input, XXH3_INTERNALBUFFER_STRIPES, secret, state->secretLimit, f_acc512, f_scramble); input += XXH3_INTERNALBUFFER_SIZE; } while (inputbuffer + sizeof(state->buffer) - XXH_STRIPE_LEN, input - XXH_STRIPE_LEN, XXH_STRIPE_LEN); } } /* Some remaining input (always) : buffer it */ XXH_ASSERT(input < bEnd); XXH_ASSERT(bEnd - input <= XXH3_INTERNALBUFFER_SIZE); XXH_ASSERT(state->bufferedSize == 0); XXH_memcpy(state->buffer, input, (size_t)(bEnd-input)); state->bufferedSize = (XXH32_hash_t)(bEnd-input); #if defined(XXH3_STREAM_USE_STACK) && XXH3_STREAM_USE_STACK >= 1 /* save stack accumulators into state */ memcpy(state->acc, acc, sizeof(acc)); #endif } return XXH_OK; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_update(XXH3_state_t* state, const void* input, size_t len) { return XXH3_update(state, (const xxh_u8*)input, len, XXH3_accumulate_512, XXH3_scrambleAcc); } XXH_FORCE_INLINE void XXH3_digest_long (XXH64_hash_t* acc, const XXH3_state_t* state, const unsigned char* secret) { /* * Digest on a local copy. This way, the state remains unaltered, and it can * continue ingesting more input afterwards. */ XXH_memcpy(acc, state->acc, sizeof(state->acc)); if (state->bufferedSize >= XXH_STRIPE_LEN) { size_t const nbStripes = (state->bufferedSize - 1) / XXH_STRIPE_LEN; size_t nbStripesSoFar = state->nbStripesSoFar; XXH3_consumeStripes(acc, &nbStripesSoFar, state->nbStripesPerBlock, state->buffer, nbStripes, secret, state->secretLimit, XXH3_accumulate_512, XXH3_scrambleAcc); /* last stripe */ XXH3_accumulate_512(acc, state->buffer + state->bufferedSize - XXH_STRIPE_LEN, secret + state->secretLimit - XXH_SECRET_LASTACC_START); } else { /* bufferedSize < XXH_STRIPE_LEN */ xxh_u8 lastStripe[XXH_STRIPE_LEN]; size_t const catchupSize = XXH_STRIPE_LEN - state->bufferedSize; XXH_ASSERT(state->bufferedSize > 0); /* there is always some input buffered */ XXH_memcpy(lastStripe, state->buffer + sizeof(state->buffer) - catchupSize, catchupSize); XXH_memcpy(lastStripe + catchupSize, state->buffer, state->bufferedSize); XXH3_accumulate_512(acc, lastStripe, secret + state->secretLimit - XXH_SECRET_LASTACC_START); } } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_digest (const XXH3_state_t* state) { const unsigned char* const secret = (state->extSecret == NULL) ? state->customSecret : state->extSecret; if (state->totalLen > XXH3_MIDSIZE_MAX) { XXH_ALIGN(XXH_ACC_ALIGN) XXH64_hash_t acc[XXH_ACC_NB]; XXH3_digest_long(acc, state, secret); return XXH3_mergeAccs(acc, secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)state->totalLen * XXH_PRIME64_1); } /* totalLen <= XXH3_MIDSIZE_MAX: digesting a short input */ if (state->useSeed) return XXH3_64bits_withSeed(state->buffer, (size_t)state->totalLen, state->seed); return XXH3_64bits_withSecret(state->buffer, (size_t)(state->totalLen), secret, state->secretLimit + XXH_STRIPE_LEN); } /* ========================================== * XXH3 128 bits (a.k.a XXH128) * ========================================== * XXH3's 128-bit variant has better mixing and strength than the 64-bit variant, * even without counting the significantly larger output size. * * For example, extra steps are taken to avoid the seed-dependent collisions * in 17-240 byte inputs (See XXH3_mix16B and XXH128_mix32B). * * This strength naturally comes at the cost of some speed, especially on short * lengths. Note that longer hashes are about as fast as the 64-bit version * due to it using only a slight modification of the 64-bit loop. * * XXH128 is also more oriented towards 64-bit machines. It is still extremely * fast for a _128-bit_ hash on 32-bit (it usually clears XXH64). */ XXH_FORCE_INLINE XXH128_hash_t XXH3_len_1to3_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { /* A doubled version of 1to3_64b with different constants. */ XXH_ASSERT(input != NULL); XXH_ASSERT(1 <= len && len <= 3); XXH_ASSERT(secret != NULL); /* * len = 1: combinedl = { input[0], 0x01, input[0], input[0] } * len = 2: combinedl = { input[1], 0x02, input[0], input[1] } * len = 3: combinedl = { input[2], 0x03, input[0], input[1] } */ { xxh_u8 const c1 = input[0]; xxh_u8 const c2 = input[len >> 1]; xxh_u8 const c3 = input[len - 1]; xxh_u32 const combinedl = ((xxh_u32)c1 <<16) | ((xxh_u32)c2 << 24) | ((xxh_u32)c3 << 0) | ((xxh_u32)len << 8); xxh_u32 const combinedh = XXH_rotl32(XXH_swap32(combinedl), 13); xxh_u64 const bitflipl = (XXH_readLE32(secret) ^ XXH_readLE32(secret+4)) + seed; xxh_u64 const bitfliph = (XXH_readLE32(secret+8) ^ XXH_readLE32(secret+12)) - seed; xxh_u64 const keyed_lo = (xxh_u64)combinedl ^ bitflipl; xxh_u64 const keyed_hi = (xxh_u64)combinedh ^ bitfliph; XXH128_hash_t h128; h128.low64 = XXH64_avalanche(keyed_lo); h128.high64 = XXH64_avalanche(keyed_hi); return h128; } } XXH_FORCE_INLINE XXH128_hash_t XXH3_len_4to8_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(4 <= len && len <= 8); seed ^= (xxh_u64)XXH_swap32((xxh_u32)seed) << 32; { xxh_u32 const input_lo = XXH_readLE32(input); xxh_u32 const input_hi = XXH_readLE32(input + len - 4); xxh_u64 const input_64 = input_lo + ((xxh_u64)input_hi << 32); xxh_u64 const bitflip = (XXH_readLE64(secret+16) ^ XXH_readLE64(secret+24)) + seed; xxh_u64 const keyed = input_64 ^ bitflip; /* Shift len to the left to ensure it is even, this avoids even multiplies. */ XXH128_hash_t m128 = XXH_mult64to128(keyed, XXH_PRIME64_1 + (len << 2)); m128.high64 += (m128.low64 << 1); m128.low64 ^= (m128.high64 >> 3); m128.low64 = XXH_xorshift64(m128.low64, 35); m128.low64 *= 0x9FB21C651E98DF25ULL; m128.low64 = XXH_xorshift64(m128.low64, 28); m128.high64 = XXH3_avalanche(m128.high64); return m128; } } XXH_FORCE_INLINE XXH128_hash_t XXH3_len_9to16_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(9 <= len && len <= 16); { xxh_u64 const bitflipl = (XXH_readLE64(secret+32) ^ XXH_readLE64(secret+40)) - seed; xxh_u64 const bitfliph = (XXH_readLE64(secret+48) ^ XXH_readLE64(secret+56)) + seed; xxh_u64 const input_lo = XXH_readLE64(input); xxh_u64 input_hi = XXH_readLE64(input + len - 8); XXH128_hash_t m128 = XXH_mult64to128(input_lo ^ input_hi ^ bitflipl, XXH_PRIME64_1); /* * Put len in the middle of m128 to ensure that the length gets mixed to * both the low and high bits in the 128x64 multiply below. */ m128.low64 += (xxh_u64)(len - 1) << 54; input_hi ^= bitfliph; /* * Add the high 32 bits of input_hi to the high 32 bits of m128, then * add the long product of the low 32 bits of input_hi and XXH_PRIME32_2 to * the high 64 bits of m128. * * The best approach to this operation is different on 32-bit and 64-bit. */ if (sizeof(void *) < sizeof(xxh_u64)) { /* 32-bit */ /* * 32-bit optimized version, which is more readable. * * On 32-bit, it removes an ADC and delays a dependency between the two * halves of m128.high64, but it generates an extra mask on 64-bit. */ m128.high64 += (input_hi & 0xFFFFFFFF00000000ULL) + XXH_mult32to64((xxh_u32)input_hi, XXH_PRIME32_2); } else { /* * 64-bit optimized (albeit more confusing) version. * * Uses some properties of addition and multiplication to remove the mask: * * Let: * a = input_hi.lo = (input_hi & 0x00000000FFFFFFFF) * b = input_hi.hi = (input_hi & 0xFFFFFFFF00000000) * c = XXH_PRIME32_2 * * a + (b * c) * Inverse Property: x + y - x == y * a + (b * (1 + c - 1)) * Distributive Property: x * (y + z) == (x * y) + (x * z) * a + (b * 1) + (b * (c - 1)) * Identity Property: x * 1 == x * a + b + (b * (c - 1)) * * Substitute a, b, and c: * input_hi.hi + input_hi.lo + ((xxh_u64)input_hi.lo * (XXH_PRIME32_2 - 1)) * * Since input_hi.hi + input_hi.lo == input_hi, we get this: * input_hi + ((xxh_u64)input_hi.lo * (XXH_PRIME32_2 - 1)) */ m128.high64 += input_hi + XXH_mult32to64((xxh_u32)input_hi, XXH_PRIME32_2 - 1); } /* m128 ^= XXH_swap64(m128 >> 64); */ m128.low64 ^= XXH_swap64(m128.high64); { /* 128x64 multiply: h128 = m128 * XXH_PRIME64_2; */ XXH128_hash_t h128 = XXH_mult64to128(m128.low64, XXH_PRIME64_2); h128.high64 += m128.high64 * XXH_PRIME64_2; h128.low64 = XXH3_avalanche(h128.low64); h128.high64 = XXH3_avalanche(h128.high64); return h128; } } } /* * Assumption: `secret` size is >= XXH3_SECRET_SIZE_MIN */ XXH_FORCE_INLINE XXH128_hash_t XXH3_len_0to16_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(len <= 16); { if (len > 8) return XXH3_len_9to16_128b(input, len, secret, seed); if (len >= 4) return XXH3_len_4to8_128b(input, len, secret, seed); if (len) return XXH3_len_1to3_128b(input, len, secret, seed); { XXH128_hash_t h128; xxh_u64 const bitflipl = XXH_readLE64(secret+64) ^ XXH_readLE64(secret+72); xxh_u64 const bitfliph = XXH_readLE64(secret+80) ^ XXH_readLE64(secret+88); h128.low64 = XXH64_avalanche(seed ^ bitflipl); h128.high64 = XXH64_avalanche( seed ^ bitfliph); return h128; } } } /* * A bit slower than XXH3_mix16B, but handles multiply by zero better. */ XXH_FORCE_INLINE XXH128_hash_t XXH128_mix32B(XXH128_hash_t acc, const xxh_u8* input_1, const xxh_u8* input_2, const xxh_u8* secret, XXH64_hash_t seed) { acc.low64 += XXH3_mix16B (input_1, secret+0, seed); acc.low64 ^= XXH_readLE64(input_2) + XXH_readLE64(input_2 + 8); acc.high64 += XXH3_mix16B (input_2, secret+16, seed); acc.high64 ^= XXH_readLE64(input_1) + XXH_readLE64(input_1 + 8); return acc; } XXH_FORCE_INLINE XXH128_hash_t XXH3_len_17to128_128b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(16 < len && len <= 128); { XXH128_hash_t acc; acc.low64 = len * XXH_PRIME64_1; acc.high64 = 0; if (len > 32) { if (len > 64) { if (len > 96) { acc = XXH128_mix32B(acc, input+48, input+len-64, secret+96, seed); } acc = XXH128_mix32B(acc, input+32, input+len-48, secret+64, seed); } acc = XXH128_mix32B(acc, input+16, input+len-32, secret+32, seed); } acc = XXH128_mix32B(acc, input, input+len-16, secret, seed); { XXH128_hash_t h128; h128.low64 = acc.low64 + acc.high64; h128.high64 = (acc.low64 * XXH_PRIME64_1) + (acc.high64 * XXH_PRIME64_4) + ((len - seed) * XXH_PRIME64_2); h128.low64 = XXH3_avalanche(h128.low64); h128.high64 = (XXH64_hash_t)0 - XXH3_avalanche(h128.high64); return h128; } } } XXH_NO_INLINE XXH128_hash_t XXH3_len_129to240_128b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(128 < len && len <= XXH3_MIDSIZE_MAX); { XXH128_hash_t acc; int const nbRounds = (int)len / 32; int i; acc.low64 = len * XXH_PRIME64_1; acc.high64 = 0; for (i=0; i<4; i++) { acc = XXH128_mix32B(acc, input + (32 * i), input + (32 * i) + 16, secret + (32 * i), seed); } acc.low64 = XXH3_avalanche(acc.low64); acc.high64 = XXH3_avalanche(acc.high64); XXH_ASSERT(nbRounds >= 4); for (i=4 ; i < nbRounds; i++) { acc = XXH128_mix32B(acc, input + (32 * i), input + (32 * i) + 16, secret + XXH3_MIDSIZE_STARTOFFSET + (32 * (i - 4)), seed); } /* last bytes */ acc = XXH128_mix32B(acc, input + len - 16, input + len - 32, secret + XXH3_SECRET_SIZE_MIN - XXH3_MIDSIZE_LASTOFFSET - 16, 0ULL - seed); { XXH128_hash_t h128; h128.low64 = acc.low64 + acc.high64; h128.high64 = (acc.low64 * XXH_PRIME64_1) + (acc.high64 * XXH_PRIME64_4) + ((len - seed) * XXH_PRIME64_2); h128.low64 = XXH3_avalanche(h128.low64); h128.high64 = (XXH64_hash_t)0 - XXH3_avalanche(h128.high64); return h128; } } } XXH_FORCE_INLINE XXH128_hash_t XXH3_hashLong_128b_internal(const void* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble) { XXH_ALIGN(XXH_ACC_ALIGN) xxh_u64 acc[XXH_ACC_NB] = XXH3_INIT_ACC; XXH3_hashLong_internal_loop(acc, (const xxh_u8*)input, len, secret, secretSize, f_acc512, f_scramble); /* converge into final hash */ XXH_STATIC_ASSERT(sizeof(acc) == 64); XXH_ASSERT(secretSize >= sizeof(acc) + XXH_SECRET_MERGEACCS_START); { XXH128_hash_t h128; h128.low64 = XXH3_mergeAccs(acc, secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)len * XXH_PRIME64_1); h128.high64 = XXH3_mergeAccs(acc, secret + secretSize - sizeof(acc) - XXH_SECRET_MERGEACCS_START, ~((xxh_u64)len * XXH_PRIME64_2)); return h128; } } /* * It's important for performance that XXH3_hashLong is not inlined. */ XXH_NO_INLINE XXH128_hash_t XXH3_hashLong_128b_default(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; (void)secret; (void)secretLen; return XXH3_hashLong_128b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_accumulate_512, XXH3_scrambleAcc); } /* * It's important for performance to pass @secretLen (when it's static) * to the compiler, so that it can properly optimize the vectorized loop. */ XXH_FORCE_INLINE XXH128_hash_t XXH3_hashLong_128b_withSecret(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; return XXH3_hashLong_128b_internal(input, len, (const xxh_u8*)secret, secretLen, XXH3_accumulate_512, XXH3_scrambleAcc); } XXH_FORCE_INLINE XXH128_hash_t XXH3_hashLong_128b_withSeed_internal(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble, XXH3_f_initCustomSecret f_initSec) { if (seed64 == 0) return XXH3_hashLong_128b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), f_acc512, f_scramble); { XXH_ALIGN(XXH_SEC_ALIGN) xxh_u8 secret[XXH_SECRET_DEFAULT_SIZE]; f_initSec(secret, seed64); return XXH3_hashLong_128b_internal(input, len, (const xxh_u8*)secret, sizeof(secret), f_acc512, f_scramble); } } /* * It's important for performance that XXH3_hashLong is not inlined. */ XXH_NO_INLINE XXH128_hash_t XXH3_hashLong_128b_withSeed(const void* input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen) { (void)secret; (void)secretLen; return XXH3_hashLong_128b_withSeed_internal(input, len, seed64, XXH3_accumulate_512, XXH3_scrambleAcc, XXH3_initCustomSecret); } typedef XXH128_hash_t (*XXH3_hashLong128_f)(const void* XXH_RESTRICT, size_t, XXH64_hash_t, const void* XXH_RESTRICT, size_t); XXH_FORCE_INLINE XXH128_hash_t XXH3_128bits_internal(const void* input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen, XXH3_hashLong128_f f_hl128) { XXH_ASSERT(secretLen >= XXH3_SECRET_SIZE_MIN); /* * If an action is to be taken if `secret` conditions are not respected, * it should be done here. * For now, it's a contract pre-condition. * Adding a check and a branch here would cost performance at every hash. */ if (len <= 16) return XXH3_len_0to16_128b((const xxh_u8*)input, len, (const xxh_u8*)secret, seed64); if (len <= 128) return XXH3_len_17to128_128b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); if (len <= XXH3_MIDSIZE_MAX) return XXH3_len_129to240_128b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); return f_hl128(input, len, seed64, secret, secretLen); } /* === Public XXH128 API === */ /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits(const void* input, size_t len) { return XXH3_128bits_internal(input, len, 0, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_128b_default); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSecret(const void* input, size_t len, const void* secret, size_t secretSize) { return XXH3_128bits_internal(input, len, 0, (const xxh_u8*)secret, secretSize, XXH3_hashLong_128b_withSecret); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSeed(const void* input, size_t len, XXH64_hash_t seed) { return XXH3_128bits_internal(input, len, seed, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_128b_withSeed); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSecretandSeed(const void* input, size_t len, const void* secret, size_t secretSize, XXH64_hash_t seed) { if (len <= XXH3_MIDSIZE_MAX) return XXH3_128bits_internal(input, len, seed, XXH3_kSecret, sizeof(XXH3_kSecret), NULL); return XXH3_hashLong_128b_withSecret(input, len, seed, secret, secretSize); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH128_hash_t XXH128(const void* input, size_t len, XXH64_hash_t seed) { return XXH3_128bits_withSeed(input, len, seed); } /* === XXH3 128-bit streaming === */ /* * All initialization and update functions are identical to 64-bit streaming variant. * The only difference is the finalization routine. */ /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset(XXH3_state_t* statePtr) { return XXH3_64bits_reset(statePtr); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecret(XXH3_state_t* statePtr, const void* secret, size_t secretSize) { return XXH3_64bits_reset_withSecret(statePtr, secret, secretSize); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSeed(XXH3_state_t* statePtr, XXH64_hash_t seed) { return XXH3_64bits_reset_withSeed(statePtr, seed); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecretandSeed(XXH3_state_t* statePtr, const void* secret, size_t secretSize, XXH64_hash_t seed) { return XXH3_64bits_reset_withSecretandSeed(statePtr, secret, secretSize, seed); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_update(XXH3_state_t* state, const void* input, size_t len) { return XXH3_update(state, (const xxh_u8*)input, len, XXH3_accumulate_512, XXH3_scrambleAcc); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_digest (const XXH3_state_t* state) { const unsigned char* const secret = (state->extSecret == NULL) ? state->customSecret : state->extSecret; if (state->totalLen > XXH3_MIDSIZE_MAX) { XXH_ALIGN(XXH_ACC_ALIGN) XXH64_hash_t acc[XXH_ACC_NB]; XXH3_digest_long(acc, state, secret); XXH_ASSERT(state->secretLimit + XXH_STRIPE_LEN >= sizeof(acc) + XXH_SECRET_MERGEACCS_START); { XXH128_hash_t h128; h128.low64 = XXH3_mergeAccs(acc, secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)state->totalLen * XXH_PRIME64_1); h128.high64 = XXH3_mergeAccs(acc, secret + state->secretLimit + XXH_STRIPE_LEN - sizeof(acc) - XXH_SECRET_MERGEACCS_START, ~((xxh_u64)state->totalLen * XXH_PRIME64_2)); return h128; } } /* len <= XXH3_MIDSIZE_MAX : short code */ if (state->seed) return XXH3_128bits_withSeed(state->buffer, (size_t)state->totalLen, state->seed); return XXH3_128bits_withSecret(state->buffer, (size_t)(state->totalLen), secret, state->secretLimit + XXH_STRIPE_LEN); } /* 128-bit utility functions */ #include /* memcmp, memcpy */ /* return : 1 is equal, 0 if different */ /*! @ingroup xxh3_family */ XXH_PUBLIC_API int XXH128_isEqual(XXH128_hash_t h1, XXH128_hash_t h2) { /* note : XXH128_hash_t is compact, it has no padding byte */ return !(memcmp(&h1, &h2, sizeof(h1))); } /* This prototype is compatible with stdlib's qsort(). * return : >0 if *h128_1 > *h128_2 * <0 if *h128_1 < *h128_2 * =0 if *h128_1 == *h128_2 */ /*! @ingroup xxh3_family */ XXH_PUBLIC_API int XXH128_cmp(const void* h128_1, const void* h128_2) { XXH128_hash_t const h1 = *(const XXH128_hash_t*)h128_1; XXH128_hash_t const h2 = *(const XXH128_hash_t*)h128_2; int const hcmp = (h1.high64 > h2.high64) - (h2.high64 > h1.high64); /* note : bets that, in most cases, hash values are different */ if (hcmp) return hcmp; return (h1.low64 > h2.low64) - (h2.low64 > h1.low64); } /*====== Canonical representation ======*/ /*! @ingroup xxh3_family */ XXH_PUBLIC_API void XXH128_canonicalFromHash(XXH128_canonical_t* dst, XXH128_hash_t hash) { XXH_STATIC_ASSERT(sizeof(XXH128_canonical_t) == sizeof(XXH128_hash_t)); if (XXH_CPU_LITTLE_ENDIAN) { hash.high64 = XXH_swap64(hash.high64); hash.low64 = XXH_swap64(hash.low64); } XXH_memcpy(dst, &hash.high64, sizeof(hash.high64)); XXH_memcpy((char*)dst + sizeof(hash.high64), &hash.low64, sizeof(hash.low64)); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH128_hash_t XXH128_hashFromCanonical(const XXH128_canonical_t* src) { XXH128_hash_t h; h.high64 = XXH_readBE64(src); h.low64 = XXH_readBE64(src->digest + 8); return h; } /* ========================================== * Secret generators * ========================================== */ #define XXH_MIN(x, y) (((x) > (y)) ? (y) : (x)) XXH_FORCE_INLINE void XXH3_combine16(void* dst, XXH128_hash_t h128) { XXH_writeLE64( dst, XXH_readLE64(dst) ^ h128.low64 ); XXH_writeLE64( (char*)dst+8, XXH_readLE64((char*)dst+8) ^ h128.high64 ); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_generateSecret(void* secretBuffer, size_t secretSize, const void* customSeed, size_t customSeedSize) { #if (XXH_DEBUGLEVEL >= 1) XXH_ASSERT(secretBuffer != NULL); XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); #else /* production mode, assert() are disabled */ if (secretBuffer == NULL) return XXH_ERROR; if (secretSize < XXH3_SECRET_SIZE_MIN) return XXH_ERROR; #endif if (customSeedSize == 0) { customSeed = XXH3_kSecret; customSeedSize = XXH_SECRET_DEFAULT_SIZE; } #if (XXH_DEBUGLEVEL >= 1) XXH_ASSERT(customSeed != NULL); #else if (customSeed == NULL) return XXH_ERROR; #endif /* Fill secretBuffer with a copy of customSeed - repeat as needed */ { size_t pos = 0; while (pos < secretSize) { size_t const toCopy = XXH_MIN((secretSize - pos), customSeedSize); memcpy((char*)secretBuffer + pos, customSeed, toCopy); pos += toCopy; } } { size_t const nbSeg16 = secretSize / 16; size_t n; XXH128_canonical_t scrambler; XXH128_canonicalFromHash(&scrambler, XXH128(customSeed, customSeedSize, 0)); for (n=0; n= 199901L) /* C99 */) # define ERR_STATIC static inline #elif defined(_MSC_VER) # define ERR_STATIC static __inline #else # define ERR_STATIC static /* this version may generate warnings for unused static functions; disable the relevant warning */ #endif /*-**************************************** * Customization (error_public.h) ******************************************/ typedef ZSTD_ErrorCode ERR_enum; #define PREFIX(name) ZSTD_error_##name /*-**************************************** * Error codes handling ******************************************/ #undef ERROR /* already defined on Visual Studio */ #define ERROR(name) ZSTD_ERROR(name) #define ZSTD_ERROR(name) ((size_t)-PREFIX(name)) ERR_STATIC unsigned ERR_isError(size_t code) { return (code > ERROR(maxCode)); } ERR_STATIC ERR_enum ERR_getErrorCode(size_t code) { if (!ERR_isError(code)) return (ERR_enum)0; return (ERR_enum) (0-code); } /* check and forward error code */ #define CHECK_V_F(e, f) size_t const e = f; if (ERR_isError(e)) return e #define CHECK_F(f) { CHECK_V_F(_var_err__, f); } /*-**************************************** * Error Strings ******************************************/ const char* ERR_getErrorString(ERR_enum code); /* error_private.c */ ERR_STATIC const char* ERR_getErrorName(size_t code) { return ERR_getErrorString(ERR_getErrorCode(code)); } /** * Ignore: this is an internal helper. * * This is a helper function to help force C99-correctness during compilation. * Under strict compilation modes, variadic macro arguments can't be empty. * However, variadic function arguments can be. Using a function therefore lets * us statically check that at least one (string) argument was passed, * independent of the compilation flags. */ static INLINE_KEYWORD UNUSED_ATTR void _force_has_format_string(const char *format, ...) { (void)format; } /** * Ignore: this is an internal helper. * * We want to force this function invocation to be syntactically correct, but * we don't want to force runtime evaluation of its arguments. */ #define _FORCE_HAS_FORMAT_STRING(...) \ if (0) { \ _force_has_format_string(__VA_ARGS__); \ } #define ERR_QUOTE(str) #str /** * Return the specified error if the condition evaluates to true. * * In debug modes, prints additional information. * In order to do that (particularly, printing the conditional that failed), * this can't just wrap RETURN_ERROR(). */ #define RETURN_ERROR_IF(cond, err, ...) \ if (cond) { \ RAWLOG(3, "%s:%d: ERROR!: check %s failed, returning %s", \ __FILE__, __LINE__, ERR_QUOTE(cond), ERR_QUOTE(ERROR(err))); \ _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \ RAWLOG(3, ": " __VA_ARGS__); \ RAWLOG(3, "\n"); \ return ERROR(err); \ } /** * Unconditionally return the specified error. * * In debug modes, prints additional information. */ #define RETURN_ERROR(err, ...) \ do { \ RAWLOG(3, "%s:%d: ERROR!: unconditional check failed, returning %s", \ __FILE__, __LINE__, ERR_QUOTE(ERROR(err))); \ _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \ RAWLOG(3, ": " __VA_ARGS__); \ RAWLOG(3, "\n"); \ return ERROR(err); \ } while(0); /** * If the provided expression evaluates to an error code, returns that error code. * * In debug modes, prints additional information. */ #define FORWARD_IF_ERROR(err, ...) \ do { \ size_t const err_code = (err); \ if (ERR_isError(err_code)) { \ RAWLOG(3, "%s:%d: ERROR!: forwarding error in %s: %s", \ __FILE__, __LINE__, ERR_QUOTE(err), ERR_getErrorName(err_code)); \ _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \ RAWLOG(3, ": " __VA_ARGS__); \ RAWLOG(3, "\n"); \ return err_code; \ } \ } while(0); #if defined (__cplusplus) } #endif #endif /* ERROR_H_MODULE */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/portability_macros.h0000644000175200007730000001046314515254731026673 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_PORTABILITY_MACROS_H #define ZSTD_PORTABILITY_MACROS_H /** * This header file contains macro definitions to support portability. * This header is shared between C and ASM code, so it MUST only * contain macro definitions. It MUST not contain any C code. * * This header ONLY defines macros to detect platforms/feature support. * */ /* compat. with non-clang compilers */ #ifndef __has_attribute #define __has_attribute(x) 0 #endif /* compat. with non-clang compilers */ #ifndef __has_builtin # define __has_builtin(x) 0 #endif /* compat. with non-clang compilers */ #ifndef __has_feature # define __has_feature(x) 0 #endif /* detects whether we are being compiled under msan */ #ifndef ZSTD_MEMORY_SANITIZER # if __has_feature(memory_sanitizer) # define ZSTD_MEMORY_SANITIZER 1 # else # define ZSTD_MEMORY_SANITIZER 0 # endif #endif /* detects whether we are being compiled under asan */ #ifndef ZSTD_ADDRESS_SANITIZER # if __has_feature(address_sanitizer) # define ZSTD_ADDRESS_SANITIZER 1 # elif defined(__SANITIZE_ADDRESS__) # define ZSTD_ADDRESS_SANITIZER 1 # else # define ZSTD_ADDRESS_SANITIZER 0 # endif #endif /* detects whether we are being compiled under dfsan */ #ifndef ZSTD_DATAFLOW_SANITIZER # if __has_feature(dataflow_sanitizer) # define ZSTD_DATAFLOW_SANITIZER 1 # else # define ZSTD_DATAFLOW_SANITIZER 0 # endif #endif /* Mark the internal assembly functions as hidden */ #ifdef __ELF__ # define ZSTD_HIDE_ASM_FUNCTION(func) .hidden func #else # define ZSTD_HIDE_ASM_FUNCTION(func) #endif /* Enable runtime BMI2 dispatch based on the CPU. * Enabled for clang & gcc >=4.8 on x86 when BMI2 isn't enabled by default. */ #ifndef DYNAMIC_BMI2 #if ((defined(__clang__) && __has_attribute(__target__)) \ || (defined(__GNUC__) \ && (__GNUC__ >= 5 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 8)))) \ && (defined(__x86_64__) || defined(_M_X64)) \ && !defined(__BMI2__) # define DYNAMIC_BMI2 1 #else # define DYNAMIC_BMI2 0 #endif #endif /** * Only enable assembly for GNUC compatible compilers, * because other platforms may not support GAS assembly syntax. * * Only enable assembly for Linux / MacOS, other platforms may * work, but they haven't been tested. This could likely be * extended to BSD systems. * * Disable assembly when MSAN is enabled, because MSAN requires * 100% of code to be instrumented to work. */ #if defined(__GNUC__) # if defined(__linux__) || defined(__linux) || defined(__APPLE__) # if ZSTD_MEMORY_SANITIZER # define ZSTD_ASM_SUPPORTED 0 # elif ZSTD_DATAFLOW_SANITIZER # define ZSTD_ASM_SUPPORTED 0 # else # define ZSTD_ASM_SUPPORTED 1 # endif # else # define ZSTD_ASM_SUPPORTED 0 # endif #else # define ZSTD_ASM_SUPPORTED 0 #endif /** * Determines whether we should enable assembly for x86-64 * with BMI2. * * Enable if all of the following conditions hold: * - ASM hasn't been explicitly disabled by defining ZSTD_DISABLE_ASM * - Assembly is supported * - We are compiling for x86-64 and either: * - DYNAMIC_BMI2 is enabled * - BMI2 is supported at compile time */ #if !defined(ZSTD_DISABLE_ASM) && \ ZSTD_ASM_SUPPORTED && \ defined(__x86_64__) && \ (DYNAMIC_BMI2 || defined(__BMI2__)) # define ZSTD_ENABLE_ASM_X86_64_BMI2 1 #else # define ZSTD_ENABLE_ASM_X86_64_BMI2 0 #endif /* * For x86 ELF targets, add .note.gnu.property section for Intel CET in * assembly sources when CET is enabled. * * Additionally, any function that may be called indirectly must begin * with ZSTD_CET_ENDBRANCH. */ #if defined(__ELF__) && (defined(__x86_64__) || defined(__i386__)) \ && defined(__has_include) # if __has_include() # include # define ZSTD_CET_ENDBRANCH _CET_ENDBR # endif #endif #ifndef ZSTD_CET_ENDBRANCH # define ZSTD_CET_ENDBRANCH #endif #endif /* ZSTD_PORTABILITY_MACROS_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/pool.h0000644000175200007730000000513614515254731023737 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef POOL_H #define POOL_H #if defined (__cplusplus) extern "C" { #endif #include "zstd_deps.h" #define ZSTD_STATIC_LINKING_ONLY /* ZSTD_customMem */ #include "../zstd.h" typedef struct POOL_ctx_s POOL_ctx; /*! POOL_create() : * Create a thread pool with at most `numThreads` threads. * `numThreads` must be at least 1. * The maximum number of queued jobs before blocking is `queueSize`. * @return : POOL_ctx pointer on success, else NULL. */ POOL_ctx* POOL_create(size_t numThreads, size_t queueSize); POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem); /*! POOL_free() : * Free a thread pool returned by POOL_create(). */ void POOL_free(POOL_ctx* ctx); /*! POOL_joinJobs() : * Waits for all queued jobs to finish executing. */ void POOL_joinJobs(POOL_ctx* ctx); /*! POOL_resize() : * Expands or shrinks pool's number of threads. * This is more efficient than releasing + creating a new context, * since it tries to preserve and re-use existing threads. * `numThreads` must be at least 1. * @return : 0 when resize was successful, * !0 (typically 1) if there is an error. * note : only numThreads can be resized, queueSize remains unchanged. */ int POOL_resize(POOL_ctx* ctx, size_t numThreads); /*! POOL_sizeof() : * @return threadpool memory usage * note : compatible with NULL (returns 0 in this case) */ size_t POOL_sizeof(const POOL_ctx* ctx); /*! POOL_function : * The function type that can be added to a thread pool. */ typedef void (*POOL_function)(void*); /*! POOL_add() : * Add the job `function(opaque)` to the thread pool. `ctx` must be valid. * Possibly blocks until there is room in the queue. * Note : The function may be executed asynchronously, * therefore, `opaque` must live until function has been completed. */ void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque); /*! POOL_tryAdd() : * Add the job `function(opaque)` to thread pool _if_ a queue slot is available. * Returns immediately even if not (does not block). * @return : 1 if successful, 0 if not. */ int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque); #if defined (__cplusplus) } #endif #endif zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/entropy_common.c0000644000175200007730000003177414515254731026040 0ustar rlaboissrlaboiss/* ****************************************************************** * Common functions of New Generation Entropy library * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy * - Public forum : https://groups.google.com/forum/#!forum/lz4c * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ /* ************************************* * Dependencies ***************************************/ #include "mem.h" #include "error_private.h" /* ERR_*, ERROR */ #define FSE_STATIC_LINKING_ONLY /* FSE_MIN_TABLELOG */ #include "fse.h" #include "huf.h" #include "bits.h" /* ZSDT_highbit32, ZSTD_countTrailingZeros32 */ /*=== Version ===*/ unsigned FSE_versionNumber(void) { return FSE_VERSION_NUMBER; } /*=== Error Management ===*/ unsigned FSE_isError(size_t code) { return ERR_isError(code); } const char* FSE_getErrorName(size_t code) { return ERR_getErrorName(code); } unsigned HUF_isError(size_t code) { return ERR_isError(code); } const char* HUF_getErrorName(size_t code) { return ERR_getErrorName(code); } /*-************************************************************** * FSE NCount encoding-decoding ****************************************************************/ FORCE_INLINE_TEMPLATE size_t FSE_readNCount_body(short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr, const void* headerBuffer, size_t hbSize) { const BYTE* const istart = (const BYTE*) headerBuffer; const BYTE* const iend = istart + hbSize; const BYTE* ip = istart; int nbBits; int remaining; int threshold; U32 bitStream; int bitCount; unsigned charnum = 0; unsigned const maxSV1 = *maxSVPtr + 1; int previous0 = 0; if (hbSize < 8) { /* This function only works when hbSize >= 8 */ char buffer[8] = {0}; ZSTD_memcpy(buffer, headerBuffer, hbSize); { size_t const countSize = FSE_readNCount(normalizedCounter, maxSVPtr, tableLogPtr, buffer, sizeof(buffer)); if (FSE_isError(countSize)) return countSize; if (countSize > hbSize) return ERROR(corruption_detected); return countSize; } } assert(hbSize >= 8); /* init */ ZSTD_memset(normalizedCounter, 0, (*maxSVPtr+1) * sizeof(normalizedCounter[0])); /* all symbols not present in NCount have a frequency of 0 */ bitStream = MEM_readLE32(ip); nbBits = (bitStream & 0xF) + FSE_MIN_TABLELOG; /* extract tableLog */ if (nbBits > FSE_TABLELOG_ABSOLUTE_MAX) return ERROR(tableLog_tooLarge); bitStream >>= 4; bitCount = 4; *tableLogPtr = nbBits; remaining = (1<> 1; while (repeats >= 12) { charnum += 3 * 12; if (LIKELY(ip <= iend-7)) { ip += 3; } else { bitCount -= (int)(8 * (iend - 7 - ip)); bitCount &= 31; ip = iend - 4; } bitStream = MEM_readLE32(ip) >> bitCount; repeats = ZSTD_countTrailingZeros32(~bitStream | 0x80000000) >> 1; } charnum += 3 * repeats; bitStream >>= 2 * repeats; bitCount += 2 * repeats; /* Add the final repeat which isn't 0b11. */ assert((bitStream & 3) < 3); charnum += bitStream & 3; bitCount += 2; /* This is an error, but break and return an error * at the end, because returning out of a loop makes * it harder for the compiler to optimize. */ if (charnum >= maxSV1) break; /* We don't need to set the normalized count to 0 * because we already memset the whole buffer to 0. */ if (LIKELY(ip <= iend-7) || (ip + (bitCount>>3) <= iend-4)) { assert((bitCount >> 3) <= 3); /* For first condition to work */ ip += bitCount>>3; bitCount &= 7; } else { bitCount -= (int)(8 * (iend - 4 - ip)); bitCount &= 31; ip = iend - 4; } bitStream = MEM_readLE32(ip) >> bitCount; } { int const max = (2*threshold-1) - remaining; int count; if ((bitStream & (threshold-1)) < (U32)max) { count = bitStream & (threshold-1); bitCount += nbBits-1; } else { count = bitStream & (2*threshold-1); if (count >= threshold) count -= max; bitCount += nbBits; } count--; /* extra accuracy */ /* When it matters (small blocks), this is a * predictable branch, because we don't use -1. */ if (count >= 0) { remaining -= count; } else { assert(count == -1); remaining += count; } normalizedCounter[charnum++] = (short)count; previous0 = !count; assert(threshold > 1); if (remaining < threshold) { /* This branch can be folded into the * threshold update condition because we * know that threshold > 1. */ if (remaining <= 1) break; nbBits = ZSTD_highbit32(remaining) + 1; threshold = 1 << (nbBits - 1); } if (charnum >= maxSV1) break; if (LIKELY(ip <= iend-7) || (ip + (bitCount>>3) <= iend-4)) { ip += bitCount>>3; bitCount &= 7; } else { bitCount -= (int)(8 * (iend - 4 - ip)); bitCount &= 31; ip = iend - 4; } bitStream = MEM_readLE32(ip) >> bitCount; } } if (remaining != 1) return ERROR(corruption_detected); /* Only possible when there are too many zeros. */ if (charnum > maxSV1) return ERROR(maxSymbolValue_tooSmall); if (bitCount > 32) return ERROR(corruption_detected); *maxSVPtr = charnum-1; ip += (bitCount+7)>>3; return ip-istart; } /* Avoids the FORCE_INLINE of the _body() function. */ static size_t FSE_readNCount_body_default( short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr, const void* headerBuffer, size_t hbSize) { return FSE_readNCount_body(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize); } #if DYNAMIC_BMI2 BMI2_TARGET_ATTRIBUTE static size_t FSE_readNCount_body_bmi2( short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr, const void* headerBuffer, size_t hbSize) { return FSE_readNCount_body(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize); } #endif size_t FSE_readNCount_bmi2( short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr, const void* headerBuffer, size_t hbSize, int bmi2) { #if DYNAMIC_BMI2 if (bmi2) { return FSE_readNCount_body_bmi2(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize); } #endif (void)bmi2; return FSE_readNCount_body_default(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize); } size_t FSE_readNCount( short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr, const void* headerBuffer, size_t hbSize) { return FSE_readNCount_bmi2(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize, /* bmi2 */ 0); } /*! HUF_readStats() : Read compact Huffman tree, saved by HUF_writeCTable(). `huffWeight` is destination buffer. `rankStats` is assumed to be a table of at least HUF_TABLELOG_MAX U32. @return : size read from `src` , or an error Code . Note : Needed by HUF_readCTable() and HUF_readDTableX?() . */ size_t HUF_readStats(BYTE* huffWeight, size_t hwSize, U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr, const void* src, size_t srcSize) { U32 wksp[HUF_READ_STATS_WORKSPACE_SIZE_U32]; return HUF_readStats_wksp(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, wksp, sizeof(wksp), /* flags */ 0); } FORCE_INLINE_TEMPLATE size_t HUF_readStats_body(BYTE* huffWeight, size_t hwSize, U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int bmi2) { U32 weightTotal; const BYTE* ip = (const BYTE*) src; size_t iSize; size_t oSize; if (!srcSize) return ERROR(srcSize_wrong); iSize = ip[0]; /* ZSTD_memset(huffWeight, 0, hwSize); *//* is not necessary, even though some analyzer complain ... */ if (iSize >= 128) { /* special header */ oSize = iSize - 127; iSize = ((oSize+1)/2); if (iSize+1 > srcSize) return ERROR(srcSize_wrong); if (oSize >= hwSize) return ERROR(corruption_detected); ip += 1; { U32 n; for (n=0; n> 4; huffWeight[n+1] = ip[n/2] & 15; } } } else { /* header compressed with FSE (normal case) */ if (iSize+1 > srcSize) return ERROR(srcSize_wrong); /* max (hwSize-1) values decoded, as last one is implied */ oSize = FSE_decompress_wksp_bmi2(huffWeight, hwSize-1, ip+1, iSize, 6, workSpace, wkspSize, bmi2); if (FSE_isError(oSize)) return oSize; } /* collect weight stats */ ZSTD_memset(rankStats, 0, (HUF_TABLELOG_MAX + 1) * sizeof(U32)); weightTotal = 0; { U32 n; for (n=0; n HUF_TABLELOG_MAX) return ERROR(corruption_detected); rankStats[huffWeight[n]]++; weightTotal += (1 << huffWeight[n]) >> 1; } } if (weightTotal == 0) return ERROR(corruption_detected); /* get last non-null symbol weight (implied, total must be 2^n) */ { U32 const tableLog = ZSTD_highbit32(weightTotal) + 1; if (tableLog > HUF_TABLELOG_MAX) return ERROR(corruption_detected); *tableLogPtr = tableLog; /* determine last weight */ { U32 const total = 1 << tableLog; U32 const rest = total - weightTotal; U32 const verif = 1 << ZSTD_highbit32(rest); U32 const lastWeight = ZSTD_highbit32(rest) + 1; if (verif != rest) return ERROR(corruption_detected); /* last value must be a clean power of 2 */ huffWeight[oSize] = (BYTE)lastWeight; rankStats[lastWeight]++; } } /* check tree construction validity */ if ((rankStats[1] < 2) || (rankStats[1] & 1)) return ERROR(corruption_detected); /* by construction : at least 2 elts of rank 1, must be even */ /* results */ *nbSymbolsPtr = (U32)(oSize+1); return iSize+1; } /* Avoids the FORCE_INLINE of the _body() function. */ static size_t HUF_readStats_body_default(BYTE* huffWeight, size_t hwSize, U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr, const void* src, size_t srcSize, void* workSpace, size_t wkspSize) { return HUF_readStats_body(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize, 0); } #if DYNAMIC_BMI2 static BMI2_TARGET_ATTRIBUTE size_t HUF_readStats_body_bmi2(BYTE* huffWeight, size_t hwSize, U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr, const void* src, size_t srcSize, void* workSpace, size_t wkspSize) { return HUF_readStats_body(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize, 1); } #endif size_t HUF_readStats_wksp(BYTE* huffWeight, size_t hwSize, U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int flags) { #if DYNAMIC_BMI2 if (flags & HUF_flags_bmi2) { return HUF_readStats_body_bmi2(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize); } #endif (void)flags; return HUF_readStats_body_default(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize); } zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/fse.h0000644000175200007730000007263614515254731023554 0ustar rlaboissrlaboiss/* ****************************************************************** * FSE : Finite State Entropy codec * Public Prototypes declaration * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ #if defined (__cplusplus) extern "C" { #endif #ifndef FSE_H #define FSE_H /*-***************************************** * Dependencies ******************************************/ #include "zstd_deps.h" /* size_t, ptrdiff_t */ /*-***************************************** * FSE_PUBLIC_API : control library symbols visibility ******************************************/ #if defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) && defined(__GNUC__) && (__GNUC__ >= 4) # define FSE_PUBLIC_API __attribute__ ((visibility ("default"))) #elif defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) /* Visual expected */ # define FSE_PUBLIC_API __declspec(dllexport) #elif defined(FSE_DLL_IMPORT) && (FSE_DLL_IMPORT==1) # define FSE_PUBLIC_API __declspec(dllimport) /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/ #else # define FSE_PUBLIC_API #endif /*------ Version ------*/ #define FSE_VERSION_MAJOR 0 #define FSE_VERSION_MINOR 9 #define FSE_VERSION_RELEASE 0 #define FSE_LIB_VERSION FSE_VERSION_MAJOR.FSE_VERSION_MINOR.FSE_VERSION_RELEASE #define FSE_QUOTE(str) #str #define FSE_EXPAND_AND_QUOTE(str) FSE_QUOTE(str) #define FSE_VERSION_STRING FSE_EXPAND_AND_QUOTE(FSE_LIB_VERSION) #define FSE_VERSION_NUMBER (FSE_VERSION_MAJOR *100*100 + FSE_VERSION_MINOR *100 + FSE_VERSION_RELEASE) FSE_PUBLIC_API unsigned FSE_versionNumber(void); /**< library version number; to be used when checking dll version */ /*-***************************************** * Tool functions ******************************************/ FSE_PUBLIC_API size_t FSE_compressBound(size_t size); /* maximum compressed size */ /* Error Management */ FSE_PUBLIC_API unsigned FSE_isError(size_t code); /* tells if a return value is an error code */ FSE_PUBLIC_API const char* FSE_getErrorName(size_t code); /* provides error code string (useful for debugging) */ /*-***************************************** * FSE detailed API ******************************************/ /*! FSE_compress() does the following: 1. count symbol occurrence from source[] into table count[] (see hist.h) 2. normalize counters so that sum(count[]) == Power_of_2 (2^tableLog) 3. save normalized counters to memory buffer using writeNCount() 4. build encoding table 'CTable' from normalized counters 5. encode the data stream using encoding table 'CTable' FSE_decompress() does the following: 1. read normalized counters with readNCount() 2. build decoding table 'DTable' from normalized counters 3. decode the data stream using decoding table 'DTable' The following API allows targeting specific sub-functions for advanced tasks. For example, it's possible to compress several blocks using the same 'CTable', or to save and provide normalized distribution using external method. */ /* *** COMPRESSION *** */ /*! FSE_optimalTableLog(): dynamically downsize 'tableLog' when conditions are met. It saves CPU time, by using smaller tables, while preserving or even improving compression ratio. @return : recommended tableLog (necessarily <= 'maxTableLog') */ FSE_PUBLIC_API unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue); /*! FSE_normalizeCount(): normalize counts so that sum(count[]) == Power_of_2 (2^tableLog) 'normalizedCounter' is a table of short, of minimum size (maxSymbolValue+1). useLowProbCount is a boolean parameter which trades off compressed size for faster header decoding. When it is set to 1, the compressed data will be slightly smaller. And when it is set to 0, FSE_readNCount() and FSE_buildDTable() will be faster. If you are compressing a small amount of data (< 2 KB) then useLowProbCount=0 is a good default, since header deserialization makes a big speed difference. Otherwise, useLowProbCount=1 is a good default, since the speed difference is small. @return : tableLog, or an errorCode, which can be tested using FSE_isError() */ FSE_PUBLIC_API size_t FSE_normalizeCount(short* normalizedCounter, unsigned tableLog, const unsigned* count, size_t srcSize, unsigned maxSymbolValue, unsigned useLowProbCount); /*! FSE_NCountWriteBound(): Provides the maximum possible size of an FSE normalized table, given 'maxSymbolValue' and 'tableLog'. Typically useful for allocation purpose. */ FSE_PUBLIC_API size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog); /*! FSE_writeNCount(): Compactly save 'normalizedCounter' into 'buffer'. @return : size of the compressed table, or an errorCode, which can be tested using FSE_isError(). */ FSE_PUBLIC_API size_t FSE_writeNCount (void* buffer, size_t bufferSize, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog); /*! Constructor and Destructor of FSE_CTable. Note that FSE_CTable size depends on 'tableLog' and 'maxSymbolValue' */ typedef unsigned FSE_CTable; /* don't allocate that. It's only meant to be more restrictive than void* */ /*! FSE_buildCTable(): Builds `ct`, which must be already allocated, using FSE_createCTable(). @return : 0, or an errorCode, which can be tested using FSE_isError() */ FSE_PUBLIC_API size_t FSE_buildCTable(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog); /*! FSE_compress_usingCTable(): Compress `src` using `ct` into `dst` which must be already allocated. @return : size of compressed data (<= `dstCapacity`), or 0 if compressed data could not fit into `dst`, or an errorCode, which can be tested using FSE_isError() */ FSE_PUBLIC_API size_t FSE_compress_usingCTable (void* dst, size_t dstCapacity, const void* src, size_t srcSize, const FSE_CTable* ct); /*! Tutorial : ---------- The first step is to count all symbols. FSE_count() does this job very fast. Result will be saved into 'count', a table of unsigned int, which must be already allocated, and have 'maxSymbolValuePtr[0]+1' cells. 'src' is a table of bytes of size 'srcSize'. All values within 'src' MUST be <= maxSymbolValuePtr[0] maxSymbolValuePtr[0] will be updated, with its real value (necessarily <= original value) FSE_count() will return the number of occurrence of the most frequent symbol. This can be used to know if there is a single symbol within 'src', and to quickly evaluate its compressibility. If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()). The next step is to normalize the frequencies. FSE_normalizeCount() will ensure that sum of frequencies is == 2 ^'tableLog'. It also guarantees a minimum of 1 to any Symbol with frequency >= 1. You can use 'tableLog'==0 to mean "use default tableLog value". If you are unsure of which tableLog value to use, you can ask FSE_optimalTableLog(), which will provide the optimal valid tableLog given sourceSize, maxSymbolValue, and a user-defined maximum (0 means "default"). The result of FSE_normalizeCount() will be saved into a table, called 'normalizedCounter', which is a table of signed short. 'normalizedCounter' must be already allocated, and have at least 'maxSymbolValue+1' cells. The return value is tableLog if everything proceeded as expected. It is 0 if there is a single symbol within distribution. If there is an error (ex: invalid tableLog value), the function will return an ErrorCode (which can be tested using FSE_isError()). 'normalizedCounter' can be saved in a compact manner to a memory area using FSE_writeNCount(). 'buffer' must be already allocated. For guaranteed success, buffer size must be at least FSE_headerBound(). The result of the function is the number of bytes written into 'buffer'. If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError(); ex : buffer size too small). 'normalizedCounter' can then be used to create the compression table 'CTable'. The space required by 'CTable' must be already allocated, using FSE_createCTable(). You can then use FSE_buildCTable() to fill 'CTable'. If there is an error, both functions will return an ErrorCode (which can be tested using FSE_isError()). 'CTable' can then be used to compress 'src', with FSE_compress_usingCTable(). Similar to FSE_count(), the convention is that 'src' is assumed to be a table of char of size 'srcSize' The function returns the size of compressed data (without header), necessarily <= `dstCapacity`. If it returns '0', compressed data could not fit into 'dst'. If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()). */ /* *** DECOMPRESSION *** */ /*! FSE_readNCount(): Read compactly saved 'normalizedCounter' from 'rBuffer'. @return : size read from 'rBuffer', or an errorCode, which can be tested using FSE_isError(). maxSymbolValuePtr[0] and tableLogPtr[0] will also be updated with their respective values */ FSE_PUBLIC_API size_t FSE_readNCount (short* normalizedCounter, unsigned* maxSymbolValuePtr, unsigned* tableLogPtr, const void* rBuffer, size_t rBuffSize); /*! FSE_readNCount_bmi2(): * Same as FSE_readNCount() but pass bmi2=1 when your CPU supports BMI2 and 0 otherwise. */ FSE_PUBLIC_API size_t FSE_readNCount_bmi2(short* normalizedCounter, unsigned* maxSymbolValuePtr, unsigned* tableLogPtr, const void* rBuffer, size_t rBuffSize, int bmi2); typedef unsigned FSE_DTable; /* don't allocate that. It's just a way to be more restrictive than void* */ /*! Tutorial : ---------- (Note : these functions only decompress FSE-compressed blocks. If block is uncompressed, use memcpy() instead If block is a single repeated byte, use memset() instead ) The first step is to obtain the normalized frequencies of symbols. This can be performed by FSE_readNCount() if it was saved using FSE_writeNCount(). 'normalizedCounter' must be already allocated, and have at least 'maxSymbolValuePtr[0]+1' cells of signed short. In practice, that means it's necessary to know 'maxSymbolValue' beforehand, or size the table to handle worst case situations (typically 256). FSE_readNCount() will provide 'tableLog' and 'maxSymbolValue'. The result of FSE_readNCount() is the number of bytes read from 'rBuffer'. Note that 'rBufferSize' must be at least 4 bytes, even if useful information is less than that. If there is an error, the function will return an error code, which can be tested using FSE_isError(). The next step is to build the decompression tables 'FSE_DTable' from 'normalizedCounter'. This is performed by the function FSE_buildDTable(). The space required by 'FSE_DTable' must be already allocated using FSE_createDTable(). If there is an error, the function will return an error code, which can be tested using FSE_isError(). `FSE_DTable` can then be used to decompress `cSrc`, with FSE_decompress_usingDTable(). `cSrcSize` must be strictly correct, otherwise decompression will fail. FSE_decompress_usingDTable() result will tell how many bytes were regenerated (<=`dstCapacity`). If there is an error, the function will return an error code, which can be tested using FSE_isError(). (ex: dst buffer too small) */ #endif /* FSE_H */ #if defined(FSE_STATIC_LINKING_ONLY) && !defined(FSE_H_FSE_STATIC_LINKING_ONLY) #define FSE_H_FSE_STATIC_LINKING_ONLY /* *** Dependency *** */ #include "bitstream.h" /* ***************************************** * Static allocation *******************************************/ /* FSE buffer bounds */ #define FSE_NCOUNTBOUND 512 #define FSE_BLOCKBOUND(size) ((size) + ((size)>>7) + 4 /* fse states */ + sizeof(size_t) /* bitContainer */) #define FSE_COMPRESSBOUND(size) (FSE_NCOUNTBOUND + FSE_BLOCKBOUND(size)) /* Macro version, useful for static allocation */ /* It is possible to statically allocate FSE CTable/DTable as a table of FSE_CTable/FSE_DTable using below macros */ #define FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) (1 + (1<<((maxTableLog)-1)) + (((maxSymbolValue)+1)*2)) #define FSE_DTABLE_SIZE_U32(maxTableLog) (1 + (1<<(maxTableLog))) /* or use the size to malloc() space directly. Pay attention to alignment restrictions though */ #define FSE_CTABLE_SIZE(maxTableLog, maxSymbolValue) (FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) * sizeof(FSE_CTable)) #define FSE_DTABLE_SIZE(maxTableLog) (FSE_DTABLE_SIZE_U32(maxTableLog) * sizeof(FSE_DTable)) /* ***************************************** * FSE advanced API ***************************************** */ unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus); /**< same as FSE_optimalTableLog(), which used `minus==2` */ size_t FSE_buildCTable_rle (FSE_CTable* ct, unsigned char symbolValue); /**< build a fake FSE_CTable, designed to compress always the same symbolValue */ /* FSE_buildCTable_wksp() : * Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`). * `wkspSize` must be >= `FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog)` of `unsigned`. * See FSE_buildCTable_wksp() for breakdown of workspace usage. */ #define FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog) (((maxSymbolValue + 2) + (1ull << (tableLog)))/2 + sizeof(U64)/sizeof(U32) /* additional 8 bytes for potential table overwrite */) #define FSE_BUILD_CTABLE_WORKSPACE_SIZE(maxSymbolValue, tableLog) (sizeof(unsigned) * FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog)) size_t FSE_buildCTable_wksp(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize); #define FSE_BUILD_DTABLE_WKSP_SIZE(maxTableLog, maxSymbolValue) (sizeof(short) * (maxSymbolValue + 1) + (1ULL << maxTableLog) + 8) #define FSE_BUILD_DTABLE_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) ((FSE_BUILD_DTABLE_WKSP_SIZE(maxTableLog, maxSymbolValue) + sizeof(unsigned) - 1) / sizeof(unsigned)) FSE_PUBLIC_API size_t FSE_buildDTable_wksp(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize); /**< Same as FSE_buildDTable(), using an externally allocated `workspace` produced with `FSE_BUILD_DTABLE_WKSP_SIZE_U32(maxSymbolValue)` */ #define FSE_DECOMPRESS_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) (FSE_DTABLE_SIZE_U32(maxTableLog) + 1 + FSE_BUILD_DTABLE_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) + (FSE_MAX_SYMBOL_VALUE + 1) / 2 + 1) #define FSE_DECOMPRESS_WKSP_SIZE(maxTableLog, maxSymbolValue) (FSE_DECOMPRESS_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) * sizeof(unsigned)) size_t FSE_decompress_wksp_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize, int bmi2); /**< same as FSE_decompress(), using an externally allocated `workSpace` produced with `FSE_DECOMPRESS_WKSP_SIZE_U32(maxLog, maxSymbolValue)`. * Set bmi2 to 1 if your CPU supports BMI2 or 0 if it doesn't */ typedef enum { FSE_repeat_none, /**< Cannot use the previous table */ FSE_repeat_check, /**< Can use the previous table but it must be checked */ FSE_repeat_valid /**< Can use the previous table and it is assumed to be valid */ } FSE_repeat; /* ***************************************** * FSE symbol compression API *******************************************/ /*! This API consists of small unitary functions, which highly benefit from being inlined. Hence their body are included in next section. */ typedef struct { ptrdiff_t value; const void* stateTable; const void* symbolTT; unsigned stateLog; } FSE_CState_t; static void FSE_initCState(FSE_CState_t* CStatePtr, const FSE_CTable* ct); static void FSE_encodeSymbol(BIT_CStream_t* bitC, FSE_CState_t* CStatePtr, unsigned symbol); static void FSE_flushCState(BIT_CStream_t* bitC, const FSE_CState_t* CStatePtr); /**< These functions are inner components of FSE_compress_usingCTable(). They allow the creation of custom streams, mixing multiple tables and bit sources. A key property to keep in mind is that encoding and decoding are done **in reverse direction**. So the first symbol you will encode is the last you will decode, like a LIFO stack. You will need a few variables to track your CStream. They are : FSE_CTable ct; // Provided by FSE_buildCTable() BIT_CStream_t bitStream; // bitStream tracking structure FSE_CState_t state; // State tracking structure (can have several) The first thing to do is to init bitStream and state. size_t errorCode = BIT_initCStream(&bitStream, dstBuffer, maxDstSize); FSE_initCState(&state, ct); Note that BIT_initCStream() can produce an error code, so its result should be tested, using FSE_isError(); You can then encode your input data, byte after byte. FSE_encodeSymbol() outputs a maximum of 'tableLog' bits at a time. Remember decoding will be done in reverse direction. FSE_encodeByte(&bitStream, &state, symbol); At any time, you can also add any bit sequence. Note : maximum allowed nbBits is 25, for compatibility with 32-bits decoders BIT_addBits(&bitStream, bitField, nbBits); The above methods don't commit data to memory, they just store it into local register, for speed. Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (size_t). Writing data to memory is a manual operation, performed by the flushBits function. BIT_flushBits(&bitStream); Your last FSE encoding operation shall be to flush your last state value(s). FSE_flushState(&bitStream, &state); Finally, you must close the bitStream. The function returns the size of CStream in bytes. If data couldn't fit into dstBuffer, it will return a 0 ( == not compressible) If there is an error, it returns an errorCode (which can be tested using FSE_isError()). size_t size = BIT_closeCStream(&bitStream); */ /* ***************************************** * FSE symbol decompression API *******************************************/ typedef struct { size_t state; const void* table; /* precise table may vary, depending on U16 */ } FSE_DState_t; static void FSE_initDState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD, const FSE_DTable* dt); static unsigned char FSE_decodeSymbol(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD); static unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr); /**< Let's now decompose FSE_decompress_usingDTable() into its unitary components. You will decode FSE-encoded symbols from the bitStream, and also any other bitFields you put in, **in reverse order**. You will need a few variables to track your bitStream. They are : BIT_DStream_t DStream; // Stream context FSE_DState_t DState; // State context. Multiple ones are possible FSE_DTable* DTablePtr; // Decoding table, provided by FSE_buildDTable() The first thing to do is to init the bitStream. errorCode = BIT_initDStream(&DStream, srcBuffer, srcSize); You should then retrieve your initial state(s) (in reverse flushing order if you have several ones) : errorCode = FSE_initDState(&DState, &DStream, DTablePtr); You can then decode your data, symbol after symbol. For information the maximum number of bits read by FSE_decodeSymbol() is 'tableLog'. Keep in mind that symbols are decoded in reverse order, like a LIFO stack (last in, first out). unsigned char symbol = FSE_decodeSymbol(&DState, &DStream); You can retrieve any bitfield you eventually stored into the bitStream (in reverse order) Note : maximum allowed nbBits is 25, for 32-bits compatibility size_t bitField = BIT_readBits(&DStream, nbBits); All above operations only read from local register (which size depends on size_t). Refueling the register from memory is manually performed by the reload method. endSignal = FSE_reloadDStream(&DStream); BIT_reloadDStream() result tells if there is still some more data to read from DStream. BIT_DStream_unfinished : there is still some data left into the DStream. BIT_DStream_endOfBuffer : Dstream reached end of buffer. Its container may no longer be completely filled. BIT_DStream_completed : Dstream reached its exact end, corresponding in general to decompression completed. BIT_DStream_tooFar : Dstream went too far. Decompression result is corrupted. When reaching end of buffer (BIT_DStream_endOfBuffer), progress slowly, notably if you decode multiple symbols per loop, to properly detect the exact end of stream. After each decoded symbol, check if DStream is fully consumed using this simple test : BIT_reloadDStream(&DStream) >= BIT_DStream_completed When it's done, verify decompression is fully completed, by checking both DStream and the relevant states. Checking if DStream has reached its end is performed by : BIT_endOfDStream(&DStream); Check also the states. There might be some symbols left there, if some high probability ones (>50%) are possible. FSE_endOfDState(&DState); */ /* ***************************************** * FSE unsafe API *******************************************/ static unsigned char FSE_decodeSymbolFast(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD); /* faster, but works only if nbBits is always >= 1 (otherwise, result will be corrupted) */ /* ***************************************** * Implementation of inlined functions *******************************************/ typedef struct { int deltaFindState; U32 deltaNbBits; } FSE_symbolCompressionTransform; /* total 8 bytes */ MEM_STATIC void FSE_initCState(FSE_CState_t* statePtr, const FSE_CTable* ct) { const void* ptr = ct; const U16* u16ptr = (const U16*) ptr; const U32 tableLog = MEM_read16(ptr); statePtr->value = (ptrdiff_t)1<stateTable = u16ptr+2; statePtr->symbolTT = ct + 1 + (tableLog ? (1<<(tableLog-1)) : 1); statePtr->stateLog = tableLog; } /*! FSE_initCState2() : * Same as FSE_initCState(), but the first symbol to include (which will be the last to be read) * uses the smallest state value possible, saving the cost of this symbol */ MEM_STATIC void FSE_initCState2(FSE_CState_t* statePtr, const FSE_CTable* ct, U32 symbol) { FSE_initCState(statePtr, ct); { const FSE_symbolCompressionTransform symbolTT = ((const FSE_symbolCompressionTransform*)(statePtr->symbolTT))[symbol]; const U16* stateTable = (const U16*)(statePtr->stateTable); U32 nbBitsOut = (U32)((symbolTT.deltaNbBits + (1<<15)) >> 16); statePtr->value = (nbBitsOut << 16) - symbolTT.deltaNbBits; statePtr->value = stateTable[(statePtr->value >> nbBitsOut) + symbolTT.deltaFindState]; } } MEM_STATIC void FSE_encodeSymbol(BIT_CStream_t* bitC, FSE_CState_t* statePtr, unsigned symbol) { FSE_symbolCompressionTransform const symbolTT = ((const FSE_symbolCompressionTransform*)(statePtr->symbolTT))[symbol]; const U16* const stateTable = (const U16*)(statePtr->stateTable); U32 const nbBitsOut = (U32)((statePtr->value + symbolTT.deltaNbBits) >> 16); BIT_addBits(bitC, statePtr->value, nbBitsOut); statePtr->value = stateTable[ (statePtr->value >> nbBitsOut) + symbolTT.deltaFindState]; } MEM_STATIC void FSE_flushCState(BIT_CStream_t* bitC, const FSE_CState_t* statePtr) { BIT_addBits(bitC, statePtr->value, statePtr->stateLog); BIT_flushBits(bitC); } /* FSE_getMaxNbBits() : * Approximate maximum cost of a symbol, in bits. * Fractional get rounded up (i.e. a symbol with a normalized frequency of 3 gives the same result as a frequency of 2) * note 1 : assume symbolValue is valid (<= maxSymbolValue) * note 2 : if freq[symbolValue]==0, @return a fake cost of tableLog+1 bits */ MEM_STATIC U32 FSE_getMaxNbBits(const void* symbolTTPtr, U32 symbolValue) { const FSE_symbolCompressionTransform* symbolTT = (const FSE_symbolCompressionTransform*) symbolTTPtr; return (symbolTT[symbolValue].deltaNbBits + ((1<<16)-1)) >> 16; } /* FSE_bitCost() : * Approximate symbol cost, as fractional value, using fixed-point format (accuracyLog fractional bits) * note 1 : assume symbolValue is valid (<= maxSymbolValue) * note 2 : if freq[symbolValue]==0, @return a fake cost of tableLog+1 bits */ MEM_STATIC U32 FSE_bitCost(const void* symbolTTPtr, U32 tableLog, U32 symbolValue, U32 accuracyLog) { const FSE_symbolCompressionTransform* symbolTT = (const FSE_symbolCompressionTransform*) symbolTTPtr; U32 const minNbBits = symbolTT[symbolValue].deltaNbBits >> 16; U32 const threshold = (minNbBits+1) << 16; assert(tableLog < 16); assert(accuracyLog < 31-tableLog); /* ensure enough room for renormalization double shift */ { U32 const tableSize = 1 << tableLog; U32 const deltaFromThreshold = threshold - (symbolTT[symbolValue].deltaNbBits + tableSize); U32 const normalizedDeltaFromThreshold = (deltaFromThreshold << accuracyLog) >> tableLog; /* linear interpolation (very approximate) */ U32 const bitMultiplier = 1 << accuracyLog; assert(symbolTT[symbolValue].deltaNbBits + tableSize <= threshold); assert(normalizedDeltaFromThreshold <= bitMultiplier); return (minNbBits+1)*bitMultiplier - normalizedDeltaFromThreshold; } } /* ====== Decompression ====== */ typedef struct { U16 tableLog; U16 fastMode; } FSE_DTableHeader; /* sizeof U32 */ typedef struct { unsigned short newState; unsigned char symbol; unsigned char nbBits; } FSE_decode_t; /* size == U32 */ MEM_STATIC void FSE_initDState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD, const FSE_DTable* dt) { const void* ptr = dt; const FSE_DTableHeader* const DTableH = (const FSE_DTableHeader*)ptr; DStatePtr->state = BIT_readBits(bitD, DTableH->tableLog); BIT_reloadDStream(bitD); DStatePtr->table = dt + 1; } MEM_STATIC BYTE FSE_peekSymbol(const FSE_DState_t* DStatePtr) { FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state]; return DInfo.symbol; } MEM_STATIC void FSE_updateState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD) { FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state]; U32 const nbBits = DInfo.nbBits; size_t const lowBits = BIT_readBits(bitD, nbBits); DStatePtr->state = DInfo.newState + lowBits; } MEM_STATIC BYTE FSE_decodeSymbol(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD) { FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state]; U32 const nbBits = DInfo.nbBits; BYTE const symbol = DInfo.symbol; size_t const lowBits = BIT_readBits(bitD, nbBits); DStatePtr->state = DInfo.newState + lowBits; return symbol; } /*! FSE_decodeSymbolFast() : unsafe, only works if no symbol has a probability > 50% */ MEM_STATIC BYTE FSE_decodeSymbolFast(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD) { FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state]; U32 const nbBits = DInfo.nbBits; BYTE const symbol = DInfo.symbol; size_t const lowBits = BIT_readBitsFast(bitD, nbBits); DStatePtr->state = DInfo.newState + lowBits; return symbol; } MEM_STATIC unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr) { return DStatePtr->state == 0; } #ifndef FSE_COMMONDEFS_ONLY /* ************************************************************** * Tuning parameters ****************************************************************/ /*!MEMORY_USAGE : * Memory usage formula : N->2^N Bytes (examples : 10 -> 1KB; 12 -> 4KB ; 16 -> 64KB; 20 -> 1MB; etc.) * Increasing memory usage improves compression ratio * Reduced memory usage can improve speed, due to cache effect * Recommended max value is 14, for 16KB, which nicely fits into Intel x86 L1 cache */ #ifndef FSE_MAX_MEMORY_USAGE # define FSE_MAX_MEMORY_USAGE 14 #endif #ifndef FSE_DEFAULT_MEMORY_USAGE # define FSE_DEFAULT_MEMORY_USAGE 13 #endif #if (FSE_DEFAULT_MEMORY_USAGE > FSE_MAX_MEMORY_USAGE) # error "FSE_DEFAULT_MEMORY_USAGE must be <= FSE_MAX_MEMORY_USAGE" #endif /*!FSE_MAX_SYMBOL_VALUE : * Maximum symbol value authorized. * Required for proper stack allocation */ #ifndef FSE_MAX_SYMBOL_VALUE # define FSE_MAX_SYMBOL_VALUE 255 #endif /* ************************************************************** * template functions type & suffix ****************************************************************/ #define FSE_FUNCTION_TYPE BYTE #define FSE_FUNCTION_EXTENSION #define FSE_DECODE_TYPE FSE_decode_t #endif /* !FSE_COMMONDEFS_ONLY */ /* *************************************************************** * Constants *****************************************************************/ #define FSE_MAX_TABLELOG (FSE_MAX_MEMORY_USAGE-2) #define FSE_MAX_TABLESIZE (1U< FSE_TABLELOG_ABSOLUTE_MAX # error "FSE_MAX_TABLELOG > FSE_TABLELOG_ABSOLUTE_MAX is not supported" #endif #define FSE_TABLESTEP(tableSize) (((tableSize)>>1) + ((tableSize)>>3) + 3) #endif /* FSE_STATIC_LINKING_ONLY */ #if defined (__cplusplus) } #endif zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/cpu.h0000644000175200007730000001056214515254731023554 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_COMMON_CPU_H #define ZSTD_COMMON_CPU_H /** * Implementation taken from folly/CpuId.h * https://github.com/facebook/folly/blob/master/folly/CpuId.h */ #include "mem.h" #ifdef _MSC_VER #include #endif typedef struct { U32 f1c; U32 f1d; U32 f7b; U32 f7c; } ZSTD_cpuid_t; MEM_STATIC ZSTD_cpuid_t ZSTD_cpuid(void) { U32 f1c = 0; U32 f1d = 0; U32 f7b = 0; U32 f7c = 0; #if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_IX86)) int reg[4]; __cpuid((int*)reg, 0); { int const n = reg[0]; if (n >= 1) { __cpuid((int*)reg, 1); f1c = (U32)reg[2]; f1d = (U32)reg[3]; } if (n >= 7) { __cpuidex((int*)reg, 7, 0); f7b = (U32)reg[1]; f7c = (U32)reg[2]; } } #elif defined(__i386__) && defined(__PIC__) && !defined(__clang__) && defined(__GNUC__) /* The following block like the normal cpuid branch below, but gcc * reserves ebx for use of its pic register so we must specially * handle the save and restore to avoid clobbering the register */ U32 n; __asm__( "pushl %%ebx\n\t" "cpuid\n\t" "popl %%ebx\n\t" : "=a"(n) : "a"(0) : "ecx", "edx"); if (n >= 1) { U32 f1a; __asm__( "pushl %%ebx\n\t" "cpuid\n\t" "popl %%ebx\n\t" : "=a"(f1a), "=c"(f1c), "=d"(f1d) : "a"(1)); } if (n >= 7) { __asm__( "pushl %%ebx\n\t" "cpuid\n\t" "movl %%ebx, %%eax\n\t" "popl %%ebx" : "=a"(f7b), "=c"(f7c) : "a"(7), "c"(0) : "edx"); } #elif defined(__x86_64__) || defined(_M_X64) || defined(__i386__) U32 n; __asm__("cpuid" : "=a"(n) : "a"(0) : "ebx", "ecx", "edx"); if (n >= 1) { U32 f1a; __asm__("cpuid" : "=a"(f1a), "=c"(f1c), "=d"(f1d) : "a"(1) : "ebx"); } if (n >= 7) { U32 f7a; __asm__("cpuid" : "=a"(f7a), "=b"(f7b), "=c"(f7c) : "a"(7), "c"(0) : "edx"); } #endif { ZSTD_cpuid_t cpuid; cpuid.f1c = f1c; cpuid.f1d = f1d; cpuid.f7b = f7b; cpuid.f7c = f7c; return cpuid; } } #define X(name, r, bit) \ MEM_STATIC int ZSTD_cpuid_##name(ZSTD_cpuid_t const cpuid) { \ return ((cpuid.r) & (1U << bit)) != 0; \ } /* cpuid(1): Processor Info and Feature Bits. */ #define C(name, bit) X(name, f1c, bit) C(sse3, 0) C(pclmuldq, 1) C(dtes64, 2) C(monitor, 3) C(dscpl, 4) C(vmx, 5) C(smx, 6) C(eist, 7) C(tm2, 8) C(ssse3, 9) C(cnxtid, 10) C(fma, 12) C(cx16, 13) C(xtpr, 14) C(pdcm, 15) C(pcid, 17) C(dca, 18) C(sse41, 19) C(sse42, 20) C(x2apic, 21) C(movbe, 22) C(popcnt, 23) C(tscdeadline, 24) C(aes, 25) C(xsave, 26) C(osxsave, 27) C(avx, 28) C(f16c, 29) C(rdrand, 30) #undef C #define D(name, bit) X(name, f1d, bit) D(fpu, 0) D(vme, 1) D(de, 2) D(pse, 3) D(tsc, 4) D(msr, 5) D(pae, 6) D(mce, 7) D(cx8, 8) D(apic, 9) D(sep, 11) D(mtrr, 12) D(pge, 13) D(mca, 14) D(cmov, 15) D(pat, 16) D(pse36, 17) D(psn, 18) D(clfsh, 19) D(ds, 21) D(acpi, 22) D(mmx, 23) D(fxsr, 24) D(sse, 25) D(sse2, 26) D(ss, 27) D(htt, 28) D(tm, 29) D(pbe, 31) #undef D /* cpuid(7): Extended Features. */ #define B(name, bit) X(name, f7b, bit) B(bmi1, 3) B(hle, 4) B(avx2, 5) B(smep, 7) B(bmi2, 8) B(erms, 9) B(invpcid, 10) B(rtm, 11) B(mpx, 14) B(avx512f, 16) B(avx512dq, 17) B(rdseed, 18) B(adx, 19) B(smap, 20) B(avx512ifma, 21) B(pcommit, 22) B(clflushopt, 23) B(clwb, 24) B(avx512pf, 26) B(avx512er, 27) B(avx512cd, 28) B(sha, 29) B(avx512bw, 30) B(avx512vl, 31) #undef B #define C(name, bit) X(name, f7c, bit) C(prefetchwt1, 0) C(avx512vbmi, 1) #undef C #undef X #endif /* ZSTD_COMMON_CPU_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/common/zstd_deps.h0000644000175200007730000000471414515254731024766 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* This file provides common libc dependencies that zstd requires. * The purpose is to allow replacing this file with a custom implementation * to compile zstd without libc support. */ /* Need: * NULL * INT_MAX * UINT_MAX * ZSTD_memcpy() * ZSTD_memset() * ZSTD_memmove() */ #ifndef ZSTD_DEPS_COMMON #define ZSTD_DEPS_COMMON #include #include #include #if defined(__GNUC__) && __GNUC__ >= 4 # define ZSTD_memcpy(d,s,l) __builtin_memcpy((d),(s),(l)) # define ZSTD_memmove(d,s,l) __builtin_memmove((d),(s),(l)) # define ZSTD_memset(p,v,l) __builtin_memset((p),(v),(l)) #else # define ZSTD_memcpy(d,s,l) memcpy((d),(s),(l)) # define ZSTD_memmove(d,s,l) memmove((d),(s),(l)) # define ZSTD_memset(p,v,l) memset((p),(v),(l)) #endif #endif /* ZSTD_DEPS_COMMON */ /* Need: * ZSTD_malloc() * ZSTD_free() * ZSTD_calloc() */ #ifdef ZSTD_DEPS_NEED_MALLOC #ifndef ZSTD_DEPS_MALLOC #define ZSTD_DEPS_MALLOC #include #define ZSTD_malloc(s) malloc(s) #define ZSTD_calloc(n,s) calloc((n), (s)) #define ZSTD_free(p) free((p)) #endif /* ZSTD_DEPS_MALLOC */ #endif /* ZSTD_DEPS_NEED_MALLOC */ /* * Provides 64-bit math support. * Need: * U64 ZSTD_div64(U64 dividend, U32 divisor) */ #ifdef ZSTD_DEPS_NEED_MATH64 #ifndef ZSTD_DEPS_MATH64 #define ZSTD_DEPS_MATH64 #define ZSTD_div64(dividend, divisor) ((dividend) / (divisor)) #endif /* ZSTD_DEPS_MATH64 */ #endif /* ZSTD_DEPS_NEED_MATH64 */ /* Need: * assert() */ #ifdef ZSTD_DEPS_NEED_ASSERT #ifndef ZSTD_DEPS_ASSERT #define ZSTD_DEPS_ASSERT #include #endif /* ZSTD_DEPS_ASSERT */ #endif /* ZSTD_DEPS_NEED_ASSERT */ /* Need: * ZSTD_DEBUG_PRINT() */ #ifdef ZSTD_DEPS_NEED_IO #ifndef ZSTD_DEPS_IO #define ZSTD_DEPS_IO #include #define ZSTD_DEBUG_PRINT(...) fprintf(stderr, __VA_ARGS__) #endif /* ZSTD_DEPS_IO */ #endif /* ZSTD_DEPS_NEED_IO */ /* Only requested when is known to be present. * Need: * intptr_t */ #ifdef ZSTD_DEPS_NEED_STDINT #ifndef ZSTD_DEPS_STDINT #define ZSTD_DEPS_STDINT #include #endif /* ZSTD_DEPS_STDINT */ #endif /* ZSTD_DEPS_NEED_STDINT */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/decompress/0000755000175200007730000000000014515254731023464 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/internal-complibs/zstd/decompress/zstd_decompress_block.c0000644000175200007730000030247114515254731030221 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* zstd_decompress_block : * this module takes care of decompressing _compressed_ block */ /*-******************************************************* * Dependencies *********************************************************/ #include "../common/zstd_deps.h" /* ZSTD_memcpy, ZSTD_memmove, ZSTD_memset */ #include "../common/compiler.h" /* prefetch */ #include "../common/cpu.h" /* bmi2 */ #include "../common/mem.h" /* low level memory routines */ #define FSE_STATIC_LINKING_ONLY #include "../common/fse.h" #include "../common/huf.h" #include "../common/zstd_internal.h" #include "zstd_decompress_internal.h" /* ZSTD_DCtx */ #include "zstd_ddict.h" /* ZSTD_DDictDictContent */ #include "zstd_decompress_block.h" #include "../common/bits.h" /* ZSTD_highbit32 */ /*_******************************************************* * Macros **********************************************************/ /* These two optional macros force the use one way or another of the two * ZSTD_decompressSequences implementations. You can't force in both directions * at the same time. */ #if defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \ defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG) #error "Cannot force the use of the short and the long ZSTD_decompressSequences variants!" #endif /*_******************************************************* * Memory operations **********************************************************/ static void ZSTD_copy4(void* dst, const void* src) { ZSTD_memcpy(dst, src, 4); } /*-************************************************************* * Block decoding ***************************************************************/ /*! ZSTD_getcBlockSize() : * Provides the size of compressed block from block header `src` */ size_t ZSTD_getcBlockSize(const void* src, size_t srcSize, blockProperties_t* bpPtr) { RETURN_ERROR_IF(srcSize < ZSTD_blockHeaderSize, srcSize_wrong, ""); { U32 const cBlockHeader = MEM_readLE24(src); U32 const cSize = cBlockHeader >> 3; bpPtr->lastBlock = cBlockHeader & 1; bpPtr->blockType = (blockType_e)((cBlockHeader >> 1) & 3); bpPtr->origSize = cSize; /* only useful for RLE */ if (bpPtr->blockType == bt_rle) return 1; RETURN_ERROR_IF(bpPtr->blockType == bt_reserved, corruption_detected, ""); return cSize; } } /* Allocate buffer for literals, either overlapping current dst, or split between dst and litExtraBuffer, or stored entirely within litExtraBuffer */ static void ZSTD_allocateLiteralsBuffer(ZSTD_DCtx* dctx, void* const dst, const size_t dstCapacity, const size_t litSize, const streaming_operation streaming, const size_t expectedWriteSize, const unsigned splitImmediately) { if (streaming == not_streaming && dstCapacity > ZSTD_BLOCKSIZE_MAX + WILDCOPY_OVERLENGTH + litSize + WILDCOPY_OVERLENGTH) { /* room for litbuffer to fit without read faulting */ dctx->litBuffer = (BYTE*)dst + ZSTD_BLOCKSIZE_MAX + WILDCOPY_OVERLENGTH; dctx->litBufferEnd = dctx->litBuffer + litSize; dctx->litBufferLocation = ZSTD_in_dst; } else if (litSize > ZSTD_LITBUFFEREXTRASIZE) { /* won't fit in litExtraBuffer, so it will be split between end of dst and extra buffer */ if (splitImmediately) { /* won't fit in litExtraBuffer, so it will be split between end of dst and extra buffer */ dctx->litBuffer = (BYTE*)dst + expectedWriteSize - litSize + ZSTD_LITBUFFEREXTRASIZE - WILDCOPY_OVERLENGTH; dctx->litBufferEnd = dctx->litBuffer + litSize - ZSTD_LITBUFFEREXTRASIZE; } else { /* initially this will be stored entirely in dst during huffman decoding, it will partially be shifted to litExtraBuffer after */ dctx->litBuffer = (BYTE*)dst + expectedWriteSize - litSize; dctx->litBufferEnd = (BYTE*)dst + expectedWriteSize; } dctx->litBufferLocation = ZSTD_split; } else { /* fits entirely within litExtraBuffer, so no split is necessary */ dctx->litBuffer = dctx->litExtraBuffer; dctx->litBufferEnd = dctx->litBuffer + litSize; dctx->litBufferLocation = ZSTD_not_in_dst; } } /* Hidden declaration for fullbench */ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx, const void* src, size_t srcSize, void* dst, size_t dstCapacity, const streaming_operation streaming); /*! ZSTD_decodeLiteralsBlock() : * Where it is possible to do so without being stomped by the output during decompression, the literals block will be stored * in the dstBuffer. If there is room to do so, it will be stored in full in the excess dst space after where the current * block will be output. Otherwise it will be stored at the end of the current dst blockspace, with a small portion being * stored in dctx->litExtraBuffer to help keep it "ahead" of the current output write. * * @return : nb of bytes read from src (< srcSize ) * note : symbol not declared but exposed for fullbench */ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx, const void* src, size_t srcSize, /* note : srcSize < BLOCKSIZE */ void* dst, size_t dstCapacity, const streaming_operation streaming) { DEBUGLOG(5, "ZSTD_decodeLiteralsBlock"); RETURN_ERROR_IF(srcSize < MIN_CBLOCK_SIZE, corruption_detected, ""); { const BYTE* const istart = (const BYTE*) src; symbolEncodingType_e const litEncType = (symbolEncodingType_e)(istart[0] & 3); switch(litEncType) { case set_repeat: DEBUGLOG(5, "set_repeat flag : re-using stats from previous compressed literals block"); RETURN_ERROR_IF(dctx->litEntropy==0, dictionary_corrupted, ""); ZSTD_FALLTHROUGH; case set_compressed: RETURN_ERROR_IF(srcSize < 5, corruption_detected, "srcSize >= MIN_CBLOCK_SIZE == 2; here we need up to 5 for case 3"); { size_t lhSize, litSize, litCSize; U32 singleStream=0; U32 const lhlCode = (istart[0] >> 2) & 3; U32 const lhc = MEM_readLE32(istart); size_t hufSuccess; size_t expectedWriteSize = MIN(ZSTD_BLOCKSIZE_MAX, dstCapacity); int const flags = 0 | (ZSTD_DCtx_get_bmi2(dctx) ? HUF_flags_bmi2 : 0) | (dctx->disableHufAsm ? HUF_flags_disableAsm : 0); switch(lhlCode) { case 0: case 1: default: /* note : default is impossible, since lhlCode into [0..3] */ /* 2 - 2 - 10 - 10 */ singleStream = !lhlCode; lhSize = 3; litSize = (lhc >> 4) & 0x3FF; litCSize = (lhc >> 14) & 0x3FF; break; case 2: /* 2 - 2 - 14 - 14 */ lhSize = 4; litSize = (lhc >> 4) & 0x3FFF; litCSize = lhc >> 18; break; case 3: /* 2 - 2 - 18 - 18 */ lhSize = 5; litSize = (lhc >> 4) & 0x3FFFF; litCSize = (lhc >> 22) + ((size_t)istart[4] << 10); break; } RETURN_ERROR_IF(litSize > 0 && dst == NULL, dstSize_tooSmall, "NULL not handled"); RETURN_ERROR_IF(litSize > ZSTD_BLOCKSIZE_MAX, corruption_detected, ""); if (!singleStream) RETURN_ERROR_IF(litSize < MIN_LITERALS_FOR_4_STREAMS, literals_headerWrong, "Not enough literals (%zu) for the 4-streams mode (min %u)", litSize, MIN_LITERALS_FOR_4_STREAMS); RETURN_ERROR_IF(litCSize + lhSize > srcSize, corruption_detected, ""); RETURN_ERROR_IF(expectedWriteSize < litSize , dstSize_tooSmall, ""); ZSTD_allocateLiteralsBuffer(dctx, dst, dstCapacity, litSize, streaming, expectedWriteSize, 0); /* prefetch huffman table if cold */ if (dctx->ddictIsCold && (litSize > 768 /* heuristic */)) { PREFETCH_AREA(dctx->HUFptr, sizeof(dctx->entropy.hufTable)); } if (litEncType==set_repeat) { if (singleStream) { hufSuccess = HUF_decompress1X_usingDTable( dctx->litBuffer, litSize, istart+lhSize, litCSize, dctx->HUFptr, flags); } else { assert(litSize >= MIN_LITERALS_FOR_4_STREAMS); hufSuccess = HUF_decompress4X_usingDTable( dctx->litBuffer, litSize, istart+lhSize, litCSize, dctx->HUFptr, flags); } } else { if (singleStream) { #if defined(HUF_FORCE_DECOMPRESS_X2) hufSuccess = HUF_decompress1X_DCtx_wksp( dctx->entropy.hufTable, dctx->litBuffer, litSize, istart+lhSize, litCSize, dctx->workspace, sizeof(dctx->workspace), flags); #else hufSuccess = HUF_decompress1X1_DCtx_wksp( dctx->entropy.hufTable, dctx->litBuffer, litSize, istart+lhSize, litCSize, dctx->workspace, sizeof(dctx->workspace), flags); #endif } else { hufSuccess = HUF_decompress4X_hufOnly_wksp( dctx->entropy.hufTable, dctx->litBuffer, litSize, istart+lhSize, litCSize, dctx->workspace, sizeof(dctx->workspace), flags); } } if (dctx->litBufferLocation == ZSTD_split) { ZSTD_memcpy(dctx->litExtraBuffer, dctx->litBufferEnd - ZSTD_LITBUFFEREXTRASIZE, ZSTD_LITBUFFEREXTRASIZE); ZSTD_memmove(dctx->litBuffer + ZSTD_LITBUFFEREXTRASIZE - WILDCOPY_OVERLENGTH, dctx->litBuffer, litSize - ZSTD_LITBUFFEREXTRASIZE); dctx->litBuffer += ZSTD_LITBUFFEREXTRASIZE - WILDCOPY_OVERLENGTH; dctx->litBufferEnd -= WILDCOPY_OVERLENGTH; } RETURN_ERROR_IF(HUF_isError(hufSuccess), corruption_detected, ""); dctx->litPtr = dctx->litBuffer; dctx->litSize = litSize; dctx->litEntropy = 1; if (litEncType==set_compressed) dctx->HUFptr = dctx->entropy.hufTable; return litCSize + lhSize; } case set_basic: { size_t litSize, lhSize; U32 const lhlCode = ((istart[0]) >> 2) & 3; size_t expectedWriteSize = MIN(ZSTD_BLOCKSIZE_MAX, dstCapacity); switch(lhlCode) { case 0: case 2: default: /* note : default is impossible, since lhlCode into [0..3] */ lhSize = 1; litSize = istart[0] >> 3; break; case 1: lhSize = 2; litSize = MEM_readLE16(istart) >> 4; break; case 3: lhSize = 3; RETURN_ERROR_IF(srcSize<3, corruption_detected, "srcSize >= MIN_CBLOCK_SIZE == 2; here we need lhSize = 3"); litSize = MEM_readLE24(istart) >> 4; break; } RETURN_ERROR_IF(litSize > 0 && dst == NULL, dstSize_tooSmall, "NULL not handled"); RETURN_ERROR_IF(expectedWriteSize < litSize, dstSize_tooSmall, ""); ZSTD_allocateLiteralsBuffer(dctx, dst, dstCapacity, litSize, streaming, expectedWriteSize, 1); if (lhSize+litSize+WILDCOPY_OVERLENGTH > srcSize) { /* risk reading beyond src buffer with wildcopy */ RETURN_ERROR_IF(litSize+lhSize > srcSize, corruption_detected, ""); if (dctx->litBufferLocation == ZSTD_split) { ZSTD_memcpy(dctx->litBuffer, istart + lhSize, litSize - ZSTD_LITBUFFEREXTRASIZE); ZSTD_memcpy(dctx->litExtraBuffer, istart + lhSize + litSize - ZSTD_LITBUFFEREXTRASIZE, ZSTD_LITBUFFEREXTRASIZE); } else { ZSTD_memcpy(dctx->litBuffer, istart + lhSize, litSize); } dctx->litPtr = dctx->litBuffer; dctx->litSize = litSize; return lhSize+litSize; } /* direct reference into compressed stream */ dctx->litPtr = istart+lhSize; dctx->litSize = litSize; dctx->litBufferEnd = dctx->litPtr + litSize; dctx->litBufferLocation = ZSTD_not_in_dst; return lhSize+litSize; } case set_rle: { U32 const lhlCode = ((istart[0]) >> 2) & 3; size_t litSize, lhSize; size_t expectedWriteSize = MIN(ZSTD_BLOCKSIZE_MAX, dstCapacity); switch(lhlCode) { case 0: case 2: default: /* note : default is impossible, since lhlCode into [0..3] */ lhSize = 1; litSize = istart[0] >> 3; break; case 1: lhSize = 2; RETURN_ERROR_IF(srcSize<3, corruption_detected, "srcSize >= MIN_CBLOCK_SIZE == 2; here we need lhSize+1 = 3"); litSize = MEM_readLE16(istart) >> 4; break; case 3: lhSize = 3; RETURN_ERROR_IF(srcSize<4, corruption_detected, "srcSize >= MIN_CBLOCK_SIZE == 2; here we need lhSize+1 = 4"); litSize = MEM_readLE24(istart) >> 4; break; } RETURN_ERROR_IF(litSize > 0 && dst == NULL, dstSize_tooSmall, "NULL not handled"); RETURN_ERROR_IF(litSize > ZSTD_BLOCKSIZE_MAX, corruption_detected, ""); RETURN_ERROR_IF(expectedWriteSize < litSize, dstSize_tooSmall, ""); ZSTD_allocateLiteralsBuffer(dctx, dst, dstCapacity, litSize, streaming, expectedWriteSize, 1); if (dctx->litBufferLocation == ZSTD_split) { ZSTD_memset(dctx->litBuffer, istart[lhSize], litSize - ZSTD_LITBUFFEREXTRASIZE); ZSTD_memset(dctx->litExtraBuffer, istart[lhSize], ZSTD_LITBUFFEREXTRASIZE); } else { ZSTD_memset(dctx->litBuffer, istart[lhSize], litSize); } dctx->litPtr = dctx->litBuffer; dctx->litSize = litSize; return lhSize+1; } default: RETURN_ERROR(corruption_detected, "impossible"); } } } /* Default FSE distribution tables. * These are pre-calculated FSE decoding tables using default distributions as defined in specification : * https://github.com/facebook/zstd/blob/release/doc/zstd_compression_format.md#default-distributions * They were generated programmatically with following method : * - start from default distributions, present in /lib/common/zstd_internal.h * - generate tables normally, using ZSTD_buildFSETable() * - printout the content of tables * - pretify output, report below, test with fuzzer to ensure it's correct */ /* Default FSE distribution table for Literal Lengths */ static const ZSTD_seqSymbol LL_defaultDTable[(1<tableLog = 0; DTableH->fastMode = 0; cell->nbBits = 0; cell->nextState = 0; assert(nbAddBits < 255); cell->nbAdditionalBits = nbAddBits; cell->baseValue = baseValue; } /* ZSTD_buildFSETable() : * generate FSE decoding table for one symbol (ll, ml or off) * cannot fail if input is valid => * all inputs are presumed validated at this stage */ FORCE_INLINE_TEMPLATE void ZSTD_buildFSETable_body(ZSTD_seqSymbol* dt, const short* normalizedCounter, unsigned maxSymbolValue, const U32* baseValue, const U8* nbAdditionalBits, unsigned tableLog, void* wksp, size_t wkspSize) { ZSTD_seqSymbol* const tableDecode = dt+1; U32 const maxSV1 = maxSymbolValue + 1; U32 const tableSize = 1 << tableLog; U16* symbolNext = (U16*)wksp; BYTE* spread = (BYTE*)(symbolNext + MaxSeq + 1); U32 highThreshold = tableSize - 1; /* Sanity Checks */ assert(maxSymbolValue <= MaxSeq); assert(tableLog <= MaxFSELog); assert(wkspSize >= ZSTD_BUILD_FSE_TABLE_WKSP_SIZE); (void)wkspSize; /* Init, lay down lowprob symbols */ { ZSTD_seqSymbol_header DTableH; DTableH.tableLog = tableLog; DTableH.fastMode = 1; { S16 const largeLimit= (S16)(1 << (tableLog-1)); U32 s; for (s=0; s= largeLimit) DTableH.fastMode=0; assert(normalizedCounter[s]>=0); symbolNext[s] = (U16)normalizedCounter[s]; } } } ZSTD_memcpy(dt, &DTableH, sizeof(DTableH)); } /* Spread symbols */ assert(tableSize <= 512); /* Specialized symbol spreading for the case when there are * no low probability (-1 count) symbols. When compressing * small blocks we avoid low probability symbols to hit this * case, since header decoding speed matters more. */ if (highThreshold == tableSize - 1) { size_t const tableMask = tableSize-1; size_t const step = FSE_TABLESTEP(tableSize); /* First lay down the symbols in order. * We use a uint64_t to lay down 8 bytes at a time. This reduces branch * misses since small blocks generally have small table logs, so nearly * all symbols have counts <= 8. We ensure we have 8 bytes at the end of * our buffer to handle the over-write. */ { U64 const add = 0x0101010101010101ull; size_t pos = 0; U64 sv = 0; U32 s; for (s=0; s=0); pos += (size_t)n; } } /* Now we spread those positions across the table. * The benefit of doing it in two stages is that we avoid the * variable size inner loop, which caused lots of branch misses. * Now we can run through all the positions without any branch misses. * We unroll the loop twice, since that is what empirically worked best. */ { size_t position = 0; size_t s; size_t const unroll = 2; assert(tableSize % unroll == 0); /* FSE_MIN_TABLELOG is 5 */ for (s = 0; s < (size_t)tableSize; s += unroll) { size_t u; for (u = 0; u < unroll; ++u) { size_t const uPosition = (position + (u * step)) & tableMask; tableDecode[uPosition].baseValue = spread[s + u]; } position = (position + (unroll * step)) & tableMask; } assert(position == 0); } } else { U32 const tableMask = tableSize-1; U32 const step = FSE_TABLESTEP(tableSize); U32 s, position = 0; for (s=0; s highThreshold)) position = (position + step) & tableMask; /* lowprob area */ } } assert(position == 0); /* position must reach all cells once, otherwise normalizedCounter is incorrect */ } /* Build Decoding table */ { U32 u; for (u=0; u max, corruption_detected, ""); { U32 const symbol = *(const BYTE*)src; U32 const baseline = baseValue[symbol]; U8 const nbBits = nbAdditionalBits[symbol]; ZSTD_buildSeqTable_rle(DTableSpace, baseline, nbBits); } *DTablePtr = DTableSpace; return 1; case set_basic : *DTablePtr = defaultTable; return 0; case set_repeat: RETURN_ERROR_IF(!flagRepeatTable, corruption_detected, ""); /* prefetch FSE table if used */ if (ddictIsCold && (nbSeq > 24 /* heuristic */)) { const void* const pStart = *DTablePtr; size_t const pSize = sizeof(ZSTD_seqSymbol) * (SEQSYMBOL_TABLE_SIZE(maxLog)); PREFETCH_AREA(pStart, pSize); } return 0; case set_compressed : { unsigned tableLog; S16 norm[MaxSeq+1]; size_t const headerSize = FSE_readNCount(norm, &max, &tableLog, src, srcSize); RETURN_ERROR_IF(FSE_isError(headerSize), corruption_detected, ""); RETURN_ERROR_IF(tableLog > maxLog, corruption_detected, ""); ZSTD_buildFSETable(DTableSpace, norm, max, baseValue, nbAdditionalBits, tableLog, wksp, wkspSize, bmi2); *DTablePtr = DTableSpace; return headerSize; } default : assert(0); RETURN_ERROR(GENERIC, "impossible"); } } size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr, const void* src, size_t srcSize) { const BYTE* const istart = (const BYTE*)src; const BYTE* const iend = istart + srcSize; const BYTE* ip = istart; int nbSeq; DEBUGLOG(5, "ZSTD_decodeSeqHeaders"); /* check */ RETURN_ERROR_IF(srcSize < MIN_SEQUENCES_SIZE, srcSize_wrong, ""); /* SeqHead */ nbSeq = *ip++; if (!nbSeq) { *nbSeqPtr=0; RETURN_ERROR_IF(srcSize != 1, srcSize_wrong, ""); return 1; } if (nbSeq > 0x7F) { if (nbSeq == 0xFF) { RETURN_ERROR_IF(ip+2 > iend, srcSize_wrong, ""); nbSeq = MEM_readLE16(ip) + LONGNBSEQ; ip+=2; } else { RETURN_ERROR_IF(ip >= iend, srcSize_wrong, ""); nbSeq = ((nbSeq-0x80)<<8) + *ip++; } } *nbSeqPtr = nbSeq; /* FSE table descriptors */ RETURN_ERROR_IF(ip+1 > iend, srcSize_wrong, ""); /* minimum possible size: 1 byte for symbol encoding types */ { symbolEncodingType_e const LLtype = (symbolEncodingType_e)(*ip >> 6); symbolEncodingType_e const OFtype = (symbolEncodingType_e)((*ip >> 4) & 3); symbolEncodingType_e const MLtype = (symbolEncodingType_e)((*ip >> 2) & 3); ip++; /* Build DTables */ { size_t const llhSize = ZSTD_buildSeqTable(dctx->entropy.LLTable, &dctx->LLTptr, LLtype, MaxLL, LLFSELog, ip, iend-ip, LL_base, LL_bits, LL_defaultDTable, dctx->fseEntropy, dctx->ddictIsCold, nbSeq, dctx->workspace, sizeof(dctx->workspace), ZSTD_DCtx_get_bmi2(dctx)); RETURN_ERROR_IF(ZSTD_isError(llhSize), corruption_detected, "ZSTD_buildSeqTable failed"); ip += llhSize; } { size_t const ofhSize = ZSTD_buildSeqTable(dctx->entropy.OFTable, &dctx->OFTptr, OFtype, MaxOff, OffFSELog, ip, iend-ip, OF_base, OF_bits, OF_defaultDTable, dctx->fseEntropy, dctx->ddictIsCold, nbSeq, dctx->workspace, sizeof(dctx->workspace), ZSTD_DCtx_get_bmi2(dctx)); RETURN_ERROR_IF(ZSTD_isError(ofhSize), corruption_detected, "ZSTD_buildSeqTable failed"); ip += ofhSize; } { size_t const mlhSize = ZSTD_buildSeqTable(dctx->entropy.MLTable, &dctx->MLTptr, MLtype, MaxML, MLFSELog, ip, iend-ip, ML_base, ML_bits, ML_defaultDTable, dctx->fseEntropy, dctx->ddictIsCold, nbSeq, dctx->workspace, sizeof(dctx->workspace), ZSTD_DCtx_get_bmi2(dctx)); RETURN_ERROR_IF(ZSTD_isError(mlhSize), corruption_detected, "ZSTD_buildSeqTable failed"); ip += mlhSize; } } return ip-istart; } typedef struct { size_t litLength; size_t matchLength; size_t offset; } seq_t; typedef struct { size_t state; const ZSTD_seqSymbol* table; } ZSTD_fseState; typedef struct { BIT_DStream_t DStream; ZSTD_fseState stateLL; ZSTD_fseState stateOffb; ZSTD_fseState stateML; size_t prevOffset[ZSTD_REP_NUM]; } seqState_t; /*! ZSTD_overlapCopy8() : * Copies 8 bytes from ip to op and updates op and ip where ip <= op. * If the offset is < 8 then the offset is spread to at least 8 bytes. * * Precondition: *ip <= *op * Postcondition: *op - *op >= 8 */ HINT_INLINE void ZSTD_overlapCopy8(BYTE** op, BYTE const** ip, size_t offset) { assert(*ip <= *op); if (offset < 8) { /* close range match, overlap */ static const U32 dec32table[] = { 0, 1, 2, 1, 4, 4, 4, 4 }; /* added */ static const int dec64table[] = { 8, 8, 8, 7, 8, 9,10,11 }; /* subtracted */ int const sub2 = dec64table[offset]; (*op)[0] = (*ip)[0]; (*op)[1] = (*ip)[1]; (*op)[2] = (*ip)[2]; (*op)[3] = (*ip)[3]; *ip += dec32table[offset]; ZSTD_copy4(*op+4, *ip); *ip -= sub2; } else { ZSTD_copy8(*op, *ip); } *ip += 8; *op += 8; assert(*op - *ip >= 8); } /*! ZSTD_safecopy() : * Specialized version of memcpy() that is allowed to READ up to WILDCOPY_OVERLENGTH past the input buffer * and write up to 16 bytes past oend_w (op >= oend_w is allowed). * This function is only called in the uncommon case where the sequence is near the end of the block. It * should be fast for a single long sequence, but can be slow for several short sequences. * * @param ovtype controls the overlap detection * - ZSTD_no_overlap: The source and destination are guaranteed to be at least WILDCOPY_VECLEN bytes apart. * - ZSTD_overlap_src_before_dst: The src and dst may overlap and may be any distance apart. * The src buffer must be before the dst buffer. */ static void ZSTD_safecopy(BYTE* op, const BYTE* const oend_w, BYTE const* ip, ptrdiff_t length, ZSTD_overlap_e ovtype) { ptrdiff_t const diff = op - ip; BYTE* const oend = op + length; assert((ovtype == ZSTD_no_overlap && (diff <= -8 || diff >= 8 || op >= oend_w)) || (ovtype == ZSTD_overlap_src_before_dst && diff >= 0)); if (length < 8) { /* Handle short lengths. */ while (op < oend) *op++ = *ip++; return; } if (ovtype == ZSTD_overlap_src_before_dst) { /* Copy 8 bytes and ensure the offset >= 8 when there can be overlap. */ assert(length >= 8); ZSTD_overlapCopy8(&op, &ip, diff); length -= 8; assert(op - ip >= 8); assert(op <= oend); } if (oend <= oend_w) { /* No risk of overwrite. */ ZSTD_wildcopy(op, ip, length, ovtype); return; } if (op <= oend_w) { /* Wildcopy until we get close to the end. */ assert(oend > oend_w); ZSTD_wildcopy(op, ip, oend_w - op, ovtype); ip += oend_w - op; op += oend_w - op; } /* Handle the leftovers. */ while (op < oend) *op++ = *ip++; } /* ZSTD_safecopyDstBeforeSrc(): * This version allows overlap with dst before src, or handles the non-overlap case with dst after src * Kept separate from more common ZSTD_safecopy case to avoid performance impact to the safecopy common case */ static void ZSTD_safecopyDstBeforeSrc(BYTE* op, BYTE const* ip, ptrdiff_t length) { ptrdiff_t const diff = op - ip; BYTE* const oend = op + length; if (length < 8 || diff > -8) { /* Handle short lengths, close overlaps, and dst not before src. */ while (op < oend) *op++ = *ip++; return; } if (op <= oend - WILDCOPY_OVERLENGTH && diff < -WILDCOPY_VECLEN) { ZSTD_wildcopy(op, ip, oend - WILDCOPY_OVERLENGTH - op, ZSTD_no_overlap); ip += oend - WILDCOPY_OVERLENGTH - op; op += oend - WILDCOPY_OVERLENGTH - op; } /* Handle the leftovers. */ while (op < oend) *op++ = *ip++; } /* ZSTD_execSequenceEnd(): * This version handles cases that are near the end of the output buffer. It requires * more careful checks to make sure there is no overflow. By separating out these hard * and unlikely cases, we can speed up the common cases. * * NOTE: This function needs to be fast for a single long sequence, but doesn't need * to be optimized for many small sequences, since those fall into ZSTD_execSequence(). */ FORCE_NOINLINE size_t ZSTD_execSequenceEnd(BYTE* op, BYTE* const oend, seq_t sequence, const BYTE** litPtr, const BYTE* const litLimit, const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd) { BYTE* const oLitEnd = op + sequence.litLength; size_t const sequenceLength = sequence.litLength + sequence.matchLength; const BYTE* const iLitEnd = *litPtr + sequence.litLength; const BYTE* match = oLitEnd - sequence.offset; BYTE* const oend_w = oend - WILDCOPY_OVERLENGTH; /* bounds checks : careful of address space overflow in 32-bit mode */ RETURN_ERROR_IF(sequenceLength > (size_t)(oend - op), dstSize_tooSmall, "last match must fit within dstBuffer"); RETURN_ERROR_IF(sequence.litLength > (size_t)(litLimit - *litPtr), corruption_detected, "try to read beyond literal buffer"); assert(op < op + sequenceLength); assert(oLitEnd < op + sequenceLength); /* copy literals */ ZSTD_safecopy(op, oend_w, *litPtr, sequence.litLength, ZSTD_no_overlap); op = oLitEnd; *litPtr = iLitEnd; /* copy Match */ if (sequence.offset > (size_t)(oLitEnd - prefixStart)) { /* offset beyond prefix */ RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart), corruption_detected, ""); match = dictEnd - (prefixStart - match); if (match + sequence.matchLength <= dictEnd) { ZSTD_memmove(oLitEnd, match, sequence.matchLength); return sequenceLength; } /* span extDict & currentPrefixSegment */ { size_t const length1 = dictEnd - match; ZSTD_memmove(oLitEnd, match, length1); op = oLitEnd + length1; sequence.matchLength -= length1; match = prefixStart; } } ZSTD_safecopy(op, oend_w, match, sequence.matchLength, ZSTD_overlap_src_before_dst); return sequenceLength; } /* ZSTD_execSequenceEndSplitLitBuffer(): * This version is intended to be used during instances where the litBuffer is still split. It is kept separate to avoid performance impact for the good case. */ FORCE_NOINLINE size_t ZSTD_execSequenceEndSplitLitBuffer(BYTE* op, BYTE* const oend, const BYTE* const oend_w, seq_t sequence, const BYTE** litPtr, const BYTE* const litLimit, const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd) { BYTE* const oLitEnd = op + sequence.litLength; size_t const sequenceLength = sequence.litLength + sequence.matchLength; const BYTE* const iLitEnd = *litPtr + sequence.litLength; const BYTE* match = oLitEnd - sequence.offset; /* bounds checks : careful of address space overflow in 32-bit mode */ RETURN_ERROR_IF(sequenceLength > (size_t)(oend - op), dstSize_tooSmall, "last match must fit within dstBuffer"); RETURN_ERROR_IF(sequence.litLength > (size_t)(litLimit - *litPtr), corruption_detected, "try to read beyond literal buffer"); assert(op < op + sequenceLength); assert(oLitEnd < op + sequenceLength); /* copy literals */ RETURN_ERROR_IF(op > *litPtr && op < *litPtr + sequence.litLength, dstSize_tooSmall, "output should not catch up to and overwrite literal buffer"); ZSTD_safecopyDstBeforeSrc(op, *litPtr, sequence.litLength); op = oLitEnd; *litPtr = iLitEnd; /* copy Match */ if (sequence.offset > (size_t)(oLitEnd - prefixStart)) { /* offset beyond prefix */ RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart), corruption_detected, ""); match = dictEnd - (prefixStart - match); if (match + sequence.matchLength <= dictEnd) { ZSTD_memmove(oLitEnd, match, sequence.matchLength); return sequenceLength; } /* span extDict & currentPrefixSegment */ { size_t const length1 = dictEnd - match; ZSTD_memmove(oLitEnd, match, length1); op = oLitEnd + length1; sequence.matchLength -= length1; match = prefixStart; } } ZSTD_safecopy(op, oend_w, match, sequence.matchLength, ZSTD_overlap_src_before_dst); return sequenceLength; } HINT_INLINE size_t ZSTD_execSequence(BYTE* op, BYTE* const oend, seq_t sequence, const BYTE** litPtr, const BYTE* const litLimit, const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd) { BYTE* const oLitEnd = op + sequence.litLength; size_t const sequenceLength = sequence.litLength + sequence.matchLength; BYTE* const oMatchEnd = op + sequenceLength; /* risk : address space overflow (32-bits) */ BYTE* const oend_w = oend - WILDCOPY_OVERLENGTH; /* risk : address space underflow on oend=NULL */ const BYTE* const iLitEnd = *litPtr + sequence.litLength; const BYTE* match = oLitEnd - sequence.offset; assert(op != NULL /* Precondition */); assert(oend_w < oend /* No underflow */); #if defined(__aarch64__) /* prefetch sequence starting from match that will be used for copy later */ PREFETCH_L1(match); #endif /* Handle edge cases in a slow path: * - Read beyond end of literals * - Match end is within WILDCOPY_OVERLIMIT of oend * - 32-bit mode and the match length overflows */ if (UNLIKELY( iLitEnd > litLimit || oMatchEnd > oend_w || (MEM_32bits() && (size_t)(oend - op) < sequenceLength + WILDCOPY_OVERLENGTH))) return ZSTD_execSequenceEnd(op, oend, sequence, litPtr, litLimit, prefixStart, virtualStart, dictEnd); /* Assumptions (everything else goes into ZSTD_execSequenceEnd()) */ assert(op <= oLitEnd /* No overflow */); assert(oLitEnd < oMatchEnd /* Non-zero match & no overflow */); assert(oMatchEnd <= oend /* No underflow */); assert(iLitEnd <= litLimit /* Literal length is in bounds */); assert(oLitEnd <= oend_w /* Can wildcopy literals */); assert(oMatchEnd <= oend_w /* Can wildcopy matches */); /* Copy Literals: * Split out litLength <= 16 since it is nearly always true. +1.6% on gcc-9. * We likely don't need the full 32-byte wildcopy. */ assert(WILDCOPY_OVERLENGTH >= 16); ZSTD_copy16(op, (*litPtr)); if (UNLIKELY(sequence.litLength > 16)) { ZSTD_wildcopy(op + 16, (*litPtr) + 16, sequence.litLength - 16, ZSTD_no_overlap); } op = oLitEnd; *litPtr = iLitEnd; /* update for next sequence */ /* Copy Match */ if (sequence.offset > (size_t)(oLitEnd - prefixStart)) { /* offset beyond prefix -> go into extDict */ RETURN_ERROR_IF(UNLIKELY(sequence.offset > (size_t)(oLitEnd - virtualStart)), corruption_detected, ""); match = dictEnd + (match - prefixStart); if (match + sequence.matchLength <= dictEnd) { ZSTD_memmove(oLitEnd, match, sequence.matchLength); return sequenceLength; } /* span extDict & currentPrefixSegment */ { size_t const length1 = dictEnd - match; ZSTD_memmove(oLitEnd, match, length1); op = oLitEnd + length1; sequence.matchLength -= length1; match = prefixStart; } } /* Match within prefix of 1 or more bytes */ assert(op <= oMatchEnd); assert(oMatchEnd <= oend_w); assert(match >= prefixStart); assert(sequence.matchLength >= 1); /* Nearly all offsets are >= WILDCOPY_VECLEN bytes, which means we can use wildcopy * without overlap checking. */ if (LIKELY(sequence.offset >= WILDCOPY_VECLEN)) { /* We bet on a full wildcopy for matches, since we expect matches to be * longer than literals (in general). In silesia, ~10% of matches are longer * than 16 bytes. */ ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength, ZSTD_no_overlap); return sequenceLength; } assert(sequence.offset < WILDCOPY_VECLEN); /* Copy 8 bytes and spread the offset to be >= 8. */ ZSTD_overlapCopy8(&op, &match, sequence.offset); /* If the match length is > 8 bytes, then continue with the wildcopy. */ if (sequence.matchLength > 8) { assert(op < oMatchEnd); ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength - 8, ZSTD_overlap_src_before_dst); } return sequenceLength; } HINT_INLINE size_t ZSTD_execSequenceSplitLitBuffer(BYTE* op, BYTE* const oend, const BYTE* const oend_w, seq_t sequence, const BYTE** litPtr, const BYTE* const litLimit, const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd) { BYTE* const oLitEnd = op + sequence.litLength; size_t const sequenceLength = sequence.litLength + sequence.matchLength; BYTE* const oMatchEnd = op + sequenceLength; /* risk : address space overflow (32-bits) */ const BYTE* const iLitEnd = *litPtr + sequence.litLength; const BYTE* match = oLitEnd - sequence.offset; assert(op != NULL /* Precondition */); assert(oend_w < oend /* No underflow */); /* Handle edge cases in a slow path: * - Read beyond end of literals * - Match end is within WILDCOPY_OVERLIMIT of oend * - 32-bit mode and the match length overflows */ if (UNLIKELY( iLitEnd > litLimit || oMatchEnd > oend_w || (MEM_32bits() && (size_t)(oend - op) < sequenceLength + WILDCOPY_OVERLENGTH))) return ZSTD_execSequenceEndSplitLitBuffer(op, oend, oend_w, sequence, litPtr, litLimit, prefixStart, virtualStart, dictEnd); /* Assumptions (everything else goes into ZSTD_execSequenceEnd()) */ assert(op <= oLitEnd /* No overflow */); assert(oLitEnd < oMatchEnd /* Non-zero match & no overflow */); assert(oMatchEnd <= oend /* No underflow */); assert(iLitEnd <= litLimit /* Literal length is in bounds */); assert(oLitEnd <= oend_w /* Can wildcopy literals */); assert(oMatchEnd <= oend_w /* Can wildcopy matches */); /* Copy Literals: * Split out litLength <= 16 since it is nearly always true. +1.6% on gcc-9. * We likely don't need the full 32-byte wildcopy. */ assert(WILDCOPY_OVERLENGTH >= 16); ZSTD_copy16(op, (*litPtr)); if (UNLIKELY(sequence.litLength > 16)) { ZSTD_wildcopy(op+16, (*litPtr)+16, sequence.litLength-16, ZSTD_no_overlap); } op = oLitEnd; *litPtr = iLitEnd; /* update for next sequence */ /* Copy Match */ if (sequence.offset > (size_t)(oLitEnd - prefixStart)) { /* offset beyond prefix -> go into extDict */ RETURN_ERROR_IF(UNLIKELY(sequence.offset > (size_t)(oLitEnd - virtualStart)), corruption_detected, ""); match = dictEnd + (match - prefixStart); if (match + sequence.matchLength <= dictEnd) { ZSTD_memmove(oLitEnd, match, sequence.matchLength); return sequenceLength; } /* span extDict & currentPrefixSegment */ { size_t const length1 = dictEnd - match; ZSTD_memmove(oLitEnd, match, length1); op = oLitEnd + length1; sequence.matchLength -= length1; match = prefixStart; } } /* Match within prefix of 1 or more bytes */ assert(op <= oMatchEnd); assert(oMatchEnd <= oend_w); assert(match >= prefixStart); assert(sequence.matchLength >= 1); /* Nearly all offsets are >= WILDCOPY_VECLEN bytes, which means we can use wildcopy * without overlap checking. */ if (LIKELY(sequence.offset >= WILDCOPY_VECLEN)) { /* We bet on a full wildcopy for matches, since we expect matches to be * longer than literals (in general). In silesia, ~10% of matches are longer * than 16 bytes. */ ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength, ZSTD_no_overlap); return sequenceLength; } assert(sequence.offset < WILDCOPY_VECLEN); /* Copy 8 bytes and spread the offset to be >= 8. */ ZSTD_overlapCopy8(&op, &match, sequence.offset); /* If the match length is > 8 bytes, then continue with the wildcopy. */ if (sequence.matchLength > 8) { assert(op < oMatchEnd); ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength-8, ZSTD_overlap_src_before_dst); } return sequenceLength; } static void ZSTD_initFseState(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, const ZSTD_seqSymbol* dt) { const void* ptr = dt; const ZSTD_seqSymbol_header* const DTableH = (const ZSTD_seqSymbol_header*)ptr; DStatePtr->state = BIT_readBits(bitD, DTableH->tableLog); DEBUGLOG(6, "ZSTD_initFseState : val=%u using %u bits", (U32)DStatePtr->state, DTableH->tableLog); BIT_reloadDStream(bitD); DStatePtr->table = dt + 1; } FORCE_INLINE_TEMPLATE void ZSTD_updateFseStateWithDInfo(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, U16 nextState, U32 nbBits) { size_t const lowBits = BIT_readBits(bitD, nbBits); DStatePtr->state = nextState + lowBits; } /* We need to add at most (ZSTD_WINDOWLOG_MAX_32 - 1) bits to read the maximum * offset bits. But we can only read at most STREAM_ACCUMULATOR_MIN_32 * bits before reloading. This value is the maximum number of bytes we read * after reloading when we are decoding long offsets. */ #define LONG_OFFSETS_MAX_EXTRA_BITS_32 \ (ZSTD_WINDOWLOG_MAX_32 > STREAM_ACCUMULATOR_MIN_32 \ ? ZSTD_WINDOWLOG_MAX_32 - STREAM_ACCUMULATOR_MIN_32 \ : 0) typedef enum { ZSTD_lo_isRegularOffset, ZSTD_lo_isLongOffset=1 } ZSTD_longOffset_e; FORCE_INLINE_TEMPLATE seq_t ZSTD_decodeSequence(seqState_t* seqState, const ZSTD_longOffset_e longOffsets) { seq_t seq; /* * ZSTD_seqSymbol is a structure with a total of 64 bits wide. So it can be * loaded in one operation and extracted its fields by simply shifting or * bit-extracting on aarch64. * GCC doesn't recognize this and generates more unnecessary ldr/ldrb/ldrh * operations that cause performance drop. This can be avoided by using this * ZSTD_memcpy hack. */ #if defined(__aarch64__) && (defined(__GNUC__) && !defined(__clang__)) ZSTD_seqSymbol llDInfoS, mlDInfoS, ofDInfoS; ZSTD_seqSymbol* const llDInfo = &llDInfoS; ZSTD_seqSymbol* const mlDInfo = &mlDInfoS; ZSTD_seqSymbol* const ofDInfo = &ofDInfoS; ZSTD_memcpy(llDInfo, seqState->stateLL.table + seqState->stateLL.state, sizeof(ZSTD_seqSymbol)); ZSTD_memcpy(mlDInfo, seqState->stateML.table + seqState->stateML.state, sizeof(ZSTD_seqSymbol)); ZSTD_memcpy(ofDInfo, seqState->stateOffb.table + seqState->stateOffb.state, sizeof(ZSTD_seqSymbol)); #else const ZSTD_seqSymbol* const llDInfo = seqState->stateLL.table + seqState->stateLL.state; const ZSTD_seqSymbol* const mlDInfo = seqState->stateML.table + seqState->stateML.state; const ZSTD_seqSymbol* const ofDInfo = seqState->stateOffb.table + seqState->stateOffb.state; #endif seq.matchLength = mlDInfo->baseValue; seq.litLength = llDInfo->baseValue; { U32 const ofBase = ofDInfo->baseValue; BYTE const llBits = llDInfo->nbAdditionalBits; BYTE const mlBits = mlDInfo->nbAdditionalBits; BYTE const ofBits = ofDInfo->nbAdditionalBits; BYTE const totalBits = llBits+mlBits+ofBits; U16 const llNext = llDInfo->nextState; U16 const mlNext = mlDInfo->nextState; U16 const ofNext = ofDInfo->nextState; U32 const llnbBits = llDInfo->nbBits; U32 const mlnbBits = mlDInfo->nbBits; U32 const ofnbBits = ofDInfo->nbBits; assert(llBits <= MaxLLBits); assert(mlBits <= MaxMLBits); assert(ofBits <= MaxOff); /* * As gcc has better branch and block analyzers, sometimes it is only * valuable to mark likeliness for clang, it gives around 3-4% of * performance. */ /* sequence */ { size_t offset; if (ofBits > 1) { ZSTD_STATIC_ASSERT(ZSTD_lo_isLongOffset == 1); ZSTD_STATIC_ASSERT(LONG_OFFSETS_MAX_EXTRA_BITS_32 == 5); ZSTD_STATIC_ASSERT(STREAM_ACCUMULATOR_MIN_32 > LONG_OFFSETS_MAX_EXTRA_BITS_32); ZSTD_STATIC_ASSERT(STREAM_ACCUMULATOR_MIN_32 - LONG_OFFSETS_MAX_EXTRA_BITS_32 >= MaxMLBits); if (MEM_32bits() && longOffsets && (ofBits >= STREAM_ACCUMULATOR_MIN_32)) { /* Always read extra bits, this keeps the logic simple, * avoids branches, and avoids accidentally reading 0 bits. */ U32 const extraBits = LONG_OFFSETS_MAX_EXTRA_BITS_32; offset = ofBase + (BIT_readBitsFast(&seqState->DStream, ofBits - extraBits) << extraBits); BIT_reloadDStream(&seqState->DStream); offset += BIT_readBitsFast(&seqState->DStream, extraBits); } else { offset = ofBase + BIT_readBitsFast(&seqState->DStream, ofBits/*>0*/); /* <= (ZSTD_WINDOWLOG_MAX-1) bits */ if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream); } seqState->prevOffset[2] = seqState->prevOffset[1]; seqState->prevOffset[1] = seqState->prevOffset[0]; seqState->prevOffset[0] = offset; } else { U32 const ll0 = (llDInfo->baseValue == 0); if (LIKELY((ofBits == 0))) { offset = seqState->prevOffset[ll0]; seqState->prevOffset[1] = seqState->prevOffset[!ll0]; seqState->prevOffset[0] = offset; } else { offset = ofBase + ll0 + BIT_readBitsFast(&seqState->DStream, 1); { size_t temp = (offset==3) ? seqState->prevOffset[0] - 1 : seqState->prevOffset[offset]; temp += !temp; /* 0 is not valid; input is corrupted; force offset to 1 */ if (offset != 1) seqState->prevOffset[2] = seqState->prevOffset[1]; seqState->prevOffset[1] = seqState->prevOffset[0]; seqState->prevOffset[0] = offset = temp; } } } seq.offset = offset; } if (mlBits > 0) seq.matchLength += BIT_readBitsFast(&seqState->DStream, mlBits/*>0*/); if (MEM_32bits() && (mlBits+llBits >= STREAM_ACCUMULATOR_MIN_32-LONG_OFFSETS_MAX_EXTRA_BITS_32)) BIT_reloadDStream(&seqState->DStream); if (MEM_64bits() && UNLIKELY(totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog))) BIT_reloadDStream(&seqState->DStream); /* Ensure there are enough bits to read the rest of data in 64-bit mode. */ ZSTD_STATIC_ASSERT(16+LLFSELog+MLFSELog+OffFSELog < STREAM_ACCUMULATOR_MIN_64); if (llBits > 0) seq.litLength += BIT_readBitsFast(&seqState->DStream, llBits/*>0*/); if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream); DEBUGLOG(6, "seq: litL=%u, matchL=%u, offset=%u", (U32)seq.litLength, (U32)seq.matchLength, (U32)seq.offset); ZSTD_updateFseStateWithDInfo(&seqState->stateLL, &seqState->DStream, llNext, llnbBits); /* <= 9 bits */ ZSTD_updateFseStateWithDInfo(&seqState->stateML, &seqState->DStream, mlNext, mlnbBits); /* <= 9 bits */ if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream); /* <= 18 bits */ ZSTD_updateFseStateWithDInfo(&seqState->stateOffb, &seqState->DStream, ofNext, ofnbBits); /* <= 8 bits */ } return seq; } #ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION MEM_STATIC int ZSTD_dictionaryIsActive(ZSTD_DCtx const* dctx, BYTE const* prefixStart, BYTE const* oLitEnd) { size_t const windowSize = dctx->fParams.windowSize; /* No dictionary used. */ if (dctx->dictContentEndForFuzzing == NULL) return 0; /* Dictionary is our prefix. */ if (prefixStart == dctx->dictContentBeginForFuzzing) return 1; /* Dictionary is not our ext-dict. */ if (dctx->dictEnd != dctx->dictContentEndForFuzzing) return 0; /* Dictionary is not within our window size. */ if ((size_t)(oLitEnd - prefixStart) >= windowSize) return 0; /* Dictionary is active. */ return 1; } MEM_STATIC void ZSTD_assertValidSequence( ZSTD_DCtx const* dctx, BYTE const* op, BYTE const* oend, seq_t const seq, BYTE const* prefixStart, BYTE const* virtualStart) { #if DEBUGLEVEL >= 1 size_t const windowSize = dctx->fParams.windowSize; size_t const sequenceSize = seq.litLength + seq.matchLength; BYTE const* const oLitEnd = op + seq.litLength; DEBUGLOG(6, "Checking sequence: litL=%u matchL=%u offset=%u", (U32)seq.litLength, (U32)seq.matchLength, (U32)seq.offset); assert(op <= oend); assert((size_t)(oend - op) >= sequenceSize); assert(sequenceSize <= ZSTD_BLOCKSIZE_MAX); if (ZSTD_dictionaryIsActive(dctx, prefixStart, oLitEnd)) { size_t const dictSize = (size_t)((char const*)dctx->dictContentEndForFuzzing - (char const*)dctx->dictContentBeginForFuzzing); /* Offset must be within the dictionary. */ assert(seq.offset <= (size_t)(oLitEnd - virtualStart)); assert(seq.offset <= windowSize + dictSize); } else { /* Offset must be within our window. */ assert(seq.offset <= windowSize); } #else (void)dctx, (void)op, (void)oend, (void)seq, (void)prefixStart, (void)virtualStart; #endif } #endif #ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG FORCE_INLINE_TEMPLATE size_t DONT_VECTORIZE ZSTD_decompressSequences_bodySplitLitBuffer( ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, const void* seqStart, size_t seqSize, int nbSeq, const ZSTD_longOffset_e isLongOffset, const int frame) { const BYTE* ip = (const BYTE*)seqStart; const BYTE* const iend = ip + seqSize; BYTE* const ostart = (BYTE*)dst; BYTE* const oend = ostart + maxDstSize; BYTE* op = ostart; const BYTE* litPtr = dctx->litPtr; const BYTE* litBufferEnd = dctx->litBufferEnd; const BYTE* const prefixStart = (const BYTE*) (dctx->prefixStart); const BYTE* const vBase = (const BYTE*) (dctx->virtualStart); const BYTE* const dictEnd = (const BYTE*) (dctx->dictEnd); DEBUGLOG(5, "ZSTD_decompressSequences_bodySplitLitBuffer"); (void)frame; /* Regen sequences */ if (nbSeq) { seqState_t seqState; dctx->fseEntropy = 1; { U32 i; for (i=0; ientropy.rep[i]; } RETURN_ERROR_IF( ERR_isError(BIT_initDStream(&seqState.DStream, ip, iend-ip)), corruption_detected, ""); ZSTD_initFseState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr); ZSTD_initFseState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr); ZSTD_initFseState(&seqState.stateML, &seqState.DStream, dctx->MLTptr); assert(dst != NULL); ZSTD_STATIC_ASSERT( BIT_DStream_unfinished < BIT_DStream_completed && BIT_DStream_endOfBuffer < BIT_DStream_completed && BIT_DStream_completed < BIT_DStream_overflow); /* decompress without overrunning litPtr begins */ { seq_t sequence = ZSTD_decodeSequence(&seqState, isLongOffset); /* Align the decompression loop to 32 + 16 bytes. * * zstd compiled with gcc-9 on an Intel i9-9900k shows 10% decompression * speed swings based on the alignment of the decompression loop. This * performance swing is caused by parts of the decompression loop falling * out of the DSB. The entire decompression loop should fit in the DSB, * when it can't we get much worse performance. You can measure if you've * hit the good case or the bad case with this perf command for some * compressed file test.zst: * * perf stat -e cycles -e instructions -e idq.all_dsb_cycles_any_uops \ * -e idq.all_mite_cycles_any_uops -- ./zstd -tq test.zst * * If you see most cycles served out of the MITE you've hit the bad case. * If you see most cycles served out of the DSB you've hit the good case. * If it is pretty even then you may be in an okay case. * * This issue has been reproduced on the following CPUs: * - Kabylake: Macbook Pro (15-inch, 2019) 2.4 GHz Intel Core i9 * Use Instruments->Counters to get DSB/MITE cycles. * I never got performance swings, but I was able to * go from the good case of mostly DSB to half of the * cycles served from MITE. * - Coffeelake: Intel i9-9900k * - Coffeelake: Intel i7-9700k * * I haven't been able to reproduce the instability or DSB misses on any * of the following CPUS: * - Haswell * - Broadwell: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GH * - Skylake * * Alignment is done for each of the three major decompression loops: * - ZSTD_decompressSequences_bodySplitLitBuffer - presplit section of the literal buffer * - ZSTD_decompressSequences_bodySplitLitBuffer - postsplit section of the literal buffer * - ZSTD_decompressSequences_body * Alignment choices are made to minimize large swings on bad cases and influence on performance * from changes external to this code, rather than to overoptimize on the current commit. * * If you are seeing performance stability this script can help test. * It tests on 4 commits in zstd where I saw performance change. * * https://gist.github.com/terrelln/9889fc06a423fd5ca6e99351564473f4 */ #if defined(__GNUC__) && defined(__x86_64__) __asm__(".p2align 6"); # if __GNUC__ >= 7 /* good for gcc-7, gcc-9, and gcc-11 */ __asm__("nop"); __asm__(".p2align 5"); __asm__("nop"); __asm__(".p2align 4"); # if __GNUC__ == 8 || __GNUC__ == 10 /* good for gcc-8 and gcc-10 */ __asm__("nop"); __asm__(".p2align 3"); # endif # endif #endif /* Handle the initial state where litBuffer is currently split between dst and litExtraBuffer */ for (; litPtr + sequence.litLength <= dctx->litBufferEnd; ) { size_t const oneSeqSize = ZSTD_execSequenceSplitLitBuffer(op, oend, litPtr + sequence.litLength - WILDCOPY_OVERLENGTH, sequence, &litPtr, litBufferEnd, prefixStart, vBase, dictEnd); #if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE) assert(!ZSTD_isError(oneSeqSize)); if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequence, prefixStart, vBase); #endif if (UNLIKELY(ZSTD_isError(oneSeqSize))) return oneSeqSize; DEBUGLOG(6, "regenerated sequence size : %u", (U32)oneSeqSize); op += oneSeqSize; if (UNLIKELY(!--nbSeq)) break; BIT_reloadDStream(&(seqState.DStream)); sequence = ZSTD_decodeSequence(&seqState, isLongOffset); } /* If there are more sequences, they will need to read literals from litExtraBuffer; copy over the remainder from dst and update litPtr and litEnd */ if (nbSeq > 0) { const size_t leftoverLit = dctx->litBufferEnd - litPtr; if (leftoverLit) { RETURN_ERROR_IF(leftoverLit > (size_t)(oend - op), dstSize_tooSmall, "remaining lit must fit within dstBuffer"); ZSTD_safecopyDstBeforeSrc(op, litPtr, leftoverLit); sequence.litLength -= leftoverLit; op += leftoverLit; } litPtr = dctx->litExtraBuffer; litBufferEnd = dctx->litExtraBuffer + ZSTD_LITBUFFEREXTRASIZE; dctx->litBufferLocation = ZSTD_not_in_dst; { size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequence, &litPtr, litBufferEnd, prefixStart, vBase, dictEnd); #if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE) assert(!ZSTD_isError(oneSeqSize)); if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequence, prefixStart, vBase); #endif if (UNLIKELY(ZSTD_isError(oneSeqSize))) return oneSeqSize; DEBUGLOG(6, "regenerated sequence size : %u", (U32)oneSeqSize); op += oneSeqSize; if (--nbSeq) BIT_reloadDStream(&(seqState.DStream)); } } } if (nbSeq > 0) /* there is remaining lit from extra buffer */ { #if defined(__GNUC__) && defined(__x86_64__) __asm__(".p2align 6"); __asm__("nop"); # if __GNUC__ != 7 /* worse for gcc-7 better for gcc-8, gcc-9, and gcc-10 and clang */ __asm__(".p2align 4"); __asm__("nop"); __asm__(".p2align 3"); # elif __GNUC__ >= 11 __asm__(".p2align 3"); # else __asm__(".p2align 5"); __asm__("nop"); __asm__(".p2align 3"); # endif #endif for (; ; ) { seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset); size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequence, &litPtr, litBufferEnd, prefixStart, vBase, dictEnd); #if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE) assert(!ZSTD_isError(oneSeqSize)); if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequence, prefixStart, vBase); #endif if (UNLIKELY(ZSTD_isError(oneSeqSize))) return oneSeqSize; DEBUGLOG(6, "regenerated sequence size : %u", (U32)oneSeqSize); op += oneSeqSize; if (UNLIKELY(!--nbSeq)) break; BIT_reloadDStream(&(seqState.DStream)); } } /* check if reached exact end */ DEBUGLOG(5, "ZSTD_decompressSequences_bodySplitLitBuffer: after decode loop, remaining nbSeq : %i", nbSeq); RETURN_ERROR_IF(nbSeq, corruption_detected, ""); RETURN_ERROR_IF(BIT_reloadDStream(&seqState.DStream) < BIT_DStream_completed, corruption_detected, ""); /* save reps for next block */ { U32 i; for (i=0; ientropy.rep[i] = (U32)(seqState.prevOffset[i]); } } /* last literal segment */ if (dctx->litBufferLocation == ZSTD_split) /* split hasn't been reached yet, first get dst then copy litExtraBuffer */ { size_t const lastLLSize = litBufferEnd - litPtr; RETURN_ERROR_IF(lastLLSize > (size_t)(oend - op), dstSize_tooSmall, ""); if (op != NULL) { ZSTD_memmove(op, litPtr, lastLLSize); op += lastLLSize; } litPtr = dctx->litExtraBuffer; litBufferEnd = dctx->litExtraBuffer + ZSTD_LITBUFFEREXTRASIZE; dctx->litBufferLocation = ZSTD_not_in_dst; } { size_t const lastLLSize = litBufferEnd - litPtr; RETURN_ERROR_IF(lastLLSize > (size_t)(oend-op), dstSize_tooSmall, ""); if (op != NULL) { ZSTD_memcpy(op, litPtr, lastLLSize); op += lastLLSize; } } return op-ostart; } FORCE_INLINE_TEMPLATE size_t DONT_VECTORIZE ZSTD_decompressSequences_body(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, const void* seqStart, size_t seqSize, int nbSeq, const ZSTD_longOffset_e isLongOffset, const int frame) { const BYTE* ip = (const BYTE*)seqStart; const BYTE* const iend = ip + seqSize; BYTE* const ostart = (BYTE*)dst; BYTE* const oend = dctx->litBufferLocation == ZSTD_not_in_dst ? ostart + maxDstSize : dctx->litBuffer; BYTE* op = ostart; const BYTE* litPtr = dctx->litPtr; const BYTE* const litEnd = litPtr + dctx->litSize; const BYTE* const prefixStart = (const BYTE*)(dctx->prefixStart); const BYTE* const vBase = (const BYTE*)(dctx->virtualStart); const BYTE* const dictEnd = (const BYTE*)(dctx->dictEnd); DEBUGLOG(5, "ZSTD_decompressSequences_body: nbSeq = %d", nbSeq); (void)frame; /* Regen sequences */ if (nbSeq) { seqState_t seqState; dctx->fseEntropy = 1; { U32 i; for (i = 0; i < ZSTD_REP_NUM; i++) seqState.prevOffset[i] = dctx->entropy.rep[i]; } RETURN_ERROR_IF( ERR_isError(BIT_initDStream(&seqState.DStream, ip, iend - ip)), corruption_detected, ""); ZSTD_initFseState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr); ZSTD_initFseState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr); ZSTD_initFseState(&seqState.stateML, &seqState.DStream, dctx->MLTptr); assert(dst != NULL); ZSTD_STATIC_ASSERT( BIT_DStream_unfinished < BIT_DStream_completed && BIT_DStream_endOfBuffer < BIT_DStream_completed && BIT_DStream_completed < BIT_DStream_overflow); #if defined(__GNUC__) && defined(__x86_64__) __asm__(".p2align 6"); __asm__("nop"); # if __GNUC__ >= 7 __asm__(".p2align 5"); __asm__("nop"); __asm__(".p2align 3"); # else __asm__(".p2align 4"); __asm__("nop"); __asm__(".p2align 3"); # endif #endif for ( ; ; ) { seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset); size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequence, &litPtr, litEnd, prefixStart, vBase, dictEnd); #if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE) assert(!ZSTD_isError(oneSeqSize)); if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequence, prefixStart, vBase); #endif if (UNLIKELY(ZSTD_isError(oneSeqSize))) return oneSeqSize; DEBUGLOG(6, "regenerated sequence size : %u", (U32)oneSeqSize); op += oneSeqSize; if (UNLIKELY(!--nbSeq)) break; BIT_reloadDStream(&(seqState.DStream)); } /* check if reached exact end */ DEBUGLOG(5, "ZSTD_decompressSequences_body: after decode loop, remaining nbSeq : %i", nbSeq); RETURN_ERROR_IF(nbSeq, corruption_detected, ""); RETURN_ERROR_IF(BIT_reloadDStream(&seqState.DStream) < BIT_DStream_completed, corruption_detected, ""); /* save reps for next block */ { U32 i; for (i=0; ientropy.rep[i] = (U32)(seqState.prevOffset[i]); } } /* last literal segment */ { size_t const lastLLSize = litEnd - litPtr; RETURN_ERROR_IF(lastLLSize > (size_t)(oend-op), dstSize_tooSmall, ""); if (op != NULL) { ZSTD_memcpy(op, litPtr, lastLLSize); op += lastLLSize; } } return op-ostart; } static size_t ZSTD_decompressSequences_default(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, const void* seqStart, size_t seqSize, int nbSeq, const ZSTD_longOffset_e isLongOffset, const int frame) { return ZSTD_decompressSequences_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame); } static size_t ZSTD_decompressSequencesSplitLitBuffer_default(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, const void* seqStart, size_t seqSize, int nbSeq, const ZSTD_longOffset_e isLongOffset, const int frame) { return ZSTD_decompressSequences_bodySplitLitBuffer(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame); } #endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */ #ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT FORCE_INLINE_TEMPLATE size_t ZSTD_prefetchMatch(size_t prefetchPos, seq_t const sequence, const BYTE* const prefixStart, const BYTE* const dictEnd) { prefetchPos += sequence.litLength; { const BYTE* const matchBase = (sequence.offset > prefetchPos) ? dictEnd : prefixStart; const BYTE* const match = matchBase + prefetchPos - sequence.offset; /* note : this operation can overflow when seq.offset is really too large, which can only happen when input is corrupted. * No consequence though : memory address is only used for prefetching, not for dereferencing */ PREFETCH_L1(match); PREFETCH_L1(match+CACHELINE_SIZE); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */ } return prefetchPos + sequence.matchLength; } /* This decoding function employs prefetching * to reduce latency impact of cache misses. * It's generally employed when block contains a significant portion of long-distance matches * or when coupled with a "cold" dictionary */ FORCE_INLINE_TEMPLATE size_t ZSTD_decompressSequencesLong_body( ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, const void* seqStart, size_t seqSize, int nbSeq, const ZSTD_longOffset_e isLongOffset, const int frame) { const BYTE* ip = (const BYTE*)seqStart; const BYTE* const iend = ip + seqSize; BYTE* const ostart = (BYTE*)dst; BYTE* const oend = dctx->litBufferLocation == ZSTD_in_dst ? dctx->litBuffer : ostart + maxDstSize; BYTE* op = ostart; const BYTE* litPtr = dctx->litPtr; const BYTE* litBufferEnd = dctx->litBufferEnd; const BYTE* const prefixStart = (const BYTE*) (dctx->prefixStart); const BYTE* const dictStart = (const BYTE*) (dctx->virtualStart); const BYTE* const dictEnd = (const BYTE*) (dctx->dictEnd); (void)frame; /* Regen sequences */ if (nbSeq) { #define STORED_SEQS 8 #define STORED_SEQS_MASK (STORED_SEQS-1) #define ADVANCED_SEQS STORED_SEQS seq_t sequences[STORED_SEQS]; int const seqAdvance = MIN(nbSeq, ADVANCED_SEQS); seqState_t seqState; int seqNb; size_t prefetchPos = (size_t)(op-prefixStart); /* track position relative to prefixStart */ dctx->fseEntropy = 1; { int i; for (i=0; ientropy.rep[i]; } assert(dst != NULL); assert(iend >= ip); RETURN_ERROR_IF( ERR_isError(BIT_initDStream(&seqState.DStream, ip, iend-ip)), corruption_detected, ""); ZSTD_initFseState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr); ZSTD_initFseState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr); ZSTD_initFseState(&seqState.stateML, &seqState.DStream, dctx->MLTptr); /* prepare in advance */ for (seqNb=0; (BIT_reloadDStream(&seqState.DStream) <= BIT_DStream_completed) && (seqNblitBufferLocation == ZSTD_split && litPtr + sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK].litLength > dctx->litBufferEnd) { /* lit buffer is reaching split point, empty out the first buffer and transition to litExtraBuffer */ const size_t leftoverLit = dctx->litBufferEnd - litPtr; if (leftoverLit) { RETURN_ERROR_IF(leftoverLit > (size_t)(oend - op), dstSize_tooSmall, "remaining lit must fit within dstBuffer"); ZSTD_safecopyDstBeforeSrc(op, litPtr, leftoverLit); sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK].litLength -= leftoverLit; op += leftoverLit; } litPtr = dctx->litExtraBuffer; litBufferEnd = dctx->litExtraBuffer + ZSTD_LITBUFFEREXTRASIZE; dctx->litBufferLocation = ZSTD_not_in_dst; oneSeqSize = ZSTD_execSequence(op, oend, sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK], &litPtr, litBufferEnd, prefixStart, dictStart, dictEnd); #if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE) assert(!ZSTD_isError(oneSeqSize)); if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK], prefixStart, dictStart); #endif if (ZSTD_isError(oneSeqSize)) return oneSeqSize; prefetchPos = ZSTD_prefetchMatch(prefetchPos, sequence, prefixStart, dictEnd); sequences[seqNb & STORED_SEQS_MASK] = sequence; op += oneSeqSize; } else { /* lit buffer is either wholly contained in first or second split, or not split at all*/ oneSeqSize = dctx->litBufferLocation == ZSTD_split ? ZSTD_execSequenceSplitLitBuffer(op, oend, litPtr + sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK].litLength - WILDCOPY_OVERLENGTH, sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK], &litPtr, litBufferEnd, prefixStart, dictStart, dictEnd) : ZSTD_execSequence(op, oend, sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK], &litPtr, litBufferEnd, prefixStart, dictStart, dictEnd); #if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE) assert(!ZSTD_isError(oneSeqSize)); if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK], prefixStart, dictStart); #endif if (ZSTD_isError(oneSeqSize)) return oneSeqSize; prefetchPos = ZSTD_prefetchMatch(prefetchPos, sequence, prefixStart, dictEnd); sequences[seqNb & STORED_SEQS_MASK] = sequence; op += oneSeqSize; } } RETURN_ERROR_IF(seqNblitBufferLocation == ZSTD_split && litPtr + sequence->litLength > dctx->litBufferEnd) { const size_t leftoverLit = dctx->litBufferEnd - litPtr; if (leftoverLit) { RETURN_ERROR_IF(leftoverLit > (size_t)(oend - op), dstSize_tooSmall, "remaining lit must fit within dstBuffer"); ZSTD_safecopyDstBeforeSrc(op, litPtr, leftoverLit); sequence->litLength -= leftoverLit; op += leftoverLit; } litPtr = dctx->litExtraBuffer; litBufferEnd = dctx->litExtraBuffer + ZSTD_LITBUFFEREXTRASIZE; dctx->litBufferLocation = ZSTD_not_in_dst; { size_t const oneSeqSize = ZSTD_execSequence(op, oend, *sequence, &litPtr, litBufferEnd, prefixStart, dictStart, dictEnd); #if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE) assert(!ZSTD_isError(oneSeqSize)); if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[seqNb&STORED_SEQS_MASK], prefixStart, dictStart); #endif if (ZSTD_isError(oneSeqSize)) return oneSeqSize; op += oneSeqSize; } } else { size_t const oneSeqSize = dctx->litBufferLocation == ZSTD_split ? ZSTD_execSequenceSplitLitBuffer(op, oend, litPtr + sequence->litLength - WILDCOPY_OVERLENGTH, *sequence, &litPtr, litBufferEnd, prefixStart, dictStart, dictEnd) : ZSTD_execSequence(op, oend, *sequence, &litPtr, litBufferEnd, prefixStart, dictStart, dictEnd); #if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE) assert(!ZSTD_isError(oneSeqSize)); if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[seqNb&STORED_SEQS_MASK], prefixStart, dictStart); #endif if (ZSTD_isError(oneSeqSize)) return oneSeqSize; op += oneSeqSize; } } /* save reps for next block */ { U32 i; for (i=0; ientropy.rep[i] = (U32)(seqState.prevOffset[i]); } } /* last literal segment */ if (dctx->litBufferLocation == ZSTD_split) /* first deplete literal buffer in dst, then copy litExtraBuffer */ { size_t const lastLLSize = litBufferEnd - litPtr; RETURN_ERROR_IF(lastLLSize > (size_t)(oend - op), dstSize_tooSmall, ""); if (op != NULL) { ZSTD_memmove(op, litPtr, lastLLSize); op += lastLLSize; } litPtr = dctx->litExtraBuffer; litBufferEnd = dctx->litExtraBuffer + ZSTD_LITBUFFEREXTRASIZE; } { size_t const lastLLSize = litBufferEnd - litPtr; RETURN_ERROR_IF(lastLLSize > (size_t)(oend-op), dstSize_tooSmall, ""); if (op != NULL) { ZSTD_memmove(op, litPtr, lastLLSize); op += lastLLSize; } } return op-ostart; } static size_t ZSTD_decompressSequencesLong_default(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, const void* seqStart, size_t seqSize, int nbSeq, const ZSTD_longOffset_e isLongOffset, const int frame) { return ZSTD_decompressSequencesLong_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame); } #endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT */ #if DYNAMIC_BMI2 #ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG static BMI2_TARGET_ATTRIBUTE size_t DONT_VECTORIZE ZSTD_decompressSequences_bmi2(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, const void* seqStart, size_t seqSize, int nbSeq, const ZSTD_longOffset_e isLongOffset, const int frame) { return ZSTD_decompressSequences_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame); } static BMI2_TARGET_ATTRIBUTE size_t DONT_VECTORIZE ZSTD_decompressSequencesSplitLitBuffer_bmi2(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, const void* seqStart, size_t seqSize, int nbSeq, const ZSTD_longOffset_e isLongOffset, const int frame) { return ZSTD_decompressSequences_bodySplitLitBuffer(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame); } #endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */ #ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT static BMI2_TARGET_ATTRIBUTE size_t ZSTD_decompressSequencesLong_bmi2(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, const void* seqStart, size_t seqSize, int nbSeq, const ZSTD_longOffset_e isLongOffset, const int frame) { return ZSTD_decompressSequencesLong_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame); } #endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT */ #endif /* DYNAMIC_BMI2 */ typedef size_t (*ZSTD_decompressSequences_t)( ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, const void* seqStart, size_t seqSize, int nbSeq, const ZSTD_longOffset_e isLongOffset, const int frame); #ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG static size_t ZSTD_decompressSequences(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, const void* seqStart, size_t seqSize, int nbSeq, const ZSTD_longOffset_e isLongOffset, const int frame) { DEBUGLOG(5, "ZSTD_decompressSequences"); #if DYNAMIC_BMI2 if (ZSTD_DCtx_get_bmi2(dctx)) { return ZSTD_decompressSequences_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame); } #endif return ZSTD_decompressSequences_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame); } static size_t ZSTD_decompressSequencesSplitLitBuffer(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, const void* seqStart, size_t seqSize, int nbSeq, const ZSTD_longOffset_e isLongOffset, const int frame) { DEBUGLOG(5, "ZSTD_decompressSequencesSplitLitBuffer"); #if DYNAMIC_BMI2 if (ZSTD_DCtx_get_bmi2(dctx)) { return ZSTD_decompressSequencesSplitLitBuffer_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame); } #endif return ZSTD_decompressSequencesSplitLitBuffer_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame); } #endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */ #ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT /* ZSTD_decompressSequencesLong() : * decompression function triggered when a minimum share of offsets is considered "long", * aka out of cache. * note : "long" definition seems overloaded here, sometimes meaning "wider than bitstream register", and sometimes meaning "farther than memory cache distance". * This function will try to mitigate main memory latency through the use of prefetching */ static size_t ZSTD_decompressSequencesLong(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, const void* seqStart, size_t seqSize, int nbSeq, const ZSTD_longOffset_e isLongOffset, const int frame) { DEBUGLOG(5, "ZSTD_decompressSequencesLong"); #if DYNAMIC_BMI2 if (ZSTD_DCtx_get_bmi2(dctx)) { return ZSTD_decompressSequencesLong_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame); } #endif return ZSTD_decompressSequencesLong_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame); } #endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT */ /** * @returns The total size of the history referenceable by zstd, including * both the prefix and the extDict. At @p op any offset larger than this * is invalid. */ static size_t ZSTD_totalHistorySize(BYTE* op, BYTE const* virtualStart) { return (size_t)(op - virtualStart); } typedef struct { unsigned longOffsetShare; unsigned maxNbAdditionalBits; } ZSTD_OffsetInfo; /* ZSTD_getOffsetInfo() : * condition : offTable must be valid * @return : "share" of long offsets (arbitrarily defined as > (1<<23)) * compared to maximum possible of (1< 22) info.longOffsetShare += 1; } assert(tableLog <= OffFSELog); info.longOffsetShare <<= (OffFSELog - tableLog); /* scale to OffFSELog */ } return info; } /** * @returns The maximum offset we can decode in one read of our bitstream, without * reloading more bits in the middle of the offset bits read. Any offsets larger * than this must use the long offset decoder. */ static size_t ZSTD_maxShortOffset(void) { if (MEM_64bits()) { /* We can decode any offset without reloading bits. * This might change if the max window size grows. */ ZSTD_STATIC_ASSERT(ZSTD_WINDOWLOG_MAX <= 31); return (size_t)-1; } else { /* The maximum offBase is (1 << (STREAM_ACCUMULATOR_MIN + 1)) - 1. * This offBase would require STREAM_ACCUMULATOR_MIN extra bits. * Then we have to subtract ZSTD_REP_NUM to get the maximum possible offset. */ size_t const maxOffbase = ((size_t)1 << (STREAM_ACCUMULATOR_MIN + 1)) - 1; size_t const maxOffset = maxOffbase - ZSTD_REP_NUM; assert(ZSTD_highbit32((U32)maxOffbase) == STREAM_ACCUMULATOR_MIN); return maxOffset; } } size_t ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const int frame, const streaming_operation streaming) { /* blockType == blockCompressed */ const BYTE* ip = (const BYTE*)src; DEBUGLOG(5, "ZSTD_decompressBlock_internal (size : %u)", (U32)srcSize); /* Note : the wording of the specification * allows compressed block to be sized exactly ZSTD_BLOCKSIZE_MAX. * This generally does not happen, as it makes little sense, * since an uncompressed block would feature same size and have no decompression cost. * Also, note that decoder from reference libzstd before < v1.5.4 * would consider this edge case as an error. * As a consequence, avoid generating compressed blocks of size ZSTD_BLOCKSIZE_MAX * for broader compatibility with the deployed ecosystem of zstd decoders */ RETURN_ERROR_IF(srcSize > ZSTD_BLOCKSIZE_MAX, srcSize_wrong, ""); /* Decode literals section */ { size_t const litCSize = ZSTD_decodeLiteralsBlock(dctx, src, srcSize, dst, dstCapacity, streaming); DEBUGLOG(5, "ZSTD_decodeLiteralsBlock : cSize=%u, nbLiterals=%zu", (U32)litCSize, dctx->litSize); if (ZSTD_isError(litCSize)) return litCSize; ip += litCSize; srcSize -= litCSize; } /* Build Decoding Tables */ { /* Compute the maximum block size, which must also work when !frame and fParams are unset. * Additionally, take the min with dstCapacity to ensure that the totalHistorySize fits in a size_t. */ size_t const blockSizeMax = MIN(dstCapacity, (frame ? dctx->fParams.blockSizeMax : ZSTD_BLOCKSIZE_MAX)); size_t const totalHistorySize = ZSTD_totalHistorySize((BYTE*)dst + blockSizeMax, (BYTE const*)dctx->virtualStart); /* isLongOffset must be true if there are long offsets. * Offsets are long if they are larger than ZSTD_maxShortOffset(). * We don't expect that to be the case in 64-bit mode. * * We check here to see if our history is large enough to allow long offsets. * If it isn't, then we can't possible have (valid) long offsets. If the offset * is invalid, then it is okay to read it incorrectly. * * If isLongOffsets is true, then we will later check our decoding table to see * if it is even possible to generate long offsets. */ ZSTD_longOffset_e isLongOffset = (ZSTD_longOffset_e)(MEM_32bits() && (totalHistorySize > ZSTD_maxShortOffset())); /* These macros control at build-time which decompressor implementation * we use. If neither is defined, we do some inspection and dispatch at * runtime. */ #if !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \ !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG) int usePrefetchDecoder = dctx->ddictIsCold; #else /* Set to 1 to avoid computing offset info if we don't need to. * Otherwise this value is ignored. */ int usePrefetchDecoder = 1; #endif int nbSeq; size_t const seqHSize = ZSTD_decodeSeqHeaders(dctx, &nbSeq, ip, srcSize); if (ZSTD_isError(seqHSize)) return seqHSize; ip += seqHSize; srcSize -= seqHSize; RETURN_ERROR_IF((dst == NULL || dstCapacity == 0) && nbSeq > 0, dstSize_tooSmall, "NULL not handled"); RETURN_ERROR_IF(MEM_64bits() && sizeof(size_t) == sizeof(void*) && (size_t)(-1) - (size_t)dst < (size_t)(1 << 20), dstSize_tooSmall, "invalid dst"); /* If we could potentially have long offsets, or we might want to use the prefetch decoder, * compute information about the share of long offsets, and the maximum nbAdditionalBits. * NOTE: could probably use a larger nbSeq limit */ if (isLongOffset || (!usePrefetchDecoder && (totalHistorySize > (1u << 24)) && (nbSeq > 8))) { ZSTD_OffsetInfo const info = ZSTD_getOffsetInfo(dctx->OFTptr, nbSeq); if (isLongOffset && info.maxNbAdditionalBits <= STREAM_ACCUMULATOR_MIN) { /* If isLongOffset, but the maximum number of additional bits that we see in our table is small * enough, then we know it is impossible to have too long an offset in this block, so we can * use the regular offset decoder. */ isLongOffset = ZSTD_lo_isRegularOffset; } if (!usePrefetchDecoder) { U32 const minShare = MEM_64bits() ? 7 : 20; /* heuristic values, correspond to 2.73% and 7.81% */ usePrefetchDecoder = (info.longOffsetShare >= minShare); } } dctx->ddictIsCold = 0; #if !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \ !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG) if (usePrefetchDecoder) { #else (void)usePrefetchDecoder; { #endif #ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT return ZSTD_decompressSequencesLong(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset, frame); #endif } #ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG /* else */ if (dctx->litBufferLocation == ZSTD_split) return ZSTD_decompressSequencesSplitLitBuffer(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset, frame); else return ZSTD_decompressSequences(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset, frame); #endif } } void ZSTD_checkContinuity(ZSTD_DCtx* dctx, const void* dst, size_t dstSize) { if (dst != dctx->previousDstEnd && dstSize > 0) { /* not contiguous */ dctx->dictEnd = dctx->previousDstEnd; dctx->virtualStart = (const char*)dst - ((const char*)(dctx->previousDstEnd) - (const char*)(dctx->prefixStart)); dctx->prefixStart = dst; dctx->previousDstEnd = dst; } } size_t ZSTD_decompressBlock_deprecated(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize) { size_t dSize; ZSTD_checkContinuity(dctx, dst, dstCapacity); dSize = ZSTD_decompressBlock_internal(dctx, dst, dstCapacity, src, srcSize, /* frame */ 0, not_streaming); dctx->previousDstEnd = (char*)dst + dSize; return dSize; } /* NOTE: Must just wrap ZSTD_decompressBlock_deprecated() */ size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize) { return ZSTD_decompressBlock_deprecated(dctx, dst, dstCapacity, src, srcSize); } zmat-0.9.9/src/blosc2/internal-complibs/zstd/decompress/zstd_decompress_block.h0000644000175200007730000000521314515254731030220 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_DEC_BLOCK_H #define ZSTD_DEC_BLOCK_H /*-******************************************************* * Dependencies *********************************************************/ #include "../common/zstd_deps.h" /* size_t */ #include "../zstd.h" /* DCtx, and some public functions */ #include "../common/zstd_internal.h" /* blockProperties_t, and some public functions */ #include "zstd_decompress_internal.h" /* ZSTD_seqSymbol */ /* === Prototypes === */ /* note: prototypes already published within `zstd.h` : * ZSTD_decompressBlock() */ /* note: prototypes already published within `zstd_internal.h` : * ZSTD_getcBlockSize() * ZSTD_decodeSeqHeaders() */ /* Streaming state is used to inform allocation of the literal buffer */ typedef enum { not_streaming = 0, is_streaming = 1 } streaming_operation; /* ZSTD_decompressBlock_internal() : * decompress block, starting at `src`, * into destination buffer `dst`. * @return : decompressed block size, * or an error code (which can be tested using ZSTD_isError()) */ size_t ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const int frame, const streaming_operation streaming); /* ZSTD_buildFSETable() : * generate FSE decoding table for one symbol (ll, ml or off) * this function must be called with valid parameters only * (dt is large enough, normalizedCounter distribution total is a power of 2, max is within range, etc.) * in which case it cannot fail. * The workspace must be 4-byte aligned and at least ZSTD_BUILD_FSE_TABLE_WKSP_SIZE bytes, which is * defined in zstd_decompress_internal.h. * Internal use only. */ void ZSTD_buildFSETable(ZSTD_seqSymbol* dt, const short* normalizedCounter, unsigned maxSymbolValue, const U32* baseValue, const U8* nbAdditionalBits, unsigned tableLog, void* wksp, size_t wkspSize, int bmi2); /* Internal definition of ZSTD_decompressBlock() to avoid deprecation warnings. */ size_t ZSTD_decompressBlock_deprecated(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); #endif /* ZSTD_DEC_BLOCK_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/decompress/zstd_decompress_internal.h0000644000175200007730000002263614515254731030752 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* zstd_decompress_internal: * objects and definitions shared within lib/decompress modules */ #ifndef ZSTD_DECOMPRESS_INTERNAL_H #define ZSTD_DECOMPRESS_INTERNAL_H /*-******************************************************* * Dependencies *********************************************************/ #include "../common/mem.h" /* BYTE, U16, U32 */ #include "../common/zstd_internal.h" /* constants : MaxLL, MaxML, MaxOff, LLFSELog, etc. */ /*-******************************************************* * Constants *********************************************************/ static UNUSED_ATTR const U32 LL_base[MaxLL+1] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 22, 24, 28, 32, 40, 48, 64, 0x80, 0x100, 0x200, 0x400, 0x800, 0x1000, 0x2000, 0x4000, 0x8000, 0x10000 }; static UNUSED_ATTR const U32 OF_base[MaxOff+1] = { 0, 1, 1, 5, 0xD, 0x1D, 0x3D, 0x7D, 0xFD, 0x1FD, 0x3FD, 0x7FD, 0xFFD, 0x1FFD, 0x3FFD, 0x7FFD, 0xFFFD, 0x1FFFD, 0x3FFFD, 0x7FFFD, 0xFFFFD, 0x1FFFFD, 0x3FFFFD, 0x7FFFFD, 0xFFFFFD, 0x1FFFFFD, 0x3FFFFFD, 0x7FFFFFD, 0xFFFFFFD, 0x1FFFFFFD, 0x3FFFFFFD, 0x7FFFFFFD }; static UNUSED_ATTR const U8 OF_bits[MaxOff+1] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31 }; static UNUSED_ATTR const U32 ML_base[MaxML+1] = { 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 37, 39, 41, 43, 47, 51, 59, 67, 83, 99, 0x83, 0x103, 0x203, 0x403, 0x803, 0x1003, 0x2003, 0x4003, 0x8003, 0x10003 }; /*-******************************************************* * Decompression types *********************************************************/ typedef struct { U32 fastMode; U32 tableLog; } ZSTD_seqSymbol_header; typedef struct { U16 nextState; BYTE nbAdditionalBits; BYTE nbBits; U32 baseValue; } ZSTD_seqSymbol; #define SEQSYMBOL_TABLE_SIZE(log) (1 + (1 << (log))) #define ZSTD_BUILD_FSE_TABLE_WKSP_SIZE (sizeof(S16) * (MaxSeq + 1) + (1u << MaxFSELog) + sizeof(U64)) #define ZSTD_BUILD_FSE_TABLE_WKSP_SIZE_U32 ((ZSTD_BUILD_FSE_TABLE_WKSP_SIZE + sizeof(U32) - 1) / sizeof(U32)) #define ZSTD_HUFFDTABLE_CAPACITY_LOG 12 typedef struct { ZSTD_seqSymbol LLTable[SEQSYMBOL_TABLE_SIZE(LLFSELog)]; /* Note : Space reserved for FSE Tables */ ZSTD_seqSymbol OFTable[SEQSYMBOL_TABLE_SIZE(OffFSELog)]; /* is also used as temporary workspace while building hufTable during DDict creation */ ZSTD_seqSymbol MLTable[SEQSYMBOL_TABLE_SIZE(MLFSELog)]; /* and therefore must be at least HUF_DECOMPRESS_WORKSPACE_SIZE large */ HUF_DTable hufTable[HUF_DTABLE_SIZE(ZSTD_HUFFDTABLE_CAPACITY_LOG)]; /* can accommodate HUF_decompress4X */ U32 rep[ZSTD_REP_NUM]; U32 workspace[ZSTD_BUILD_FSE_TABLE_WKSP_SIZE_U32]; } ZSTD_entropyDTables_t; typedef enum { ZSTDds_getFrameHeaderSize, ZSTDds_decodeFrameHeader, ZSTDds_decodeBlockHeader, ZSTDds_decompressBlock, ZSTDds_decompressLastBlock, ZSTDds_checkChecksum, ZSTDds_decodeSkippableHeader, ZSTDds_skipFrame } ZSTD_dStage; typedef enum { zdss_init=0, zdss_loadHeader, zdss_read, zdss_load, zdss_flush } ZSTD_dStreamStage; typedef enum { ZSTD_use_indefinitely = -1, /* Use the dictionary indefinitely */ ZSTD_dont_use = 0, /* Do not use the dictionary (if one exists free it) */ ZSTD_use_once = 1 /* Use the dictionary once and set to ZSTD_dont_use */ } ZSTD_dictUses_e; /* Hashset for storing references to multiple ZSTD_DDict within ZSTD_DCtx */ typedef struct { const ZSTD_DDict** ddictPtrTable; size_t ddictPtrTableSize; size_t ddictPtrCount; } ZSTD_DDictHashSet; #ifndef ZSTD_DECODER_INTERNAL_BUFFER # define ZSTD_DECODER_INTERNAL_BUFFER (1 << 16) #endif #define ZSTD_LBMIN 64 #define ZSTD_LBMAX (128 << 10) /* extra buffer, compensates when dst is not large enough to store litBuffer */ #define ZSTD_LITBUFFEREXTRASIZE BOUNDED(ZSTD_LBMIN, ZSTD_DECODER_INTERNAL_BUFFER, ZSTD_LBMAX) typedef enum { ZSTD_not_in_dst = 0, /* Stored entirely within litExtraBuffer */ ZSTD_in_dst = 1, /* Stored entirely within dst (in memory after current output write) */ ZSTD_split = 2 /* Split between litExtraBuffer and dst */ } ZSTD_litLocation_e; struct ZSTD_DCtx_s { const ZSTD_seqSymbol* LLTptr; const ZSTD_seqSymbol* MLTptr; const ZSTD_seqSymbol* OFTptr; const HUF_DTable* HUFptr; ZSTD_entropyDTables_t entropy; U32 workspace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32]; /* space needed when building huffman tables */ const void* previousDstEnd; /* detect continuity */ const void* prefixStart; /* start of current segment */ const void* virtualStart; /* virtual start of previous segment if it was just before current one */ const void* dictEnd; /* end of previous segment */ size_t expected; ZSTD_frameHeader fParams; U64 processedCSize; U64 decodedSize; blockType_e bType; /* used in ZSTD_decompressContinue(), store blockType between block header decoding and block decompression stages */ ZSTD_dStage stage; U32 litEntropy; U32 fseEntropy; XXH64_state_t xxhState; size_t headerSize; ZSTD_format_e format; ZSTD_forceIgnoreChecksum_e forceIgnoreChecksum; /* User specified: if == 1, will ignore checksums in compressed frame. Default == 0 */ U32 validateChecksum; /* if == 1, will validate checksum. Is == 1 if (fParams.checksumFlag == 1) and (forceIgnoreChecksum == 0). */ const BYTE* litPtr; ZSTD_customMem customMem; size_t litSize; size_t rleSize; size_t staticSize; #if DYNAMIC_BMI2 != 0 int bmi2; /* == 1 if the CPU supports BMI2 and 0 otherwise. CPU support is determined dynamically once per context lifetime. */ #endif /* dictionary */ ZSTD_DDict* ddictLocal; const ZSTD_DDict* ddict; /* set by ZSTD_initDStream_usingDDict(), or ZSTD_DCtx_refDDict() */ U32 dictID; int ddictIsCold; /* if == 1 : dictionary is "new" for working context, and presumed "cold" (not in cpu cache) */ ZSTD_dictUses_e dictUses; ZSTD_DDictHashSet* ddictSet; /* Hash set for multiple ddicts */ ZSTD_refMultipleDDicts_e refMultipleDDicts; /* User specified: if == 1, will allow references to multiple DDicts. Default == 0 (disabled) */ int disableHufAsm; /* streaming */ ZSTD_dStreamStage streamStage; char* inBuff; size_t inBuffSize; size_t inPos; size_t maxWindowSize; char* outBuff; size_t outBuffSize; size_t outStart; size_t outEnd; size_t lhSize; #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1) void* legacyContext; U32 previousLegacyVersion; U32 legacyVersion; #endif U32 hostageByte; int noForwardProgress; ZSTD_bufferMode_e outBufferMode; ZSTD_outBuffer expectedOutBuffer; /* workspace */ BYTE* litBuffer; const BYTE* litBufferEnd; ZSTD_litLocation_e litBufferLocation; BYTE litExtraBuffer[ZSTD_LITBUFFEREXTRASIZE + WILDCOPY_OVERLENGTH]; /* literal buffer can be split between storage within dst and within this scratch buffer */ BYTE headerBuffer[ZSTD_FRAMEHEADERSIZE_MAX]; size_t oversizedDuration; #ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION void const* dictContentBeginForFuzzing; void const* dictContentEndForFuzzing; #endif /* Tracing */ #if ZSTD_TRACE ZSTD_TraceCtx traceCtx; #endif }; /* typedef'd to ZSTD_DCtx within "zstd.h" */ MEM_STATIC int ZSTD_DCtx_get_bmi2(const struct ZSTD_DCtx_s *dctx) { #if DYNAMIC_BMI2 != 0 return dctx->bmi2; #else (void)dctx; return 0; #endif } /*-******************************************************* * Shared internal functions *********************************************************/ /*! ZSTD_loadDEntropy() : * dict : must point at beginning of a valid zstd dictionary. * @return : size of dictionary header (size of magic number + dict ID + entropy tables) */ size_t ZSTD_loadDEntropy(ZSTD_entropyDTables_t* entropy, const void* const dict, size_t const dictSize); /*! ZSTD_checkContinuity() : * check if next `dst` follows previous position, where decompression ended. * If yes, do nothing (continue on current segment). * If not, classify previous segment as "external dictionary", and start a new segment. * This function cannot fail. */ void ZSTD_checkContinuity(ZSTD_DCtx* dctx, const void* dst, size_t dstSize); #endif /* ZSTD_DECOMPRESS_INTERNAL_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/decompress/zstd_decompress.c0000644000175200007730000030256514515254731027053 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* *************************************************************** * Tuning parameters *****************************************************************/ /*! * HEAPMODE : * Select how default decompression function ZSTD_decompress() allocates its context, * on stack (0), or into heap (1, default; requires malloc()). * Note that functions with explicit context such as ZSTD_decompressDCtx() are unaffected. */ #ifndef ZSTD_HEAPMODE # define ZSTD_HEAPMODE 1 #endif /*! * LEGACY_SUPPORT : * if set to 1+, ZSTD_decompress() can decode older formats (v0.1+) */ #ifndef ZSTD_LEGACY_SUPPORT # define ZSTD_LEGACY_SUPPORT 0 #endif /*! * MAXWINDOWSIZE_DEFAULT : * maximum window size accepted by DStream __by default__. * Frames requiring more memory will be rejected. * It's possible to set a different limit using ZSTD_DCtx_setMaxWindowSize(). */ #ifndef ZSTD_MAXWINDOWSIZE_DEFAULT # define ZSTD_MAXWINDOWSIZE_DEFAULT (((U32)1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT) + 1) #endif /*! * NO_FORWARD_PROGRESS_MAX : * maximum allowed nb of calls to ZSTD_decompressStream() * without any forward progress * (defined as: no byte read from input, and no byte flushed to output) * before triggering an error. */ #ifndef ZSTD_NO_FORWARD_PROGRESS_MAX # define ZSTD_NO_FORWARD_PROGRESS_MAX 16 #endif /*-******************************************************* * Dependencies *********************************************************/ #include "../common/allocations.h" /* ZSTD_customMalloc, ZSTD_customCalloc, ZSTD_customFree */ #include "../common/zstd_deps.h" /* ZSTD_memcpy, ZSTD_memmove, ZSTD_memset */ #include "../common/mem.h" /* low level memory routines */ #define FSE_STATIC_LINKING_ONLY #include "../common/fse.h" #include "../common/huf.h" #include "../common/xxhash.h" /* XXH64_reset, XXH64_update, XXH64_digest, XXH64 */ #include "../common/zstd_internal.h" /* blockProperties_t */ #include "zstd_decompress_internal.h" /* ZSTD_DCtx */ #include "zstd_ddict.h" /* ZSTD_DDictDictContent */ #include "zstd_decompress_block.h" /* ZSTD_decompressBlock_internal */ #include "../common/bits.h" /* ZSTD_highbit32 */ #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1) # include "../legacy/zstd_legacy.h" #endif /************************************* * Multiple DDicts Hashset internals * *************************************/ #define DDICT_HASHSET_MAX_LOAD_FACTOR_COUNT_MULT 4 #define DDICT_HASHSET_MAX_LOAD_FACTOR_SIZE_MULT 3 /* These two constants represent SIZE_MULT/COUNT_MULT load factor without using a float. * Currently, that means a 0.75 load factor. * So, if count * COUNT_MULT / size * SIZE_MULT != 0, then we've exceeded * the load factor of the ddict hash set. */ #define DDICT_HASHSET_TABLE_BASE_SIZE 64 #define DDICT_HASHSET_RESIZE_FACTOR 2 /* Hash function to determine starting position of dict insertion within the table * Returns an index between [0, hashSet->ddictPtrTableSize] */ static size_t ZSTD_DDictHashSet_getIndex(const ZSTD_DDictHashSet* hashSet, U32 dictID) { const U64 hash = XXH64(&dictID, sizeof(U32), 0); /* DDict ptr table size is a multiple of 2, use size - 1 as mask to get index within [0, hashSet->ddictPtrTableSize) */ return hash & (hashSet->ddictPtrTableSize - 1); } /* Adds DDict to a hashset without resizing it. * If inserting a DDict with a dictID that already exists in the set, replaces the one in the set. * Returns 0 if successful, or a zstd error code if something went wrong. */ static size_t ZSTD_DDictHashSet_emplaceDDict(ZSTD_DDictHashSet* hashSet, const ZSTD_DDict* ddict) { const U32 dictID = ZSTD_getDictID_fromDDict(ddict); size_t idx = ZSTD_DDictHashSet_getIndex(hashSet, dictID); const size_t idxRangeMask = hashSet->ddictPtrTableSize - 1; RETURN_ERROR_IF(hashSet->ddictPtrCount == hashSet->ddictPtrTableSize, GENERIC, "Hash set is full!"); DEBUGLOG(4, "Hashed index: for dictID: %u is %zu", dictID, idx); while (hashSet->ddictPtrTable[idx] != NULL) { /* Replace existing ddict if inserting ddict with same dictID */ if (ZSTD_getDictID_fromDDict(hashSet->ddictPtrTable[idx]) == dictID) { DEBUGLOG(4, "DictID already exists, replacing rather than adding"); hashSet->ddictPtrTable[idx] = ddict; return 0; } idx &= idxRangeMask; idx++; } DEBUGLOG(4, "Final idx after probing for dictID %u is: %zu", dictID, idx); hashSet->ddictPtrTable[idx] = ddict; hashSet->ddictPtrCount++; return 0; } /* Expands hash table by factor of DDICT_HASHSET_RESIZE_FACTOR and * rehashes all values, allocates new table, frees old table. * Returns 0 on success, otherwise a zstd error code. */ static size_t ZSTD_DDictHashSet_expand(ZSTD_DDictHashSet* hashSet, ZSTD_customMem customMem) { size_t newTableSize = hashSet->ddictPtrTableSize * DDICT_HASHSET_RESIZE_FACTOR; const ZSTD_DDict** newTable = (const ZSTD_DDict**)ZSTD_customCalloc(sizeof(ZSTD_DDict*) * newTableSize, customMem); const ZSTD_DDict** oldTable = hashSet->ddictPtrTable; size_t oldTableSize = hashSet->ddictPtrTableSize; size_t i; DEBUGLOG(4, "Expanding DDict hash table! Old size: %zu new size: %zu", oldTableSize, newTableSize); RETURN_ERROR_IF(!newTable, memory_allocation, "Expanded hashset allocation failed!"); hashSet->ddictPtrTable = newTable; hashSet->ddictPtrTableSize = newTableSize; hashSet->ddictPtrCount = 0; for (i = 0; i < oldTableSize; ++i) { if (oldTable[i] != NULL) { FORWARD_IF_ERROR(ZSTD_DDictHashSet_emplaceDDict(hashSet, oldTable[i]), ""); } } ZSTD_customFree((void*)oldTable, customMem); DEBUGLOG(4, "Finished re-hash"); return 0; } /* Fetches a DDict with the given dictID * Returns the ZSTD_DDict* with the requested dictID. If it doesn't exist, then returns NULL. */ static const ZSTD_DDict* ZSTD_DDictHashSet_getDDict(ZSTD_DDictHashSet* hashSet, U32 dictID) { size_t idx = ZSTD_DDictHashSet_getIndex(hashSet, dictID); const size_t idxRangeMask = hashSet->ddictPtrTableSize - 1; DEBUGLOG(4, "Hashed index: for dictID: %u is %zu", dictID, idx); for (;;) { size_t currDictID = ZSTD_getDictID_fromDDict(hashSet->ddictPtrTable[idx]); if (currDictID == dictID || currDictID == 0) { /* currDictID == 0 implies a NULL ddict entry */ break; } else { idx &= idxRangeMask; /* Goes to start of table when we reach the end */ idx++; } } DEBUGLOG(4, "Final idx after probing for dictID %u is: %zu", dictID, idx); return hashSet->ddictPtrTable[idx]; } /* Allocates space for and returns a ddict hash set * The hash set's ZSTD_DDict* table has all values automatically set to NULL to begin with. * Returns NULL if allocation failed. */ static ZSTD_DDictHashSet* ZSTD_createDDictHashSet(ZSTD_customMem customMem) { ZSTD_DDictHashSet* ret = (ZSTD_DDictHashSet*)ZSTD_customMalloc(sizeof(ZSTD_DDictHashSet), customMem); DEBUGLOG(4, "Allocating new hash set"); if (!ret) return NULL; ret->ddictPtrTable = (const ZSTD_DDict**)ZSTD_customCalloc(DDICT_HASHSET_TABLE_BASE_SIZE * sizeof(ZSTD_DDict*), customMem); if (!ret->ddictPtrTable) { ZSTD_customFree(ret, customMem); return NULL; } ret->ddictPtrTableSize = DDICT_HASHSET_TABLE_BASE_SIZE; ret->ddictPtrCount = 0; return ret; } /* Frees the table of ZSTD_DDict* within a hashset, then frees the hashset itself. * Note: The ZSTD_DDict* within the table are NOT freed. */ static void ZSTD_freeDDictHashSet(ZSTD_DDictHashSet* hashSet, ZSTD_customMem customMem) { DEBUGLOG(4, "Freeing ddict hash set"); if (hashSet && hashSet->ddictPtrTable) { ZSTD_customFree((void*)hashSet->ddictPtrTable, customMem); } if (hashSet) { ZSTD_customFree(hashSet, customMem); } } /* Public function: Adds a DDict into the ZSTD_DDictHashSet, possibly triggering a resize of the hash set. * Returns 0 on success, or a ZSTD error. */ static size_t ZSTD_DDictHashSet_addDDict(ZSTD_DDictHashSet* hashSet, const ZSTD_DDict* ddict, ZSTD_customMem customMem) { DEBUGLOG(4, "Adding dict ID: %u to hashset with - Count: %zu Tablesize: %zu", ZSTD_getDictID_fromDDict(ddict), hashSet->ddictPtrCount, hashSet->ddictPtrTableSize); if (hashSet->ddictPtrCount * DDICT_HASHSET_MAX_LOAD_FACTOR_COUNT_MULT / hashSet->ddictPtrTableSize * DDICT_HASHSET_MAX_LOAD_FACTOR_SIZE_MULT != 0) { FORWARD_IF_ERROR(ZSTD_DDictHashSet_expand(hashSet, customMem), ""); } FORWARD_IF_ERROR(ZSTD_DDictHashSet_emplaceDDict(hashSet, ddict), ""); return 0; } /*-************************************************************* * Context management ***************************************************************/ size_t ZSTD_sizeof_DCtx (const ZSTD_DCtx* dctx) { if (dctx==NULL) return 0; /* support sizeof NULL */ return sizeof(*dctx) + ZSTD_sizeof_DDict(dctx->ddictLocal) + dctx->inBuffSize + dctx->outBuffSize; } size_t ZSTD_estimateDCtxSize(void) { return sizeof(ZSTD_DCtx); } static size_t ZSTD_startingInputLength(ZSTD_format_e format) { size_t const startingInputLength = ZSTD_FRAMEHEADERSIZE_PREFIX(format); /* only supports formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless */ assert( (format == ZSTD_f_zstd1) || (format == ZSTD_f_zstd1_magicless) ); return startingInputLength; } static void ZSTD_DCtx_resetParameters(ZSTD_DCtx* dctx) { assert(dctx->streamStage == zdss_init); dctx->format = ZSTD_f_zstd1; dctx->maxWindowSize = ZSTD_MAXWINDOWSIZE_DEFAULT; dctx->outBufferMode = ZSTD_bm_buffered; dctx->forceIgnoreChecksum = ZSTD_d_validateChecksum; dctx->refMultipleDDicts = ZSTD_rmd_refSingleDDict; dctx->disableHufAsm = 0; } static void ZSTD_initDCtx_internal(ZSTD_DCtx* dctx) { dctx->staticSize = 0; dctx->ddict = NULL; dctx->ddictLocal = NULL; dctx->dictEnd = NULL; dctx->ddictIsCold = 0; dctx->dictUses = ZSTD_dont_use; dctx->inBuff = NULL; dctx->inBuffSize = 0; dctx->outBuffSize = 0; dctx->streamStage = zdss_init; #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1) dctx->legacyContext = NULL; dctx->previousLegacyVersion = 0; #endif dctx->noForwardProgress = 0; dctx->oversizedDuration = 0; #if DYNAMIC_BMI2 dctx->bmi2 = ZSTD_cpuSupportsBmi2(); #endif dctx->ddictSet = NULL; ZSTD_DCtx_resetParameters(dctx); #ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION dctx->dictContentEndForFuzzing = NULL; #endif } ZSTD_DCtx* ZSTD_initStaticDCtx(void *workspace, size_t workspaceSize) { ZSTD_DCtx* const dctx = (ZSTD_DCtx*) workspace; if ((size_t)workspace & 7) return NULL; /* 8-aligned */ if (workspaceSize < sizeof(ZSTD_DCtx)) return NULL; /* minimum size */ ZSTD_initDCtx_internal(dctx); dctx->staticSize = workspaceSize; dctx->inBuff = (char*)(dctx+1); return dctx; } static ZSTD_DCtx* ZSTD_createDCtx_internal(ZSTD_customMem customMem) { if ((!customMem.customAlloc) ^ (!customMem.customFree)) return NULL; { ZSTD_DCtx* const dctx = (ZSTD_DCtx*)ZSTD_customMalloc(sizeof(*dctx), customMem); if (!dctx) return NULL; dctx->customMem = customMem; ZSTD_initDCtx_internal(dctx); return dctx; } } ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem) { return ZSTD_createDCtx_internal(customMem); } ZSTD_DCtx* ZSTD_createDCtx(void) { DEBUGLOG(3, "ZSTD_createDCtx"); return ZSTD_createDCtx_internal(ZSTD_defaultCMem); } static void ZSTD_clearDict(ZSTD_DCtx* dctx) { ZSTD_freeDDict(dctx->ddictLocal); dctx->ddictLocal = NULL; dctx->ddict = NULL; dctx->dictUses = ZSTD_dont_use; } size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx) { if (dctx==NULL) return 0; /* support free on NULL */ RETURN_ERROR_IF(dctx->staticSize, memory_allocation, "not compatible with static DCtx"); { ZSTD_customMem const cMem = dctx->customMem; ZSTD_clearDict(dctx); ZSTD_customFree(dctx->inBuff, cMem); dctx->inBuff = NULL; #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1) if (dctx->legacyContext) ZSTD_freeLegacyStreamContext(dctx->legacyContext, dctx->previousLegacyVersion); #endif if (dctx->ddictSet) { ZSTD_freeDDictHashSet(dctx->ddictSet, cMem); dctx->ddictSet = NULL; } ZSTD_customFree(dctx, cMem); return 0; } } /* no longer useful */ void ZSTD_copyDCtx(ZSTD_DCtx* dstDCtx, const ZSTD_DCtx* srcDCtx) { size_t const toCopy = (size_t)((char*)(&dstDCtx->inBuff) - (char*)dstDCtx); ZSTD_memcpy(dstDCtx, srcDCtx, toCopy); /* no need to copy workspace */ } /* Given a dctx with a digested frame params, re-selects the correct ZSTD_DDict based on * the requested dict ID from the frame. If there exists a reference to the correct ZSTD_DDict, then * accordingly sets the ddict to be used to decompress the frame. * * If no DDict is found, then no action is taken, and the ZSTD_DCtx::ddict remains as-is. * * ZSTD_d_refMultipleDDicts must be enabled for this function to be called. */ static void ZSTD_DCtx_selectFrameDDict(ZSTD_DCtx* dctx) { assert(dctx->refMultipleDDicts && dctx->ddictSet); DEBUGLOG(4, "Adjusting DDict based on requested dict ID from frame"); if (dctx->ddict) { const ZSTD_DDict* frameDDict = ZSTD_DDictHashSet_getDDict(dctx->ddictSet, dctx->fParams.dictID); if (frameDDict) { DEBUGLOG(4, "DDict found!"); ZSTD_clearDict(dctx); dctx->dictID = dctx->fParams.dictID; dctx->ddict = frameDDict; dctx->dictUses = ZSTD_use_indefinitely; } } } /*-************************************************************* * Frame header decoding ***************************************************************/ /*! ZSTD_isFrame() : * Tells if the content of `buffer` starts with a valid Frame Identifier. * Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0. * Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled. * Note 3 : Skippable Frame Identifiers are considered valid. */ unsigned ZSTD_isFrame(const void* buffer, size_t size) { if (size < ZSTD_FRAMEIDSIZE) return 0; { U32 const magic = MEM_readLE32(buffer); if (magic == ZSTD_MAGICNUMBER) return 1; if ((magic & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) return 1; } #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1) if (ZSTD_isLegacy(buffer, size)) return 1; #endif return 0; } /*! ZSTD_isSkippableFrame() : * Tells if the content of `buffer` starts with a valid Frame Identifier for a skippable frame. * Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0. */ unsigned ZSTD_isSkippableFrame(const void* buffer, size_t size) { if (size < ZSTD_FRAMEIDSIZE) return 0; { U32 const magic = MEM_readLE32(buffer); if ((magic & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) return 1; } return 0; } /** ZSTD_frameHeaderSize_internal() : * srcSize must be large enough to reach header size fields. * note : only works for formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless. * @return : size of the Frame Header * or an error code, which can be tested with ZSTD_isError() */ static size_t ZSTD_frameHeaderSize_internal(const void* src, size_t srcSize, ZSTD_format_e format) { size_t const minInputSize = ZSTD_startingInputLength(format); RETURN_ERROR_IF(srcSize < minInputSize, srcSize_wrong, ""); { BYTE const fhd = ((const BYTE*)src)[minInputSize-1]; U32 const dictID= fhd & 3; U32 const singleSegment = (fhd >> 5) & 1; U32 const fcsId = fhd >> 6; return minInputSize + !singleSegment + ZSTD_did_fieldSize[dictID] + ZSTD_fcs_fieldSize[fcsId] + (singleSegment && !fcsId); } } /** ZSTD_frameHeaderSize() : * srcSize must be >= ZSTD_frameHeaderSize_prefix. * @return : size of the Frame Header, * or an error code (if srcSize is too small) */ size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize) { return ZSTD_frameHeaderSize_internal(src, srcSize, ZSTD_f_zstd1); } /** ZSTD_getFrameHeader_advanced() : * decode Frame Header, or require larger `srcSize`. * note : only works for formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless * @return : 0, `zfhPtr` is correctly filled, * >0, `srcSize` is too small, value is wanted `srcSize` amount, ** or an error code, which can be tested using ZSTD_isError() */ size_t ZSTD_getFrameHeader_advanced(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize, ZSTD_format_e format) { const BYTE* ip = (const BYTE*)src; size_t const minInputSize = ZSTD_startingInputLength(format); DEBUGLOG(5, "ZSTD_getFrameHeader_advanced: minInputSize = %zu, srcSize = %zu", minInputSize, srcSize); if (srcSize > 0) { /* note : technically could be considered an assert(), since it's an invalid entry */ RETURN_ERROR_IF(src==NULL, GENERIC, "invalid parameter : src==NULL, but srcSize>0"); } if (srcSize < minInputSize) { if (srcSize > 0 && format != ZSTD_f_zstd1_magicless) { /* when receiving less than @minInputSize bytes, * control these bytes at least correspond to a supported magic number * in order to error out early if they don't. **/ size_t const toCopy = MIN(4, srcSize); unsigned char hbuf[4]; MEM_writeLE32(hbuf, ZSTD_MAGICNUMBER); assert(src != NULL); ZSTD_memcpy(hbuf, src, toCopy); if ( MEM_readLE32(hbuf) != ZSTD_MAGICNUMBER ) { /* not a zstd frame : let's check if it's a skippable frame */ MEM_writeLE32(hbuf, ZSTD_MAGIC_SKIPPABLE_START); ZSTD_memcpy(hbuf, src, toCopy); if ((MEM_readLE32(hbuf) & ZSTD_MAGIC_SKIPPABLE_MASK) != ZSTD_MAGIC_SKIPPABLE_START) { RETURN_ERROR(prefix_unknown, "first bytes don't correspond to any supported magic number"); } } } return minInputSize; } ZSTD_memset(zfhPtr, 0, sizeof(*zfhPtr)); /* not strictly necessary, but static analyzers may not understand that zfhPtr will be read only if return value is zero, since they are 2 different signals */ if ( (format != ZSTD_f_zstd1_magicless) && (MEM_readLE32(src) != ZSTD_MAGICNUMBER) ) { if ((MEM_readLE32(src) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) { /* skippable frame */ if (srcSize < ZSTD_SKIPPABLEHEADERSIZE) return ZSTD_SKIPPABLEHEADERSIZE; /* magic number + frame length */ ZSTD_memset(zfhPtr, 0, sizeof(*zfhPtr)); zfhPtr->frameContentSize = MEM_readLE32((const char *)src + ZSTD_FRAMEIDSIZE); zfhPtr->frameType = ZSTD_skippableFrame; return 0; } RETURN_ERROR(prefix_unknown, ""); } /* ensure there is enough `srcSize` to fully read/decode frame header */ { size_t const fhsize = ZSTD_frameHeaderSize_internal(src, srcSize, format); if (srcSize < fhsize) return fhsize; zfhPtr->headerSize = (U32)fhsize; } { BYTE const fhdByte = ip[minInputSize-1]; size_t pos = minInputSize; U32 const dictIDSizeCode = fhdByte&3; U32 const checksumFlag = (fhdByte>>2)&1; U32 const singleSegment = (fhdByte>>5)&1; U32 const fcsID = fhdByte>>6; U64 windowSize = 0; U32 dictID = 0; U64 frameContentSize = ZSTD_CONTENTSIZE_UNKNOWN; RETURN_ERROR_IF((fhdByte & 0x08) != 0, frameParameter_unsupported, "reserved bits, must be zero"); if (!singleSegment) { BYTE const wlByte = ip[pos++]; U32 const windowLog = (wlByte >> 3) + ZSTD_WINDOWLOG_ABSOLUTEMIN; RETURN_ERROR_IF(windowLog > ZSTD_WINDOWLOG_MAX, frameParameter_windowTooLarge, ""); windowSize = (1ULL << windowLog); windowSize += (windowSize >> 3) * (wlByte&7); } switch(dictIDSizeCode) { default: assert(0); /* impossible */ ZSTD_FALLTHROUGH; case 0 : break; case 1 : dictID = ip[pos]; pos++; break; case 2 : dictID = MEM_readLE16(ip+pos); pos+=2; break; case 3 : dictID = MEM_readLE32(ip+pos); pos+=4; break; } switch(fcsID) { default: assert(0); /* impossible */ ZSTD_FALLTHROUGH; case 0 : if (singleSegment) frameContentSize = ip[pos]; break; case 1 : frameContentSize = MEM_readLE16(ip+pos)+256; break; case 2 : frameContentSize = MEM_readLE32(ip+pos); break; case 3 : frameContentSize = MEM_readLE64(ip+pos); break; } if (singleSegment) windowSize = frameContentSize; zfhPtr->frameType = ZSTD_frame; zfhPtr->frameContentSize = frameContentSize; zfhPtr->windowSize = windowSize; zfhPtr->blockSizeMax = (unsigned) MIN(windowSize, ZSTD_BLOCKSIZE_MAX); zfhPtr->dictID = dictID; zfhPtr->checksumFlag = checksumFlag; } return 0; } /** ZSTD_getFrameHeader() : * decode Frame Header, or require larger `srcSize`. * note : this function does not consume input, it only reads it. * @return : 0, `zfhPtr` is correctly filled, * >0, `srcSize` is too small, value is wanted `srcSize` amount, * or an error code, which can be tested using ZSTD_isError() */ size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize) { return ZSTD_getFrameHeader_advanced(zfhPtr, src, srcSize, ZSTD_f_zstd1); } /** ZSTD_getFrameContentSize() : * compatible with legacy mode * @return : decompressed size of the single frame pointed to be `src` if known, otherwise * - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined * - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small) */ unsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize) { #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1) if (ZSTD_isLegacy(src, srcSize)) { unsigned long long const ret = ZSTD_getDecompressedSize_legacy(src, srcSize); return ret == 0 ? ZSTD_CONTENTSIZE_UNKNOWN : ret; } #endif { ZSTD_frameHeader zfh; if (ZSTD_getFrameHeader(&zfh, src, srcSize) != 0) return ZSTD_CONTENTSIZE_ERROR; if (zfh.frameType == ZSTD_skippableFrame) { return 0; } else { return zfh.frameContentSize; } } } static size_t readSkippableFrameSize(void const* src, size_t srcSize) { size_t const skippableHeaderSize = ZSTD_SKIPPABLEHEADERSIZE; U32 sizeU32; RETURN_ERROR_IF(srcSize < ZSTD_SKIPPABLEHEADERSIZE, srcSize_wrong, ""); sizeU32 = MEM_readLE32((BYTE const*)src + ZSTD_FRAMEIDSIZE); RETURN_ERROR_IF((U32)(sizeU32 + ZSTD_SKIPPABLEHEADERSIZE) < sizeU32, frameParameter_unsupported, ""); { size_t const skippableSize = skippableHeaderSize + sizeU32; RETURN_ERROR_IF(skippableSize > srcSize, srcSize_wrong, ""); return skippableSize; } } /*! ZSTD_readSkippableFrame() : * Retrieves content of a skippable frame, and writes it to dst buffer. * * The parameter magicVariant will receive the magicVariant that was supplied when the frame was written, * i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START. This can be NULL if the caller is not interested * in the magicVariant. * * Returns an error if destination buffer is not large enough, or if this is not a valid skippable frame. * * @return : number of bytes written or a ZSTD error. */ size_t ZSTD_readSkippableFrame(void* dst, size_t dstCapacity, unsigned* magicVariant, /* optional, can be NULL */ const void* src, size_t srcSize) { RETURN_ERROR_IF(srcSize < ZSTD_SKIPPABLEHEADERSIZE, srcSize_wrong, ""); { U32 const magicNumber = MEM_readLE32(src); size_t skippableFrameSize = readSkippableFrameSize(src, srcSize); size_t skippableContentSize = skippableFrameSize - ZSTD_SKIPPABLEHEADERSIZE; /* check input validity */ RETURN_ERROR_IF(!ZSTD_isSkippableFrame(src, srcSize), frameParameter_unsupported, ""); RETURN_ERROR_IF(skippableFrameSize < ZSTD_SKIPPABLEHEADERSIZE || skippableFrameSize > srcSize, srcSize_wrong, ""); RETURN_ERROR_IF(skippableContentSize > dstCapacity, dstSize_tooSmall, ""); /* deliver payload */ if (skippableContentSize > 0 && dst != NULL) ZSTD_memcpy(dst, (const BYTE *)src + ZSTD_SKIPPABLEHEADERSIZE, skippableContentSize); if (magicVariant != NULL) *magicVariant = magicNumber - ZSTD_MAGIC_SKIPPABLE_START; return skippableContentSize; } } /** ZSTD_findDecompressedSize() : * `srcSize` must be the exact length of some number of ZSTD compressed and/or * skippable frames * note: compatible with legacy mode * @return : decompressed size of the frames contained */ unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize) { unsigned long long totalDstSize = 0; while (srcSize >= ZSTD_startingInputLength(ZSTD_f_zstd1)) { U32 const magicNumber = MEM_readLE32(src); if ((magicNumber & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) { size_t const skippableSize = readSkippableFrameSize(src, srcSize); if (ZSTD_isError(skippableSize)) return ZSTD_CONTENTSIZE_ERROR; assert(skippableSize <= srcSize); src = (const BYTE *)src + skippableSize; srcSize -= skippableSize; continue; } { unsigned long long const fcs = ZSTD_getFrameContentSize(src, srcSize); if (fcs >= ZSTD_CONTENTSIZE_ERROR) return fcs; if (totalDstSize + fcs < totalDstSize) return ZSTD_CONTENTSIZE_ERROR; /* check for overflow */ totalDstSize += fcs; } /* skip to next frame */ { size_t const frameSrcSize = ZSTD_findFrameCompressedSize(src, srcSize); if (ZSTD_isError(frameSrcSize)) return ZSTD_CONTENTSIZE_ERROR; assert(frameSrcSize <= srcSize); src = (const BYTE *)src + frameSrcSize; srcSize -= frameSrcSize; } } /* while (srcSize >= ZSTD_frameHeaderSize_prefix) */ if (srcSize) return ZSTD_CONTENTSIZE_ERROR; return totalDstSize; } /** ZSTD_getDecompressedSize() : * compatible with legacy mode * @return : decompressed size if known, 0 otherwise note : 0 can mean any of the following : - frame content is empty - decompressed size field is not present in frame header - frame header unknown / not supported - frame header not complete (`srcSize` too small) */ unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize) { unsigned long long const ret = ZSTD_getFrameContentSize(src, srcSize); ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_ERROR < ZSTD_CONTENTSIZE_UNKNOWN); return (ret >= ZSTD_CONTENTSIZE_ERROR) ? 0 : ret; } /** ZSTD_decodeFrameHeader() : * `headerSize` must be the size provided by ZSTD_frameHeaderSize(). * If multiple DDict references are enabled, also will choose the correct DDict to use. * @return : 0 if success, or an error code, which can be tested using ZSTD_isError() */ static size_t ZSTD_decodeFrameHeader(ZSTD_DCtx* dctx, const void* src, size_t headerSize) { size_t const result = ZSTD_getFrameHeader_advanced(&(dctx->fParams), src, headerSize, dctx->format); if (ZSTD_isError(result)) return result; /* invalid header */ RETURN_ERROR_IF(result>0, srcSize_wrong, "headerSize too small"); /* Reference DDict requested by frame if dctx references multiple ddicts */ if (dctx->refMultipleDDicts == ZSTD_rmd_refMultipleDDicts && dctx->ddictSet) { ZSTD_DCtx_selectFrameDDict(dctx); } #ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION /* Skip the dictID check in fuzzing mode, because it makes the search * harder. */ RETURN_ERROR_IF(dctx->fParams.dictID && (dctx->dictID != dctx->fParams.dictID), dictionary_wrong, ""); #endif dctx->validateChecksum = (dctx->fParams.checksumFlag && !dctx->forceIgnoreChecksum) ? 1 : 0; if (dctx->validateChecksum) XXH64_reset(&dctx->xxhState, 0); dctx->processedCSize += headerSize; return 0; } static ZSTD_frameSizeInfo ZSTD_errorFrameSizeInfo(size_t ret) { ZSTD_frameSizeInfo frameSizeInfo; frameSizeInfo.compressedSize = ret; frameSizeInfo.decompressedBound = ZSTD_CONTENTSIZE_ERROR; return frameSizeInfo; } static ZSTD_frameSizeInfo ZSTD_findFrameSizeInfo(const void* src, size_t srcSize) { ZSTD_frameSizeInfo frameSizeInfo; ZSTD_memset(&frameSizeInfo, 0, sizeof(ZSTD_frameSizeInfo)); #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1) if (ZSTD_isLegacy(src, srcSize)) return ZSTD_findFrameSizeInfoLegacy(src, srcSize); #endif if ((srcSize >= ZSTD_SKIPPABLEHEADERSIZE) && (MEM_readLE32(src) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) { frameSizeInfo.compressedSize = readSkippableFrameSize(src, srcSize); assert(ZSTD_isError(frameSizeInfo.compressedSize) || frameSizeInfo.compressedSize <= srcSize); return frameSizeInfo; } else { const BYTE* ip = (const BYTE*)src; const BYTE* const ipstart = ip; size_t remainingSize = srcSize; size_t nbBlocks = 0; ZSTD_frameHeader zfh; /* Extract Frame Header */ { size_t const ret = ZSTD_getFrameHeader(&zfh, src, srcSize); if (ZSTD_isError(ret)) return ZSTD_errorFrameSizeInfo(ret); if (ret > 0) return ZSTD_errorFrameSizeInfo(ERROR(srcSize_wrong)); } ip += zfh.headerSize; remainingSize -= zfh.headerSize; /* Iterate over each block */ while (1) { blockProperties_t blockProperties; size_t const cBlockSize = ZSTD_getcBlockSize(ip, remainingSize, &blockProperties); if (ZSTD_isError(cBlockSize)) return ZSTD_errorFrameSizeInfo(cBlockSize); if (ZSTD_blockHeaderSize + cBlockSize > remainingSize) return ZSTD_errorFrameSizeInfo(ERROR(srcSize_wrong)); ip += ZSTD_blockHeaderSize + cBlockSize; remainingSize -= ZSTD_blockHeaderSize + cBlockSize; nbBlocks++; if (blockProperties.lastBlock) break; } /* Final frame content checksum */ if (zfh.checksumFlag) { if (remainingSize < 4) return ZSTD_errorFrameSizeInfo(ERROR(srcSize_wrong)); ip += 4; } frameSizeInfo.nbBlocks = nbBlocks; frameSizeInfo.compressedSize = (size_t)(ip - ipstart); frameSizeInfo.decompressedBound = (zfh.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN) ? zfh.frameContentSize : (unsigned long long)nbBlocks * zfh.blockSizeMax; return frameSizeInfo; } } /** ZSTD_findFrameCompressedSize() : * compatible with legacy mode * `src` must point to the start of a ZSTD frame, ZSTD legacy frame, or skippable frame * `srcSize` must be at least as large as the frame contained * @return : the compressed size of the frame starting at `src` */ size_t ZSTD_findFrameCompressedSize(const void *src, size_t srcSize) { ZSTD_frameSizeInfo const frameSizeInfo = ZSTD_findFrameSizeInfo(src, srcSize); return frameSizeInfo.compressedSize; } /** ZSTD_decompressBound() : * compatible with legacy mode * `src` must point to the start of a ZSTD frame or a skippeable frame * `srcSize` must be at least as large as the frame contained * @return : the maximum decompressed size of the compressed source */ unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize) { unsigned long long bound = 0; /* Iterate over each frame */ while (srcSize > 0) { ZSTD_frameSizeInfo const frameSizeInfo = ZSTD_findFrameSizeInfo(src, srcSize); size_t const compressedSize = frameSizeInfo.compressedSize; unsigned long long const decompressedBound = frameSizeInfo.decompressedBound; if (ZSTD_isError(compressedSize) || decompressedBound == ZSTD_CONTENTSIZE_ERROR) return ZSTD_CONTENTSIZE_ERROR; assert(srcSize >= compressedSize); src = (const BYTE*)src + compressedSize; srcSize -= compressedSize; bound += decompressedBound; } return bound; } size_t ZSTD_decompressionMargin(void const* src, size_t srcSize) { size_t margin = 0; unsigned maxBlockSize = 0; /* Iterate over each frame */ while (srcSize > 0) { ZSTD_frameSizeInfo const frameSizeInfo = ZSTD_findFrameSizeInfo(src, srcSize); size_t const compressedSize = frameSizeInfo.compressedSize; unsigned long long const decompressedBound = frameSizeInfo.decompressedBound; ZSTD_frameHeader zfh; FORWARD_IF_ERROR(ZSTD_getFrameHeader(&zfh, src, srcSize), ""); if (ZSTD_isError(compressedSize) || decompressedBound == ZSTD_CONTENTSIZE_ERROR) return ERROR(corruption_detected); if (zfh.frameType == ZSTD_frame) { /* Add the frame header to our margin */ margin += zfh.headerSize; /* Add the checksum to our margin */ margin += zfh.checksumFlag ? 4 : 0; /* Add 3 bytes per block */ margin += 3 * frameSizeInfo.nbBlocks; /* Compute the max block size */ maxBlockSize = MAX(maxBlockSize, zfh.blockSizeMax); } else { assert(zfh.frameType == ZSTD_skippableFrame); /* Add the entire skippable frame size to our margin. */ margin += compressedSize; } assert(srcSize >= compressedSize); src = (const BYTE*)src + compressedSize; srcSize -= compressedSize; } /* Add the max block size back to the margin. */ margin += maxBlockSize; return margin; } /*-************************************************************* * Frame decoding ***************************************************************/ /** ZSTD_insertBlock() : * insert `src` block into `dctx` history. Useful to track uncompressed blocks. */ size_t ZSTD_insertBlock(ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize) { DEBUGLOG(5, "ZSTD_insertBlock: %u bytes", (unsigned)blockSize); ZSTD_checkContinuity(dctx, blockStart, blockSize); dctx->previousDstEnd = (const char*)blockStart + blockSize; return blockSize; } static size_t ZSTD_copyRawBlock(void* dst, size_t dstCapacity, const void* src, size_t srcSize) { DEBUGLOG(5, "ZSTD_copyRawBlock"); RETURN_ERROR_IF(srcSize > dstCapacity, dstSize_tooSmall, ""); if (dst == NULL) { if (srcSize == 0) return 0; RETURN_ERROR(dstBuffer_null, ""); } ZSTD_memmove(dst, src, srcSize); return srcSize; } static size_t ZSTD_setRleBlock(void* dst, size_t dstCapacity, BYTE b, size_t regenSize) { RETURN_ERROR_IF(regenSize > dstCapacity, dstSize_tooSmall, ""); if (dst == NULL) { if (regenSize == 0) return 0; RETURN_ERROR(dstBuffer_null, ""); } ZSTD_memset(dst, b, regenSize); return regenSize; } static void ZSTD_DCtx_trace_end(ZSTD_DCtx const* dctx, U64 uncompressedSize, U64 compressedSize, unsigned streaming) { #if ZSTD_TRACE if (dctx->traceCtx && ZSTD_trace_decompress_end != NULL) { ZSTD_Trace trace; ZSTD_memset(&trace, 0, sizeof(trace)); trace.version = ZSTD_VERSION_NUMBER; trace.streaming = streaming; if (dctx->ddict) { trace.dictionaryID = ZSTD_getDictID_fromDDict(dctx->ddict); trace.dictionarySize = ZSTD_DDict_dictSize(dctx->ddict); trace.dictionaryIsCold = dctx->ddictIsCold; } trace.uncompressedSize = (size_t)uncompressedSize; trace.compressedSize = (size_t)compressedSize; trace.dctx = dctx; ZSTD_trace_decompress_end(dctx->traceCtx, &trace); } #else (void)dctx; (void)uncompressedSize; (void)compressedSize; (void)streaming; #endif } /*! ZSTD_decompressFrame() : * @dctx must be properly initialized * will update *srcPtr and *srcSizePtr, * to make *srcPtr progress by one frame. */ static size_t ZSTD_decompressFrame(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void** srcPtr, size_t *srcSizePtr) { const BYTE* const istart = (const BYTE*)(*srcPtr); const BYTE* ip = istart; BYTE* const ostart = (BYTE*)dst; BYTE* const oend = dstCapacity != 0 ? ostart + dstCapacity : ostart; BYTE* op = ostart; size_t remainingSrcSize = *srcSizePtr; DEBUGLOG(4, "ZSTD_decompressFrame (srcSize:%i)", (int)*srcSizePtr); /* check */ RETURN_ERROR_IF( remainingSrcSize < ZSTD_FRAMEHEADERSIZE_MIN(dctx->format)+ZSTD_blockHeaderSize, srcSize_wrong, ""); /* Frame Header */ { size_t const frameHeaderSize = ZSTD_frameHeaderSize_internal( ip, ZSTD_FRAMEHEADERSIZE_PREFIX(dctx->format), dctx->format); if (ZSTD_isError(frameHeaderSize)) return frameHeaderSize; RETURN_ERROR_IF(remainingSrcSize < frameHeaderSize+ZSTD_blockHeaderSize, srcSize_wrong, ""); FORWARD_IF_ERROR( ZSTD_decodeFrameHeader(dctx, ip, frameHeaderSize) , ""); ip += frameHeaderSize; remainingSrcSize -= frameHeaderSize; } /* Loop on each block */ while (1) { BYTE* oBlockEnd = oend; size_t decodedSize; blockProperties_t blockProperties; size_t const cBlockSize = ZSTD_getcBlockSize(ip, remainingSrcSize, &blockProperties); if (ZSTD_isError(cBlockSize)) return cBlockSize; ip += ZSTD_blockHeaderSize; remainingSrcSize -= ZSTD_blockHeaderSize; RETURN_ERROR_IF(cBlockSize > remainingSrcSize, srcSize_wrong, ""); if (ip >= op && ip < oBlockEnd) { /* We are decompressing in-place. Limit the output pointer so that we * don't overwrite the block that we are currently reading. This will * fail decompression if the input & output pointers aren't spaced * far enough apart. * * This is important to set, even when the pointers are far enough * apart, because ZSTD_decompressBlock_internal() can decide to store * literals in the output buffer, after the block it is decompressing. * Since we don't want anything to overwrite our input, we have to tell * ZSTD_decompressBlock_internal to never write past ip. * * See ZSTD_allocateLiteralsBuffer() for reference. */ oBlockEnd = op + (ip - op); } switch(blockProperties.blockType) { case bt_compressed: decodedSize = ZSTD_decompressBlock_internal(dctx, op, (size_t)(oBlockEnd-op), ip, cBlockSize, /* frame */ 1, not_streaming); break; case bt_raw : /* Use oend instead of oBlockEnd because this function is safe to overlap. It uses memmove. */ decodedSize = ZSTD_copyRawBlock(op, (size_t)(oend-op), ip, cBlockSize); break; case bt_rle : decodedSize = ZSTD_setRleBlock(op, (size_t)(oBlockEnd-op), *ip, blockProperties.origSize); break; case bt_reserved : default: RETURN_ERROR(corruption_detected, "invalid block type"); } if (ZSTD_isError(decodedSize)) return decodedSize; if (dctx->validateChecksum) XXH64_update(&dctx->xxhState, op, decodedSize); if (decodedSize != 0) op += decodedSize; assert(ip != NULL); ip += cBlockSize; remainingSrcSize -= cBlockSize; if (blockProperties.lastBlock) break; } if (dctx->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN) { RETURN_ERROR_IF((U64)(op-ostart) != dctx->fParams.frameContentSize, corruption_detected, ""); } if (dctx->fParams.checksumFlag) { /* Frame content checksum verification */ RETURN_ERROR_IF(remainingSrcSize<4, checksum_wrong, ""); if (!dctx->forceIgnoreChecksum) { U32 const checkCalc = (U32)XXH64_digest(&dctx->xxhState); U32 checkRead; checkRead = MEM_readLE32(ip); RETURN_ERROR_IF(checkRead != checkCalc, checksum_wrong, ""); } ip += 4; remainingSrcSize -= 4; } ZSTD_DCtx_trace_end(dctx, (U64)(op-ostart), (U64)(ip-istart), /* streaming */ 0); /* Allow caller to get size read */ DEBUGLOG(4, "ZSTD_decompressFrame: decompressed frame of size %zi, consuming %zi bytes of input", op-ostart, ip - (const BYTE*)*srcPtr); *srcPtr = ip; *srcSizePtr = remainingSrcSize; return (size_t)(op-ostart); } static size_t ZSTD_decompressMultiFrame(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const void* dict, size_t dictSize, const ZSTD_DDict* ddict) { void* const dststart = dst; int moreThan1Frame = 0; DEBUGLOG(5, "ZSTD_decompressMultiFrame"); assert(dict==NULL || ddict==NULL); /* either dict or ddict set, not both */ if (ddict) { dict = ZSTD_DDict_dictContent(ddict); dictSize = ZSTD_DDict_dictSize(ddict); } while (srcSize >= ZSTD_startingInputLength(dctx->format)) { #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1) if (ZSTD_isLegacy(src, srcSize)) { size_t decodedSize; size_t const frameSize = ZSTD_findFrameCompressedSizeLegacy(src, srcSize); if (ZSTD_isError(frameSize)) return frameSize; RETURN_ERROR_IF(dctx->staticSize, memory_allocation, "legacy support is not compatible with static dctx"); decodedSize = ZSTD_decompressLegacy(dst, dstCapacity, src, frameSize, dict, dictSize); if (ZSTD_isError(decodedSize)) return decodedSize; assert(decodedSize <= dstCapacity); dst = (BYTE*)dst + decodedSize; dstCapacity -= decodedSize; src = (const BYTE*)src + frameSize; srcSize -= frameSize; continue; } #endif if (srcSize >= 4) { U32 const magicNumber = MEM_readLE32(src); DEBUGLOG(5, "reading magic number %08X", (unsigned)magicNumber); if ((magicNumber & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) { /* skippable frame detected : skip it */ size_t const skippableSize = readSkippableFrameSize(src, srcSize); FORWARD_IF_ERROR(skippableSize, "invalid skippable frame"); assert(skippableSize <= srcSize); src = (const BYTE *)src + skippableSize; srcSize -= skippableSize; continue; /* check next frame */ } } if (ddict) { /* we were called from ZSTD_decompress_usingDDict */ FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDDict(dctx, ddict), ""); } else { /* this will initialize correctly with no dict if dict == NULL, so * use this in all cases but ddict */ FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDict(dctx, dict, dictSize), ""); } ZSTD_checkContinuity(dctx, dst, dstCapacity); { const size_t res = ZSTD_decompressFrame(dctx, dst, dstCapacity, &src, &srcSize); RETURN_ERROR_IF( (ZSTD_getErrorCode(res) == ZSTD_error_prefix_unknown) && (moreThan1Frame==1), srcSize_wrong, "At least one frame successfully completed, " "but following bytes are garbage: " "it's more likely to be a srcSize error, " "specifying more input bytes than size of frame(s). " "Note: one could be unlucky, it might be a corruption error instead, " "happening right at the place where we expect zstd magic bytes. " "But this is _much_ less likely than a srcSize field error."); if (ZSTD_isError(res)) return res; assert(res <= dstCapacity); if (res != 0) dst = (BYTE*)dst + res; dstCapacity -= res; } moreThan1Frame = 1; } /* while (srcSize >= ZSTD_frameHeaderSize_prefix) */ RETURN_ERROR_IF(srcSize, srcSize_wrong, "input not entirely consumed"); return (size_t)((BYTE*)dst - (BYTE*)dststart); } size_t ZSTD_decompress_usingDict(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const void* dict, size_t dictSize) { return ZSTD_decompressMultiFrame(dctx, dst, dstCapacity, src, srcSize, dict, dictSize, NULL); } static ZSTD_DDict const* ZSTD_getDDict(ZSTD_DCtx* dctx) { switch (dctx->dictUses) { default: assert(0 /* Impossible */); ZSTD_FALLTHROUGH; case ZSTD_dont_use: ZSTD_clearDict(dctx); return NULL; case ZSTD_use_indefinitely: return dctx->ddict; case ZSTD_use_once: dctx->dictUses = ZSTD_dont_use; return dctx->ddict; } } size_t ZSTD_decompressDCtx(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize) { return ZSTD_decompress_usingDDict(dctx, dst, dstCapacity, src, srcSize, ZSTD_getDDict(dctx)); } size_t ZSTD_decompress(void* dst, size_t dstCapacity, const void* src, size_t srcSize) { #if defined(ZSTD_HEAPMODE) && (ZSTD_HEAPMODE>=1) size_t regenSize; ZSTD_DCtx* const dctx = ZSTD_createDCtx_internal(ZSTD_defaultCMem); RETURN_ERROR_IF(dctx==NULL, memory_allocation, "NULL pointer!"); regenSize = ZSTD_decompressDCtx(dctx, dst, dstCapacity, src, srcSize); ZSTD_freeDCtx(dctx); return regenSize; #else /* stack mode */ ZSTD_DCtx dctx; ZSTD_initDCtx_internal(&dctx); return ZSTD_decompressDCtx(&dctx, dst, dstCapacity, src, srcSize); #endif } /*-************************************** * Advanced Streaming Decompression API * Bufferless and synchronous ****************************************/ size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx* dctx) { return dctx->expected; } /** * Similar to ZSTD_nextSrcSizeToDecompress(), but when a block input can be streamed, we * allow taking a partial block as the input. Currently only raw uncompressed blocks can * be streamed. * * For blocks that can be streamed, this allows us to reduce the latency until we produce * output, and avoid copying the input. * * @param inputSize - The total amount of input that the caller currently has. */ static size_t ZSTD_nextSrcSizeToDecompressWithInputSize(ZSTD_DCtx* dctx, size_t inputSize) { if (!(dctx->stage == ZSTDds_decompressBlock || dctx->stage == ZSTDds_decompressLastBlock)) return dctx->expected; if (dctx->bType != bt_raw) return dctx->expected; return BOUNDED(1, inputSize, dctx->expected); } ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx) { switch(dctx->stage) { default: /* should not happen */ assert(0); ZSTD_FALLTHROUGH; case ZSTDds_getFrameHeaderSize: ZSTD_FALLTHROUGH; case ZSTDds_decodeFrameHeader: return ZSTDnit_frameHeader; case ZSTDds_decodeBlockHeader: return ZSTDnit_blockHeader; case ZSTDds_decompressBlock: return ZSTDnit_block; case ZSTDds_decompressLastBlock: return ZSTDnit_lastBlock; case ZSTDds_checkChecksum: return ZSTDnit_checksum; case ZSTDds_decodeSkippableHeader: ZSTD_FALLTHROUGH; case ZSTDds_skipFrame: return ZSTDnit_skippableFrame; } } static int ZSTD_isSkipFrame(ZSTD_DCtx* dctx) { return dctx->stage == ZSTDds_skipFrame; } /** ZSTD_decompressContinue() : * srcSize : must be the exact nb of bytes expected (see ZSTD_nextSrcSizeToDecompress()) * @return : nb of bytes generated into `dst` (necessarily <= `dstCapacity) * or an error code, which can be tested using ZSTD_isError() */ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize) { DEBUGLOG(5, "ZSTD_decompressContinue (srcSize:%u)", (unsigned)srcSize); /* Sanity check */ RETURN_ERROR_IF(srcSize != ZSTD_nextSrcSizeToDecompressWithInputSize(dctx, srcSize), srcSize_wrong, "not allowed"); ZSTD_checkContinuity(dctx, dst, dstCapacity); dctx->processedCSize += srcSize; switch (dctx->stage) { case ZSTDds_getFrameHeaderSize : assert(src != NULL); if (dctx->format == ZSTD_f_zstd1) { /* allows header */ assert(srcSize >= ZSTD_FRAMEIDSIZE); /* to read skippable magic number */ if ((MEM_readLE32(src) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) { /* skippable frame */ ZSTD_memcpy(dctx->headerBuffer, src, srcSize); dctx->expected = ZSTD_SKIPPABLEHEADERSIZE - srcSize; /* remaining to load to get full skippable frame header */ dctx->stage = ZSTDds_decodeSkippableHeader; return 0; } } dctx->headerSize = ZSTD_frameHeaderSize_internal(src, srcSize, dctx->format); if (ZSTD_isError(dctx->headerSize)) return dctx->headerSize; ZSTD_memcpy(dctx->headerBuffer, src, srcSize); dctx->expected = dctx->headerSize - srcSize; dctx->stage = ZSTDds_decodeFrameHeader; return 0; case ZSTDds_decodeFrameHeader: assert(src != NULL); ZSTD_memcpy(dctx->headerBuffer + (dctx->headerSize - srcSize), src, srcSize); FORWARD_IF_ERROR(ZSTD_decodeFrameHeader(dctx, dctx->headerBuffer, dctx->headerSize), ""); dctx->expected = ZSTD_blockHeaderSize; dctx->stage = ZSTDds_decodeBlockHeader; return 0; case ZSTDds_decodeBlockHeader: { blockProperties_t bp; size_t const cBlockSize = ZSTD_getcBlockSize(src, ZSTD_blockHeaderSize, &bp); if (ZSTD_isError(cBlockSize)) return cBlockSize; RETURN_ERROR_IF(cBlockSize > dctx->fParams.blockSizeMax, corruption_detected, "Block Size Exceeds Maximum"); dctx->expected = cBlockSize; dctx->bType = bp.blockType; dctx->rleSize = bp.origSize; if (cBlockSize) { dctx->stage = bp.lastBlock ? ZSTDds_decompressLastBlock : ZSTDds_decompressBlock; return 0; } /* empty block */ if (bp.lastBlock) { if (dctx->fParams.checksumFlag) { dctx->expected = 4; dctx->stage = ZSTDds_checkChecksum; } else { dctx->expected = 0; /* end of frame */ dctx->stage = ZSTDds_getFrameHeaderSize; } } else { dctx->expected = ZSTD_blockHeaderSize; /* jump to next header */ dctx->stage = ZSTDds_decodeBlockHeader; } return 0; } case ZSTDds_decompressLastBlock: case ZSTDds_decompressBlock: DEBUGLOG(5, "ZSTD_decompressContinue: case ZSTDds_decompressBlock"); { size_t rSize; switch(dctx->bType) { case bt_compressed: DEBUGLOG(5, "ZSTD_decompressContinue: case bt_compressed"); rSize = ZSTD_decompressBlock_internal(dctx, dst, dstCapacity, src, srcSize, /* frame */ 1, is_streaming); dctx->expected = 0; /* Streaming not supported */ break; case bt_raw : assert(srcSize <= dctx->expected); rSize = ZSTD_copyRawBlock(dst, dstCapacity, src, srcSize); FORWARD_IF_ERROR(rSize, "ZSTD_copyRawBlock failed"); assert(rSize == srcSize); dctx->expected -= rSize; break; case bt_rle : rSize = ZSTD_setRleBlock(dst, dstCapacity, *(const BYTE*)src, dctx->rleSize); dctx->expected = 0; /* Streaming not supported */ break; case bt_reserved : /* should never happen */ default: RETURN_ERROR(corruption_detected, "invalid block type"); } FORWARD_IF_ERROR(rSize, ""); RETURN_ERROR_IF(rSize > dctx->fParams.blockSizeMax, corruption_detected, "Decompressed Block Size Exceeds Maximum"); DEBUGLOG(5, "ZSTD_decompressContinue: decoded size from block : %u", (unsigned)rSize); dctx->decodedSize += rSize; if (dctx->validateChecksum) XXH64_update(&dctx->xxhState, dst, rSize); dctx->previousDstEnd = (char*)dst + rSize; /* Stay on the same stage until we are finished streaming the block. */ if (dctx->expected > 0) { return rSize; } if (dctx->stage == ZSTDds_decompressLastBlock) { /* end of frame */ DEBUGLOG(4, "ZSTD_decompressContinue: decoded size from frame : %u", (unsigned)dctx->decodedSize); RETURN_ERROR_IF( dctx->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN && dctx->decodedSize != dctx->fParams.frameContentSize, corruption_detected, ""); if (dctx->fParams.checksumFlag) { /* another round for frame checksum */ dctx->expected = 4; dctx->stage = ZSTDds_checkChecksum; } else { ZSTD_DCtx_trace_end(dctx, dctx->decodedSize, dctx->processedCSize, /* streaming */ 1); dctx->expected = 0; /* ends here */ dctx->stage = ZSTDds_getFrameHeaderSize; } } else { dctx->stage = ZSTDds_decodeBlockHeader; dctx->expected = ZSTD_blockHeaderSize; } return rSize; } case ZSTDds_checkChecksum: assert(srcSize == 4); /* guaranteed by dctx->expected */ { if (dctx->validateChecksum) { U32 const h32 = (U32)XXH64_digest(&dctx->xxhState); U32 const check32 = MEM_readLE32(src); DEBUGLOG(4, "ZSTD_decompressContinue: checksum : calculated %08X :: %08X read", (unsigned)h32, (unsigned)check32); RETURN_ERROR_IF(check32 != h32, checksum_wrong, ""); } ZSTD_DCtx_trace_end(dctx, dctx->decodedSize, dctx->processedCSize, /* streaming */ 1); dctx->expected = 0; dctx->stage = ZSTDds_getFrameHeaderSize; return 0; } case ZSTDds_decodeSkippableHeader: assert(src != NULL); assert(srcSize <= ZSTD_SKIPPABLEHEADERSIZE); ZSTD_memcpy(dctx->headerBuffer + (ZSTD_SKIPPABLEHEADERSIZE - srcSize), src, srcSize); /* complete skippable header */ dctx->expected = MEM_readLE32(dctx->headerBuffer + ZSTD_FRAMEIDSIZE); /* note : dctx->expected can grow seriously large, beyond local buffer size */ dctx->stage = ZSTDds_skipFrame; return 0; case ZSTDds_skipFrame: dctx->expected = 0; dctx->stage = ZSTDds_getFrameHeaderSize; return 0; default: assert(0); /* impossible */ RETURN_ERROR(GENERIC, "impossible to reach"); /* some compilers require default to do something */ } } static size_t ZSTD_refDictContent(ZSTD_DCtx* dctx, const void* dict, size_t dictSize) { dctx->dictEnd = dctx->previousDstEnd; dctx->virtualStart = (const char*)dict - ((const char*)(dctx->previousDstEnd) - (const char*)(dctx->prefixStart)); dctx->prefixStart = dict; dctx->previousDstEnd = (const char*)dict + dictSize; #ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION dctx->dictContentBeginForFuzzing = dctx->prefixStart; dctx->dictContentEndForFuzzing = dctx->previousDstEnd; #endif return 0; } /*! ZSTD_loadDEntropy() : * dict : must point at beginning of a valid zstd dictionary. * @return : size of entropy tables read */ size_t ZSTD_loadDEntropy(ZSTD_entropyDTables_t* entropy, const void* const dict, size_t const dictSize) { const BYTE* dictPtr = (const BYTE*)dict; const BYTE* const dictEnd = dictPtr + dictSize; RETURN_ERROR_IF(dictSize <= 8, dictionary_corrupted, "dict is too small"); assert(MEM_readLE32(dict) == ZSTD_MAGIC_DICTIONARY); /* dict must be valid */ dictPtr += 8; /* skip header = magic + dictID */ ZSTD_STATIC_ASSERT(offsetof(ZSTD_entropyDTables_t, OFTable) == offsetof(ZSTD_entropyDTables_t, LLTable) + sizeof(entropy->LLTable)); ZSTD_STATIC_ASSERT(offsetof(ZSTD_entropyDTables_t, MLTable) == offsetof(ZSTD_entropyDTables_t, OFTable) + sizeof(entropy->OFTable)); ZSTD_STATIC_ASSERT(sizeof(entropy->LLTable) + sizeof(entropy->OFTable) + sizeof(entropy->MLTable) >= HUF_DECOMPRESS_WORKSPACE_SIZE); { void* const workspace = &entropy->LLTable; /* use fse tables as temporary workspace; implies fse tables are grouped together */ size_t const workspaceSize = sizeof(entropy->LLTable) + sizeof(entropy->OFTable) + sizeof(entropy->MLTable); #ifdef HUF_FORCE_DECOMPRESS_X1 /* in minimal huffman, we always use X1 variants */ size_t const hSize = HUF_readDTableX1_wksp(entropy->hufTable, dictPtr, dictEnd - dictPtr, workspace, workspaceSize, /* flags */ 0); #else size_t const hSize = HUF_readDTableX2_wksp(entropy->hufTable, dictPtr, (size_t)(dictEnd - dictPtr), workspace, workspaceSize, /* flags */ 0); #endif RETURN_ERROR_IF(HUF_isError(hSize), dictionary_corrupted, ""); dictPtr += hSize; } { short offcodeNCount[MaxOff+1]; unsigned offcodeMaxValue = MaxOff, offcodeLog; size_t const offcodeHeaderSize = FSE_readNCount(offcodeNCount, &offcodeMaxValue, &offcodeLog, dictPtr, (size_t)(dictEnd-dictPtr)); RETURN_ERROR_IF(FSE_isError(offcodeHeaderSize), dictionary_corrupted, ""); RETURN_ERROR_IF(offcodeMaxValue > MaxOff, dictionary_corrupted, ""); RETURN_ERROR_IF(offcodeLog > OffFSELog, dictionary_corrupted, ""); ZSTD_buildFSETable( entropy->OFTable, offcodeNCount, offcodeMaxValue, OF_base, OF_bits, offcodeLog, entropy->workspace, sizeof(entropy->workspace), /* bmi2 */0); dictPtr += offcodeHeaderSize; } { short matchlengthNCount[MaxML+1]; unsigned matchlengthMaxValue = MaxML, matchlengthLog; size_t const matchlengthHeaderSize = FSE_readNCount(matchlengthNCount, &matchlengthMaxValue, &matchlengthLog, dictPtr, (size_t)(dictEnd-dictPtr)); RETURN_ERROR_IF(FSE_isError(matchlengthHeaderSize), dictionary_corrupted, ""); RETURN_ERROR_IF(matchlengthMaxValue > MaxML, dictionary_corrupted, ""); RETURN_ERROR_IF(matchlengthLog > MLFSELog, dictionary_corrupted, ""); ZSTD_buildFSETable( entropy->MLTable, matchlengthNCount, matchlengthMaxValue, ML_base, ML_bits, matchlengthLog, entropy->workspace, sizeof(entropy->workspace), /* bmi2 */ 0); dictPtr += matchlengthHeaderSize; } { short litlengthNCount[MaxLL+1]; unsigned litlengthMaxValue = MaxLL, litlengthLog; size_t const litlengthHeaderSize = FSE_readNCount(litlengthNCount, &litlengthMaxValue, &litlengthLog, dictPtr, (size_t)(dictEnd-dictPtr)); RETURN_ERROR_IF(FSE_isError(litlengthHeaderSize), dictionary_corrupted, ""); RETURN_ERROR_IF(litlengthMaxValue > MaxLL, dictionary_corrupted, ""); RETURN_ERROR_IF(litlengthLog > LLFSELog, dictionary_corrupted, ""); ZSTD_buildFSETable( entropy->LLTable, litlengthNCount, litlengthMaxValue, LL_base, LL_bits, litlengthLog, entropy->workspace, sizeof(entropy->workspace), /* bmi2 */ 0); dictPtr += litlengthHeaderSize; } RETURN_ERROR_IF(dictPtr+12 > dictEnd, dictionary_corrupted, ""); { int i; size_t const dictContentSize = (size_t)(dictEnd - (dictPtr+12)); for (i=0; i<3; i++) { U32 const rep = MEM_readLE32(dictPtr); dictPtr += 4; RETURN_ERROR_IF(rep==0 || rep > dictContentSize, dictionary_corrupted, ""); entropy->rep[i] = rep; } } return (size_t)(dictPtr - (const BYTE*)dict); } static size_t ZSTD_decompress_insertDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize) { if (dictSize < 8) return ZSTD_refDictContent(dctx, dict, dictSize); { U32 const magic = MEM_readLE32(dict); if (magic != ZSTD_MAGIC_DICTIONARY) { return ZSTD_refDictContent(dctx, dict, dictSize); /* pure content mode */ } } dctx->dictID = MEM_readLE32((const char*)dict + ZSTD_FRAMEIDSIZE); /* load entropy tables */ { size_t const eSize = ZSTD_loadDEntropy(&dctx->entropy, dict, dictSize); RETURN_ERROR_IF(ZSTD_isError(eSize), dictionary_corrupted, ""); dict = (const char*)dict + eSize; dictSize -= eSize; } dctx->litEntropy = dctx->fseEntropy = 1; /* reference dictionary content */ return ZSTD_refDictContent(dctx, dict, dictSize); } size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx) { assert(dctx != NULL); #if ZSTD_TRACE dctx->traceCtx = (ZSTD_trace_decompress_begin != NULL) ? ZSTD_trace_decompress_begin(dctx) : 0; #endif dctx->expected = ZSTD_startingInputLength(dctx->format); /* dctx->format must be properly set */ dctx->stage = ZSTDds_getFrameHeaderSize; dctx->processedCSize = 0; dctx->decodedSize = 0; dctx->previousDstEnd = NULL; dctx->prefixStart = NULL; dctx->virtualStart = NULL; dctx->dictEnd = NULL; dctx->entropy.hufTable[0] = (HUF_DTable)((ZSTD_HUFFDTABLE_CAPACITY_LOG)*0x1000001); /* cover both little and big endian */ dctx->litEntropy = dctx->fseEntropy = 0; dctx->dictID = 0; dctx->bType = bt_reserved; ZSTD_STATIC_ASSERT(sizeof(dctx->entropy.rep) == sizeof(repStartValue)); ZSTD_memcpy(dctx->entropy.rep, repStartValue, sizeof(repStartValue)); /* initial repcodes */ dctx->LLTptr = dctx->entropy.LLTable; dctx->MLTptr = dctx->entropy.MLTable; dctx->OFTptr = dctx->entropy.OFTable; dctx->HUFptr = dctx->entropy.hufTable; return 0; } size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx* dctx, const void* dict, size_t dictSize) { FORWARD_IF_ERROR( ZSTD_decompressBegin(dctx) , ""); if (dict && dictSize) RETURN_ERROR_IF( ZSTD_isError(ZSTD_decompress_insertDictionary(dctx, dict, dictSize)), dictionary_corrupted, ""); return 0; } /* ====== ZSTD_DDict ====== */ size_t ZSTD_decompressBegin_usingDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict) { DEBUGLOG(4, "ZSTD_decompressBegin_usingDDict"); assert(dctx != NULL); if (ddict) { const char* const dictStart = (const char*)ZSTD_DDict_dictContent(ddict); size_t const dictSize = ZSTD_DDict_dictSize(ddict); const void* const dictEnd = dictStart + dictSize; dctx->ddictIsCold = (dctx->dictEnd != dictEnd); DEBUGLOG(4, "DDict is %s", dctx->ddictIsCold ? "~cold~" : "hot!"); } FORWARD_IF_ERROR( ZSTD_decompressBegin(dctx) , ""); if (ddict) { /* NULL ddict is equivalent to no dictionary */ ZSTD_copyDDictParameters(dctx, ddict); } return 0; } /*! ZSTD_getDictID_fromDict() : * Provides the dictID stored within dictionary. * if @return == 0, the dictionary is not conformant with Zstandard specification. * It can still be loaded, but as a content-only dictionary. */ unsigned ZSTD_getDictID_fromDict(const void* dict, size_t dictSize) { if (dictSize < 8) return 0; if (MEM_readLE32(dict) != ZSTD_MAGIC_DICTIONARY) return 0; return MEM_readLE32((const char*)dict + ZSTD_FRAMEIDSIZE); } /*! ZSTD_getDictID_fromFrame() : * Provides the dictID required to decompress frame stored within `src`. * If @return == 0, the dictID could not be decoded. * This could for one of the following reasons : * - The frame does not require a dictionary (most common case). * - The frame was built with dictID intentionally removed. * Needed dictionary is a hidden piece of information. * Note : this use case also happens when using a non-conformant dictionary. * - `srcSize` is too small, and as a result, frame header could not be decoded. * Note : possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`. * - This is not a Zstandard frame. * When identifying the exact failure cause, it's possible to use * ZSTD_getFrameHeader(), which will provide a more precise error code. */ unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize) { ZSTD_frameHeader zfp = { 0, 0, 0, ZSTD_frame, 0, 0, 0, 0, 0 }; size_t const hError = ZSTD_getFrameHeader(&zfp, src, srcSize); if (ZSTD_isError(hError)) return 0; return zfp.dictID; } /*! ZSTD_decompress_usingDDict() : * Decompression using a pre-digested Dictionary * Use dictionary without significant overhead. */ size_t ZSTD_decompress_usingDDict(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const ZSTD_DDict* ddict) { /* pass content and size in case legacy frames are encountered */ return ZSTD_decompressMultiFrame(dctx, dst, dstCapacity, src, srcSize, NULL, 0, ddict); } /*===================================== * Streaming decompression *====================================*/ ZSTD_DStream* ZSTD_createDStream(void) { DEBUGLOG(3, "ZSTD_createDStream"); return ZSTD_createDCtx_internal(ZSTD_defaultCMem); } ZSTD_DStream* ZSTD_initStaticDStream(void *workspace, size_t workspaceSize) { return ZSTD_initStaticDCtx(workspace, workspaceSize); } ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem) { return ZSTD_createDCtx_internal(customMem); } size_t ZSTD_freeDStream(ZSTD_DStream* zds) { return ZSTD_freeDCtx(zds); } /* *** Initialization *** */ size_t ZSTD_DStreamInSize(void) { return ZSTD_BLOCKSIZE_MAX + ZSTD_blockHeaderSize; } size_t ZSTD_DStreamOutSize(void) { return ZSTD_BLOCKSIZE_MAX; } size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType) { RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, ""); ZSTD_clearDict(dctx); if (dict && dictSize != 0) { dctx->ddictLocal = ZSTD_createDDict_advanced(dict, dictSize, dictLoadMethod, dictContentType, dctx->customMem); RETURN_ERROR_IF(dctx->ddictLocal == NULL, memory_allocation, "NULL pointer!"); dctx->ddict = dctx->ddictLocal; dctx->dictUses = ZSTD_use_indefinitely; } return 0; } size_t ZSTD_DCtx_loadDictionary_byReference(ZSTD_DCtx* dctx, const void* dict, size_t dictSize) { return ZSTD_DCtx_loadDictionary_advanced(dctx, dict, dictSize, ZSTD_dlm_byRef, ZSTD_dct_auto); } size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize) { return ZSTD_DCtx_loadDictionary_advanced(dctx, dict, dictSize, ZSTD_dlm_byCopy, ZSTD_dct_auto); } size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType) { FORWARD_IF_ERROR(ZSTD_DCtx_loadDictionary_advanced(dctx, prefix, prefixSize, ZSTD_dlm_byRef, dictContentType), ""); dctx->dictUses = ZSTD_use_once; return 0; } size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize) { return ZSTD_DCtx_refPrefix_advanced(dctx, prefix, prefixSize, ZSTD_dct_rawContent); } /* ZSTD_initDStream_usingDict() : * return : expected size, aka ZSTD_startingInputLength(). * this function cannot fail */ size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize) { DEBUGLOG(4, "ZSTD_initDStream_usingDict"); FORWARD_IF_ERROR( ZSTD_DCtx_reset(zds, ZSTD_reset_session_only) , ""); FORWARD_IF_ERROR( ZSTD_DCtx_loadDictionary(zds, dict, dictSize) , ""); return ZSTD_startingInputLength(zds->format); } /* note : this variant can't fail */ size_t ZSTD_initDStream(ZSTD_DStream* zds) { DEBUGLOG(4, "ZSTD_initDStream"); FORWARD_IF_ERROR(ZSTD_DCtx_reset(zds, ZSTD_reset_session_only), ""); FORWARD_IF_ERROR(ZSTD_DCtx_refDDict(zds, NULL), ""); return ZSTD_startingInputLength(zds->format); } /* ZSTD_initDStream_usingDDict() : * ddict will just be referenced, and must outlive decompression session * this function cannot fail */ size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* dctx, const ZSTD_DDict* ddict) { DEBUGLOG(4, "ZSTD_initDStream_usingDDict"); FORWARD_IF_ERROR( ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only) , ""); FORWARD_IF_ERROR( ZSTD_DCtx_refDDict(dctx, ddict) , ""); return ZSTD_startingInputLength(dctx->format); } /* ZSTD_resetDStream() : * return : expected size, aka ZSTD_startingInputLength(). * this function cannot fail */ size_t ZSTD_resetDStream(ZSTD_DStream* dctx) { DEBUGLOG(4, "ZSTD_resetDStream"); FORWARD_IF_ERROR(ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only), ""); return ZSTD_startingInputLength(dctx->format); } size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict) { RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, ""); ZSTD_clearDict(dctx); if (ddict) { dctx->ddict = ddict; dctx->dictUses = ZSTD_use_indefinitely; if (dctx->refMultipleDDicts == ZSTD_rmd_refMultipleDDicts) { if (dctx->ddictSet == NULL) { dctx->ddictSet = ZSTD_createDDictHashSet(dctx->customMem); if (!dctx->ddictSet) { RETURN_ERROR(memory_allocation, "Failed to allocate memory for hash set!"); } } assert(!dctx->staticSize); /* Impossible: ddictSet cannot have been allocated if static dctx */ FORWARD_IF_ERROR(ZSTD_DDictHashSet_addDDict(dctx->ddictSet, ddict, dctx->customMem), ""); } } return 0; } /* ZSTD_DCtx_setMaxWindowSize() : * note : no direct equivalence in ZSTD_DCtx_setParameter, * since this version sets windowSize, and the other sets windowLog */ size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize) { ZSTD_bounds const bounds = ZSTD_dParam_getBounds(ZSTD_d_windowLogMax); size_t const min = (size_t)1 << bounds.lowerBound; size_t const max = (size_t)1 << bounds.upperBound; RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, ""); RETURN_ERROR_IF(maxWindowSize < min, parameter_outOfBound, ""); RETURN_ERROR_IF(maxWindowSize > max, parameter_outOfBound, ""); dctx->maxWindowSize = maxWindowSize; return 0; } size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format) { return ZSTD_DCtx_setParameter(dctx, ZSTD_d_format, (int)format); } ZSTD_bounds ZSTD_dParam_getBounds(ZSTD_dParameter dParam) { ZSTD_bounds bounds = { 0, 0, 0 }; switch(dParam) { case ZSTD_d_windowLogMax: bounds.lowerBound = ZSTD_WINDOWLOG_ABSOLUTEMIN; bounds.upperBound = ZSTD_WINDOWLOG_MAX; return bounds; case ZSTD_d_format: bounds.lowerBound = (int)ZSTD_f_zstd1; bounds.upperBound = (int)ZSTD_f_zstd1_magicless; ZSTD_STATIC_ASSERT(ZSTD_f_zstd1 < ZSTD_f_zstd1_magicless); return bounds; case ZSTD_d_stableOutBuffer: bounds.lowerBound = (int)ZSTD_bm_buffered; bounds.upperBound = (int)ZSTD_bm_stable; return bounds; case ZSTD_d_forceIgnoreChecksum: bounds.lowerBound = (int)ZSTD_d_validateChecksum; bounds.upperBound = (int)ZSTD_d_ignoreChecksum; return bounds; case ZSTD_d_refMultipleDDicts: bounds.lowerBound = (int)ZSTD_rmd_refSingleDDict; bounds.upperBound = (int)ZSTD_rmd_refMultipleDDicts; return bounds; case ZSTD_d_disableHuffmanAssembly: bounds.lowerBound = 0; bounds.upperBound = 1; return bounds; default:; } bounds.error = ERROR(parameter_unsupported); return bounds; } /* ZSTD_dParam_withinBounds: * @return 1 if value is within dParam bounds, * 0 otherwise */ static int ZSTD_dParam_withinBounds(ZSTD_dParameter dParam, int value) { ZSTD_bounds const bounds = ZSTD_dParam_getBounds(dParam); if (ZSTD_isError(bounds.error)) return 0; if (value < bounds.lowerBound) return 0; if (value > bounds.upperBound) return 0; return 1; } #define CHECK_DBOUNDS(p,v) { \ RETURN_ERROR_IF(!ZSTD_dParam_withinBounds(p, v), parameter_outOfBound, ""); \ } size_t ZSTD_DCtx_getParameter(ZSTD_DCtx* dctx, ZSTD_dParameter param, int* value) { switch (param) { case ZSTD_d_windowLogMax: *value = (int)ZSTD_highbit32((U32)dctx->maxWindowSize); return 0; case ZSTD_d_format: *value = (int)dctx->format; return 0; case ZSTD_d_stableOutBuffer: *value = (int)dctx->outBufferMode; return 0; case ZSTD_d_forceIgnoreChecksum: *value = (int)dctx->forceIgnoreChecksum; return 0; case ZSTD_d_refMultipleDDicts: *value = (int)dctx->refMultipleDDicts; return 0; case ZSTD_d_disableHuffmanAssembly: *value = (int)dctx->disableHufAsm; return 0; default:; } RETURN_ERROR(parameter_unsupported, ""); } size_t ZSTD_DCtx_setParameter(ZSTD_DCtx* dctx, ZSTD_dParameter dParam, int value) { RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, ""); switch(dParam) { case ZSTD_d_windowLogMax: if (value == 0) value = ZSTD_WINDOWLOG_LIMIT_DEFAULT; CHECK_DBOUNDS(ZSTD_d_windowLogMax, value); dctx->maxWindowSize = ((size_t)1) << value; return 0; case ZSTD_d_format: CHECK_DBOUNDS(ZSTD_d_format, value); dctx->format = (ZSTD_format_e)value; return 0; case ZSTD_d_stableOutBuffer: CHECK_DBOUNDS(ZSTD_d_stableOutBuffer, value); dctx->outBufferMode = (ZSTD_bufferMode_e)value; return 0; case ZSTD_d_forceIgnoreChecksum: CHECK_DBOUNDS(ZSTD_d_forceIgnoreChecksum, value); dctx->forceIgnoreChecksum = (ZSTD_forceIgnoreChecksum_e)value; return 0; case ZSTD_d_refMultipleDDicts: CHECK_DBOUNDS(ZSTD_d_refMultipleDDicts, value); if (dctx->staticSize != 0) { RETURN_ERROR(parameter_unsupported, "Static dctx does not support multiple DDicts!"); } dctx->refMultipleDDicts = (ZSTD_refMultipleDDicts_e)value; return 0; case ZSTD_d_disableHuffmanAssembly: CHECK_DBOUNDS(ZSTD_d_disableHuffmanAssembly, value); dctx->disableHufAsm = value != 0; return 0; default:; } RETURN_ERROR(parameter_unsupported, ""); } size_t ZSTD_DCtx_reset(ZSTD_DCtx* dctx, ZSTD_ResetDirective reset) { if ( (reset == ZSTD_reset_session_only) || (reset == ZSTD_reset_session_and_parameters) ) { dctx->streamStage = zdss_init; dctx->noForwardProgress = 0; } if ( (reset == ZSTD_reset_parameters) || (reset == ZSTD_reset_session_and_parameters) ) { RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, ""); ZSTD_clearDict(dctx); ZSTD_DCtx_resetParameters(dctx); } return 0; } size_t ZSTD_sizeof_DStream(const ZSTD_DStream* dctx) { return ZSTD_sizeof_DCtx(dctx); } size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long long frameContentSize) { size_t const blockSize = (size_t) MIN(windowSize, ZSTD_BLOCKSIZE_MAX); /* space is needed to store the litbuffer after the output of a given block without stomping the extDict of a previous run, as well as to cover both windows against wildcopy*/ unsigned long long const neededRBSize = windowSize + blockSize + ZSTD_BLOCKSIZE_MAX + (WILDCOPY_OVERLENGTH * 2); unsigned long long const neededSize = MIN(frameContentSize, neededRBSize); size_t const minRBSize = (size_t) neededSize; RETURN_ERROR_IF((unsigned long long)minRBSize != neededSize, frameParameter_windowTooLarge, ""); return minRBSize; } size_t ZSTD_estimateDStreamSize(size_t windowSize) { size_t const blockSize = MIN(windowSize, ZSTD_BLOCKSIZE_MAX); size_t const inBuffSize = blockSize; /* no block can be larger */ size_t const outBuffSize = ZSTD_decodingBufferSize_min(windowSize, ZSTD_CONTENTSIZE_UNKNOWN); return ZSTD_estimateDCtxSize() + inBuffSize + outBuffSize; } size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize) { U32 const windowSizeMax = 1U << ZSTD_WINDOWLOG_MAX; /* note : should be user-selectable, but requires an additional parameter (or a dctx) */ ZSTD_frameHeader zfh; size_t const err = ZSTD_getFrameHeader(&zfh, src, srcSize); if (ZSTD_isError(err)) return err; RETURN_ERROR_IF(err>0, srcSize_wrong, ""); RETURN_ERROR_IF(zfh.windowSize > windowSizeMax, frameParameter_windowTooLarge, ""); return ZSTD_estimateDStreamSize((size_t)zfh.windowSize); } /* ***** Decompression ***** */ static int ZSTD_DCtx_isOverflow(ZSTD_DStream* zds, size_t const neededInBuffSize, size_t const neededOutBuffSize) { return (zds->inBuffSize + zds->outBuffSize) >= (neededInBuffSize + neededOutBuffSize) * ZSTD_WORKSPACETOOLARGE_FACTOR; } static void ZSTD_DCtx_updateOversizedDuration(ZSTD_DStream* zds, size_t const neededInBuffSize, size_t const neededOutBuffSize) { if (ZSTD_DCtx_isOverflow(zds, neededInBuffSize, neededOutBuffSize)) zds->oversizedDuration++; else zds->oversizedDuration = 0; } static int ZSTD_DCtx_isOversizedTooLong(ZSTD_DStream* zds) { return zds->oversizedDuration >= ZSTD_WORKSPACETOOLARGE_MAXDURATION; } /* Checks that the output buffer hasn't changed if ZSTD_obm_stable is used. */ static size_t ZSTD_checkOutBuffer(ZSTD_DStream const* zds, ZSTD_outBuffer const* output) { ZSTD_outBuffer const expect = zds->expectedOutBuffer; /* No requirement when ZSTD_obm_stable is not enabled. */ if (zds->outBufferMode != ZSTD_bm_stable) return 0; /* Any buffer is allowed in zdss_init, this must be the same for every other call until * the context is reset. */ if (zds->streamStage == zdss_init) return 0; /* The buffer must match our expectation exactly. */ if (expect.dst == output->dst && expect.pos == output->pos && expect.size == output->size) return 0; RETURN_ERROR(dstBuffer_wrong, "ZSTD_d_stableOutBuffer enabled but output differs!"); } /* Calls ZSTD_decompressContinue() with the right parameters for ZSTD_decompressStream() * and updates the stage and the output buffer state. This call is extracted so it can be * used both when reading directly from the ZSTD_inBuffer, and in buffered input mode. * NOTE: You must break after calling this function since the streamStage is modified. */ static size_t ZSTD_decompressContinueStream( ZSTD_DStream* zds, char** op, char* oend, void const* src, size_t srcSize) { int const isSkipFrame = ZSTD_isSkipFrame(zds); if (zds->outBufferMode == ZSTD_bm_buffered) { size_t const dstSize = isSkipFrame ? 0 : zds->outBuffSize - zds->outStart; size_t const decodedSize = ZSTD_decompressContinue(zds, zds->outBuff + zds->outStart, dstSize, src, srcSize); FORWARD_IF_ERROR(decodedSize, ""); if (!decodedSize && !isSkipFrame) { zds->streamStage = zdss_read; } else { zds->outEnd = zds->outStart + decodedSize; zds->streamStage = zdss_flush; } } else { /* Write directly into the output buffer */ size_t const dstSize = isSkipFrame ? 0 : (size_t)(oend - *op); size_t const decodedSize = ZSTD_decompressContinue(zds, *op, dstSize, src, srcSize); FORWARD_IF_ERROR(decodedSize, ""); *op += decodedSize; /* Flushing is not needed. */ zds->streamStage = zdss_read; assert(*op <= oend); assert(zds->outBufferMode == ZSTD_bm_stable); } return 0; } size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inBuffer* input) { const char* const src = (const char*)input->src; const char* const istart = input->pos != 0 ? src + input->pos : src; const char* const iend = input->size != 0 ? src + input->size : src; const char* ip = istart; char* const dst = (char*)output->dst; char* const ostart = output->pos != 0 ? dst + output->pos : dst; char* const oend = output->size != 0 ? dst + output->size : dst; char* op = ostart; U32 someMoreWork = 1; DEBUGLOG(5, "ZSTD_decompressStream"); RETURN_ERROR_IF( input->pos > input->size, srcSize_wrong, "forbidden. in: pos: %u vs size: %u", (U32)input->pos, (U32)input->size); RETURN_ERROR_IF( output->pos > output->size, dstSize_tooSmall, "forbidden. out: pos: %u vs size: %u", (U32)output->pos, (U32)output->size); DEBUGLOG(5, "input size : %u", (U32)(input->size - input->pos)); FORWARD_IF_ERROR(ZSTD_checkOutBuffer(zds, output), ""); while (someMoreWork) { switch(zds->streamStage) { case zdss_init : DEBUGLOG(5, "stage zdss_init => transparent reset "); zds->streamStage = zdss_loadHeader; zds->lhSize = zds->inPos = zds->outStart = zds->outEnd = 0; #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1) zds->legacyVersion = 0; #endif zds->hostageByte = 0; zds->expectedOutBuffer = *output; ZSTD_FALLTHROUGH; case zdss_loadHeader : DEBUGLOG(5, "stage zdss_loadHeader (srcSize : %u)", (U32)(iend - ip)); #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1) if (zds->legacyVersion) { RETURN_ERROR_IF(zds->staticSize, memory_allocation, "legacy support is incompatible with static dctx"); { size_t const hint = ZSTD_decompressLegacyStream(zds->legacyContext, zds->legacyVersion, output, input); if (hint==0) zds->streamStage = zdss_init; return hint; } } #endif { size_t const hSize = ZSTD_getFrameHeader_advanced(&zds->fParams, zds->headerBuffer, zds->lhSize, zds->format); if (zds->refMultipleDDicts && zds->ddictSet) { ZSTD_DCtx_selectFrameDDict(zds); } if (ZSTD_isError(hSize)) { #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1) U32 const legacyVersion = ZSTD_isLegacy(istart, iend-istart); if (legacyVersion) { ZSTD_DDict const* const ddict = ZSTD_getDDict(zds); const void* const dict = ddict ? ZSTD_DDict_dictContent(ddict) : NULL; size_t const dictSize = ddict ? ZSTD_DDict_dictSize(ddict) : 0; DEBUGLOG(5, "ZSTD_decompressStream: detected legacy version v0.%u", legacyVersion); RETURN_ERROR_IF(zds->staticSize, memory_allocation, "legacy support is incompatible with static dctx"); FORWARD_IF_ERROR(ZSTD_initLegacyStream(&zds->legacyContext, zds->previousLegacyVersion, legacyVersion, dict, dictSize), ""); zds->legacyVersion = zds->previousLegacyVersion = legacyVersion; { size_t const hint = ZSTD_decompressLegacyStream(zds->legacyContext, legacyVersion, output, input); if (hint==0) zds->streamStage = zdss_init; /* or stay in stage zdss_loadHeader */ return hint; } } #endif return hSize; /* error */ } if (hSize != 0) { /* need more input */ size_t const toLoad = hSize - zds->lhSize; /* if hSize!=0, hSize > zds->lhSize */ size_t const remainingInput = (size_t)(iend-ip); assert(iend >= ip); if (toLoad > remainingInput) { /* not enough input to load full header */ if (remainingInput > 0) { ZSTD_memcpy(zds->headerBuffer + zds->lhSize, ip, remainingInput); zds->lhSize += remainingInput; } input->pos = input->size; /* check first few bytes */ FORWARD_IF_ERROR( ZSTD_getFrameHeader_advanced(&zds->fParams, zds->headerBuffer, zds->lhSize, zds->format), "First few bytes detected incorrect" ); /* return hint input size */ return (MAX((size_t)ZSTD_FRAMEHEADERSIZE_MIN(zds->format), hSize) - zds->lhSize) + ZSTD_blockHeaderSize; /* remaining header bytes + next block header */ } assert(ip != NULL); ZSTD_memcpy(zds->headerBuffer + zds->lhSize, ip, toLoad); zds->lhSize = hSize; ip += toLoad; break; } } /* check for single-pass mode opportunity */ if (zds->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN && zds->fParams.frameType != ZSTD_skippableFrame && (U64)(size_t)(oend-op) >= zds->fParams.frameContentSize) { size_t const cSize = ZSTD_findFrameCompressedSize(istart, (size_t)(iend-istart)); if (cSize <= (size_t)(iend-istart)) { /* shortcut : using single-pass mode */ size_t const decompressedSize = ZSTD_decompress_usingDDict(zds, op, (size_t)(oend-op), istart, cSize, ZSTD_getDDict(zds)); if (ZSTD_isError(decompressedSize)) return decompressedSize; DEBUGLOG(4, "shortcut to single-pass ZSTD_decompress_usingDDict()") assert(istart != NULL); ip = istart + cSize; op = op ? op + decompressedSize : op; /* can occur if frameContentSize = 0 (empty frame) */ zds->expected = 0; zds->streamStage = zdss_init; someMoreWork = 0; break; } } /* Check output buffer is large enough for ZSTD_odm_stable. */ if (zds->outBufferMode == ZSTD_bm_stable && zds->fParams.frameType != ZSTD_skippableFrame && zds->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN && (U64)(size_t)(oend-op) < zds->fParams.frameContentSize) { RETURN_ERROR(dstSize_tooSmall, "ZSTD_obm_stable passed but ZSTD_outBuffer is too small"); } /* Consume header (see ZSTDds_decodeFrameHeader) */ DEBUGLOG(4, "Consume header"); FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDDict(zds, ZSTD_getDDict(zds)), ""); if ((MEM_readLE32(zds->headerBuffer) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) { /* skippable frame */ zds->expected = MEM_readLE32(zds->headerBuffer + ZSTD_FRAMEIDSIZE); zds->stage = ZSTDds_skipFrame; } else { FORWARD_IF_ERROR(ZSTD_decodeFrameHeader(zds, zds->headerBuffer, zds->lhSize), ""); zds->expected = ZSTD_blockHeaderSize; zds->stage = ZSTDds_decodeBlockHeader; } /* control buffer memory usage */ DEBUGLOG(4, "Control max memory usage (%u KB <= max %u KB)", (U32)(zds->fParams.windowSize >>10), (U32)(zds->maxWindowSize >> 10) ); zds->fParams.windowSize = MAX(zds->fParams.windowSize, 1U << ZSTD_WINDOWLOG_ABSOLUTEMIN); RETURN_ERROR_IF(zds->fParams.windowSize > zds->maxWindowSize, frameParameter_windowTooLarge, ""); /* Adapt buffer sizes to frame header instructions */ { size_t const neededInBuffSize = MAX(zds->fParams.blockSizeMax, 4 /* frame checksum */); size_t const neededOutBuffSize = zds->outBufferMode == ZSTD_bm_buffered ? ZSTD_decodingBufferSize_min(zds->fParams.windowSize, zds->fParams.frameContentSize) : 0; ZSTD_DCtx_updateOversizedDuration(zds, neededInBuffSize, neededOutBuffSize); { int const tooSmall = (zds->inBuffSize < neededInBuffSize) || (zds->outBuffSize < neededOutBuffSize); int const tooLarge = ZSTD_DCtx_isOversizedTooLong(zds); if (tooSmall || tooLarge) { size_t const bufferSize = neededInBuffSize + neededOutBuffSize; DEBUGLOG(4, "inBuff : from %u to %u", (U32)zds->inBuffSize, (U32)neededInBuffSize); DEBUGLOG(4, "outBuff : from %u to %u", (U32)zds->outBuffSize, (U32)neededOutBuffSize); if (zds->staticSize) { /* static DCtx */ DEBUGLOG(4, "staticSize : %u", (U32)zds->staticSize); assert(zds->staticSize >= sizeof(ZSTD_DCtx)); /* controlled at init */ RETURN_ERROR_IF( bufferSize > zds->staticSize - sizeof(ZSTD_DCtx), memory_allocation, ""); } else { ZSTD_customFree(zds->inBuff, zds->customMem); zds->inBuffSize = 0; zds->outBuffSize = 0; zds->inBuff = (char*)ZSTD_customMalloc(bufferSize, zds->customMem); RETURN_ERROR_IF(zds->inBuff == NULL, memory_allocation, ""); } zds->inBuffSize = neededInBuffSize; zds->outBuff = zds->inBuff + zds->inBuffSize; zds->outBuffSize = neededOutBuffSize; } } } zds->streamStage = zdss_read; ZSTD_FALLTHROUGH; case zdss_read: DEBUGLOG(5, "stage zdss_read"); { size_t const neededInSize = ZSTD_nextSrcSizeToDecompressWithInputSize(zds, (size_t)(iend - ip)); DEBUGLOG(5, "neededInSize = %u", (U32)neededInSize); if (neededInSize==0) { /* end of frame */ zds->streamStage = zdss_init; someMoreWork = 0; break; } if ((size_t)(iend-ip) >= neededInSize) { /* decode directly from src */ FORWARD_IF_ERROR(ZSTD_decompressContinueStream(zds, &op, oend, ip, neededInSize), ""); assert(ip != NULL); ip += neededInSize; /* Function modifies the stage so we must break */ break; } } if (ip==iend) { someMoreWork = 0; break; } /* no more input */ zds->streamStage = zdss_load; ZSTD_FALLTHROUGH; case zdss_load: { size_t const neededInSize = ZSTD_nextSrcSizeToDecompress(zds); size_t const toLoad = neededInSize - zds->inPos; int const isSkipFrame = ZSTD_isSkipFrame(zds); size_t loadedSize; /* At this point we shouldn't be decompressing a block that we can stream. */ assert(neededInSize == ZSTD_nextSrcSizeToDecompressWithInputSize(zds, (size_t)(iend - ip))); if (isSkipFrame) { loadedSize = MIN(toLoad, (size_t)(iend-ip)); } else { RETURN_ERROR_IF(toLoad > zds->inBuffSize - zds->inPos, corruption_detected, "should never happen"); loadedSize = ZSTD_limitCopy(zds->inBuff + zds->inPos, toLoad, ip, (size_t)(iend-ip)); } if (loadedSize != 0) { /* ip may be NULL */ ip += loadedSize; zds->inPos += loadedSize; } if (loadedSize < toLoad) { someMoreWork = 0; break; } /* not enough input, wait for more */ /* decode loaded input */ zds->inPos = 0; /* input is consumed */ FORWARD_IF_ERROR(ZSTD_decompressContinueStream(zds, &op, oend, zds->inBuff, neededInSize), ""); /* Function modifies the stage so we must break */ break; } case zdss_flush: { size_t const toFlushSize = zds->outEnd - zds->outStart; size_t const flushedSize = ZSTD_limitCopy(op, (size_t)(oend-op), zds->outBuff + zds->outStart, toFlushSize); op = op ? op + flushedSize : op; zds->outStart += flushedSize; if (flushedSize == toFlushSize) { /* flush completed */ zds->streamStage = zdss_read; if ( (zds->outBuffSize < zds->fParams.frameContentSize) && (zds->outStart + zds->fParams.blockSizeMax > zds->outBuffSize) ) { DEBUGLOG(5, "restart filling outBuff from beginning (left:%i, needed:%u)", (int)(zds->outBuffSize - zds->outStart), (U32)zds->fParams.blockSizeMax); zds->outStart = zds->outEnd = 0; } break; } } /* cannot complete flush */ someMoreWork = 0; break; default: assert(0); /* impossible */ RETURN_ERROR(GENERIC, "impossible to reach"); /* some compilers require default to do something */ } } /* result */ input->pos = (size_t)(ip - (const char*)(input->src)); output->pos = (size_t)(op - (char*)(output->dst)); /* Update the expected output buffer for ZSTD_obm_stable. */ zds->expectedOutBuffer = *output; if ((ip==istart) && (op==ostart)) { /* no forward progress */ zds->noForwardProgress ++; if (zds->noForwardProgress >= ZSTD_NO_FORWARD_PROGRESS_MAX) { RETURN_ERROR_IF(op==oend, noForwardProgress_destFull, ""); RETURN_ERROR_IF(ip==iend, noForwardProgress_inputEmpty, ""); assert(0); } } else { zds->noForwardProgress = 0; } { size_t nextSrcSizeHint = ZSTD_nextSrcSizeToDecompress(zds); if (!nextSrcSizeHint) { /* frame fully decoded */ if (zds->outEnd == zds->outStart) { /* output fully flushed */ if (zds->hostageByte) { if (input->pos >= input->size) { /* can't release hostage (not present) */ zds->streamStage = zdss_read; return 1; } input->pos++; /* release hostage */ } /* zds->hostageByte */ return 0; } /* zds->outEnd == zds->outStart */ if (!zds->hostageByte) { /* output not fully flushed; keep last byte as hostage; will be released when all output is flushed */ input->pos--; /* note : pos > 0, otherwise, impossible to finish reading last block */ zds->hostageByte=1; } return 1; } /* nextSrcSizeHint==0 */ nextSrcSizeHint += ZSTD_blockHeaderSize * (ZSTD_nextInputType(zds) == ZSTDnit_block); /* preload header of next block */ assert(zds->inPos <= nextSrcSizeHint); nextSrcSizeHint -= zds->inPos; /* part already loaded*/ return nextSrcSizeHint; } } size_t ZSTD_decompressStream_simpleArgs ( ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, size_t* dstPos, const void* src, size_t srcSize, size_t* srcPos) { ZSTD_outBuffer output; ZSTD_inBuffer input; output.dst = dst; output.size = dstCapacity; output.pos = *dstPos; input.src = src; input.size = srcSize; input.pos = *srcPos; { size_t const cErr = ZSTD_decompressStream(dctx, &output, &input); *dstPos = output.pos; *srcPos = input.pos; return cErr; } } zmat-0.9.9/src/blosc2/internal-complibs/zstd/decompress/zstd_ddict.h0000644000175200007730000000244714515254731025777 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_DDICT_H #define ZSTD_DDICT_H /*-******************************************************* * Dependencies *********************************************************/ #include "../common/zstd_deps.h" /* size_t */ #include "../zstd.h" /* ZSTD_DDict, and several public functions */ /*-******************************************************* * Interface *********************************************************/ /* note: several prototypes are already published in `zstd.h` : * ZSTD_createDDict() * ZSTD_createDDict_byReference() * ZSTD_createDDict_advanced() * ZSTD_freeDDict() * ZSTD_initStaticDDict() * ZSTD_sizeof_DDict() * ZSTD_estimateDDictSize() * ZSTD_getDictID_fromDict() */ const void* ZSTD_DDict_dictContent(const ZSTD_DDict* ddict); size_t ZSTD_DDict_dictSize(const ZSTD_DDict* ddict); void ZSTD_copyDDictParameters(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict); #endif /* ZSTD_DDICT_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/decompress/zstd_ddict.c0000644000175200007730000002173614515254731025774 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* zstd_ddict.c : * concentrates all logic that needs to know the internals of ZSTD_DDict object */ /*-******************************************************* * Dependencies *********************************************************/ #include "../common/allocations.h" /* ZSTD_customMalloc, ZSTD_customFree */ #include "../common/zstd_deps.h" /* ZSTD_memcpy, ZSTD_memmove, ZSTD_memset */ #include "../common/cpu.h" /* bmi2 */ #include "../common/mem.h" /* low level memory routines */ #define FSE_STATIC_LINKING_ONLY #include "../common/fse.h" #include "../common/huf.h" #include "zstd_decompress_internal.h" #include "zstd_ddict.h" #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1) # include "../legacy/zstd_legacy.h" #endif /*-******************************************************* * Types *********************************************************/ struct ZSTD_DDict_s { void* dictBuffer; const void* dictContent; size_t dictSize; ZSTD_entropyDTables_t entropy; U32 dictID; U32 entropyPresent; ZSTD_customMem cMem; }; /* typedef'd to ZSTD_DDict within "zstd.h" */ const void* ZSTD_DDict_dictContent(const ZSTD_DDict* ddict) { assert(ddict != NULL); return ddict->dictContent; } size_t ZSTD_DDict_dictSize(const ZSTD_DDict* ddict) { assert(ddict != NULL); return ddict->dictSize; } void ZSTD_copyDDictParameters(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict) { DEBUGLOG(4, "ZSTD_copyDDictParameters"); assert(dctx != NULL); assert(ddict != NULL); dctx->dictID = ddict->dictID; dctx->prefixStart = ddict->dictContent; dctx->virtualStart = ddict->dictContent; dctx->dictEnd = (const BYTE*)ddict->dictContent + ddict->dictSize; dctx->previousDstEnd = dctx->dictEnd; #ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION dctx->dictContentBeginForFuzzing = dctx->prefixStart; dctx->dictContentEndForFuzzing = dctx->previousDstEnd; #endif if (ddict->entropyPresent) { dctx->litEntropy = 1; dctx->fseEntropy = 1; dctx->LLTptr = ddict->entropy.LLTable; dctx->MLTptr = ddict->entropy.MLTable; dctx->OFTptr = ddict->entropy.OFTable; dctx->HUFptr = ddict->entropy.hufTable; dctx->entropy.rep[0] = ddict->entropy.rep[0]; dctx->entropy.rep[1] = ddict->entropy.rep[1]; dctx->entropy.rep[2] = ddict->entropy.rep[2]; } else { dctx->litEntropy = 0; dctx->fseEntropy = 0; } } static size_t ZSTD_loadEntropy_intoDDict(ZSTD_DDict* ddict, ZSTD_dictContentType_e dictContentType) { ddict->dictID = 0; ddict->entropyPresent = 0; if (dictContentType == ZSTD_dct_rawContent) return 0; if (ddict->dictSize < 8) { if (dictContentType == ZSTD_dct_fullDict) return ERROR(dictionary_corrupted); /* only accept specified dictionaries */ return 0; /* pure content mode */ } { U32 const magic = MEM_readLE32(ddict->dictContent); if (magic != ZSTD_MAGIC_DICTIONARY) { if (dictContentType == ZSTD_dct_fullDict) return ERROR(dictionary_corrupted); /* only accept specified dictionaries */ return 0; /* pure content mode */ } } ddict->dictID = MEM_readLE32((const char*)ddict->dictContent + ZSTD_FRAMEIDSIZE); /* load entropy tables */ RETURN_ERROR_IF(ZSTD_isError(ZSTD_loadDEntropy( &ddict->entropy, ddict->dictContent, ddict->dictSize)), dictionary_corrupted, ""); ddict->entropyPresent = 1; return 0; } static size_t ZSTD_initDDict_internal(ZSTD_DDict* ddict, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType) { if ((dictLoadMethod == ZSTD_dlm_byRef) || (!dict) || (!dictSize)) { ddict->dictBuffer = NULL; ddict->dictContent = dict; if (!dict) dictSize = 0; } else { void* const internalBuffer = ZSTD_customMalloc(dictSize, ddict->cMem); ddict->dictBuffer = internalBuffer; ddict->dictContent = internalBuffer; if (!internalBuffer) return ERROR(memory_allocation); ZSTD_memcpy(internalBuffer, dict, dictSize); } ddict->dictSize = dictSize; ddict->entropy.hufTable[0] = (HUF_DTable)((ZSTD_HUFFDTABLE_CAPACITY_LOG)*0x1000001); /* cover both little and big endian */ /* parse dictionary content */ FORWARD_IF_ERROR( ZSTD_loadEntropy_intoDDict(ddict, dictContentType) , ""); return 0; } ZSTD_DDict* ZSTD_createDDict_advanced(const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType, ZSTD_customMem customMem) { if ((!customMem.customAlloc) ^ (!customMem.customFree)) return NULL; { ZSTD_DDict* const ddict = (ZSTD_DDict*) ZSTD_customMalloc(sizeof(ZSTD_DDict), customMem); if (ddict == NULL) return NULL; ddict->cMem = customMem; { size_t const initResult = ZSTD_initDDict_internal(ddict, dict, dictSize, dictLoadMethod, dictContentType); if (ZSTD_isError(initResult)) { ZSTD_freeDDict(ddict); return NULL; } } return ddict; } } /*! ZSTD_createDDict() : * Create a digested dictionary, to start decompression without startup delay. * `dict` content is copied inside DDict. * Consequently, `dict` can be released after `ZSTD_DDict` creation */ ZSTD_DDict* ZSTD_createDDict(const void* dict, size_t dictSize) { ZSTD_customMem const allocator = { NULL, NULL, NULL }; return ZSTD_createDDict_advanced(dict, dictSize, ZSTD_dlm_byCopy, ZSTD_dct_auto, allocator); } /*! ZSTD_createDDict_byReference() : * Create a digested dictionary, to start decompression without startup delay. * Dictionary content is simply referenced, it will be accessed during decompression. * Warning : dictBuffer must outlive DDict (DDict must be freed before dictBuffer) */ ZSTD_DDict* ZSTD_createDDict_byReference(const void* dictBuffer, size_t dictSize) { ZSTD_customMem const allocator = { NULL, NULL, NULL }; return ZSTD_createDDict_advanced(dictBuffer, dictSize, ZSTD_dlm_byRef, ZSTD_dct_auto, allocator); } const ZSTD_DDict* ZSTD_initStaticDDict( void* sBuffer, size_t sBufferSize, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType) { size_t const neededSpace = sizeof(ZSTD_DDict) + (dictLoadMethod == ZSTD_dlm_byRef ? 0 : dictSize); ZSTD_DDict* const ddict = (ZSTD_DDict*)sBuffer; assert(sBuffer != NULL); assert(dict != NULL); if ((size_t)sBuffer & 7) return NULL; /* 8-aligned */ if (sBufferSize < neededSpace) return NULL; if (dictLoadMethod == ZSTD_dlm_byCopy) { ZSTD_memcpy(ddict+1, dict, dictSize); /* local copy */ dict = ddict+1; } if (ZSTD_isError( ZSTD_initDDict_internal(ddict, dict, dictSize, ZSTD_dlm_byRef, dictContentType) )) return NULL; return ddict; } size_t ZSTD_freeDDict(ZSTD_DDict* ddict) { if (ddict==NULL) return 0; /* support free on NULL */ { ZSTD_customMem const cMem = ddict->cMem; ZSTD_customFree(ddict->dictBuffer, cMem); ZSTD_customFree(ddict, cMem); return 0; } } /*! ZSTD_estimateDDictSize() : * Estimate amount of memory that will be needed to create a dictionary for decompression. * Note : dictionary created by reference using ZSTD_dlm_byRef are smaller */ size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod) { return sizeof(ZSTD_DDict) + (dictLoadMethod == ZSTD_dlm_byRef ? 0 : dictSize); } size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict) { if (ddict==NULL) return 0; /* support sizeof on NULL */ return sizeof(*ddict) + (ddict->dictBuffer ? ddict->dictSize : 0) ; } /*! ZSTD_getDictID_fromDDict() : * Provides the dictID of the dictionary loaded into `ddict`. * If @return == 0, the dictionary is not conformant to Zstandard specification, or empty. * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */ unsigned ZSTD_getDictID_fromDDict(const ZSTD_DDict* ddict) { if (ddict==NULL) return 0; return ddict->dictID; } zmat-0.9.9/src/blosc2/internal-complibs/zstd/decompress/huf_decompress.c0000644000175200007730000021774514515254731026656 0ustar rlaboissrlaboiss/* ****************************************************************** * huff0 huffman decoder, * part of Finite State Entropy library * Copyright (c) Meta Platforms, Inc. and affiliates. * * You can contact the author at : * - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ /* ************************************************************** * Dependencies ****************************************************************/ #include "../common/zstd_deps.h" /* ZSTD_memcpy, ZSTD_memset */ #include "../common/compiler.h" #include "../common/bitstream.h" /* BIT_* */ #include "../common/fse.h" /* to compress headers */ #include "../common/huf.h" #include "../common/error_private.h" #include "../common/zstd_internal.h" #include "../common/bits.h" /* ZSTD_highbit32, ZSTD_countTrailingZeros64 */ /* ************************************************************** * Constants ****************************************************************/ #define HUF_DECODER_FAST_TABLELOG 11 /* ************************************************************** * Macros ****************************************************************/ /* These two optional macros force the use one way or another of the two * Huffman decompression implementations. You can't force in both directions * at the same time. */ #if defined(HUF_FORCE_DECOMPRESS_X1) && \ defined(HUF_FORCE_DECOMPRESS_X2) #error "Cannot force the use of the X1 and X2 decoders at the same time!" #endif /* When DYNAMIC_BMI2 is enabled, fast decoders are only called when bmi2 is * supported at runtime, so we can add the BMI2 target attribute. * When it is disabled, we will still get BMI2 if it is enabled statically. */ #if DYNAMIC_BMI2 # define HUF_FAST_BMI2_ATTRS BMI2_TARGET_ATTRIBUTE #else # define HUF_FAST_BMI2_ATTRS #endif #ifdef __cplusplus # define HUF_EXTERN_C extern "C" #else # define HUF_EXTERN_C #endif #define HUF_ASM_DECL HUF_EXTERN_C #if DYNAMIC_BMI2 # define HUF_NEED_BMI2_FUNCTION 1 #else # define HUF_NEED_BMI2_FUNCTION 0 #endif /* ************************************************************** * Error Management ****************************************************************/ #define HUF_isError ERR_isError /* ************************************************************** * Byte alignment for workSpace management ****************************************************************/ #define HUF_ALIGN(x, a) HUF_ALIGN_MASK((x), (a) - 1) #define HUF_ALIGN_MASK(x, mask) (((x) + (mask)) & ~(mask)) /* ************************************************************** * BMI2 Variant Wrappers ****************************************************************/ typedef size_t (*HUF_DecompressUsingDTableFn)(void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable); #if DYNAMIC_BMI2 #define HUF_DGEN(fn) \ \ static size_t fn##_default( \ void* dst, size_t dstSize, \ const void* cSrc, size_t cSrcSize, \ const HUF_DTable* DTable) \ { \ return fn##_body(dst, dstSize, cSrc, cSrcSize, DTable); \ } \ \ static BMI2_TARGET_ATTRIBUTE size_t fn##_bmi2( \ void* dst, size_t dstSize, \ const void* cSrc, size_t cSrcSize, \ const HUF_DTable* DTable) \ { \ return fn##_body(dst, dstSize, cSrc, cSrcSize, DTable); \ } \ \ static size_t fn(void* dst, size_t dstSize, void const* cSrc, \ size_t cSrcSize, HUF_DTable const* DTable, int flags) \ { \ if (flags & HUF_flags_bmi2) { \ return fn##_bmi2(dst, dstSize, cSrc, cSrcSize, DTable); \ } \ return fn##_default(dst, dstSize, cSrc, cSrcSize, DTable); \ } #else #define HUF_DGEN(fn) \ static size_t fn(void* dst, size_t dstSize, void const* cSrc, \ size_t cSrcSize, HUF_DTable const* DTable, int flags) \ { \ (void)flags; \ return fn##_body(dst, dstSize, cSrc, cSrcSize, DTable); \ } #endif /*-***************************/ /* generic DTableDesc */ /*-***************************/ typedef struct { BYTE maxTableLog; BYTE tableType; BYTE tableLog; BYTE reserved; } DTableDesc; static DTableDesc HUF_getDTableDesc(const HUF_DTable* table) { DTableDesc dtd; ZSTD_memcpy(&dtd, table, sizeof(dtd)); return dtd; } static size_t HUF_initFastDStream(BYTE const* ip) { BYTE const lastByte = ip[7]; size_t const bitsConsumed = lastByte ? 8 - ZSTD_highbit32(lastByte) : 0; size_t const value = MEM_readLEST(ip) | 1; assert(bitsConsumed <= 8); assert(sizeof(size_t) == 8); return value << bitsConsumed; } /** * The input/output arguments to the Huffman fast decoding loop: * * ip [in/out] - The input pointers, must be updated to reflect what is consumed. * op [in/out] - The output pointers, must be updated to reflect what is written. * bits [in/out] - The bitstream containers, must be updated to reflect the current state. * dt [in] - The decoding table. * ilimit [in] - The input limit, stop when any input pointer is below ilimit. * oend [in] - The end of the output stream. op[3] must not cross oend. * iend [in] - The end of each input stream. ip[i] may cross iend[i], * as long as it is above ilimit, but that indicates corruption. */ typedef struct { BYTE const* ip[4]; BYTE* op[4]; U64 bits[4]; void const* dt; BYTE const* ilimit; BYTE* oend; BYTE const* iend[4]; } HUF_DecompressFastArgs; typedef void (*HUF_DecompressFastLoopFn)(HUF_DecompressFastArgs*); /** * Initializes args for the fast decoding loop. * @returns 1 on success * 0 if the fallback implementation should be used. * Or an error code on failure. */ static size_t HUF_DecompressFastArgs_init(HUF_DecompressFastArgs* args, void* dst, size_t dstSize, void const* src, size_t srcSize, const HUF_DTable* DTable) { void const* dt = DTable + 1; U32 const dtLog = HUF_getDTableDesc(DTable).tableLog; const BYTE* const ilimit = (const BYTE*)src + 6 + 8; BYTE* const oend = (BYTE*)dst + dstSize; /* The fast decoding loop assumes 64-bit little-endian. * This condition is false on x32. */ if (!MEM_isLittleEndian() || MEM_32bits()) return 0; /* strict minimum : jump table + 1 byte per stream */ if (srcSize < 10) return ERROR(corruption_detected); /* Must have at least 8 bytes per stream because we don't handle initializing smaller bit containers. * If table log is not correct at this point, fallback to the old decoder. * On small inputs we don't have enough data to trigger the fast loop, so use the old decoder. */ if (dtLog != HUF_DECODER_FAST_TABLELOG) return 0; /* Read the jump table. */ { const BYTE* const istart = (const BYTE*)src; size_t const length1 = MEM_readLE16(istart); size_t const length2 = MEM_readLE16(istart+2); size_t const length3 = MEM_readLE16(istart+4); size_t const length4 = srcSize - (length1 + length2 + length3 + 6); args->iend[0] = istart + 6; /* jumpTable */ args->iend[1] = args->iend[0] + length1; args->iend[2] = args->iend[1] + length2; args->iend[3] = args->iend[2] + length3; /* HUF_initFastDStream() requires this, and this small of an input * won't benefit from the ASM loop anyways. * length1 must be >= 16 so that ip[0] >= ilimit before the loop * starts. */ if (length1 < 16 || length2 < 8 || length3 < 8 || length4 < 8) return 0; if (length4 > srcSize) return ERROR(corruption_detected); /* overflow */ } /* ip[] contains the position that is currently loaded into bits[]. */ args->ip[0] = args->iend[1] - sizeof(U64); args->ip[1] = args->iend[2] - sizeof(U64); args->ip[2] = args->iend[3] - sizeof(U64); args->ip[3] = (BYTE const*)src + srcSize - sizeof(U64); /* op[] contains the output pointers. */ args->op[0] = (BYTE*)dst; args->op[1] = args->op[0] + (dstSize+3)/4; args->op[2] = args->op[1] + (dstSize+3)/4; args->op[3] = args->op[2] + (dstSize+3)/4; /* No point to call the ASM loop for tiny outputs. */ if (args->op[3] >= oend) return 0; /* bits[] is the bit container. * It is read from the MSB down to the LSB. * It is shifted left as it is read, and zeros are * shifted in. After the lowest valid bit a 1 is * set, so that CountTrailingZeros(bits[]) can be used * to count how many bits we've consumed. */ args->bits[0] = HUF_initFastDStream(args->ip[0]); args->bits[1] = HUF_initFastDStream(args->ip[1]); args->bits[2] = HUF_initFastDStream(args->ip[2]); args->bits[3] = HUF_initFastDStream(args->ip[3]); /* If ip[] >= ilimit, it is guaranteed to be safe to * reload bits[]. It may be beyond its section, but is * guaranteed to be valid (>= istart). */ args->ilimit = ilimit; args->oend = oend; args->dt = dt; return 1; } static size_t HUF_initRemainingDStream(BIT_DStream_t* bit, HUF_DecompressFastArgs const* args, int stream, BYTE* segmentEnd) { /* Validate that we haven't overwritten. */ if (args->op[stream] > segmentEnd) return ERROR(corruption_detected); /* Validate that we haven't read beyond iend[]. * Note that ip[] may be < iend[] because the MSB is * the next bit to read, and we may have consumed 100% * of the stream, so down to iend[i] - 8 is valid. */ if (args->ip[stream] < args->iend[stream] - 8) return ERROR(corruption_detected); /* Construct the BIT_DStream_t. */ assert(sizeof(size_t) == 8); bit->bitContainer = MEM_readLEST(args->ip[stream]); bit->bitsConsumed = ZSTD_countTrailingZeros64(args->bits[stream]); bit->start = (const char*)args->iend[0]; bit->limitPtr = bit->start + sizeof(size_t); bit->ptr = (const char*)args->ip[stream]; return 0; } #ifndef HUF_FORCE_DECOMPRESS_X2 /*-***************************/ /* single-symbol decoding */ /*-***************************/ typedef struct { BYTE nbBits; BYTE byte; } HUF_DEltX1; /* single-symbol decoding */ /** * Packs 4 HUF_DEltX1 structs into a U64. This is used to lay down 4 entries at * a time. */ static U64 HUF_DEltX1_set4(BYTE symbol, BYTE nbBits) { U64 D4; if (MEM_isLittleEndian()) { D4 = (U64)((symbol << 8) + nbBits); } else { D4 = (U64)(symbol + (nbBits << 8)); } assert(D4 < (1U << 16)); D4 *= 0x0001000100010001ULL; return D4; } /** * Increase the tableLog to targetTableLog and rescales the stats. * If tableLog > targetTableLog this is a no-op. * @returns New tableLog */ static U32 HUF_rescaleStats(BYTE* huffWeight, U32* rankVal, U32 nbSymbols, U32 tableLog, U32 targetTableLog) { if (tableLog > targetTableLog) return tableLog; if (tableLog < targetTableLog) { U32 const scale = targetTableLog - tableLog; U32 s; /* Increase the weight for all non-zero probability symbols by scale. */ for (s = 0; s < nbSymbols; ++s) { huffWeight[s] += (BYTE)((huffWeight[s] == 0) ? 0 : scale); } /* Update rankVal to reflect the new weights. * All weights except 0 get moved to weight + scale. * Weights [1, scale] are empty. */ for (s = targetTableLog; s > scale; --s) { rankVal[s] = rankVal[s - scale]; } for (s = scale; s > 0; --s) { rankVal[s] = 0; } } return targetTableLog; } typedef struct { U32 rankVal[HUF_TABLELOG_ABSOLUTEMAX + 1]; U32 rankStart[HUF_TABLELOG_ABSOLUTEMAX + 1]; U32 statsWksp[HUF_READ_STATS_WORKSPACE_SIZE_U32]; BYTE symbols[HUF_SYMBOLVALUE_MAX + 1]; BYTE huffWeight[HUF_SYMBOLVALUE_MAX + 1]; } HUF_ReadDTableX1_Workspace; size_t HUF_readDTableX1_wksp(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int flags) { U32 tableLog = 0; U32 nbSymbols = 0; size_t iSize; void* const dtPtr = DTable + 1; HUF_DEltX1* const dt = (HUF_DEltX1*)dtPtr; HUF_ReadDTableX1_Workspace* wksp = (HUF_ReadDTableX1_Workspace*)workSpace; DEBUG_STATIC_ASSERT(HUF_DECOMPRESS_WORKSPACE_SIZE >= sizeof(*wksp)); if (sizeof(*wksp) > wkspSize) return ERROR(tableLog_tooLarge); DEBUG_STATIC_ASSERT(sizeof(DTableDesc) == sizeof(HUF_DTable)); /* ZSTD_memset(huffWeight, 0, sizeof(huffWeight)); */ /* is not necessary, even though some analyzer complain ... */ iSize = HUF_readStats_wksp(wksp->huffWeight, HUF_SYMBOLVALUE_MAX + 1, wksp->rankVal, &nbSymbols, &tableLog, src, srcSize, wksp->statsWksp, sizeof(wksp->statsWksp), flags); if (HUF_isError(iSize)) return iSize; /* Table header */ { DTableDesc dtd = HUF_getDTableDesc(DTable); U32 const maxTableLog = dtd.maxTableLog + 1; U32 const targetTableLog = MIN(maxTableLog, HUF_DECODER_FAST_TABLELOG); tableLog = HUF_rescaleStats(wksp->huffWeight, wksp->rankVal, nbSymbols, tableLog, targetTableLog); if (tableLog > (U32)(dtd.maxTableLog+1)) return ERROR(tableLog_tooLarge); /* DTable too small, Huffman tree cannot fit in */ dtd.tableType = 0; dtd.tableLog = (BYTE)tableLog; ZSTD_memcpy(DTable, &dtd, sizeof(dtd)); } /* Compute symbols and rankStart given rankVal: * * rankVal already contains the number of values of each weight. * * symbols contains the symbols ordered by weight. First are the rankVal[0] * weight 0 symbols, followed by the rankVal[1] weight 1 symbols, and so on. * symbols[0] is filled (but unused) to avoid a branch. * * rankStart contains the offset where each rank belongs in the DTable. * rankStart[0] is not filled because there are no entries in the table for * weight 0. */ { int n; U32 nextRankStart = 0; int const unroll = 4; int const nLimit = (int)nbSymbols - unroll + 1; for (n=0; n<(int)tableLog+1; n++) { U32 const curr = nextRankStart; nextRankStart += wksp->rankVal[n]; wksp->rankStart[n] = curr; } for (n=0; n < nLimit; n += unroll) { int u; for (u=0; u < unroll; ++u) { size_t const w = wksp->huffWeight[n+u]; wksp->symbols[wksp->rankStart[w]++] = (BYTE)(n+u); } } for (; n < (int)nbSymbols; ++n) { size_t const w = wksp->huffWeight[n]; wksp->symbols[wksp->rankStart[w]++] = (BYTE)n; } } /* fill DTable * We fill all entries of each weight in order. * That way length is a constant for each iteration of the outer loop. * We can switch based on the length to a different inner loop which is * optimized for that particular case. */ { U32 w; int symbol = wksp->rankVal[0]; int rankStart = 0; for (w=1; wrankVal[w]; int const length = (1 << w) >> 1; int uStart = rankStart; BYTE const nbBits = (BYTE)(tableLog + 1 - w); int s; int u; switch (length) { case 1: for (s=0; ssymbols[symbol + s]; D.nbBits = nbBits; dt[uStart] = D; uStart += 1; } break; case 2: for (s=0; ssymbols[symbol + s]; D.nbBits = nbBits; dt[uStart+0] = D; dt[uStart+1] = D; uStart += 2; } break; case 4: for (s=0; ssymbols[symbol + s], nbBits); MEM_write64(dt + uStart, D4); uStart += 4; } break; case 8: for (s=0; ssymbols[symbol + s], nbBits); MEM_write64(dt + uStart, D4); MEM_write64(dt + uStart + 4, D4); uStart += 8; } break; default: for (s=0; ssymbols[symbol + s], nbBits); for (u=0; u < length; u += 16) { MEM_write64(dt + uStart + u + 0, D4); MEM_write64(dt + uStart + u + 4, D4); MEM_write64(dt + uStart + u + 8, D4); MEM_write64(dt + uStart + u + 12, D4); } assert(u == length); uStart += length; } break; } symbol += symbolCount; rankStart += symbolCount * length; } } return iSize; } FORCE_INLINE_TEMPLATE BYTE HUF_decodeSymbolX1(BIT_DStream_t* Dstream, const HUF_DEltX1* dt, const U32 dtLog) { size_t const val = BIT_lookBitsFast(Dstream, dtLog); /* note : dtLog >= 1 */ BYTE const c = dt[val].byte; BIT_skipBits(Dstream, dt[val].nbBits); return c; } #define HUF_DECODE_SYMBOLX1_0(ptr, DStreamPtr) \ *ptr++ = HUF_decodeSymbolX1(DStreamPtr, dt, dtLog) #define HUF_DECODE_SYMBOLX1_1(ptr, DStreamPtr) \ if (MEM_64bits() || (HUF_TABLELOG_MAX<=12)) \ HUF_DECODE_SYMBOLX1_0(ptr, DStreamPtr) #define HUF_DECODE_SYMBOLX1_2(ptr, DStreamPtr) \ if (MEM_64bits()) \ HUF_DECODE_SYMBOLX1_0(ptr, DStreamPtr) HINT_INLINE size_t HUF_decodeStreamX1(BYTE* p, BIT_DStream_t* const bitDPtr, BYTE* const pEnd, const HUF_DEltX1* const dt, const U32 dtLog) { BYTE* const pStart = p; /* up to 4 symbols at a time */ if ((pEnd - p) > 3) { while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd-3)) { HUF_DECODE_SYMBOLX1_2(p, bitDPtr); HUF_DECODE_SYMBOLX1_1(p, bitDPtr); HUF_DECODE_SYMBOLX1_2(p, bitDPtr); HUF_DECODE_SYMBOLX1_0(p, bitDPtr); } } else { BIT_reloadDStream(bitDPtr); } /* [0-3] symbols remaining */ if (MEM_32bits()) while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd)) HUF_DECODE_SYMBOLX1_0(p, bitDPtr); /* no more data to retrieve from bitstream, no need to reload */ while (p < pEnd) HUF_DECODE_SYMBOLX1_0(p, bitDPtr); return (size_t)(pEnd-pStart); } FORCE_INLINE_TEMPLATE size_t HUF_decompress1X1_usingDTable_internal_body( void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable) { BYTE* op = (BYTE*)dst; BYTE* const oend = op + dstSize; const void* dtPtr = DTable + 1; const HUF_DEltX1* const dt = (const HUF_DEltX1*)dtPtr; BIT_DStream_t bitD; DTableDesc const dtd = HUF_getDTableDesc(DTable); U32 const dtLog = dtd.tableLog; CHECK_F( BIT_initDStream(&bitD, cSrc, cSrcSize) ); HUF_decodeStreamX1(op, &bitD, oend, dt, dtLog); if (!BIT_endOfDStream(&bitD)) return ERROR(corruption_detected); return dstSize; } /* HUF_decompress4X1_usingDTable_internal_body(): * Conditions : * @dstSize >= 6 */ FORCE_INLINE_TEMPLATE size_t HUF_decompress4X1_usingDTable_internal_body( void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable) { /* Check */ if (cSrcSize < 10) return ERROR(corruption_detected); /* strict minimum : jump table + 1 byte per stream */ { const BYTE* const istart = (const BYTE*) cSrc; BYTE* const ostart = (BYTE*) dst; BYTE* const oend = ostart + dstSize; BYTE* const olimit = oend - 3; const void* const dtPtr = DTable + 1; const HUF_DEltX1* const dt = (const HUF_DEltX1*)dtPtr; /* Init */ BIT_DStream_t bitD1; BIT_DStream_t bitD2; BIT_DStream_t bitD3; BIT_DStream_t bitD4; size_t const length1 = MEM_readLE16(istart); size_t const length2 = MEM_readLE16(istart+2); size_t const length3 = MEM_readLE16(istart+4); size_t const length4 = cSrcSize - (length1 + length2 + length3 + 6); const BYTE* const istart1 = istart + 6; /* jumpTable */ const BYTE* const istart2 = istart1 + length1; const BYTE* const istart3 = istart2 + length2; const BYTE* const istart4 = istart3 + length3; const size_t segmentSize = (dstSize+3) / 4; BYTE* const opStart2 = ostart + segmentSize; BYTE* const opStart3 = opStart2 + segmentSize; BYTE* const opStart4 = opStart3 + segmentSize; BYTE* op1 = ostart; BYTE* op2 = opStart2; BYTE* op3 = opStart3; BYTE* op4 = opStart4; DTableDesc const dtd = HUF_getDTableDesc(DTable); U32 const dtLog = dtd.tableLog; U32 endSignal = 1; if (length4 > cSrcSize) return ERROR(corruption_detected); /* overflow */ if (opStart4 > oend) return ERROR(corruption_detected); /* overflow */ if (dstSize < 6) return ERROR(corruption_detected); /* stream 4-split doesn't work */ CHECK_F( BIT_initDStream(&bitD1, istart1, length1) ); CHECK_F( BIT_initDStream(&bitD2, istart2, length2) ); CHECK_F( BIT_initDStream(&bitD3, istart3, length3) ); CHECK_F( BIT_initDStream(&bitD4, istart4, length4) ); /* up to 16 symbols per loop (4 symbols per stream) in 64-bit mode */ if ((size_t)(oend - op4) >= sizeof(size_t)) { for ( ; (endSignal) & (op4 < olimit) ; ) { HUF_DECODE_SYMBOLX1_2(op1, &bitD1); HUF_DECODE_SYMBOLX1_2(op2, &bitD2); HUF_DECODE_SYMBOLX1_2(op3, &bitD3); HUF_DECODE_SYMBOLX1_2(op4, &bitD4); HUF_DECODE_SYMBOLX1_1(op1, &bitD1); HUF_DECODE_SYMBOLX1_1(op2, &bitD2); HUF_DECODE_SYMBOLX1_1(op3, &bitD3); HUF_DECODE_SYMBOLX1_1(op4, &bitD4); HUF_DECODE_SYMBOLX1_2(op1, &bitD1); HUF_DECODE_SYMBOLX1_2(op2, &bitD2); HUF_DECODE_SYMBOLX1_2(op3, &bitD3); HUF_DECODE_SYMBOLX1_2(op4, &bitD4); HUF_DECODE_SYMBOLX1_0(op1, &bitD1); HUF_DECODE_SYMBOLX1_0(op2, &bitD2); HUF_DECODE_SYMBOLX1_0(op3, &bitD3); HUF_DECODE_SYMBOLX1_0(op4, &bitD4); endSignal &= BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished; endSignal &= BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished; endSignal &= BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished; endSignal &= BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished; } } /* check corruption */ /* note : should not be necessary : op# advance in lock step, and we control op4. * but curiously, binary generated by gcc 7.2 & 7.3 with -mbmi2 runs faster when >=1 test is present */ if (op1 > opStart2) return ERROR(corruption_detected); if (op2 > opStart3) return ERROR(corruption_detected); if (op3 > opStart4) return ERROR(corruption_detected); /* note : op4 supposed already verified within main loop */ /* finish bitStreams one by one */ HUF_decodeStreamX1(op1, &bitD1, opStart2, dt, dtLog); HUF_decodeStreamX1(op2, &bitD2, opStart3, dt, dtLog); HUF_decodeStreamX1(op3, &bitD3, opStart4, dt, dtLog); HUF_decodeStreamX1(op4, &bitD4, oend, dt, dtLog); /* check */ { U32 const endCheck = BIT_endOfDStream(&bitD1) & BIT_endOfDStream(&bitD2) & BIT_endOfDStream(&bitD3) & BIT_endOfDStream(&bitD4); if (!endCheck) return ERROR(corruption_detected); } /* decoded size */ return dstSize; } } #if HUF_NEED_BMI2_FUNCTION static BMI2_TARGET_ATTRIBUTE size_t HUF_decompress4X1_usingDTable_internal_bmi2(void* dst, size_t dstSize, void const* cSrc, size_t cSrcSize, HUF_DTable const* DTable) { return HUF_decompress4X1_usingDTable_internal_body(dst, dstSize, cSrc, cSrcSize, DTable); } #endif static size_t HUF_decompress4X1_usingDTable_internal_default(void* dst, size_t dstSize, void const* cSrc, size_t cSrcSize, HUF_DTable const* DTable) { return HUF_decompress4X1_usingDTable_internal_body(dst, dstSize, cSrc, cSrcSize, DTable); } #if ZSTD_ENABLE_ASM_X86_64_BMI2 HUF_ASM_DECL void HUF_decompress4X1_usingDTable_internal_fast_asm_loop(HUF_DecompressFastArgs* args) ZSTDLIB_HIDDEN; #endif static HUF_FAST_BMI2_ATTRS void HUF_decompress4X1_usingDTable_internal_fast_c_loop(HUF_DecompressFastArgs* args) { U64 bits[4]; BYTE const* ip[4]; BYTE* op[4]; U16 const* const dtable = (U16 const*)args->dt; BYTE* const oend = args->oend; BYTE const* const ilimit = args->ilimit; /* Copy the arguments to local variables */ ZSTD_memcpy(&bits, &args->bits, sizeof(bits)); ZSTD_memcpy((void*)(&ip), &args->ip, sizeof(ip)); ZSTD_memcpy(&op, &args->op, sizeof(op)); assert(MEM_isLittleEndian()); assert(!MEM_32bits()); for (;;) { BYTE* olimit; int stream; int symbol; /* Assert loop preconditions */ #ifndef NDEBUG for (stream = 0; stream < 4; ++stream) { assert(op[stream] <= (stream == 3 ? oend : op[stream + 1])); assert(ip[stream] >= ilimit); } #endif /* Compute olimit */ { /* Each iteration produces 5 output symbols per stream */ size_t const oiters = (size_t)(oend - op[3]) / 5; /* Each iteration consumes up to 11 bits * 5 = 55 bits < 7 bytes * per stream. */ size_t const iiters = (size_t)(ip[0] - ilimit) / 7; /* We can safely run iters iterations before running bounds checks */ size_t const iters = MIN(oiters, iiters); size_t const symbols = iters * 5; /* We can simply check that op[3] < olimit, instead of checking all * of our bounds, since we can't hit the other bounds until we've run * iters iterations, which only happens when op[3] == olimit. */ olimit = op[3] + symbols; /* Exit fast decoding loop once we get close to the end. */ if (op[3] + 20 > olimit) break; /* Exit the decoding loop if any input pointer has crossed the * previous one. This indicates corruption, and a precondition * to our loop is that ip[i] >= ip[0]. */ for (stream = 1; stream < 4; ++stream) { if (ip[stream] < ip[stream - 1]) goto _out; } } #ifndef NDEBUG for (stream = 1; stream < 4; ++stream) { assert(ip[stream] >= ip[stream - 1]); } #endif do { /* Decode 5 symbols in each of the 4 streams */ for (symbol = 0; symbol < 5; ++symbol) { for (stream = 0; stream < 4; ++stream) { int const index = (int)(bits[stream] >> 53); int const entry = (int)dtable[index]; bits[stream] <<= (entry & 63); op[stream][symbol] = (BYTE)((entry >> 8) & 0xFF); } } /* Reload the bitstreams */ for (stream = 0; stream < 4; ++stream) { int const ctz = ZSTD_countTrailingZeros64(bits[stream]); int const nbBits = ctz & 7; int const nbBytes = ctz >> 3; op[stream] += 5; ip[stream] -= nbBytes; bits[stream] = MEM_read64(ip[stream]) | 1; bits[stream] <<= nbBits; } } while (op[3] < olimit); } _out: /* Save the final values of each of the state variables back to args. */ ZSTD_memcpy(&args->bits, &bits, sizeof(bits)); ZSTD_memcpy((void*)(&args->ip), &ip, sizeof(ip)); ZSTD_memcpy(&args->op, &op, sizeof(op)); } /** * @returns @p dstSize on success (>= 6) * 0 if the fallback implementation should be used * An error if an error occurred */ static HUF_FAST_BMI2_ATTRS size_t HUF_decompress4X1_usingDTable_internal_fast( void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, HUF_DecompressFastLoopFn loopFn) { void const* dt = DTable + 1; const BYTE* const iend = (const BYTE*)cSrc + 6; BYTE* const oend = (BYTE*)dst + dstSize; HUF_DecompressFastArgs args; { size_t const ret = HUF_DecompressFastArgs_init(&args, dst, dstSize, cSrc, cSrcSize, DTable); FORWARD_IF_ERROR(ret, "Failed to init fast loop args"); if (ret == 0) return 0; } assert(args.ip[0] >= args.ilimit); loopFn(&args); /* Our loop guarantees that ip[] >= ilimit and that we haven't * overwritten any op[]. */ assert(args.ip[0] >= iend); assert(args.ip[1] >= iend); assert(args.ip[2] >= iend); assert(args.ip[3] >= iend); assert(args.op[3] <= oend); (void)iend; /* finish bit streams one by one. */ { size_t const segmentSize = (dstSize+3) / 4; BYTE* segmentEnd = (BYTE*)dst; int i; for (i = 0; i < 4; ++i) { BIT_DStream_t bit; if (segmentSize <= (size_t)(oend - segmentEnd)) segmentEnd += segmentSize; else segmentEnd = oend; FORWARD_IF_ERROR(HUF_initRemainingDStream(&bit, &args, i, segmentEnd), "corruption"); /* Decompress and validate that we've produced exactly the expected length. */ args.op[i] += HUF_decodeStreamX1(args.op[i], &bit, segmentEnd, (HUF_DEltX1 const*)dt, HUF_DECODER_FAST_TABLELOG); if (args.op[i] != segmentEnd) return ERROR(corruption_detected); } } /* decoded size */ assert(dstSize != 0); return dstSize; } HUF_DGEN(HUF_decompress1X1_usingDTable_internal) static size_t HUF_decompress4X1_usingDTable_internal(void* dst, size_t dstSize, void const* cSrc, size_t cSrcSize, HUF_DTable const* DTable, int flags) { HUF_DecompressUsingDTableFn fallbackFn = HUF_decompress4X1_usingDTable_internal_default; HUF_DecompressFastLoopFn loopFn = HUF_decompress4X1_usingDTable_internal_fast_c_loop; #if DYNAMIC_BMI2 if (flags & HUF_flags_bmi2) { fallbackFn = HUF_decompress4X1_usingDTable_internal_bmi2; # if ZSTD_ENABLE_ASM_X86_64_BMI2 if (!(flags & HUF_flags_disableAsm)) { loopFn = HUF_decompress4X1_usingDTable_internal_fast_asm_loop; } # endif } else { return fallbackFn(dst, dstSize, cSrc, cSrcSize, DTable); } #endif #if ZSTD_ENABLE_ASM_X86_64_BMI2 && defined(__BMI2__) if (!(flags & HUF_flags_disableAsm)) { loopFn = HUF_decompress4X1_usingDTable_internal_fast_asm_loop; } #endif if (!(flags & HUF_flags_disableFast)) { size_t const ret = HUF_decompress4X1_usingDTable_internal_fast(dst, dstSize, cSrc, cSrcSize, DTable, loopFn); if (ret != 0) return ret; } return fallbackFn(dst, dstSize, cSrc, cSrcSize, DTable); } static size_t HUF_decompress4X1_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags) { const BYTE* ip = (const BYTE*) cSrc; size_t const hSize = HUF_readDTableX1_wksp(dctx, cSrc, cSrcSize, workSpace, wkspSize, flags); if (HUF_isError(hSize)) return hSize; if (hSize >= cSrcSize) return ERROR(srcSize_wrong); ip += hSize; cSrcSize -= hSize; return HUF_decompress4X1_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx, flags); } #endif /* HUF_FORCE_DECOMPRESS_X2 */ #ifndef HUF_FORCE_DECOMPRESS_X1 /* *************************/ /* double-symbols decoding */ /* *************************/ typedef struct { U16 sequence; BYTE nbBits; BYTE length; } HUF_DEltX2; /* double-symbols decoding */ typedef struct { BYTE symbol; } sortedSymbol_t; typedef U32 rankValCol_t[HUF_TABLELOG_MAX + 1]; typedef rankValCol_t rankVal_t[HUF_TABLELOG_MAX]; /** * Constructs a HUF_DEltX2 in a U32. */ static U32 HUF_buildDEltX2U32(U32 symbol, U32 nbBits, U32 baseSeq, int level) { U32 seq; DEBUG_STATIC_ASSERT(offsetof(HUF_DEltX2, sequence) == 0); DEBUG_STATIC_ASSERT(offsetof(HUF_DEltX2, nbBits) == 2); DEBUG_STATIC_ASSERT(offsetof(HUF_DEltX2, length) == 3); DEBUG_STATIC_ASSERT(sizeof(HUF_DEltX2) == sizeof(U32)); if (MEM_isLittleEndian()) { seq = level == 1 ? symbol : (baseSeq + (symbol << 8)); return seq + (nbBits << 16) + ((U32)level << 24); } else { seq = level == 1 ? (symbol << 8) : ((baseSeq << 8) + symbol); return (seq << 16) + (nbBits << 8) + (U32)level; } } /** * Constructs a HUF_DEltX2. */ static HUF_DEltX2 HUF_buildDEltX2(U32 symbol, U32 nbBits, U32 baseSeq, int level) { HUF_DEltX2 DElt; U32 const val = HUF_buildDEltX2U32(symbol, nbBits, baseSeq, level); DEBUG_STATIC_ASSERT(sizeof(DElt) == sizeof(val)); ZSTD_memcpy(&DElt, &val, sizeof(val)); return DElt; } /** * Constructs 2 HUF_DEltX2s and packs them into a U64. */ static U64 HUF_buildDEltX2U64(U32 symbol, U32 nbBits, U16 baseSeq, int level) { U32 DElt = HUF_buildDEltX2U32(symbol, nbBits, baseSeq, level); return (U64)DElt + ((U64)DElt << 32); } /** * Fills the DTable rank with all the symbols from [begin, end) that are each * nbBits long. * * @param DTableRank The start of the rank in the DTable. * @param begin The first symbol to fill (inclusive). * @param end The last symbol to fill (exclusive). * @param nbBits Each symbol is nbBits long. * @param tableLog The table log. * @param baseSeq If level == 1 { 0 } else { the first level symbol } * @param level The level in the table. Must be 1 or 2. */ static void HUF_fillDTableX2ForWeight( HUF_DEltX2* DTableRank, sortedSymbol_t const* begin, sortedSymbol_t const* end, U32 nbBits, U32 tableLog, U16 baseSeq, int const level) { U32 const length = 1U << ((tableLog - nbBits) & 0x1F /* quiet static-analyzer */); const sortedSymbol_t* ptr; assert(level >= 1 && level <= 2); switch (length) { case 1: for (ptr = begin; ptr != end; ++ptr) { HUF_DEltX2 const DElt = HUF_buildDEltX2(ptr->symbol, nbBits, baseSeq, level); *DTableRank++ = DElt; } break; case 2: for (ptr = begin; ptr != end; ++ptr) { HUF_DEltX2 const DElt = HUF_buildDEltX2(ptr->symbol, nbBits, baseSeq, level); DTableRank[0] = DElt; DTableRank[1] = DElt; DTableRank += 2; } break; case 4: for (ptr = begin; ptr != end; ++ptr) { U64 const DEltX2 = HUF_buildDEltX2U64(ptr->symbol, nbBits, baseSeq, level); ZSTD_memcpy(DTableRank + 0, &DEltX2, sizeof(DEltX2)); ZSTD_memcpy(DTableRank + 2, &DEltX2, sizeof(DEltX2)); DTableRank += 4; } break; case 8: for (ptr = begin; ptr != end; ++ptr) { U64 const DEltX2 = HUF_buildDEltX2U64(ptr->symbol, nbBits, baseSeq, level); ZSTD_memcpy(DTableRank + 0, &DEltX2, sizeof(DEltX2)); ZSTD_memcpy(DTableRank + 2, &DEltX2, sizeof(DEltX2)); ZSTD_memcpy(DTableRank + 4, &DEltX2, sizeof(DEltX2)); ZSTD_memcpy(DTableRank + 6, &DEltX2, sizeof(DEltX2)); DTableRank += 8; } break; default: for (ptr = begin; ptr != end; ++ptr) { U64 const DEltX2 = HUF_buildDEltX2U64(ptr->symbol, nbBits, baseSeq, level); HUF_DEltX2* const DTableRankEnd = DTableRank + length; for (; DTableRank != DTableRankEnd; DTableRank += 8) { ZSTD_memcpy(DTableRank + 0, &DEltX2, sizeof(DEltX2)); ZSTD_memcpy(DTableRank + 2, &DEltX2, sizeof(DEltX2)); ZSTD_memcpy(DTableRank + 4, &DEltX2, sizeof(DEltX2)); ZSTD_memcpy(DTableRank + 6, &DEltX2, sizeof(DEltX2)); } } break; } } /* HUF_fillDTableX2Level2() : * `rankValOrigin` must be a table of at least (HUF_TABLELOG_MAX + 1) U32 */ static void HUF_fillDTableX2Level2(HUF_DEltX2* DTable, U32 targetLog, const U32 consumedBits, const U32* rankVal, const int minWeight, const int maxWeight1, const sortedSymbol_t* sortedSymbols, U32 const* rankStart, U32 nbBitsBaseline, U16 baseSeq) { /* Fill skipped values (all positions up to rankVal[minWeight]). * These are positions only get a single symbol because the combined weight * is too large. */ if (minWeight>1) { U32 const length = 1U << ((targetLog - consumedBits) & 0x1F /* quiet static-analyzer */); U64 const DEltX2 = HUF_buildDEltX2U64(baseSeq, consumedBits, /* baseSeq */ 0, /* level */ 1); int const skipSize = rankVal[minWeight]; assert(length > 1); assert((U32)skipSize < length); switch (length) { case 2: assert(skipSize == 1); ZSTD_memcpy(DTable, &DEltX2, sizeof(DEltX2)); break; case 4: assert(skipSize <= 4); ZSTD_memcpy(DTable + 0, &DEltX2, sizeof(DEltX2)); ZSTD_memcpy(DTable + 2, &DEltX2, sizeof(DEltX2)); break; default: { int i; for (i = 0; i < skipSize; i += 8) { ZSTD_memcpy(DTable + i + 0, &DEltX2, sizeof(DEltX2)); ZSTD_memcpy(DTable + i + 2, &DEltX2, sizeof(DEltX2)); ZSTD_memcpy(DTable + i + 4, &DEltX2, sizeof(DEltX2)); ZSTD_memcpy(DTable + i + 6, &DEltX2, sizeof(DEltX2)); } } } } /* Fill each of the second level symbols by weight. */ { int w; for (w = minWeight; w < maxWeight1; ++w) { int const begin = rankStart[w]; int const end = rankStart[w+1]; U32 const nbBits = nbBitsBaseline - w; U32 const totalBits = nbBits + consumedBits; HUF_fillDTableX2ForWeight( DTable + rankVal[w], sortedSymbols + begin, sortedSymbols + end, totalBits, targetLog, baseSeq, /* level */ 2); } } } static void HUF_fillDTableX2(HUF_DEltX2* DTable, const U32 targetLog, const sortedSymbol_t* sortedList, const U32* rankStart, rankValCol_t* rankValOrigin, const U32 maxWeight, const U32 nbBitsBaseline) { U32* const rankVal = rankValOrigin[0]; const int scaleLog = nbBitsBaseline - targetLog; /* note : targetLog >= srcLog, hence scaleLog <= 1 */ const U32 minBits = nbBitsBaseline - maxWeight; int w; int const wEnd = (int)maxWeight + 1; /* Fill DTable in order of weight. */ for (w = 1; w < wEnd; ++w) { int const begin = (int)rankStart[w]; int const end = (int)rankStart[w+1]; U32 const nbBits = nbBitsBaseline - w; if (targetLog-nbBits >= minBits) { /* Enough room for a second symbol. */ int start = rankVal[w]; U32 const length = 1U << ((targetLog - nbBits) & 0x1F /* quiet static-analyzer */); int minWeight = nbBits + scaleLog; int s; if (minWeight < 1) minWeight = 1; /* Fill the DTable for every symbol of weight w. * These symbols get at least 1 second symbol. */ for (s = begin; s != end; ++s) { HUF_fillDTableX2Level2( DTable + start, targetLog, nbBits, rankValOrigin[nbBits], minWeight, wEnd, sortedList, rankStart, nbBitsBaseline, sortedList[s].symbol); start += length; } } else { /* Only a single symbol. */ HUF_fillDTableX2ForWeight( DTable + rankVal[w], sortedList + begin, sortedList + end, nbBits, targetLog, /* baseSeq */ 0, /* level */ 1); } } } typedef struct { rankValCol_t rankVal[HUF_TABLELOG_MAX]; U32 rankStats[HUF_TABLELOG_MAX + 1]; U32 rankStart0[HUF_TABLELOG_MAX + 3]; sortedSymbol_t sortedSymbol[HUF_SYMBOLVALUE_MAX + 1]; BYTE weightList[HUF_SYMBOLVALUE_MAX + 1]; U32 calleeWksp[HUF_READ_STATS_WORKSPACE_SIZE_U32]; } HUF_ReadDTableX2_Workspace; size_t HUF_readDTableX2_wksp(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int flags) { U32 tableLog, maxW, nbSymbols; DTableDesc dtd = HUF_getDTableDesc(DTable); U32 maxTableLog = dtd.maxTableLog; size_t iSize; void* dtPtr = DTable+1; /* force compiler to avoid strict-aliasing */ HUF_DEltX2* const dt = (HUF_DEltX2*)dtPtr; U32 *rankStart; HUF_ReadDTableX2_Workspace* const wksp = (HUF_ReadDTableX2_Workspace*)workSpace; if (sizeof(*wksp) > wkspSize) return ERROR(GENERIC); rankStart = wksp->rankStart0 + 1; ZSTD_memset(wksp->rankStats, 0, sizeof(wksp->rankStats)); ZSTD_memset(wksp->rankStart0, 0, sizeof(wksp->rankStart0)); DEBUG_STATIC_ASSERT(sizeof(HUF_DEltX2) == sizeof(HUF_DTable)); /* if compiler fails here, assertion is wrong */ if (maxTableLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge); /* ZSTD_memset(weightList, 0, sizeof(weightList)); */ /* is not necessary, even though some analyzer complain ... */ iSize = HUF_readStats_wksp(wksp->weightList, HUF_SYMBOLVALUE_MAX + 1, wksp->rankStats, &nbSymbols, &tableLog, src, srcSize, wksp->calleeWksp, sizeof(wksp->calleeWksp), flags); if (HUF_isError(iSize)) return iSize; /* check result */ if (tableLog > maxTableLog) return ERROR(tableLog_tooLarge); /* DTable can't fit code depth */ if (tableLog <= HUF_DECODER_FAST_TABLELOG && maxTableLog > HUF_DECODER_FAST_TABLELOG) maxTableLog = HUF_DECODER_FAST_TABLELOG; /* find maxWeight */ for (maxW = tableLog; wksp->rankStats[maxW]==0; maxW--) {} /* necessarily finds a solution before 0 */ /* Get start index of each weight */ { U32 w, nextRankStart = 0; for (w=1; wrankStats[w]; rankStart[w] = curr; } rankStart[0] = nextRankStart; /* put all 0w symbols at the end of sorted list*/ rankStart[maxW+1] = nextRankStart; } /* sort symbols by weight */ { U32 s; for (s=0; sweightList[s]; U32 const r = rankStart[w]++; wksp->sortedSymbol[r].symbol = (BYTE)s; } rankStart[0] = 0; /* forget 0w symbols; this is beginning of weight(1) */ } /* Build rankVal */ { U32* const rankVal0 = wksp->rankVal[0]; { int const rescale = (maxTableLog-tableLog) - 1; /* tableLog <= maxTableLog */ U32 nextRankVal = 0; U32 w; for (w=1; wrankStats[w] << (w+rescale); rankVal0[w] = curr; } } { U32 const minBits = tableLog+1 - maxW; U32 consumed; for (consumed = minBits; consumed < maxTableLog - minBits + 1; consumed++) { U32* const rankValPtr = wksp->rankVal[consumed]; U32 w; for (w = 1; w < maxW+1; w++) { rankValPtr[w] = rankVal0[w] >> consumed; } } } } HUF_fillDTableX2(dt, maxTableLog, wksp->sortedSymbol, wksp->rankStart0, wksp->rankVal, maxW, tableLog+1); dtd.tableLog = (BYTE)maxTableLog; dtd.tableType = 1; ZSTD_memcpy(DTable, &dtd, sizeof(dtd)); return iSize; } FORCE_INLINE_TEMPLATE U32 HUF_decodeSymbolX2(void* op, BIT_DStream_t* DStream, const HUF_DEltX2* dt, const U32 dtLog) { size_t const val = BIT_lookBitsFast(DStream, dtLog); /* note : dtLog >= 1 */ ZSTD_memcpy(op, &dt[val].sequence, 2); BIT_skipBits(DStream, dt[val].nbBits); return dt[val].length; } FORCE_INLINE_TEMPLATE U32 HUF_decodeLastSymbolX2(void* op, BIT_DStream_t* DStream, const HUF_DEltX2* dt, const U32 dtLog) { size_t const val = BIT_lookBitsFast(DStream, dtLog); /* note : dtLog >= 1 */ ZSTD_memcpy(op, &dt[val].sequence, 1); if (dt[val].length==1) { BIT_skipBits(DStream, dt[val].nbBits); } else { if (DStream->bitsConsumed < (sizeof(DStream->bitContainer)*8)) { BIT_skipBits(DStream, dt[val].nbBits); if (DStream->bitsConsumed > (sizeof(DStream->bitContainer)*8)) /* ugly hack; works only because it's the last symbol. Note : can't easily extract nbBits from just this symbol */ DStream->bitsConsumed = (sizeof(DStream->bitContainer)*8); } } return 1; } #define HUF_DECODE_SYMBOLX2_0(ptr, DStreamPtr) \ ptr += HUF_decodeSymbolX2(ptr, DStreamPtr, dt, dtLog) #define HUF_DECODE_SYMBOLX2_1(ptr, DStreamPtr) \ if (MEM_64bits() || (HUF_TABLELOG_MAX<=12)) \ ptr += HUF_decodeSymbolX2(ptr, DStreamPtr, dt, dtLog) #define HUF_DECODE_SYMBOLX2_2(ptr, DStreamPtr) \ if (MEM_64bits()) \ ptr += HUF_decodeSymbolX2(ptr, DStreamPtr, dt, dtLog) HINT_INLINE size_t HUF_decodeStreamX2(BYTE* p, BIT_DStream_t* bitDPtr, BYTE* const pEnd, const HUF_DEltX2* const dt, const U32 dtLog) { BYTE* const pStart = p; /* up to 8 symbols at a time */ if ((size_t)(pEnd - p) >= sizeof(bitDPtr->bitContainer)) { if (dtLog <= 11 && MEM_64bits()) { /* up to 10 symbols at a time */ while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd-9)) { HUF_DECODE_SYMBOLX2_0(p, bitDPtr); HUF_DECODE_SYMBOLX2_0(p, bitDPtr); HUF_DECODE_SYMBOLX2_0(p, bitDPtr); HUF_DECODE_SYMBOLX2_0(p, bitDPtr); HUF_DECODE_SYMBOLX2_0(p, bitDPtr); } } else { /* up to 8 symbols at a time */ while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd-(sizeof(bitDPtr->bitContainer)-1))) { HUF_DECODE_SYMBOLX2_2(p, bitDPtr); HUF_DECODE_SYMBOLX2_1(p, bitDPtr); HUF_DECODE_SYMBOLX2_2(p, bitDPtr); HUF_DECODE_SYMBOLX2_0(p, bitDPtr); } } } else { BIT_reloadDStream(bitDPtr); } /* closer to end : up to 2 symbols at a time */ if ((size_t)(pEnd - p) >= 2) { while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p <= pEnd-2)) HUF_DECODE_SYMBOLX2_0(p, bitDPtr); while (p <= pEnd-2) HUF_DECODE_SYMBOLX2_0(p, bitDPtr); /* no need to reload : reached the end of DStream */ } if (p < pEnd) p += HUF_decodeLastSymbolX2(p, bitDPtr, dt, dtLog); return p-pStart; } FORCE_INLINE_TEMPLATE size_t HUF_decompress1X2_usingDTable_internal_body( void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable) { BIT_DStream_t bitD; /* Init */ CHECK_F( BIT_initDStream(&bitD, cSrc, cSrcSize) ); /* decode */ { BYTE* const ostart = (BYTE*) dst; BYTE* const oend = ostart + dstSize; const void* const dtPtr = DTable+1; /* force compiler to not use strict-aliasing */ const HUF_DEltX2* const dt = (const HUF_DEltX2*)dtPtr; DTableDesc const dtd = HUF_getDTableDesc(DTable); HUF_decodeStreamX2(ostart, &bitD, oend, dt, dtd.tableLog); } /* check */ if (!BIT_endOfDStream(&bitD)) return ERROR(corruption_detected); /* decoded size */ return dstSize; } /* HUF_decompress4X2_usingDTable_internal_body(): * Conditions: * @dstSize >= 6 */ FORCE_INLINE_TEMPLATE size_t HUF_decompress4X2_usingDTable_internal_body( void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable) { if (cSrcSize < 10) return ERROR(corruption_detected); /* strict minimum : jump table + 1 byte per stream */ { const BYTE* const istart = (const BYTE*) cSrc; BYTE* const ostart = (BYTE*) dst; BYTE* const oend = ostart + dstSize; BYTE* const olimit = oend - (sizeof(size_t)-1); const void* const dtPtr = DTable+1; const HUF_DEltX2* const dt = (const HUF_DEltX2*)dtPtr; /* Init */ BIT_DStream_t bitD1; BIT_DStream_t bitD2; BIT_DStream_t bitD3; BIT_DStream_t bitD4; size_t const length1 = MEM_readLE16(istart); size_t const length2 = MEM_readLE16(istart+2); size_t const length3 = MEM_readLE16(istart+4); size_t const length4 = cSrcSize - (length1 + length2 + length3 + 6); const BYTE* const istart1 = istart + 6; /* jumpTable */ const BYTE* const istart2 = istart1 + length1; const BYTE* const istart3 = istart2 + length2; const BYTE* const istart4 = istart3 + length3; size_t const segmentSize = (dstSize+3) / 4; BYTE* const opStart2 = ostart + segmentSize; BYTE* const opStart3 = opStart2 + segmentSize; BYTE* const opStart4 = opStart3 + segmentSize; BYTE* op1 = ostart; BYTE* op2 = opStart2; BYTE* op3 = opStart3; BYTE* op4 = opStart4; U32 endSignal = 1; DTableDesc const dtd = HUF_getDTableDesc(DTable); U32 const dtLog = dtd.tableLog; if (length4 > cSrcSize) return ERROR(corruption_detected); /* overflow */ if (opStart4 > oend) return ERROR(corruption_detected); /* overflow */ if (dstSize < 6) return ERROR(corruption_detected); /* stream 4-split doesn't work */ CHECK_F( BIT_initDStream(&bitD1, istart1, length1) ); CHECK_F( BIT_initDStream(&bitD2, istart2, length2) ); CHECK_F( BIT_initDStream(&bitD3, istart3, length3) ); CHECK_F( BIT_initDStream(&bitD4, istart4, length4) ); /* 16-32 symbols per loop (4-8 symbols per stream) */ if ((size_t)(oend - op4) >= sizeof(size_t)) { for ( ; (endSignal) & (op4 < olimit); ) { #if defined(__clang__) && (defined(__x86_64__) || defined(__i386__)) HUF_DECODE_SYMBOLX2_2(op1, &bitD1); HUF_DECODE_SYMBOLX2_1(op1, &bitD1); HUF_DECODE_SYMBOLX2_2(op1, &bitD1); HUF_DECODE_SYMBOLX2_0(op1, &bitD1); HUF_DECODE_SYMBOLX2_2(op2, &bitD2); HUF_DECODE_SYMBOLX2_1(op2, &bitD2); HUF_DECODE_SYMBOLX2_2(op2, &bitD2); HUF_DECODE_SYMBOLX2_0(op2, &bitD2); endSignal &= BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished; endSignal &= BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished; HUF_DECODE_SYMBOLX2_2(op3, &bitD3); HUF_DECODE_SYMBOLX2_1(op3, &bitD3); HUF_DECODE_SYMBOLX2_2(op3, &bitD3); HUF_DECODE_SYMBOLX2_0(op3, &bitD3); HUF_DECODE_SYMBOLX2_2(op4, &bitD4); HUF_DECODE_SYMBOLX2_1(op4, &bitD4); HUF_DECODE_SYMBOLX2_2(op4, &bitD4); HUF_DECODE_SYMBOLX2_0(op4, &bitD4); endSignal &= BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished; endSignal &= BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished; #else HUF_DECODE_SYMBOLX2_2(op1, &bitD1); HUF_DECODE_SYMBOLX2_2(op2, &bitD2); HUF_DECODE_SYMBOLX2_2(op3, &bitD3); HUF_DECODE_SYMBOLX2_2(op4, &bitD4); HUF_DECODE_SYMBOLX2_1(op1, &bitD1); HUF_DECODE_SYMBOLX2_1(op2, &bitD2); HUF_DECODE_SYMBOLX2_1(op3, &bitD3); HUF_DECODE_SYMBOLX2_1(op4, &bitD4); HUF_DECODE_SYMBOLX2_2(op1, &bitD1); HUF_DECODE_SYMBOLX2_2(op2, &bitD2); HUF_DECODE_SYMBOLX2_2(op3, &bitD3); HUF_DECODE_SYMBOLX2_2(op4, &bitD4); HUF_DECODE_SYMBOLX2_0(op1, &bitD1); HUF_DECODE_SYMBOLX2_0(op2, &bitD2); HUF_DECODE_SYMBOLX2_0(op3, &bitD3); HUF_DECODE_SYMBOLX2_0(op4, &bitD4); endSignal = (U32)LIKELY((U32) (BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished) & (BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished) & (BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished) & (BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished)); #endif } } /* check corruption */ if (op1 > opStart2) return ERROR(corruption_detected); if (op2 > opStart3) return ERROR(corruption_detected); if (op3 > opStart4) return ERROR(corruption_detected); /* note : op4 already verified within main loop */ /* finish bitStreams one by one */ HUF_decodeStreamX2(op1, &bitD1, opStart2, dt, dtLog); HUF_decodeStreamX2(op2, &bitD2, opStart3, dt, dtLog); HUF_decodeStreamX2(op3, &bitD3, opStart4, dt, dtLog); HUF_decodeStreamX2(op4, &bitD4, oend, dt, dtLog); /* check */ { U32 const endCheck = BIT_endOfDStream(&bitD1) & BIT_endOfDStream(&bitD2) & BIT_endOfDStream(&bitD3) & BIT_endOfDStream(&bitD4); if (!endCheck) return ERROR(corruption_detected); } /* decoded size */ return dstSize; } } #if HUF_NEED_BMI2_FUNCTION static BMI2_TARGET_ATTRIBUTE size_t HUF_decompress4X2_usingDTable_internal_bmi2(void* dst, size_t dstSize, void const* cSrc, size_t cSrcSize, HUF_DTable const* DTable) { return HUF_decompress4X2_usingDTable_internal_body(dst, dstSize, cSrc, cSrcSize, DTable); } #endif static size_t HUF_decompress4X2_usingDTable_internal_default(void* dst, size_t dstSize, void const* cSrc, size_t cSrcSize, HUF_DTable const* DTable) { return HUF_decompress4X2_usingDTable_internal_body(dst, dstSize, cSrc, cSrcSize, DTable); } #if ZSTD_ENABLE_ASM_X86_64_BMI2 HUF_ASM_DECL void HUF_decompress4X2_usingDTable_internal_fast_asm_loop(HUF_DecompressFastArgs* args) ZSTDLIB_HIDDEN; #endif static HUF_FAST_BMI2_ATTRS void HUF_decompress4X2_usingDTable_internal_fast_c_loop(HUF_DecompressFastArgs* args) { U64 bits[4]; BYTE const* ip[4]; BYTE* op[4]; BYTE* oend[4]; HUF_DEltX2 const* const dtable = (HUF_DEltX2 const*)args->dt; BYTE const* const ilimit = args->ilimit; /* Copy the arguments to local registers. */ ZSTD_memcpy(&bits, &args->bits, sizeof(bits)); ZSTD_memcpy((void*)(&ip), &args->ip, sizeof(ip)); ZSTD_memcpy(&op, &args->op, sizeof(op)); oend[0] = op[1]; oend[1] = op[2]; oend[2] = op[3]; oend[3] = args->oend; assert(MEM_isLittleEndian()); assert(!MEM_32bits()); for (;;) { BYTE* olimit; int stream; int symbol; /* Assert loop preconditions */ #ifndef NDEBUG for (stream = 0; stream < 4; ++stream) { assert(op[stream] <= oend[stream]); assert(ip[stream] >= ilimit); } #endif /* Compute olimit */ { /* Each loop does 5 table lookups for each of the 4 streams. * Each table lookup consumes up to 11 bits of input, and produces * up to 2 bytes of output. */ /* We can consume up to 7 bytes of input per iteration per stream. * We also know that each input pointer is >= ip[0]. So we can run * iters loops before running out of input. */ size_t iters = (size_t)(ip[0] - ilimit) / 7; /* Each iteration can produce up to 10 bytes of output per stream. * Each output stream my advance at different rates. So take the * minimum number of safe iterations among all the output streams. */ for (stream = 0; stream < 4; ++stream) { size_t const oiters = (size_t)(oend[stream] - op[stream]) / 10; iters = MIN(iters, oiters); } /* Each iteration produces at least 5 output symbols. So until * op[3] crosses olimit, we know we haven't executed iters * iterations yet. This saves us maintaining an iters counter, * at the expense of computing the remaining # of iterations * more frequently. */ olimit = op[3] + (iters * 5); /* Exit the fast decoding loop if we are too close to the end. */ if (op[3] + 10 > olimit) break; /* Exit the decoding loop if any input pointer has crossed the * previous one. This indicates corruption, and a precondition * to our loop is that ip[i] >= ip[0]. */ for (stream = 1; stream < 4; ++stream) { if (ip[stream] < ip[stream - 1]) goto _out; } } #ifndef NDEBUG for (stream = 1; stream < 4; ++stream) { assert(ip[stream] >= ip[stream - 1]); } #endif do { /* Do 5 table lookups for each of the first 3 streams */ for (symbol = 0; symbol < 5; ++symbol) { for (stream = 0; stream < 3; ++stream) { int const index = (int)(bits[stream] >> 53); HUF_DEltX2 const entry = dtable[index]; MEM_write16(op[stream], entry.sequence); bits[stream] <<= (entry.nbBits); op[stream] += (entry.length); } } /* Do 1 table lookup from the final stream */ { int const index = (int)(bits[3] >> 53); HUF_DEltX2 const entry = dtable[index]; MEM_write16(op[3], entry.sequence); bits[3] <<= (entry.nbBits); op[3] += (entry.length); } /* Do 4 table lookups from the final stream & reload bitstreams */ for (stream = 0; stream < 4; ++stream) { /* Do a table lookup from the final stream. * This is interleaved with the reloading to reduce register * pressure. This shouldn't be necessary, but compilers can * struggle with codegen with high register pressure. */ { int const index = (int)(bits[3] >> 53); HUF_DEltX2 const entry = dtable[index]; MEM_write16(op[3], entry.sequence); bits[3] <<= (entry.nbBits); op[3] += (entry.length); } /* Reload the bistreams. The final bitstream must be reloaded * after the 5th symbol was decoded. */ { int const ctz = ZSTD_countTrailingZeros64(bits[stream]); int const nbBits = ctz & 7; int const nbBytes = ctz >> 3; ip[stream] -= nbBytes; bits[stream] = MEM_read64(ip[stream]) | 1; bits[stream] <<= nbBits; } } } while (op[3] < olimit); } _out: /* Save the final values of each of the state variables back to args. */ ZSTD_memcpy(&args->bits, &bits, sizeof(bits)); ZSTD_memcpy((void*)(&args->ip), &ip, sizeof(ip)); ZSTD_memcpy(&args->op, &op, sizeof(op)); } static HUF_FAST_BMI2_ATTRS size_t HUF_decompress4X2_usingDTable_internal_fast( void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, HUF_DecompressFastLoopFn loopFn) { void const* dt = DTable + 1; const BYTE* const iend = (const BYTE*)cSrc + 6; BYTE* const oend = (BYTE*)dst + dstSize; HUF_DecompressFastArgs args; { size_t const ret = HUF_DecompressFastArgs_init(&args, dst, dstSize, cSrc, cSrcSize, DTable); FORWARD_IF_ERROR(ret, "Failed to init asm args"); if (ret == 0) return 0; } assert(args.ip[0] >= args.ilimit); loopFn(&args); /* note : op4 already verified within main loop */ assert(args.ip[0] >= iend); assert(args.ip[1] >= iend); assert(args.ip[2] >= iend); assert(args.ip[3] >= iend); assert(args.op[3] <= oend); (void)iend; /* finish bitStreams one by one */ { size_t const segmentSize = (dstSize+3) / 4; BYTE* segmentEnd = (BYTE*)dst; int i; for (i = 0; i < 4; ++i) { BIT_DStream_t bit; if (segmentSize <= (size_t)(oend - segmentEnd)) segmentEnd += segmentSize; else segmentEnd = oend; FORWARD_IF_ERROR(HUF_initRemainingDStream(&bit, &args, i, segmentEnd), "corruption"); args.op[i] += HUF_decodeStreamX2(args.op[i], &bit, segmentEnd, (HUF_DEltX2 const*)dt, HUF_DECODER_FAST_TABLELOG); if (args.op[i] != segmentEnd) return ERROR(corruption_detected); } } /* decoded size */ return dstSize; } static size_t HUF_decompress4X2_usingDTable_internal(void* dst, size_t dstSize, void const* cSrc, size_t cSrcSize, HUF_DTable const* DTable, int flags) { HUF_DecompressUsingDTableFn fallbackFn = HUF_decompress4X2_usingDTable_internal_default; HUF_DecompressFastLoopFn loopFn = HUF_decompress4X2_usingDTable_internal_fast_c_loop; #if DYNAMIC_BMI2 if (flags & HUF_flags_bmi2) { fallbackFn = HUF_decompress4X2_usingDTable_internal_bmi2; # if ZSTD_ENABLE_ASM_X86_64_BMI2 if (!(flags & HUF_flags_disableAsm)) { loopFn = HUF_decompress4X2_usingDTable_internal_fast_asm_loop; } # endif } else { return fallbackFn(dst, dstSize, cSrc, cSrcSize, DTable); } #endif #if ZSTD_ENABLE_ASM_X86_64_BMI2 && defined(__BMI2__) if (!(flags & HUF_flags_disableAsm)) { loopFn = HUF_decompress4X2_usingDTable_internal_fast_asm_loop; } #endif if (!(flags & HUF_flags_disableFast)) { size_t const ret = HUF_decompress4X2_usingDTable_internal_fast(dst, dstSize, cSrc, cSrcSize, DTable, loopFn); if (ret != 0) return ret; } return fallbackFn(dst, dstSize, cSrc, cSrcSize, DTable); } HUF_DGEN(HUF_decompress1X2_usingDTable_internal) size_t HUF_decompress1X2_DCtx_wksp(HUF_DTable* DCtx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags) { const BYTE* ip = (const BYTE*) cSrc; size_t const hSize = HUF_readDTableX2_wksp(DCtx, cSrc, cSrcSize, workSpace, wkspSize, flags); if (HUF_isError(hSize)) return hSize; if (hSize >= cSrcSize) return ERROR(srcSize_wrong); ip += hSize; cSrcSize -= hSize; return HUF_decompress1X2_usingDTable_internal(dst, dstSize, ip, cSrcSize, DCtx, flags); } static size_t HUF_decompress4X2_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags) { const BYTE* ip = (const BYTE*) cSrc; size_t hSize = HUF_readDTableX2_wksp(dctx, cSrc, cSrcSize, workSpace, wkspSize, flags); if (HUF_isError(hSize)) return hSize; if (hSize >= cSrcSize) return ERROR(srcSize_wrong); ip += hSize; cSrcSize -= hSize; return HUF_decompress4X2_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx, flags); } #endif /* HUF_FORCE_DECOMPRESS_X1 */ /* ***********************************/ /* Universal decompression selectors */ /* ***********************************/ #if !defined(HUF_FORCE_DECOMPRESS_X1) && !defined(HUF_FORCE_DECOMPRESS_X2) typedef struct { U32 tableTime; U32 decode256Time; } algo_time_t; static const algo_time_t algoTime[16 /* Quantization */][2 /* single, double */] = { /* single, double, quad */ {{0,0}, {1,1}}, /* Q==0 : impossible */ {{0,0}, {1,1}}, /* Q==1 : impossible */ {{ 150,216}, { 381,119}}, /* Q == 2 : 12-18% */ {{ 170,205}, { 514,112}}, /* Q == 3 : 18-25% */ {{ 177,199}, { 539,110}}, /* Q == 4 : 25-32% */ {{ 197,194}, { 644,107}}, /* Q == 5 : 32-38% */ {{ 221,192}, { 735,107}}, /* Q == 6 : 38-44% */ {{ 256,189}, { 881,106}}, /* Q == 7 : 44-50% */ {{ 359,188}, {1167,109}}, /* Q == 8 : 50-56% */ {{ 582,187}, {1570,114}}, /* Q == 9 : 56-62% */ {{ 688,187}, {1712,122}}, /* Q ==10 : 62-69% */ {{ 825,186}, {1965,136}}, /* Q ==11 : 69-75% */ {{ 976,185}, {2131,150}}, /* Q ==12 : 75-81% */ {{1180,186}, {2070,175}}, /* Q ==13 : 81-87% */ {{1377,185}, {1731,202}}, /* Q ==14 : 87-93% */ {{1412,185}, {1695,202}}, /* Q ==15 : 93-99% */ }; #endif /** HUF_selectDecoder() : * Tells which decoder is likely to decode faster, * based on a set of pre-computed metrics. * @return : 0==HUF_decompress4X1, 1==HUF_decompress4X2 . * Assumption : 0 < dstSize <= 128 KB */ U32 HUF_selectDecoder (size_t dstSize, size_t cSrcSize) { assert(dstSize > 0); assert(dstSize <= 128*1024); #if defined(HUF_FORCE_DECOMPRESS_X1) (void)dstSize; (void)cSrcSize; return 0; #elif defined(HUF_FORCE_DECOMPRESS_X2) (void)dstSize; (void)cSrcSize; return 1; #else /* decoder timing evaluation */ { U32 const Q = (cSrcSize >= dstSize) ? 15 : (U32)(cSrcSize * 16 / dstSize); /* Q < 16 */ U32 const D256 = (U32)(dstSize >> 8); U32 const DTime0 = algoTime[Q][0].tableTime + (algoTime[Q][0].decode256Time * D256); U32 DTime1 = algoTime[Q][1].tableTime + (algoTime[Q][1].decode256Time * D256); DTime1 += DTime1 >> 5; /* small advantage to algorithm using less memory, to reduce cache eviction */ return DTime1 < DTime0; } #endif } size_t HUF_decompress1X_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags) { /* validation checks */ if (dstSize == 0) return ERROR(dstSize_tooSmall); if (cSrcSize > dstSize) return ERROR(corruption_detected); /* invalid */ if (cSrcSize == dstSize) { ZSTD_memcpy(dst, cSrc, dstSize); return dstSize; } /* not compressed */ if (cSrcSize == 1) { ZSTD_memset(dst, *(const BYTE*)cSrc, dstSize); return dstSize; } /* RLE */ { U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize); #if defined(HUF_FORCE_DECOMPRESS_X1) (void)algoNb; assert(algoNb == 0); return HUF_decompress1X1_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, flags); #elif defined(HUF_FORCE_DECOMPRESS_X2) (void)algoNb; assert(algoNb == 1); return HUF_decompress1X2_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, flags); #else return algoNb ? HUF_decompress1X2_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, flags): HUF_decompress1X1_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, flags); #endif } } size_t HUF_decompress1X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int flags) { DTableDesc const dtd = HUF_getDTableDesc(DTable); #if defined(HUF_FORCE_DECOMPRESS_X1) (void)dtd; assert(dtd.tableType == 0); return HUF_decompress1X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, flags); #elif defined(HUF_FORCE_DECOMPRESS_X2) (void)dtd; assert(dtd.tableType == 1); return HUF_decompress1X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, flags); #else return dtd.tableType ? HUF_decompress1X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, flags) : HUF_decompress1X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, flags); #endif } #ifndef HUF_FORCE_DECOMPRESS_X2 size_t HUF_decompress1X1_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags) { const BYTE* ip = (const BYTE*) cSrc; size_t const hSize = HUF_readDTableX1_wksp(dctx, cSrc, cSrcSize, workSpace, wkspSize, flags); if (HUF_isError(hSize)) return hSize; if (hSize >= cSrcSize) return ERROR(srcSize_wrong); ip += hSize; cSrcSize -= hSize; return HUF_decompress1X1_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx, flags); } #endif size_t HUF_decompress4X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int flags) { DTableDesc const dtd = HUF_getDTableDesc(DTable); #if defined(HUF_FORCE_DECOMPRESS_X1) (void)dtd; assert(dtd.tableType == 0); return HUF_decompress4X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, flags); #elif defined(HUF_FORCE_DECOMPRESS_X2) (void)dtd; assert(dtd.tableType == 1); return HUF_decompress4X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, flags); #else return dtd.tableType ? HUF_decompress4X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, flags) : HUF_decompress4X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, flags); #endif } size_t HUF_decompress4X_hufOnly_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags) { /* validation checks */ if (dstSize == 0) return ERROR(dstSize_tooSmall); if (cSrcSize == 0) return ERROR(corruption_detected); { U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize); #if defined(HUF_FORCE_DECOMPRESS_X1) (void)algoNb; assert(algoNb == 0); return HUF_decompress4X1_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, flags); #elif defined(HUF_FORCE_DECOMPRESS_X2) (void)algoNb; assert(algoNb == 1); return HUF_decompress4X2_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, flags); #else return algoNb ? HUF_decompress4X2_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, flags) : HUF_decompress4X1_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, flags); #endif } } zmat-0.9.9/src/blosc2/internal-complibs/zstd/decompress/huf_decompress_amd64.S0000644000175200007730000003357714515254731027630 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #include "../common/portability_macros.h" /* Stack marking * ref: https://wiki.gentoo.org/wiki/Hardened/GNU_stack_quickstart */ #if defined(__ELF__) && defined(__GNUC__) .section .note.GNU-stack,"",%progbits #endif #if ZSTD_ENABLE_ASM_X86_64_BMI2 /* Calling convention: * * %rdi contains the first argument: HUF_DecompressAsmArgs*. * %rbp isn't maintained (no frame pointer). * %rsp contains the stack pointer that grows down. * No red-zone is assumed, only addresses >= %rsp are used. * All register contents are preserved. * * TODO: Support Windows calling convention. */ ZSTD_HIDE_ASM_FUNCTION(HUF_decompress4X1_usingDTable_internal_fast_asm_loop) ZSTD_HIDE_ASM_FUNCTION(HUF_decompress4X2_usingDTable_internal_fast_asm_loop) ZSTD_HIDE_ASM_FUNCTION(_HUF_decompress4X2_usingDTable_internal_fast_asm_loop) ZSTD_HIDE_ASM_FUNCTION(_HUF_decompress4X1_usingDTable_internal_fast_asm_loop) .global HUF_decompress4X1_usingDTable_internal_fast_asm_loop .global HUF_decompress4X2_usingDTable_internal_fast_asm_loop .global _HUF_decompress4X1_usingDTable_internal_fast_asm_loop .global _HUF_decompress4X2_usingDTable_internal_fast_asm_loop .text /* Sets up register mappings for clarity. * op[], bits[], dtable & ip[0] each get their own register. * ip[1,2,3] & olimit alias var[]. * %rax is a scratch register. */ #define op0 rsi #define op1 rbx #define op2 rcx #define op3 rdi #define ip0 r8 #define ip1 r9 #define ip2 r10 #define ip3 r11 #define bits0 rbp #define bits1 rdx #define bits2 r12 #define bits3 r13 #define dtable r14 #define olimit r15 /* var[] aliases ip[1,2,3] & olimit * ip[1,2,3] are saved every iteration. * olimit is only used in compute_olimit. */ #define var0 r15 #define var1 r9 #define var2 r10 #define var3 r11 /* 32-bit var registers */ #define vard0 r15d #define vard1 r9d #define vard2 r10d #define vard3 r11d /* Calls X(N) for each stream 0, 1, 2, 3. */ #define FOR_EACH_STREAM(X) \ X(0); \ X(1); \ X(2); \ X(3) /* Calls X(N, idx) for each stream 0, 1, 2, 3. */ #define FOR_EACH_STREAM_WITH_INDEX(X, idx) \ X(0, idx); \ X(1, idx); \ X(2, idx); \ X(3, idx) /* Define both _HUF_* & HUF_* symbols because MacOS * C symbols are prefixed with '_' & Linux symbols aren't. */ _HUF_decompress4X1_usingDTable_internal_fast_asm_loop: HUF_decompress4X1_usingDTable_internal_fast_asm_loop: ZSTD_CET_ENDBRANCH /* Save all registers - even if they are callee saved for simplicity. */ push %rax push %rbx push %rcx push %rdx push %rbp push %rsi push %rdi push %r8 push %r9 push %r10 push %r11 push %r12 push %r13 push %r14 push %r15 /* Read HUF_DecompressAsmArgs* args from %rax */ movq %rdi, %rax movq 0(%rax), %ip0 movq 8(%rax), %ip1 movq 16(%rax), %ip2 movq 24(%rax), %ip3 movq 32(%rax), %op0 movq 40(%rax), %op1 movq 48(%rax), %op2 movq 56(%rax), %op3 movq 64(%rax), %bits0 movq 72(%rax), %bits1 movq 80(%rax), %bits2 movq 88(%rax), %bits3 movq 96(%rax), %dtable push %rax /* argument */ push 104(%rax) /* ilimit */ push 112(%rax) /* oend */ push %olimit /* olimit space */ subq $24, %rsp .L_4X1_compute_olimit: /* Computes how many iterations we can do safely * %r15, %rax may be clobbered * rbx, rdx must be saved * op3 & ip0 mustn't be clobbered */ movq %rbx, 0(%rsp) movq %rdx, 8(%rsp) movq 32(%rsp), %rax /* rax = oend */ subq %op3, %rax /* rax = oend - op3 */ /* r15 = (oend - op3) / 5 */ movabsq $-3689348814741910323, %rdx mulq %rdx movq %rdx, %r15 shrq $2, %r15 movq %ip0, %rax /* rax = ip0 */ movq 40(%rsp), %rdx /* rdx = ilimit */ subq %rdx, %rax /* rax = ip0 - ilimit */ movq %rax, %rbx /* rbx = ip0 - ilimit */ /* rdx = (ip0 - ilimit) / 7 */ movabsq $2635249153387078803, %rdx mulq %rdx subq %rdx, %rbx shrq %rbx addq %rbx, %rdx shrq $2, %rdx /* r15 = min(%rdx, %r15) */ cmpq %rdx, %r15 cmova %rdx, %r15 /* r15 = r15 * 5 */ leaq (%r15, %r15, 4), %r15 /* olimit = op3 + r15 */ addq %op3, %olimit movq 8(%rsp), %rdx movq 0(%rsp), %rbx /* If (op3 + 20 > olimit) */ movq %op3, %rax /* rax = op3 */ addq $20, %rax /* rax = op3 + 20 */ cmpq %rax, %olimit /* op3 + 20 > olimit */ jb .L_4X1_exit /* If (ip1 < ip0) go to exit */ cmpq %ip0, %ip1 jb .L_4X1_exit /* If (ip2 < ip1) go to exit */ cmpq %ip1, %ip2 jb .L_4X1_exit /* If (ip3 < ip2) go to exit */ cmpq %ip2, %ip3 jb .L_4X1_exit /* Reads top 11 bits from bits[n] * Loads dt[bits[n]] into var[n] */ #define GET_NEXT_DELT(n) \ movq $53, %var##n; \ shrxq %var##n, %bits##n, %var##n; \ movzwl (%dtable,%var##n,2),%vard##n /* var[n] must contain the DTable entry computed with GET_NEXT_DELT * Moves var[n] to %rax * bits[n] <<= var[n] & 63 * op[n][idx] = %rax >> 8 * %ah is a way to access bits [8, 16) of %rax */ #define DECODE_FROM_DELT(n, idx) \ movq %var##n, %rax; \ shlxq %var##n, %bits##n, %bits##n; \ movb %ah, idx(%op##n) /* Assumes GET_NEXT_DELT has been called. * Calls DECODE_FROM_DELT then GET_NEXT_DELT */ #define DECODE_AND_GET_NEXT(n, idx) \ DECODE_FROM_DELT(n, idx); \ GET_NEXT_DELT(n) \ /* // ctz & nbBytes is stored in bits[n] * // nbBits is stored in %rax * ctz = CTZ[bits[n]] * nbBits = ctz & 7 * nbBytes = ctz >> 3 * op[n] += 5 * ip[n] -= nbBytes * // Note: x86-64 is little-endian ==> no bswap * bits[n] = MEM_readST(ip[n]) | 1 * bits[n] <<= nbBits */ #define RELOAD_BITS(n) \ bsfq %bits##n, %bits##n; \ movq %bits##n, %rax; \ andq $7, %rax; \ shrq $3, %bits##n; \ leaq 5(%op##n), %op##n; \ subq %bits##n, %ip##n; \ movq (%ip##n), %bits##n; \ orq $1, %bits##n; \ shlx %rax, %bits##n, %bits##n /* Store clobbered variables on the stack */ movq %olimit, 24(%rsp) movq %ip1, 0(%rsp) movq %ip2, 8(%rsp) movq %ip3, 16(%rsp) /* Call GET_NEXT_DELT for each stream */ FOR_EACH_STREAM(GET_NEXT_DELT) .p2align 6 .L_4X1_loop_body: /* Decode 5 symbols in each of the 4 streams (20 total) * Must have called GET_NEXT_DELT for each stream */ FOR_EACH_STREAM_WITH_INDEX(DECODE_AND_GET_NEXT, 0) FOR_EACH_STREAM_WITH_INDEX(DECODE_AND_GET_NEXT, 1) FOR_EACH_STREAM_WITH_INDEX(DECODE_AND_GET_NEXT, 2) FOR_EACH_STREAM_WITH_INDEX(DECODE_AND_GET_NEXT, 3) FOR_EACH_STREAM_WITH_INDEX(DECODE_FROM_DELT, 4) /* Load ip[1,2,3] from stack (var[] aliases them) * ip[] is needed for RELOAD_BITS * Each will be stored back to the stack after RELOAD */ movq 0(%rsp), %ip1 movq 8(%rsp), %ip2 movq 16(%rsp), %ip3 /* Reload each stream & fetch the next table entry * to prepare for the next iteration */ RELOAD_BITS(0) GET_NEXT_DELT(0) RELOAD_BITS(1) movq %ip1, 0(%rsp) GET_NEXT_DELT(1) RELOAD_BITS(2) movq %ip2, 8(%rsp) GET_NEXT_DELT(2) RELOAD_BITS(3) movq %ip3, 16(%rsp) GET_NEXT_DELT(3) /* If op3 < olimit: continue the loop */ cmp %op3, 24(%rsp) ja .L_4X1_loop_body /* Reload ip[1,2,3] from stack */ movq 0(%rsp), %ip1 movq 8(%rsp), %ip2 movq 16(%rsp), %ip3 /* Re-compute olimit */ jmp .L_4X1_compute_olimit #undef GET_NEXT_DELT #undef DECODE_FROM_DELT #undef DECODE #undef RELOAD_BITS .L_4X1_exit: addq $24, %rsp /* Restore stack (oend & olimit) */ pop %rax /* olimit */ pop %rax /* oend */ pop %rax /* ilimit */ pop %rax /* arg */ /* Save ip / op / bits */ movq %ip0, 0(%rax) movq %ip1, 8(%rax) movq %ip2, 16(%rax) movq %ip3, 24(%rax) movq %op0, 32(%rax) movq %op1, 40(%rax) movq %op2, 48(%rax) movq %op3, 56(%rax) movq %bits0, 64(%rax) movq %bits1, 72(%rax) movq %bits2, 80(%rax) movq %bits3, 88(%rax) /* Restore registers */ pop %r15 pop %r14 pop %r13 pop %r12 pop %r11 pop %r10 pop %r9 pop %r8 pop %rdi pop %rsi pop %rbp pop %rdx pop %rcx pop %rbx pop %rax ret _HUF_decompress4X2_usingDTable_internal_fast_asm_loop: HUF_decompress4X2_usingDTable_internal_fast_asm_loop: ZSTD_CET_ENDBRANCH /* Save all registers - even if they are callee saved for simplicity. */ push %rax push %rbx push %rcx push %rdx push %rbp push %rsi push %rdi push %r8 push %r9 push %r10 push %r11 push %r12 push %r13 push %r14 push %r15 movq %rdi, %rax movq 0(%rax), %ip0 movq 8(%rax), %ip1 movq 16(%rax), %ip2 movq 24(%rax), %ip3 movq 32(%rax), %op0 movq 40(%rax), %op1 movq 48(%rax), %op2 movq 56(%rax), %op3 movq 64(%rax), %bits0 movq 72(%rax), %bits1 movq 80(%rax), %bits2 movq 88(%rax), %bits3 movq 96(%rax), %dtable push %rax /* argument */ push %rax /* olimit */ push 104(%rax) /* ilimit */ movq 112(%rax), %rax push %rax /* oend3 */ movq %op3, %rax push %rax /* oend2 */ movq %op2, %rax push %rax /* oend1 */ movq %op1, %rax push %rax /* oend0 */ /* Scratch space */ subq $8, %rsp .L_4X2_compute_olimit: /* Computes how many iterations we can do safely * %r15, %rax may be clobbered * rdx must be saved * op[1,2,3,4] & ip0 mustn't be clobbered */ movq %rdx, 0(%rsp) /* We can consume up to 7 input bytes each iteration. */ movq %ip0, %rax /* rax = ip0 */ movq 40(%rsp), %rdx /* rdx = ilimit */ subq %rdx, %rax /* rax = ip0 - ilimit */ movq %rax, %r15 /* r15 = ip0 - ilimit */ /* rdx = rax / 7 */ movabsq $2635249153387078803, %rdx mulq %rdx subq %rdx, %r15 shrq %r15 addq %r15, %rdx shrq $2, %rdx /* r15 = (ip0 - ilimit) / 7 */ movq %rdx, %r15 /* r15 = min(r15, min(oend0 - op0, oend1 - op1, oend2 - op2, oend3 - op3) / 10) */ movq 8(%rsp), %rax /* rax = oend0 */ subq %op0, %rax /* rax = oend0 - op0 */ movq 16(%rsp), %rdx /* rdx = oend1 */ subq %op1, %rdx /* rdx = oend1 - op1 */ cmpq %rax, %rdx cmova %rax, %rdx /* rdx = min(%rdx, %rax) */ movq 24(%rsp), %rax /* rax = oend2 */ subq %op2, %rax /* rax = oend2 - op2 */ cmpq %rax, %rdx cmova %rax, %rdx /* rdx = min(%rdx, %rax) */ movq 32(%rsp), %rax /* rax = oend3 */ subq %op3, %rax /* rax = oend3 - op3 */ cmpq %rax, %rdx cmova %rax, %rdx /* rdx = min(%rdx, %rax) */ movabsq $-3689348814741910323, %rax mulq %rdx shrq $3, %rdx /* rdx = rdx / 10 */ /* r15 = min(%rdx, %r15) */ cmpq %rdx, %r15 cmova %rdx, %r15 /* olimit = op3 + 5 * r15 */ movq %r15, %rax leaq (%op3, %rax, 4), %olimit addq %rax, %olimit movq 0(%rsp), %rdx /* If (op3 + 10 > olimit) */ movq %op3, %rax /* rax = op3 */ addq $10, %rax /* rax = op3 + 10 */ cmpq %rax, %olimit /* op3 + 10 > olimit */ jb .L_4X2_exit /* If (ip1 < ip0) go to exit */ cmpq %ip0, %ip1 jb .L_4X2_exit /* If (ip2 < ip1) go to exit */ cmpq %ip1, %ip2 jb .L_4X2_exit /* If (ip3 < ip2) go to exit */ cmpq %ip2, %ip3 jb .L_4X2_exit #define DECODE(n, idx) \ movq %bits##n, %rax; \ shrq $53, %rax; \ movzwl 0(%dtable,%rax,4),%r8d; \ movzbl 2(%dtable,%rax,4),%r15d; \ movzbl 3(%dtable,%rax,4),%eax; \ movw %r8w, (%op##n); \ shlxq %r15, %bits##n, %bits##n; \ addq %rax, %op##n #define RELOAD_BITS(n) \ bsfq %bits##n, %bits##n; \ movq %bits##n, %rax; \ shrq $3, %bits##n; \ andq $7, %rax; \ subq %bits##n, %ip##n; \ movq (%ip##n), %bits##n; \ orq $1, %bits##n; \ shlxq %rax, %bits##n, %bits##n movq %olimit, 48(%rsp) .p2align 6 .L_4X2_loop_body: /* We clobber r8, so store it on the stack */ movq %r8, 0(%rsp) /* Decode 5 symbols from each of the 4 streams (20 symbols total). */ FOR_EACH_STREAM_WITH_INDEX(DECODE, 0) FOR_EACH_STREAM_WITH_INDEX(DECODE, 1) FOR_EACH_STREAM_WITH_INDEX(DECODE, 2) FOR_EACH_STREAM_WITH_INDEX(DECODE, 3) FOR_EACH_STREAM_WITH_INDEX(DECODE, 4) /* Reload r8 */ movq 0(%rsp), %r8 FOR_EACH_STREAM(RELOAD_BITS) cmp %op3, 48(%rsp) ja .L_4X2_loop_body jmp .L_4X2_compute_olimit #undef DECODE #undef RELOAD_BITS .L_4X2_exit: addq $8, %rsp /* Restore stack (oend & olimit) */ pop %rax /* oend0 */ pop %rax /* oend1 */ pop %rax /* oend2 */ pop %rax /* oend3 */ pop %rax /* ilimit */ pop %rax /* olimit */ pop %rax /* arg */ /* Save ip / op / bits */ movq %ip0, 0(%rax) movq %ip1, 8(%rax) movq %ip2, 16(%rax) movq %ip3, 24(%rax) movq %op0, 32(%rax) movq %op1, 40(%rax) movq %op2, 48(%rax) movq %op3, 56(%rax) movq %bits0, 64(%rax) movq %bits1, 72(%rax) movq %bits2, 80(%rax) movq %bits3, 88(%rax) /* Restore registers */ pop %r15 pop %r14 pop %r13 pop %r12 pop %r11 pop %r10 pop %r9 pop %r8 pop %rdi pop %rsi pop %rbp pop %rdx pop %rcx pop %rbx pop %rax ret #endif zmat-0.9.9/src/blosc2/internal-complibs/zstd/libzstd.pc.in0000644000175200007730000000071314515254731023725 0ustar rlaboissrlaboiss# ZSTD - standard compression algorithm # Copyright (c) Meta Platforms, Inc. and affiliates. # BSD 2-Clause License (https://opensource.org/licenses/bsd-license.php) prefix=@PREFIX@ exec_prefix=@EXEC_PREFIX@ includedir=@INCLUDEDIR@ libdir=@LIBDIR@ Name: zstd Description: fast lossless compression algorithm library URL: https://facebook.github.io/zstd/ Version: @VERSION@ Libs: -L${libdir} -lzstd Libs.private: @LIBS_PRIVATE@ Cflags: -I${includedir} zmat-0.9.9/src/blosc2/internal-complibs/zstd/dll/0000755000175200007730000000000014515254731022073 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/internal-complibs/zstd/dll/example/0000755000175200007730000000000014515254731023526 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/internal-complibs/zstd/dll/example/fullbench-dll.vcxproj0000644000175200007730000002357114515254731027666 0ustar rlaboissrlaboiss Debug Win32 Debug x64 Release Win32 Release x64 {00000000-1CC8-4FD7-9281-6B8DBB9D3DF8} Win32Proj fullbench-dll $(SolutionDir)bin\$(Platform)_$(Configuration)\ $(SolutionDir)bin\obj\$(RootNamespace)_$(Platform)_$(Configuration)\ Application true MultiByte Application true MultiByte Application false true MultiByte Application false true MultiByte true $(IncludePath);$(SolutionDir)..\..\lib;$(SolutionDir)..\..\programs;$(SolutionDir)..\..\lib\legacy;$(SolutionDir)..\..\lib\common;$(UniversalCRT_IncludePath); false true $(IncludePath);$(SolutionDir)..\..\lib;$(SolutionDir)..\..\programs;$(SolutionDir)..\..\lib\legacy;$(SolutionDir)..\..\lib\common;$(UniversalCRT_IncludePath); false false $(IncludePath);$(SolutionDir)..\..\lib;$(SolutionDir)..\..\programs;$(SolutionDir)..\..\lib\legacy;$(SolutionDir)..\..\lib\common;$(UniversalCRT_IncludePath); false false $(IncludePath);$(SolutionDir)..\..\lib;$(SolutionDir)..\..\programs;$(SolutionDir)..\..\lib\legacy;$(SolutionDir)..\..\lib\common;$(UniversalCRT_IncludePath); false Level4 Disabled WIN32;_DEBUG;_CONSOLE;ZSTD_DLL_IMPORT=1;%(PreprocessorDefinitions) true false ..\include Console true $(SolutionDir)..\dll;%(AdditionalLibraryDirectories) libzstd.lib;%(AdditionalDependencies) false Level4 Disabled WIN32;_DEBUG;_CONSOLE;ZSTD_DLL_IMPORT=1;%(PreprocessorDefinitions) true false ..\include Console true $(SolutionDir)..\dll;%(AdditionalLibraryDirectories) libzstd.lib;%(AdditionalDependencies) Level4 MaxSpeed true true WIN32;_DEBUG;_CONSOLE;ZSTD_DLL_IMPORT=1;%(PreprocessorDefinitions) false ..\include false MultiThreaded Console true true true $(SolutionDir)..\dll;%(AdditionalLibraryDirectories) libzstd.lib;%(AdditionalDependencies) false Level4 MaxSpeed true true WIN32;_DEBUG;_CONSOLE;ZSTD_DLL_IMPORT=1;%(PreprocessorDefinitions) false false ..\include MultiThreaded Console true true true $(SolutionDir)..\dll;%(AdditionalLibraryDirectories) libzstd.lib;%(AdditionalDependencies) zmat-0.9.9/src/blosc2/internal-complibs/zstd/dll/example/Makefile0000644000175200007730000000270114515254731025166 0ustar rlaboissrlaboiss# ################################################################ # Copyright (c) Meta Platforms, Inc. and affiliates. # All rights reserved. # # This source code is licensed under both the BSD-style license (found in the # LICENSE file in the root directory of this source tree) and the GPLv2 (found # in the COPYING file in the root directory of this source tree). # You may select, at your option, one of the above-listed licenses. # ################################################################ VOID := /dev/null ZSTDDIR := ../include LIBDIR := ../static DLLDIR := ../dll CFLAGS ?= -O3 # can select custom flags. For example : CFLAGS="-O2 -g" make CFLAGS += -Wall -Wextra -Wundef -Wcast-qual -Wcast-align -Wshadow -Wswitch-enum \ -Wdeclaration-after-statement -Wstrict-prototypes \ -Wpointer-arith -Wstrict-aliasing=1 CFLAGS += $(MOREFLAGS) CPPFLAGS:= -I$(ZSTDDIR) -DXXH_NAMESPACE=ZSTD_ FLAGS := $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) # Define *.exe as extension for Windows systems ifneq (,$(filter Windows%,$(OS))) EXT =.exe else EXT = endif .PHONY: default fullbench-dll fullbench-lib default: all all: fullbench-dll fullbench-lib fullbench-lib: fullbench.c datagen.c $(CC) $(FLAGS) $^ -o $@$(EXT) $(LIBDIR)/libzstd_static.lib fullbench-dll: fullbench.c datagen.c $(CC) $(FLAGS) $^ -o $@$(EXT) -DZSTD_DLL_IMPORT=1 $(DLLDIR)/libzstd.dll clean: @$(RM) fullbench-dll$(EXT) fullbench-lib$(EXT) \ @echo Cleaning completed zmat-0.9.9/src/blosc2/internal-complibs/zstd/dll/example/fullbench-dll.sln0000644000175200007730000000236514515254731026765 0ustar rlaboissrlaboissMicrosoft Visual Studio Solution File, Format Version 12.00 # Visual Studio Express 2012 for Windows Desktop Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "fullbench-dll", "fullbench-dll.vcxproj", "{13992FD2-077E-4954-B065-A428198201A9}" EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Win32 = Debug|Win32 Debug|x64 = Debug|x64 Release|Win32 = Release|Win32 Release|x64 = Release|x64 EndGlobalSection GlobalSection(ProjectConfigurationPlatforms) = postSolution {13992FD2-077E-4954-B065-A428198201A9}.Debug|Win32.ActiveCfg = Debug|Win32 {13992FD2-077E-4954-B065-A428198201A9}.Debug|Win32.Build.0 = Debug|Win32 {13992FD2-077E-4954-B065-A428198201A9}.Debug|x64.ActiveCfg = Debug|x64 {13992FD2-077E-4954-B065-A428198201A9}.Debug|x64.Build.0 = Debug|x64 {13992FD2-077E-4954-B065-A428198201A9}.Release|Win32.ActiveCfg = Release|Win32 {13992FD2-077E-4954-B065-A428198201A9}.Release|Win32.Build.0 = Release|Win32 {13992FD2-077E-4954-B065-A428198201A9}.Release|x64.ActiveCfg = Release|x64 {13992FD2-077E-4954-B065-A428198201A9}.Release|x64.Build.0 = Release|x64 EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE EndGlobalSection EndGlobal zmat-0.9.9/src/blosc2/internal-complibs/zstd/dll/example/README.md0000644000175200007730000000561414515254731025013 0ustar rlaboissrlaboiss# ZSTD Windows binary package ## The package contents - `zstd.exe` : Command Line Utility, supporting gzip-like arguments - `dll\libzstd.dll` : The ZSTD dynamic library (DLL) - `dll\libzstd.lib` : The import library of the ZSTD dynamic library (DLL) for Visual C++ - `example\` : The example of usage of the ZSTD library - `include\` : Header files required by the ZSTD library - `static\libzstd_static.lib` : The static ZSTD library (LIB) ## Usage of Command Line Interface Command Line Interface (CLI) supports gzip-like arguments. By default CLI takes an input file and compresses it to an output file: Usage: zstd [arg] [input] [output] The full list of commands for CLI can be obtained with `-h` or `-H`. The ratio can be improved with commands from `-3` to `-16` but higher levels also have slower compression. CLI includes in-memory compression benchmark module with compression levels starting from `-b` and ending with `-e` with iteration time of `-i` seconds. CLI supports aggregation of parameters i.e. `-b1`, `-e18`, and `-i1` can be joined into `-b1e18i1`. ## The example of usage of static and dynamic ZSTD libraries with gcc/MinGW Use `cd example` and `make` to build `fullbench-dll` and `fullbench-lib`. `fullbench-dll` uses a dynamic ZSTD library from the `dll` directory. `fullbench-lib` uses a static ZSTD library from the `lib` directory. ## Using ZSTD DLL with gcc/MinGW The header files from `include\` and the dynamic library `dll\libzstd.dll` are required to compile a project using gcc/MinGW. The dynamic library has to be added to linking options. It means that if a project that uses ZSTD consists of a single `test-dll.c` file it should be linked with `dll\libzstd.dll`. For example: gcc $(CFLAGS) -Iinclude\ test-dll.c -o test-dll dll\libzstd.dll The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`. ## The example of usage of static and dynamic ZSTD libraries with Visual C++ Open `example\fullbench-dll.sln` to compile `fullbench-dll` that uses a dynamic ZSTD library from the `dll` directory. The solution works with Visual C++ 2010 or newer. When one will open the solution with Visual C++ newer than 2010 then the solution will be upgraded to the current version. ## Using ZSTD DLL with Visual C++ The header files from `include\` and the import library `dll\libzstd.lib` are required to compile a project using Visual C++. 1. The path to header files should be added to `Additional Include Directories` that can be found in project properties `C/C++` then `General`. 2. The import library has to be added to `Additional Dependencies` that can be found in project properties `Linker` then `Input`. If one will provide only the name `libzstd.lib` without a full path to the library the directory has to be added to `Linker\General\Additional Library Directories`. The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`. zmat-0.9.9/src/blosc2/internal-complibs/zstd/.gitignore0000644000175200007730000000006014515254731023304 0ustar rlaboissrlaboiss# make install artefact libzstd.pc libzstd-nomt zmat-0.9.9/src/blosc2/internal-complibs/zstd/legacy/0000755000175200007730000000000014515254731022564 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/internal-complibs/zstd/legacy/zstd_legacy.h0000644000175200007730000003375614515254731025263 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZSTD_LEGACY_H #define ZSTD_LEGACY_H #if defined (__cplusplus) extern "C" { #endif /* ************************************* * Includes ***************************************/ #include "../common/mem.h" /* MEM_STATIC */ #include "../common/error_private.h" /* ERROR */ #include "../common/zstd_internal.h" /* ZSTD_inBuffer, ZSTD_outBuffer, ZSTD_frameSizeInfo */ #if !defined (ZSTD_LEGACY_SUPPORT) || (ZSTD_LEGACY_SUPPORT == 0) # undef ZSTD_LEGACY_SUPPORT # define ZSTD_LEGACY_SUPPORT 8 #endif #if (ZSTD_LEGACY_SUPPORT <= 1) # include "zstd_v01.h" #endif #if (ZSTD_LEGACY_SUPPORT <= 2) # include "zstd_v02.h" #endif #if (ZSTD_LEGACY_SUPPORT <= 3) # include "zstd_v03.h" #endif #if (ZSTD_LEGACY_SUPPORT <= 4) # include "zstd_v04.h" #endif #if (ZSTD_LEGACY_SUPPORT <= 5) # include "zstd_v05.h" #endif #if (ZSTD_LEGACY_SUPPORT <= 6) # include "zstd_v06.h" #endif #if (ZSTD_LEGACY_SUPPORT <= 7) # include "zstd_v07.h" #endif /** ZSTD_isLegacy() : @return : > 0 if supported by legacy decoder. 0 otherwise. return value is the version. */ MEM_STATIC unsigned ZSTD_isLegacy(const void* src, size_t srcSize) { U32 magicNumberLE; if (srcSize<4) return 0; magicNumberLE = MEM_readLE32(src); switch(magicNumberLE) { #if (ZSTD_LEGACY_SUPPORT <= 1) case ZSTDv01_magicNumberLE:return 1; #endif #if (ZSTD_LEGACY_SUPPORT <= 2) case ZSTDv02_magicNumber : return 2; #endif #if (ZSTD_LEGACY_SUPPORT <= 3) case ZSTDv03_magicNumber : return 3; #endif #if (ZSTD_LEGACY_SUPPORT <= 4) case ZSTDv04_magicNumber : return 4; #endif #if (ZSTD_LEGACY_SUPPORT <= 5) case ZSTDv05_MAGICNUMBER : return 5; #endif #if (ZSTD_LEGACY_SUPPORT <= 6) case ZSTDv06_MAGICNUMBER : return 6; #endif #if (ZSTD_LEGACY_SUPPORT <= 7) case ZSTDv07_MAGICNUMBER : return 7; #endif default : return 0; } } MEM_STATIC unsigned long long ZSTD_getDecompressedSize_legacy(const void* src, size_t srcSize) { U32 const version = ZSTD_isLegacy(src, srcSize); if (version < 5) return 0; /* no decompressed size in frame header, or not a legacy format */ #if (ZSTD_LEGACY_SUPPORT <= 5) if (version==5) { ZSTDv05_parameters fParams; size_t const frResult = ZSTDv05_getFrameParams(&fParams, src, srcSize); if (frResult != 0) return 0; return fParams.srcSize; } #endif #if (ZSTD_LEGACY_SUPPORT <= 6) if (version==6) { ZSTDv06_frameParams fParams; size_t const frResult = ZSTDv06_getFrameParams(&fParams, src, srcSize); if (frResult != 0) return 0; return fParams.frameContentSize; } #endif #if (ZSTD_LEGACY_SUPPORT <= 7) if (version==7) { ZSTDv07_frameParams fParams; size_t const frResult = ZSTDv07_getFrameParams(&fParams, src, srcSize); if (frResult != 0) return 0; return fParams.frameContentSize; } #endif return 0; /* should not be possible */ } MEM_STATIC size_t ZSTD_decompressLegacy( void* dst, size_t dstCapacity, const void* src, size_t compressedSize, const void* dict,size_t dictSize) { U32 const version = ZSTD_isLegacy(src, compressedSize); (void)dst; (void)dstCapacity; (void)dict; (void)dictSize; /* unused when ZSTD_LEGACY_SUPPORT >= 8 */ switch(version) { #if (ZSTD_LEGACY_SUPPORT <= 1) case 1 : return ZSTDv01_decompress(dst, dstCapacity, src, compressedSize); #endif #if (ZSTD_LEGACY_SUPPORT <= 2) case 2 : return ZSTDv02_decompress(dst, dstCapacity, src, compressedSize); #endif #if (ZSTD_LEGACY_SUPPORT <= 3) case 3 : return ZSTDv03_decompress(dst, dstCapacity, src, compressedSize); #endif #if (ZSTD_LEGACY_SUPPORT <= 4) case 4 : return ZSTDv04_decompress(dst, dstCapacity, src, compressedSize); #endif #if (ZSTD_LEGACY_SUPPORT <= 5) case 5 : { size_t result; ZSTDv05_DCtx* const zd = ZSTDv05_createDCtx(); if (zd==NULL) return ERROR(memory_allocation); result = ZSTDv05_decompress_usingDict(zd, dst, dstCapacity, src, compressedSize, dict, dictSize); ZSTDv05_freeDCtx(zd); return result; } #endif #if (ZSTD_LEGACY_SUPPORT <= 6) case 6 : { size_t result; ZSTDv06_DCtx* const zd = ZSTDv06_createDCtx(); if (zd==NULL) return ERROR(memory_allocation); result = ZSTDv06_decompress_usingDict(zd, dst, dstCapacity, src, compressedSize, dict, dictSize); ZSTDv06_freeDCtx(zd); return result; } #endif #if (ZSTD_LEGACY_SUPPORT <= 7) case 7 : { size_t result; ZSTDv07_DCtx* const zd = ZSTDv07_createDCtx(); if (zd==NULL) return ERROR(memory_allocation); result = ZSTDv07_decompress_usingDict(zd, dst, dstCapacity, src, compressedSize, dict, dictSize); ZSTDv07_freeDCtx(zd); return result; } #endif default : return ERROR(prefix_unknown); } } MEM_STATIC ZSTD_frameSizeInfo ZSTD_findFrameSizeInfoLegacy(const void *src, size_t srcSize) { ZSTD_frameSizeInfo frameSizeInfo; U32 const version = ZSTD_isLegacy(src, srcSize); switch(version) { #if (ZSTD_LEGACY_SUPPORT <= 1) case 1 : ZSTDv01_findFrameSizeInfoLegacy(src, srcSize, &frameSizeInfo.compressedSize, &frameSizeInfo.decompressedBound); break; #endif #if (ZSTD_LEGACY_SUPPORT <= 2) case 2 : ZSTDv02_findFrameSizeInfoLegacy(src, srcSize, &frameSizeInfo.compressedSize, &frameSizeInfo.decompressedBound); break; #endif #if (ZSTD_LEGACY_SUPPORT <= 3) case 3 : ZSTDv03_findFrameSizeInfoLegacy(src, srcSize, &frameSizeInfo.compressedSize, &frameSizeInfo.decompressedBound); break; #endif #if (ZSTD_LEGACY_SUPPORT <= 4) case 4 : ZSTDv04_findFrameSizeInfoLegacy(src, srcSize, &frameSizeInfo.compressedSize, &frameSizeInfo.decompressedBound); break; #endif #if (ZSTD_LEGACY_SUPPORT <= 5) case 5 : ZSTDv05_findFrameSizeInfoLegacy(src, srcSize, &frameSizeInfo.compressedSize, &frameSizeInfo.decompressedBound); break; #endif #if (ZSTD_LEGACY_SUPPORT <= 6) case 6 : ZSTDv06_findFrameSizeInfoLegacy(src, srcSize, &frameSizeInfo.compressedSize, &frameSizeInfo.decompressedBound); break; #endif #if (ZSTD_LEGACY_SUPPORT <= 7) case 7 : ZSTDv07_findFrameSizeInfoLegacy(src, srcSize, &frameSizeInfo.compressedSize, &frameSizeInfo.decompressedBound); break; #endif default : frameSizeInfo.compressedSize = ERROR(prefix_unknown); frameSizeInfo.decompressedBound = ZSTD_CONTENTSIZE_ERROR; break; } if (!ZSTD_isError(frameSizeInfo.compressedSize) && frameSizeInfo.compressedSize > srcSize) { frameSizeInfo.compressedSize = ERROR(srcSize_wrong); frameSizeInfo.decompressedBound = ZSTD_CONTENTSIZE_ERROR; } /* In all cases, decompressedBound == nbBlocks * ZSTD_BLOCKSIZE_MAX. * So we can compute nbBlocks without having to change every function. */ if (frameSizeInfo.decompressedBound != ZSTD_CONTENTSIZE_ERROR) { assert((frameSizeInfo.decompressedBound & (ZSTD_BLOCKSIZE_MAX - 1)) == 0); frameSizeInfo.nbBlocks = (size_t)(frameSizeInfo.decompressedBound / ZSTD_BLOCKSIZE_MAX); } return frameSizeInfo; } MEM_STATIC size_t ZSTD_findFrameCompressedSizeLegacy(const void *src, size_t srcSize) { ZSTD_frameSizeInfo frameSizeInfo = ZSTD_findFrameSizeInfoLegacy(src, srcSize); return frameSizeInfo.compressedSize; } MEM_STATIC size_t ZSTD_freeLegacyStreamContext(void* legacyContext, U32 version) { switch(version) { default : case 1 : case 2 : case 3 : (void)legacyContext; return ERROR(version_unsupported); #if (ZSTD_LEGACY_SUPPORT <= 4) case 4 : return ZBUFFv04_freeDCtx((ZBUFFv04_DCtx*)legacyContext); #endif #if (ZSTD_LEGACY_SUPPORT <= 5) case 5 : return ZBUFFv05_freeDCtx((ZBUFFv05_DCtx*)legacyContext); #endif #if (ZSTD_LEGACY_SUPPORT <= 6) case 6 : return ZBUFFv06_freeDCtx((ZBUFFv06_DCtx*)legacyContext); #endif #if (ZSTD_LEGACY_SUPPORT <= 7) case 7 : return ZBUFFv07_freeDCtx((ZBUFFv07_DCtx*)legacyContext); #endif } } MEM_STATIC size_t ZSTD_initLegacyStream(void** legacyContext, U32 prevVersion, U32 newVersion, const void* dict, size_t dictSize) { DEBUGLOG(5, "ZSTD_initLegacyStream for v0.%u", newVersion); if (prevVersion != newVersion) ZSTD_freeLegacyStreamContext(*legacyContext, prevVersion); switch(newVersion) { default : case 1 : case 2 : case 3 : (void)dict; (void)dictSize; return 0; #if (ZSTD_LEGACY_SUPPORT <= 4) case 4 : { ZBUFFv04_DCtx* dctx = (prevVersion != newVersion) ? ZBUFFv04_createDCtx() : (ZBUFFv04_DCtx*)*legacyContext; if (dctx==NULL) return ERROR(memory_allocation); ZBUFFv04_decompressInit(dctx); ZBUFFv04_decompressWithDictionary(dctx, dict, dictSize); *legacyContext = dctx; return 0; } #endif #if (ZSTD_LEGACY_SUPPORT <= 5) case 5 : { ZBUFFv05_DCtx* dctx = (prevVersion != newVersion) ? ZBUFFv05_createDCtx() : (ZBUFFv05_DCtx*)*legacyContext; if (dctx==NULL) return ERROR(memory_allocation); ZBUFFv05_decompressInitDictionary(dctx, dict, dictSize); *legacyContext = dctx; return 0; } #endif #if (ZSTD_LEGACY_SUPPORT <= 6) case 6 : { ZBUFFv06_DCtx* dctx = (prevVersion != newVersion) ? ZBUFFv06_createDCtx() : (ZBUFFv06_DCtx*)*legacyContext; if (dctx==NULL) return ERROR(memory_allocation); ZBUFFv06_decompressInitDictionary(dctx, dict, dictSize); *legacyContext = dctx; return 0; } #endif #if (ZSTD_LEGACY_SUPPORT <= 7) case 7 : { ZBUFFv07_DCtx* dctx = (prevVersion != newVersion) ? ZBUFFv07_createDCtx() : (ZBUFFv07_DCtx*)*legacyContext; if (dctx==NULL) return ERROR(memory_allocation); ZBUFFv07_decompressInitDictionary(dctx, dict, dictSize); *legacyContext = dctx; return 0; } #endif } } MEM_STATIC size_t ZSTD_decompressLegacyStream(void* legacyContext, U32 version, ZSTD_outBuffer* output, ZSTD_inBuffer* input) { DEBUGLOG(5, "ZSTD_decompressLegacyStream for v0.%u", version); switch(version) { default : case 1 : case 2 : case 3 : (void)legacyContext; (void)output; (void)input; return ERROR(version_unsupported); #if (ZSTD_LEGACY_SUPPORT <= 4) case 4 : { ZBUFFv04_DCtx* dctx = (ZBUFFv04_DCtx*) legacyContext; const void* src = (const char*)input->src + input->pos; size_t readSize = input->size - input->pos; void* dst = (char*)output->dst + output->pos; size_t decodedSize = output->size - output->pos; size_t const hintSize = ZBUFFv04_decompressContinue(dctx, dst, &decodedSize, src, &readSize); output->pos += decodedSize; input->pos += readSize; return hintSize; } #endif #if (ZSTD_LEGACY_SUPPORT <= 5) case 5 : { ZBUFFv05_DCtx* dctx = (ZBUFFv05_DCtx*) legacyContext; const void* src = (const char*)input->src + input->pos; size_t readSize = input->size - input->pos; void* dst = (char*)output->dst + output->pos; size_t decodedSize = output->size - output->pos; size_t const hintSize = ZBUFFv05_decompressContinue(dctx, dst, &decodedSize, src, &readSize); output->pos += decodedSize; input->pos += readSize; return hintSize; } #endif #if (ZSTD_LEGACY_SUPPORT <= 6) case 6 : { ZBUFFv06_DCtx* dctx = (ZBUFFv06_DCtx*) legacyContext; const void* src = (const char*)input->src + input->pos; size_t readSize = input->size - input->pos; void* dst = (char*)output->dst + output->pos; size_t decodedSize = output->size - output->pos; size_t const hintSize = ZBUFFv06_decompressContinue(dctx, dst, &decodedSize, src, &readSize); output->pos += decodedSize; input->pos += readSize; return hintSize; } #endif #if (ZSTD_LEGACY_SUPPORT <= 7) case 7 : { ZBUFFv07_DCtx* dctx = (ZBUFFv07_DCtx*) legacyContext; const void* src = (const char*)input->src + input->pos; size_t readSize = input->size - input->pos; void* dst = (char*)output->dst + output->pos; size_t decodedSize = output->size - output->pos; size_t const hintSize = ZBUFFv07_decompressContinue(dctx, dst, &decodedSize, src, &readSize); output->pos += decodedSize; input->pos += readSize; return hintSize; } #endif } } #if defined (__cplusplus) } #endif #endif /* ZSTD_LEGACY_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/libzstd.mk0000644000175200007730000001665014515254731023334 0ustar rlaboissrlaboiss# ################################################################ # Copyright (c) Meta Platforms, Inc. and affiliates. # All rights reserved. # # This source code is licensed under both the BSD-style license (found in the # LICENSE file in the root directory of this source tree) and the GPLv2 (found # in the COPYING file in the root directory of this source tree). # You may select, at your option, one of the above-listed licenses. # ################################################################ ################################################################## # Input Variables ################################################################## # Zstd lib directory LIBZSTD ?= ./ # ZSTD_LIB_MINIFY is a helper variable that # configures a bunch of other variables to space-optimized defaults. ZSTD_LIB_MINIFY ?= 0 # Legacy support ifneq ($(ZSTD_LIB_MINIFY), 0) ZSTD_LEGACY_SUPPORT ?= 0 else ZSTD_LEGACY_SUPPORT ?= 5 endif ZSTD_LEGACY_MULTITHREADED_API ?= 0 # Build size optimizations ifneq ($(ZSTD_LIB_MINIFY), 0) HUF_FORCE_DECOMPRESS_X1 ?= 1 HUF_FORCE_DECOMPRESS_X2 ?= 0 ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT ?= 1 ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG ?= 0 ZSTD_NO_INLINE ?= 1 ZSTD_STRIP_ERROR_STRINGS ?= 1 else HUF_FORCE_DECOMPRESS_X1 ?= 0 HUF_FORCE_DECOMPRESS_X2 ?= 0 ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT ?= 0 ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG ?= 0 ZSTD_NO_INLINE ?= 0 ZSTD_STRIP_ERROR_STRINGS ?= 0 endif # Assembly support ZSTD_NO_ASM ?= 0 ################################################################## # libzstd helpers ################################################################## VOID ?= /dev/null # Make 4.3 doesn't support '\#' anymore (https://lwn.net/Articles/810071/) NUM_SYMBOL := \# # define silent mode as default (verbose mode with V=1 or VERBOSE=1) $(V)$(VERBOSE).SILENT: # When cross-compiling from linux to windows, # one might need to specify TARGET_SYSTEM as "Windows." # Building from Fedora fails without it. # (but Ubuntu and Debian don't need to set anything) TARGET_SYSTEM ?= $(OS) # Version numbers LIBVER_SRC := $(LIBZSTD)/zstd.h LIBVER_MAJOR_SCRIPT:=`sed -n '/define ZSTD_VERSION_MAJOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < $(LIBVER_SRC)` LIBVER_MINOR_SCRIPT:=`sed -n '/define ZSTD_VERSION_MINOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < $(LIBVER_SRC)` LIBVER_PATCH_SCRIPT:=`sed -n '/define ZSTD_VERSION_RELEASE/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < $(LIBVER_SRC)` LIBVER_SCRIPT:= $(LIBVER_MAJOR_SCRIPT).$(LIBVER_MINOR_SCRIPT).$(LIBVER_PATCH_SCRIPT) LIBVER_MAJOR := $(shell echo $(LIBVER_MAJOR_SCRIPT)) LIBVER_MINOR := $(shell echo $(LIBVER_MINOR_SCRIPT)) LIBVER_PATCH := $(shell echo $(LIBVER_PATCH_SCRIPT)) LIBVER := $(shell echo $(LIBVER_SCRIPT)) CCVER := $(shell $(CC) --version) ZSTD_VERSION?= $(LIBVER) ifneq ($(ZSTD_LIB_MINIFY), 0) HAVE_CC_OZ ?= $(shell echo "" | $(CC) -Oz -x c -c - -o /dev/null 2> /dev/null && echo 1 || echo 0) ifneq ($(HAVE_CC_OZ), 0) # Some compilers (clang) support an even more space-optimized setting. CFLAGS += -Oz else CFLAGS += -Os endif CFLAGS += -fno-stack-protector -fomit-frame-pointer -fno-ident \ -DDYNAMIC_BMI2=0 -DNDEBUG else CFLAGS ?= -O3 endif DEBUGLEVEL ?= 0 CPPFLAGS += -DXXH_NAMESPACE=ZSTD_ -DDEBUGLEVEL=$(DEBUGLEVEL) ifeq ($(TARGET_SYSTEM),Windows_NT) # MinGW assumed CPPFLAGS += -D__USE_MINGW_ANSI_STDIO # compatibility with %zu formatting endif DEBUGFLAGS= -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \ -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \ -Wstrict-prototypes -Wundef -Wpointer-arith \ -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings \ -Wredundant-decls -Wmissing-prototypes -Wc++-compat CFLAGS += $(DEBUGFLAGS) $(MOREFLAGS) ASFLAGS += $(DEBUGFLAGS) $(MOREFLAGS) $(CFLAGS) LDFLAGS += $(MOREFLAGS) FLAGS = $(CPPFLAGS) $(CFLAGS) $(ASFLAGS) $(LDFLAGS) ifndef ALREADY_APPENDED_NOEXECSTACK export ALREADY_APPENDED_NOEXECSTACK := 1 ifeq ($(shell echo "int main(int argc, char* argv[]) { (void)argc; (void)argv; return 0; }" | $(CC) $(FLAGS) -z noexecstack -x c -Werror - -o $(VOID) 2>$(VOID) && echo 1 || echo 0),1) LDFLAGS += -z noexecstack endif ifeq ($(shell echo | $(CC) $(FLAGS) -Wa,--noexecstack -x assembler -Werror -c - -o $(VOID) 2>$(VOID) && echo 1 || echo 0),1) CFLAGS += -Wa,--noexecstack # CFLAGS are also added to ASFLAGS else ifeq ($(shell echo | $(CC) $(FLAGS) -Qunused-arguments -Wa,--noexecstack -x assembler -Werror -c - -o $(VOID) 2>$(VOID) && echo 1 || echo 0),1) # See e.g.: https://github.com/android/ndk/issues/171 CFLAGS += -Qunused-arguments -Wa,--noexecstack # CFLAGS are also added to ASFLAGS endif endif ifeq ($(shell echo "int main(int argc, char* argv[]) { (void)argc; (void)argv; return 0; }" | $(CC) $(FLAGS) -z cet-report=error -x c -Werror - -o $(VOID) 2>$(VOID) && echo 1 || echo 0),1) LDFLAGS += -z cet-report=error endif HAVE_COLORNEVER = $(shell echo a | grep --color=never a > /dev/null 2> /dev/null && echo 1 || echo 0) GREP_OPTIONS ?= ifeq ($(HAVE_COLORNEVER), 1) GREP_OPTIONS += --color=never endif GREP = grep $(GREP_OPTIONS) ZSTD_COMMON_FILES := $(sort $(wildcard $(LIBZSTD)/common/*.c)) ZSTD_COMPRESS_FILES := $(sort $(wildcard $(LIBZSTD)/compress/*.c)) ZSTD_DECOMPRESS_FILES := $(sort $(wildcard $(LIBZSTD)/decompress/*.c)) ZSTD_DICTBUILDER_FILES := $(sort $(wildcard $(LIBZSTD)/dictBuilder/*.c)) ZSTD_DEPRECATED_FILES := $(sort $(wildcard $(LIBZSTD)/deprecated/*.c)) ZSTD_LEGACY_FILES := ZSTD_DECOMPRESS_AMD64_ASM_FILES := $(sort $(wildcard $(LIBZSTD)/decompress/*_amd64.S)) ifneq ($(ZSTD_NO_ASM), 0) CPPFLAGS += -DZSTD_DISABLE_ASM else # Unconditionally add the ASM files they are disabled by # macros in the .S file. ZSTD_DECOMPRESS_FILES += $(ZSTD_DECOMPRESS_AMD64_ASM_FILES) endif ifneq ($(HUF_FORCE_DECOMPRESS_X1), 0) CFLAGS += -DHUF_FORCE_DECOMPRESS_X1 endif ifneq ($(HUF_FORCE_DECOMPRESS_X2), 0) CFLAGS += -DHUF_FORCE_DECOMPRESS_X2 endif ifneq ($(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT), 0) CFLAGS += -DZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT endif ifneq ($(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG), 0) CFLAGS += -DZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG endif ifneq ($(ZSTD_NO_INLINE), 0) CFLAGS += -DZSTD_NO_INLINE endif ifneq ($(ZSTD_STRIP_ERROR_STRINGS), 0) CFLAGS += -DZSTD_STRIP_ERROR_STRINGS endif ifneq ($(ZSTD_LEGACY_MULTITHREADED_API), 0) CFLAGS += -DZSTD_LEGACY_MULTITHREADED_API endif ifneq ($(ZSTD_LEGACY_SUPPORT), 0) ifeq ($(shell test $(ZSTD_LEGACY_SUPPORT) -lt 8; echo $$?), 0) ZSTD_LEGACY_FILES += $(shell ls $(LIBZSTD)/legacy/*.c | $(GREP) 'v0[$(ZSTD_LEGACY_SUPPORT)-7]') endif endif CPPFLAGS += -DZSTD_LEGACY_SUPPORT=$(ZSTD_LEGACY_SUPPORT) UNAME := $(shell uname) ifndef BUILD_DIR ifeq ($(UNAME), Darwin) ifeq ($(shell md5 < /dev/null > /dev/null; echo $$?), 0) HASH ?= md5 endif else ifeq ($(UNAME), FreeBSD) HASH ?= gmd5sum else ifeq ($(UNAME), NetBSD) HASH ?= md5 -n else ifeq ($(UNAME), OpenBSD) HASH ?= md5 endif HASH ?= md5sum HASH_DIR = conf_$(shell echo $(CC) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(ZSTD_FILES) | $(HASH) | cut -f 1 -d " " ) HAVE_HASH :=$(shell echo 1 | $(HASH) > /dev/null && echo 1 || echo 0) ifeq ($(HAVE_HASH),0) $(info warning : could not find HASH ($(HASH)), needed to differentiate builds using different flags) BUILD_DIR := obj/generic_noconf endif endif # BUILD_DIR ZSTD_SUBDIR := $(LIBZSTD)/common $(LIBZSTD)/compress $(LIBZSTD)/decompress $(LIBZSTD)/dictBuilder $(LIBZSTD)/legacy $(LIBZSTD)/deprecated vpath %.c $(ZSTD_SUBDIR) vpath %.S $(ZSTD_SUBDIR) zmat-0.9.9/src/blosc2/internal-complibs/zstd/dictBuilder/0000755000175200007730000000000014515254731023552 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/internal-complibs/zstd/dictBuilder/divsufsort.h0000644000175200007730000000456314515254731026143 0ustar rlaboissrlaboiss/* * divsufsort.h for libdivsufsort-lite * Copyright (c) 2003-2008 Yuta Mori All Rights Reserved. * * Permission is hereby granted, free of charge, to any person * obtaining a copy of this software and associated documentation * files (the "Software"), to deal in the Software without * restriction, including without limitation the rights to use, * copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the * Software is furnished to do so, subject to the following * conditions: * * The above copyright notice and this permission notice shall be * included in all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR * OTHER DEALINGS IN THE SOFTWARE. */ #ifndef _DIVSUFSORT_H #define _DIVSUFSORT_H 1 #ifdef __cplusplus extern "C" { #endif /* __cplusplus */ /*- Prototypes -*/ /** * Constructs the suffix array of a given string. * @param T [0..n-1] The input string. * @param SA [0..n-1] The output array of suffixes. * @param n The length of the given string. * @param openMP enables OpenMP optimization. * @return 0 if no error occurred, -1 or -2 otherwise. */ int divsufsort(const unsigned char *T, int *SA, int n, int openMP); /** * Constructs the burrows-wheeler transformed string of a given string. * @param T [0..n-1] The input string. * @param U [0..n-1] The output string. (can be T) * @param A [0..n-1] The temporary array. (can be NULL) * @param n The length of the given string. * @param num_indexes The length of secondary indexes array. (can be NULL) * @param indexes The secondary indexes array. (can be NULL) * @param openMP enables OpenMP optimization. * @return The primary index if no error occurred, -1 or -2 otherwise. */ int divbwt(const unsigned char *T, unsigned char *U, int *A, int n, unsigned char * num_indexes, int * indexes, int openMP); #ifdef __cplusplus } /* extern "C" */ #endif /* __cplusplus */ #endif /* _DIVSUFSORT_H */ zmat-0.9.9/src/blosc2/internal-complibs/zstd/dictBuilder/zdict.c0000644000175200007730000012575014515254731025045 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /*-************************************** * Tuning parameters ****************************************/ #define MINRATIO 4 /* minimum nb of apparition to be selected in dictionary */ #define ZDICT_MAX_SAMPLES_SIZE (2000U << 20) #define ZDICT_MIN_SAMPLES_SIZE (ZDICT_CONTENTSIZE_MIN * MINRATIO) /*-************************************** * Compiler Options ****************************************/ /* Unix Large Files support (>4GB) */ #define _FILE_OFFSET_BITS 64 #if (defined(__sun__) && (!defined(__LP64__))) /* Sun Solaris 32-bits requires specific definitions */ # ifndef _LARGEFILE_SOURCE # define _LARGEFILE_SOURCE # endif #elif ! defined(__LP64__) /* No point defining Large file for 64 bit */ # ifndef _LARGEFILE64_SOURCE # define _LARGEFILE64_SOURCE # endif #endif /*-************************************* * Dependencies ***************************************/ #include /* malloc, free */ #include /* memset */ #include /* fprintf, fopen, ftello64 */ #include /* clock */ #ifndef ZDICT_STATIC_LINKING_ONLY # define ZDICT_STATIC_LINKING_ONLY #endif #include "../common/mem.h" /* read */ #include "../common/fse.h" /* FSE_normalizeCount, FSE_writeNCount */ #include "../common/huf.h" /* HUF_buildCTable, HUF_writeCTable */ #include "../common/zstd_internal.h" /* includes zstd.h */ #include "../common/xxhash.h" /* XXH64 */ #include "../compress/zstd_compress_internal.h" /* ZSTD_loadCEntropy() */ #include "../zdict.h" #include "divsufsort.h" #include "../common/bits.h" /* ZSTD_NbCommonBytes */ /*-************************************* * Constants ***************************************/ #define KB *(1 <<10) #define MB *(1 <<20) #define GB *(1U<<30) #define DICTLISTSIZE_DEFAULT 10000 #define NOISELENGTH 32 static const U32 g_selectivity_default = 9; /*-************************************* * Console display ***************************************/ #undef DISPLAY #define DISPLAY(...) { fprintf(stderr, __VA_ARGS__); fflush( stderr ); } #undef DISPLAYLEVEL #define DISPLAYLEVEL(l, ...) if (notificationLevel>=l) { DISPLAY(__VA_ARGS__); } /* 0 : no display; 1: errors; 2: default; 3: details; 4: debug */ static clock_t ZDICT_clockSpan(clock_t nPrevious) { return clock() - nPrevious; } static void ZDICT_printHex(const void* ptr, size_t length) { const BYTE* const b = (const BYTE*)ptr; size_t u; for (u=0; u126) c = '.'; /* non-printable char */ DISPLAY("%c", c); } } /*-******************************************************** * Helper functions **********************************************************/ unsigned ZDICT_isError(size_t errorCode) { return ERR_isError(errorCode); } const char* ZDICT_getErrorName(size_t errorCode) { return ERR_getErrorName(errorCode); } unsigned ZDICT_getDictID(const void* dictBuffer, size_t dictSize) { if (dictSize < 8) return 0; if (MEM_readLE32(dictBuffer) != ZSTD_MAGIC_DICTIONARY) return 0; return MEM_readLE32((const char*)dictBuffer + 4); } size_t ZDICT_getDictHeaderSize(const void* dictBuffer, size_t dictSize) { size_t headerSize; if (dictSize <= 8 || MEM_readLE32(dictBuffer) != ZSTD_MAGIC_DICTIONARY) return ERROR(dictionary_corrupted); { ZSTD_compressedBlockState_t* bs = (ZSTD_compressedBlockState_t*)malloc(sizeof(ZSTD_compressedBlockState_t)); U32* wksp = (U32*)malloc(HUF_WORKSPACE_SIZE); if (!bs || !wksp) { headerSize = ERROR(memory_allocation); } else { ZSTD_reset_compressedBlockState(bs); headerSize = ZSTD_loadCEntropy(bs, wksp, dictBuffer, dictSize); } free(bs); free(wksp); } return headerSize; } /*-******************************************************** * Dictionary training functions **********************************************************/ /*! ZDICT_count() : Count the nb of common bytes between 2 pointers. Note : this function presumes end of buffer followed by noisy guard band. */ static size_t ZDICT_count(const void* pIn, const void* pMatch) { const char* const pStart = (const char*)pIn; for (;;) { size_t const diff = MEM_readST(pMatch) ^ MEM_readST(pIn); if (!diff) { pIn = (const char*)pIn+sizeof(size_t); pMatch = (const char*)pMatch+sizeof(size_t); continue; } pIn = (const char*)pIn+ZSTD_NbCommonBytes(diff); return (size_t)((const char*)pIn - pStart); } } typedef struct { U32 pos; U32 length; U32 savings; } dictItem; static void ZDICT_initDictItem(dictItem* d) { d->pos = 1; d->length = 0; d->savings = (U32)(-1); } #define LLIMIT 64 /* heuristic determined experimentally */ #define MINMATCHLENGTH 7 /* heuristic determined experimentally */ static dictItem ZDICT_analyzePos( BYTE* doneMarks, const int* suffix, U32 start, const void* buffer, U32 minRatio, U32 notificationLevel) { U32 lengthList[LLIMIT] = {0}; U32 cumulLength[LLIMIT] = {0}; U32 savings[LLIMIT] = {0}; const BYTE* b = (const BYTE*)buffer; size_t maxLength = LLIMIT; size_t pos = (size_t)suffix[start]; U32 end = start; dictItem solution; /* init */ memset(&solution, 0, sizeof(solution)); doneMarks[pos] = 1; /* trivial repetition cases */ if ( (MEM_read16(b+pos+0) == MEM_read16(b+pos+2)) ||(MEM_read16(b+pos+1) == MEM_read16(b+pos+3)) ||(MEM_read16(b+pos+2) == MEM_read16(b+pos+4)) ) { /* skip and mark segment */ U16 const pattern16 = MEM_read16(b+pos+4); U32 u, patternEnd = 6; while (MEM_read16(b+pos+patternEnd) == pattern16) patternEnd+=2 ; if (b[pos+patternEnd] == b[pos+patternEnd-1]) patternEnd++; for (u=1; u= MINMATCHLENGTH); } /* look backward */ { size_t length; do { length = ZDICT_count(b + pos, b + *(suffix+start-1)); if (length >=MINMATCHLENGTH) start--; } while(length >= MINMATCHLENGTH); } /* exit if not found a minimum nb of repetitions */ if (end-start < minRatio) { U32 idx; for(idx=start; idx= %i at pos %7u ", (unsigned)(end-start), MINMATCHLENGTH, (unsigned)pos); DISPLAYLEVEL(4, "\n"); for (mml = MINMATCHLENGTH ; ; mml++) { BYTE currentChar = 0; U32 currentCount = 0; U32 currentID = refinedStart; U32 id; U32 selectedCount = 0; U32 selectedID = currentID; for (id =refinedStart; id < refinedEnd; id++) { if (b[suffix[id] + mml] != currentChar) { if (currentCount > selectedCount) { selectedCount = currentCount; selectedID = currentID; } currentID = id; currentChar = b[ suffix[id] + mml]; currentCount = 0; } currentCount ++; } if (currentCount > selectedCount) { /* for last */ selectedCount = currentCount; selectedID = currentID; } if (selectedCount < minRatio) break; refinedStart = selectedID; refinedEnd = refinedStart + selectedCount; } /* evaluate gain based on new dict */ start = refinedStart; pos = suffix[refinedStart]; end = start; memset(lengthList, 0, sizeof(lengthList)); /* look forward */ { size_t length; do { end++; length = ZDICT_count(b + pos, b + suffix[end]); if (length >= LLIMIT) length = LLIMIT-1; lengthList[length]++; } while (length >=MINMATCHLENGTH); } /* look backward */ { size_t length = MINMATCHLENGTH; while ((length >= MINMATCHLENGTH) & (start > 0)) { length = ZDICT_count(b + pos, b + suffix[start - 1]); if (length >= LLIMIT) length = LLIMIT - 1; lengthList[length]++; if (length >= MINMATCHLENGTH) start--; } } /* largest useful length */ memset(cumulLength, 0, sizeof(cumulLength)); cumulLength[maxLength-1] = lengthList[maxLength-1]; for (i=(int)(maxLength-2); i>=0; i--) cumulLength[i] = cumulLength[i+1] + lengthList[i]; for (i=LLIMIT-1; i>=MINMATCHLENGTH; i--) if (cumulLength[i]>=minRatio) break; maxLength = i; /* reduce maxLength in case of final into repetitive data */ { U32 l = (U32)maxLength; BYTE const c = b[pos + maxLength-1]; while (b[pos+l-2]==c) l--; maxLength = l; } if (maxLength < MINMATCHLENGTH) return solution; /* skip : no long-enough solution */ /* calculate savings */ savings[5] = 0; for (i=MINMATCHLENGTH; i<=(int)maxLength; i++) savings[i] = savings[i-1] + (lengthList[i] * (i-3)); DISPLAYLEVEL(4, "Selected dict at position %u, of length %u : saves %u (ratio: %.2f) \n", (unsigned)pos, (unsigned)maxLength, (unsigned)savings[maxLength], (double)savings[maxLength] / (double)maxLength); solution.pos = (U32)pos; solution.length = (U32)maxLength; solution.savings = savings[maxLength]; /* mark positions done */ { U32 id; for (id=start; id solution.length) length = solution.length; } pEnd = (U32)(testedPos + length); for (p=testedPos; ppos; const U32 eltEnd = elt.pos + elt.length; const char* const buf = (const char*) buffer; /* tail overlap */ U32 u; for (u=1; u elt.pos) && (table[u].pos <= eltEnd)) { /* overlap, existing > new */ /* append */ U32 const addedLength = table[u].pos - elt.pos; table[u].length += addedLength; table[u].pos = elt.pos; table[u].savings += elt.savings * addedLength / elt.length; /* rough approx */ table[u].savings += elt.length / 8; /* rough approx bonus */ elt = table[u]; /* sort : improve rank */ while ((u>1) && (table[u-1].savings < elt.savings)) table[u] = table[u-1], u--; table[u] = elt; return u; } } /* front overlap */ for (u=1; u= elt.pos) && (table[u].pos < elt.pos)) { /* overlap, existing < new */ /* append */ int const addedLength = (int)eltEnd - (int)(table[u].pos + table[u].length); table[u].savings += elt.length / 8; /* rough approx bonus */ if (addedLength > 0) { /* otherwise, elt fully included into existing */ table[u].length += addedLength; table[u].savings += elt.savings * addedLength / elt.length; /* rough approx */ } /* sort : improve rank */ elt = table[u]; while ((u>1) && (table[u-1].savings < elt.savings)) table[u] = table[u-1], u--; table[u] = elt; return u; } if (MEM_read64(buf + table[u].pos) == MEM_read64(buf + elt.pos + 1)) { if (isIncluded(buf + table[u].pos, buf + elt.pos + 1, table[u].length)) { size_t const addedLength = MAX( (int)elt.length - (int)table[u].length , 1 ); table[u].pos = elt.pos; table[u].savings += (U32)(elt.savings * addedLength / elt.length); table[u].length = MIN(elt.length, table[u].length + 1); return u; } } } return 0; } static void ZDICT_removeDictItem(dictItem* table, U32 id) { /* convention : table[0].pos stores nb of elts */ U32 const max = table[0].pos; U32 u; if (!id) return; /* protection, should never happen */ for (u=id; upos--; } static void ZDICT_insertDictItem(dictItem* table, U32 maxSize, dictItem elt, const void* buffer) { /* merge if possible */ U32 mergeId = ZDICT_tryMerge(table, elt, 0, buffer); if (mergeId) { U32 newMerge = 1; while (newMerge) { newMerge = ZDICT_tryMerge(table, table[mergeId], mergeId, buffer); if (newMerge) ZDICT_removeDictItem(table, mergeId); mergeId = newMerge; } return; } /* insert */ { U32 current; U32 nextElt = table->pos; if (nextElt >= maxSize) nextElt = maxSize-1; current = nextElt-1; while (table[current].savings < elt.savings) { table[current+1] = table[current]; current--; } table[current+1] = elt; table->pos = nextElt+1; } } static U32 ZDICT_dictSize(const dictItem* dictList) { U32 u, dictSize = 0; for (u=1; u=l) { \ if (ZDICT_clockSpan(displayClock) > refreshRate) \ { displayClock = clock(); DISPLAY(__VA_ARGS__); \ if (notificationLevel>=4) fflush(stderr); } } /* init */ DISPLAYLEVEL(2, "\r%70s\r", ""); /* clean display line */ if (!suffix0 || !reverseSuffix || !doneMarks || !filePos) { result = ERROR(memory_allocation); goto _cleanup; } if (minRatio < MINRATIO) minRatio = MINRATIO; memset(doneMarks, 0, bufferSize+16); /* limit sample set size (divsufsort limitation)*/ if (bufferSize > ZDICT_MAX_SAMPLES_SIZE) DISPLAYLEVEL(3, "sample set too large : reduced to %u MB ...\n", (unsigned)(ZDICT_MAX_SAMPLES_SIZE>>20)); while (bufferSize > ZDICT_MAX_SAMPLES_SIZE) bufferSize -= fileSizes[--nbFiles]; /* sort */ DISPLAYLEVEL(2, "sorting %u files of total size %u MB ...\n", nbFiles, (unsigned)(bufferSize>>20)); { int const divSuftSortResult = divsufsort((const unsigned char*)buffer, suffix, (int)bufferSize, 0); if (divSuftSortResult != 0) { result = ERROR(GENERIC); goto _cleanup; } } suffix[bufferSize] = (int)bufferSize; /* leads into noise */ suffix0[0] = (int)bufferSize; /* leads into noise */ /* build reverse suffix sort */ { size_t pos; for (pos=0; pos < bufferSize; pos++) reverseSuffix[suffix[pos]] = (U32)pos; /* note filePos tracks borders between samples. It's not used at this stage, but planned to become useful in a later update */ filePos[0] = 0; for (pos=1; pos> 21); } } typedef struct { ZSTD_CDict* dict; /* dictionary */ ZSTD_CCtx* zc; /* working context */ void* workPlace; /* must be ZSTD_BLOCKSIZE_MAX allocated */ } EStats_ress_t; #define MAXREPOFFSET 1024 static void ZDICT_countEStats(EStats_ress_t esr, const ZSTD_parameters* params, unsigned* countLit, unsigned* offsetcodeCount, unsigned* matchlengthCount, unsigned* litlengthCount, U32* repOffsets, const void* src, size_t srcSize, U32 notificationLevel) { size_t const blockSizeMax = MIN (ZSTD_BLOCKSIZE_MAX, 1 << params->cParams.windowLog); size_t cSize; if (srcSize > blockSizeMax) srcSize = blockSizeMax; /* protection vs large samples */ { size_t const errorCode = ZSTD_compressBegin_usingCDict_deprecated(esr.zc, esr.dict); if (ZSTD_isError(errorCode)) { DISPLAYLEVEL(1, "warning : ZSTD_compressBegin_usingCDict failed \n"); return; } } cSize = ZSTD_compressBlock_deprecated(esr.zc, esr.workPlace, ZSTD_BLOCKSIZE_MAX, src, srcSize); if (ZSTD_isError(cSize)) { DISPLAYLEVEL(3, "warning : could not compress sample size %u \n", (unsigned)srcSize); return; } if (cSize) { /* if == 0; block is not compressible */ const seqStore_t* const seqStorePtr = ZSTD_getSeqStore(esr.zc); /* literals stats */ { const BYTE* bytePtr; for(bytePtr = seqStorePtr->litStart; bytePtr < seqStorePtr->lit; bytePtr++) countLit[*bytePtr]++; } /* seqStats */ { U32 const nbSeq = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart); ZSTD_seqToCodes(seqStorePtr); { const BYTE* codePtr = seqStorePtr->ofCode; U32 u; for (u=0; umlCode; U32 u; for (u=0; ullCode; U32 u; for (u=0; u= 2) { /* rep offsets */ const seqDef* const seq = seqStorePtr->sequencesStart; U32 offset1 = seq[0].offBase - ZSTD_REP_NUM; U32 offset2 = seq[1].offBase - ZSTD_REP_NUM; if (offset1 >= MAXREPOFFSET) offset1 = 0; if (offset2 >= MAXREPOFFSET) offset2 = 0; repOffsets[offset1] += 3; repOffsets[offset2] += 1; } } } } static size_t ZDICT_totalSampleSize(const size_t* fileSizes, unsigned nbFiles) { size_t total=0; unsigned u; for (u=0; u0; u--) { offsetCount_t tmp; if (table[u-1].count >= table[u].count) break; tmp = table[u-1]; table[u-1] = table[u]; table[u] = tmp; } } /* ZDICT_flatLit() : * rewrite `countLit` to contain a mostly flat but still compressible distribution of literals. * necessary to avoid generating a non-compressible distribution that HUF_writeCTable() cannot encode. */ static void ZDICT_flatLit(unsigned* countLit) { int u; for (u=1; u<256; u++) countLit[u] = 2; countLit[0] = 4; countLit[253] = 1; countLit[254] = 1; } #define OFFCODE_MAX 30 /* only applicable to first block */ static size_t ZDICT_analyzeEntropy(void* dstBuffer, size_t maxDstSize, int compressionLevel, const void* srcBuffer, const size_t* fileSizes, unsigned nbFiles, const void* dictBuffer, size_t dictBufferSize, unsigned notificationLevel) { unsigned countLit[256]; HUF_CREATE_STATIC_CTABLE(hufTable, 255); unsigned offcodeCount[OFFCODE_MAX+1]; short offcodeNCount[OFFCODE_MAX+1]; U32 offcodeMax = ZSTD_highbit32((U32)(dictBufferSize + 128 KB)); unsigned matchLengthCount[MaxML+1]; short matchLengthNCount[MaxML+1]; unsigned litLengthCount[MaxLL+1]; short litLengthNCount[MaxLL+1]; U32 repOffset[MAXREPOFFSET]; offsetCount_t bestRepOffset[ZSTD_REP_NUM+1]; EStats_ress_t esr = { NULL, NULL, NULL }; ZSTD_parameters params; U32 u, huffLog = 11, Offlog = OffFSELog, mlLog = MLFSELog, llLog = LLFSELog, total; size_t pos = 0, errorCode; size_t eSize = 0; size_t const totalSrcSize = ZDICT_totalSampleSize(fileSizes, nbFiles); size_t const averageSampleSize = totalSrcSize / (nbFiles + !nbFiles); BYTE* dstPtr = (BYTE*)dstBuffer; U32 wksp[HUF_CTABLE_WORKSPACE_SIZE_U32]; /* init */ DEBUGLOG(4, "ZDICT_analyzeEntropy"); if (offcodeMax>OFFCODE_MAX) { eSize = ERROR(dictionaryCreation_failed); goto _cleanup; } /* too large dictionary */ for (u=0; u<256; u++) countLit[u] = 1; /* any character must be described */ for (u=0; u<=offcodeMax; u++) offcodeCount[u] = 1; for (u=0; u<=MaxML; u++) matchLengthCount[u] = 1; for (u=0; u<=MaxLL; u++) litLengthCount[u] = 1; memset(repOffset, 0, sizeof(repOffset)); repOffset[1] = repOffset[4] = repOffset[8] = 1; memset(bestRepOffset, 0, sizeof(bestRepOffset)); if (compressionLevel==0) compressionLevel = ZSTD_CLEVEL_DEFAULT; params = ZSTD_getParams(compressionLevel, averageSampleSize, dictBufferSize); esr.dict = ZSTD_createCDict_advanced(dictBuffer, dictBufferSize, ZSTD_dlm_byRef, ZSTD_dct_rawContent, params.cParams, ZSTD_defaultCMem); esr.zc = ZSTD_createCCtx(); esr.workPlace = malloc(ZSTD_BLOCKSIZE_MAX); if (!esr.dict || !esr.zc || !esr.workPlace) { eSize = ERROR(memory_allocation); DISPLAYLEVEL(1, "Not enough memory \n"); goto _cleanup; } /* collect stats on all samples */ for (u=0; u= 4) { /* writeStats */ DISPLAYLEVEL(4, "Offset Code Frequencies : \n"); for (u=0; u<=offcodeMax; u++) { DISPLAYLEVEL(4, "%2u :%7u \n", u, offcodeCount[u]); } } /* analyze, build stats, starting with literals */ { size_t maxNbBits = HUF_buildCTable_wksp(hufTable, countLit, 255, huffLog, wksp, sizeof(wksp)); if (HUF_isError(maxNbBits)) { eSize = maxNbBits; DISPLAYLEVEL(1, " HUF_buildCTable error \n"); goto _cleanup; } if (maxNbBits==8) { /* not compressible : will fail on HUF_writeCTable() */ DISPLAYLEVEL(2, "warning : pathological dataset : literals are not compressible : samples are noisy or too regular \n"); ZDICT_flatLit(countLit); /* replace distribution by a fake "mostly flat but still compressible" distribution, that HUF_writeCTable() can encode */ maxNbBits = HUF_buildCTable_wksp(hufTable, countLit, 255, huffLog, wksp, sizeof(wksp)); assert(maxNbBits==9); } huffLog = (U32)maxNbBits; } /* looking for most common first offsets */ { U32 offset; for (offset=1; offset dictBufferCapacity) { dictContentSize = dictBufferCapacity - hSize; } /* Pad the dictionary content with zeros if it is too small */ if (dictContentSize < minContentSize) { RETURN_ERROR_IF(hSize + minContentSize > dictBufferCapacity, dstSize_tooSmall, "dictBufferCapacity too small to fit max repcode"); paddingSize = minContentSize - dictContentSize; } else { paddingSize = 0; } { size_t const dictSize = hSize + paddingSize + dictContentSize; /* The dictionary consists of the header, optional padding, and the content. * The padding comes before the content because the "best" position in the * dictionary is the last byte. */ BYTE* const outDictHeader = (BYTE*)dictBuffer; BYTE* const outDictPadding = outDictHeader + hSize; BYTE* const outDictContent = outDictPadding + paddingSize; assert(dictSize <= dictBufferCapacity); assert(outDictContent + dictContentSize == (BYTE*)dictBuffer + dictSize); /* First copy the customDictContent into its final location. * `customDictContent` and `dictBuffer` may overlap, so we must * do this before any other writes into the output buffer. * Then copy the header & padding into the output buffer. */ memmove(outDictContent, customDictContent, dictContentSize); memcpy(outDictHeader, header, hSize); memset(outDictPadding, 0, paddingSize); return dictSize; } } static size_t ZDICT_addEntropyTablesFromBuffer_advanced( void* dictBuffer, size_t dictContentSize, size_t dictBufferCapacity, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples, ZDICT_params_t params) { int const compressionLevel = (params.compressionLevel == 0) ? ZSTD_CLEVEL_DEFAULT : params.compressionLevel; U32 const notificationLevel = params.notificationLevel; size_t hSize = 8; /* calculate entropy tables */ DISPLAYLEVEL(2, "\r%70s\r", ""); /* clean display line */ DISPLAYLEVEL(2, "statistics ... \n"); { size_t const eSize = ZDICT_analyzeEntropy((char*)dictBuffer+hSize, dictBufferCapacity-hSize, compressionLevel, samplesBuffer, samplesSizes, nbSamples, (char*)dictBuffer + dictBufferCapacity - dictContentSize, dictContentSize, notificationLevel); if (ZDICT_isError(eSize)) return eSize; hSize += eSize; } /* add dictionary header (after entropy tables) */ MEM_writeLE32(dictBuffer, ZSTD_MAGIC_DICTIONARY); { U64 const randomID = XXH64((char*)dictBuffer + dictBufferCapacity - dictContentSize, dictContentSize, 0); U32 const compliantID = (randomID % ((1U<<31)-32768)) + 32768; U32 const dictID = params.dictID ? params.dictID : compliantID; MEM_writeLE32((char*)dictBuffer+4, dictID); } if (hSize + dictContentSize < dictBufferCapacity) memmove((char*)dictBuffer + hSize, (char*)dictBuffer + dictBufferCapacity - dictContentSize, dictContentSize); return MIN(dictBufferCapacity, hSize+dictContentSize); } /*! ZDICT_trainFromBuffer_unsafe_legacy() : * Warning : `samplesBuffer` must be followed by noisy guard band !!! * @return : size of dictionary, or an error code which can be tested with ZDICT_isError() */ static size_t ZDICT_trainFromBuffer_unsafe_legacy( void* dictBuffer, size_t maxDictSize, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples, ZDICT_legacy_params_t params) { U32 const dictListSize = MAX(MAX(DICTLISTSIZE_DEFAULT, nbSamples), (U32)(maxDictSize/16)); dictItem* const dictList = (dictItem*)malloc(dictListSize * sizeof(*dictList)); unsigned const selectivity = params.selectivityLevel == 0 ? g_selectivity_default : params.selectivityLevel; unsigned const minRep = (selectivity > 30) ? MINRATIO : nbSamples >> selectivity; size_t const targetDictSize = maxDictSize; size_t const samplesBuffSize = ZDICT_totalSampleSize(samplesSizes, nbSamples); size_t dictSize = 0; U32 const notificationLevel = params.zParams.notificationLevel; /* checks */ if (!dictList) return ERROR(memory_allocation); if (maxDictSize < ZDICT_DICTSIZE_MIN) { free(dictList); return ERROR(dstSize_tooSmall); } /* requested dictionary size is too small */ if (samplesBuffSize < ZDICT_MIN_SAMPLES_SIZE) { free(dictList); return ERROR(dictionaryCreation_failed); } /* not enough source to create dictionary */ /* init */ ZDICT_initDictItem(dictList); /* build dictionary */ ZDICT_trainBuffer_legacy(dictList, dictListSize, samplesBuffer, samplesBuffSize, samplesSizes, nbSamples, minRep, notificationLevel); /* display best matches */ if (params.zParams.notificationLevel>= 3) { unsigned const nb = MIN(25, dictList[0].pos); unsigned const dictContentSize = ZDICT_dictSize(dictList); unsigned u; DISPLAYLEVEL(3, "\n %u segments found, of total size %u \n", (unsigned)dictList[0].pos-1, dictContentSize); DISPLAYLEVEL(3, "list %u best segments \n", nb-1); for (u=1; u samplesBuffSize) || ((pos + length) > samplesBuffSize)) { free(dictList); return ERROR(GENERIC); /* should never happen */ } DISPLAYLEVEL(3, "%3u:%3u bytes at pos %8u, savings %7u bytes |", u, length, pos, (unsigned)dictList[u].savings); ZDICT_printHex((const char*)samplesBuffer+pos, printedLength); DISPLAYLEVEL(3, "| \n"); } } /* create dictionary */ { unsigned dictContentSize = ZDICT_dictSize(dictList); if (dictContentSize < ZDICT_CONTENTSIZE_MIN) { free(dictList); return ERROR(dictionaryCreation_failed); } /* dictionary content too small */ if (dictContentSize < targetDictSize/4) { DISPLAYLEVEL(2, "! warning : selected content significantly smaller than requested (%u < %u) \n", dictContentSize, (unsigned)maxDictSize); if (samplesBuffSize < 10 * targetDictSize) DISPLAYLEVEL(2, "! consider increasing the number of samples (total size : %u MB)\n", (unsigned)(samplesBuffSize>>20)); if (minRep > MINRATIO) { DISPLAYLEVEL(2, "! consider increasing selectivity to produce larger dictionary (-s%u) \n", selectivity+1); DISPLAYLEVEL(2, "! note : larger dictionaries are not necessarily better, test its efficiency on samples \n"); } } if ((dictContentSize > targetDictSize*3) && (nbSamples > 2*MINRATIO) && (selectivity>1)) { unsigned proposedSelectivity = selectivity-1; while ((nbSamples >> proposedSelectivity) <= MINRATIO) { proposedSelectivity--; } DISPLAYLEVEL(2, "! note : calculated dictionary significantly larger than requested (%u > %u) \n", dictContentSize, (unsigned)maxDictSize); DISPLAYLEVEL(2, "! consider increasing dictionary size, or produce denser dictionary (-s%u) \n", proposedSelectivity); DISPLAYLEVEL(2, "! always test dictionary efficiency on real samples \n"); } /* limit dictionary size */ { U32 const max = dictList->pos; /* convention : nb of useful elts within dictList */ U32 currentSize = 0; U32 n; for (n=1; n targetDictSize) { currentSize -= dictList[n].length; break; } } dictList->pos = n; dictContentSize = currentSize; } /* build dict content */ { U32 u; BYTE* ptr = (BYTE*)dictBuffer + maxDictSize; for (u=1; upos; u++) { U32 l = dictList[u].length; ptr -= l; if (ptr<(BYTE*)dictBuffer) { free(dictList); return ERROR(GENERIC); } /* should not happen */ memcpy(ptr, (const char*)samplesBuffer+dictList[u].pos, l); } } dictSize = ZDICT_addEntropyTablesFromBuffer_advanced(dictBuffer, dictContentSize, maxDictSize, samplesBuffer, samplesSizes, nbSamples, params.zParams); } /* clean up */ free(dictList); return dictSize; } /* ZDICT_trainFromBuffer_legacy() : * issue : samplesBuffer need to be followed by a noisy guard band. * work around : duplicate the buffer, and add the noise */ size_t ZDICT_trainFromBuffer_legacy(void* dictBuffer, size_t dictBufferCapacity, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples, ZDICT_legacy_params_t params) { size_t result; void* newBuff; size_t const sBuffSize = ZDICT_totalSampleSize(samplesSizes, nbSamples); if (sBuffSize < ZDICT_MIN_SAMPLES_SIZE) return 0; /* not enough content => no dictionary */ newBuff = malloc(sBuffSize + NOISELENGTH); if (!newBuff) return ERROR(memory_allocation); memcpy(newBuff, samplesBuffer, sBuffSize); ZDICT_fillNoise((char*)newBuff + sBuffSize, NOISELENGTH); /* guard band, for end of buffer condition */ result = ZDICT_trainFromBuffer_unsafe_legacy(dictBuffer, dictBufferCapacity, newBuff, samplesSizes, nbSamples, params); free(newBuff); return result; } size_t ZDICT_trainFromBuffer(void* dictBuffer, size_t dictBufferCapacity, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples) { ZDICT_fastCover_params_t params; DEBUGLOG(3, "ZDICT_trainFromBuffer"); memset(¶ms, 0, sizeof(params)); params.d = 8; params.steps = 4; /* Use default level since no compression level information is available */ params.zParams.compressionLevel = ZSTD_CLEVEL_DEFAULT; #if defined(DEBUGLEVEL) && (DEBUGLEVEL>=1) params.zParams.notificationLevel = DEBUGLEVEL; #endif return ZDICT_optimizeTrainFromBuffer_fastCover(dictBuffer, dictBufferCapacity, samplesBuffer, samplesSizes, nbSamples, ¶ms); } size_t ZDICT_addEntropyTablesFromBuffer(void* dictBuffer, size_t dictContentSize, size_t dictBufferCapacity, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples) { ZDICT_params_t params; memset(¶ms, 0, sizeof(params)); return ZDICT_addEntropyTablesFromBuffer_advanced(dictBuffer, dictContentSize, dictBufferCapacity, samplesBuffer, samplesSizes, nbSamples, params); } zmat-0.9.9/src/blosc2/internal-complibs/zstd/dictBuilder/fastcover.c0000644000175200007730000006762114515254731025726 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /*-************************************* * Dependencies ***************************************/ #include /* fprintf */ #include /* malloc, free, qsort */ #include /* memset */ #include /* clock */ #ifndef ZDICT_STATIC_LINKING_ONLY # define ZDICT_STATIC_LINKING_ONLY #endif #include "../common/mem.h" /* read */ #include "../common/pool.h" #include "../common/threading.h" #include "../common/zstd_internal.h" /* includes zstd.h */ #include "../compress/zstd_compress_internal.h" /* ZSTD_hash*() */ #include "../zdict.h" #include "cover.h" /*-************************************* * Constants ***************************************/ /** * There are 32bit indexes used to ref samples, so limit samples size to 4GB * on 64bit builds. * For 32bit builds we choose 1 GB. * Most 32bit platforms have 2GB user-mode addressable space and we allocate a large * contiguous buffer, so 1GB is already a high limit. */ #define FASTCOVER_MAX_SAMPLES_SIZE (sizeof(size_t) == 8 ? ((unsigned)-1) : ((unsigned)1 GB)) #define FASTCOVER_MAX_F 31 #define FASTCOVER_MAX_ACCEL 10 #define FASTCOVER_DEFAULT_SPLITPOINT 0.75 #define DEFAULT_F 20 #define DEFAULT_ACCEL 1 /*-************************************* * Console display ***************************************/ #ifndef LOCALDISPLAYLEVEL static int g_displayLevel = 0; #endif #undef DISPLAY #define DISPLAY(...) \ { \ fprintf(stderr, __VA_ARGS__); \ fflush(stderr); \ } #undef LOCALDISPLAYLEVEL #define LOCALDISPLAYLEVEL(displayLevel, l, ...) \ if (displayLevel >= l) { \ DISPLAY(__VA_ARGS__); \ } /* 0 : no display; 1: errors; 2: default; 3: details; 4: debug */ #undef DISPLAYLEVEL #define DISPLAYLEVEL(l, ...) LOCALDISPLAYLEVEL(g_displayLevel, l, __VA_ARGS__) #ifndef LOCALDISPLAYUPDATE static const clock_t g_refreshRate = CLOCKS_PER_SEC * 15 / 100; static clock_t g_time = 0; #endif #undef LOCALDISPLAYUPDATE #define LOCALDISPLAYUPDATE(displayLevel, l, ...) \ if (displayLevel >= l) { \ if ((clock() - g_time > g_refreshRate) || (displayLevel >= 4)) { \ g_time = clock(); \ DISPLAY(__VA_ARGS__); \ } \ } #undef DISPLAYUPDATE #define DISPLAYUPDATE(l, ...) LOCALDISPLAYUPDATE(g_displayLevel, l, __VA_ARGS__) /*-************************************* * Hash Functions ***************************************/ /** * Hash the d-byte value pointed to by p and mod 2^f into the frequency vector */ static size_t FASTCOVER_hashPtrToIndex(const void* p, U32 f, unsigned d) { if (d == 6) { return ZSTD_hash6Ptr(p, f); } return ZSTD_hash8Ptr(p, f); } /*-************************************* * Acceleration ***************************************/ typedef struct { unsigned finalize; /* Percentage of training samples used for ZDICT_finalizeDictionary */ unsigned skip; /* Number of dmer skipped between each dmer counted in computeFrequency */ } FASTCOVER_accel_t; static const FASTCOVER_accel_t FASTCOVER_defaultAccelParameters[FASTCOVER_MAX_ACCEL+1] = { { 100, 0 }, /* accel = 0, should not happen because accel = 0 defaults to accel = 1 */ { 100, 0 }, /* accel = 1 */ { 50, 1 }, /* accel = 2 */ { 34, 2 }, /* accel = 3 */ { 25, 3 }, /* accel = 4 */ { 20, 4 }, /* accel = 5 */ { 17, 5 }, /* accel = 6 */ { 14, 6 }, /* accel = 7 */ { 13, 7 }, /* accel = 8 */ { 11, 8 }, /* accel = 9 */ { 10, 9 }, /* accel = 10 */ }; /*-************************************* * Context ***************************************/ typedef struct { const BYTE *samples; size_t *offsets; const size_t *samplesSizes; size_t nbSamples; size_t nbTrainSamples; size_t nbTestSamples; size_t nbDmers; U32 *freqs; unsigned d; unsigned f; FASTCOVER_accel_t accelParams; } FASTCOVER_ctx_t; /*-************************************* * Helper functions ***************************************/ /** * Selects the best segment in an epoch. * Segments of are scored according to the function: * * Let F(d) be the frequency of all dmers with hash value d. * Let S_i be hash value of the dmer at position i of segment S which has length k. * * Score(S) = F(S_1) + F(S_2) + ... + F(S_{k-d+1}) * * Once the dmer with hash value d is in the dictionary we set F(d) = 0. */ static COVER_segment_t FASTCOVER_selectSegment(const FASTCOVER_ctx_t *ctx, U32 *freqs, U32 begin, U32 end, ZDICT_cover_params_t parameters, U16* segmentFreqs) { /* Constants */ const U32 k = parameters.k; const U32 d = parameters.d; const U32 f = ctx->f; const U32 dmersInK = k - d + 1; /* Try each segment (activeSegment) and save the best (bestSegment) */ COVER_segment_t bestSegment = {0, 0, 0}; COVER_segment_t activeSegment; /* Reset the activeDmers in the segment */ /* The activeSegment starts at the beginning of the epoch. */ activeSegment.begin = begin; activeSegment.end = begin; activeSegment.score = 0; /* Slide the activeSegment through the whole epoch. * Save the best segment in bestSegment. */ while (activeSegment.end < end) { /* Get hash value of current dmer */ const size_t idx = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.end, f, d); /* Add frequency of this index to score if this is the first occurrence of index in active segment */ if (segmentFreqs[idx] == 0) { activeSegment.score += freqs[idx]; } /* Increment end of segment and segmentFreqs*/ activeSegment.end += 1; segmentFreqs[idx] += 1; /* If the window is now too large, drop the first position */ if (activeSegment.end - activeSegment.begin == dmersInK + 1) { /* Get hash value of the dmer to be eliminated from active segment */ const size_t delIndex = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.begin, f, d); segmentFreqs[delIndex] -= 1; /* Subtract frequency of this index from score if this is the last occurrence of this index in active segment */ if (segmentFreqs[delIndex] == 0) { activeSegment.score -= freqs[delIndex]; } /* Increment start of segment */ activeSegment.begin += 1; } /* If this segment is the best so far save it */ if (activeSegment.score > bestSegment.score) { bestSegment = activeSegment; } } /* Zero out rest of segmentFreqs array */ while (activeSegment.begin < end) { const size_t delIndex = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.begin, f, d); segmentFreqs[delIndex] -= 1; activeSegment.begin += 1; } { /* Zero the frequency of hash value of each dmer covered by the chosen segment. */ U32 pos; for (pos = bestSegment.begin; pos != bestSegment.end; ++pos) { const size_t i = FASTCOVER_hashPtrToIndex(ctx->samples + pos, f, d); freqs[i] = 0; } } return bestSegment; } static int FASTCOVER_checkParameters(ZDICT_cover_params_t parameters, size_t maxDictSize, unsigned f, unsigned accel) { /* k, d, and f are required parameters */ if (parameters.d == 0 || parameters.k == 0) { return 0; } /* d has to be 6 or 8 */ if (parameters.d != 6 && parameters.d != 8) { return 0; } /* k <= maxDictSize */ if (parameters.k > maxDictSize) { return 0; } /* d <= k */ if (parameters.d > parameters.k) { return 0; } /* 0 < f <= FASTCOVER_MAX_F*/ if (f > FASTCOVER_MAX_F || f == 0) { return 0; } /* 0 < splitPoint <= 1 */ if (parameters.splitPoint <= 0 || parameters.splitPoint > 1) { return 0; } /* 0 < accel <= 10 */ if (accel > 10 || accel == 0) { return 0; } return 1; } /** * Clean up a context initialized with `FASTCOVER_ctx_init()`. */ static void FASTCOVER_ctx_destroy(FASTCOVER_ctx_t* ctx) { if (!ctx) return; free(ctx->freqs); ctx->freqs = NULL; free(ctx->offsets); ctx->offsets = NULL; } /** * Calculate for frequency of hash value of each dmer in ctx->samples */ static void FASTCOVER_computeFrequency(U32* freqs, const FASTCOVER_ctx_t* ctx) { const unsigned f = ctx->f; const unsigned d = ctx->d; const unsigned skip = ctx->accelParams.skip; const unsigned readLength = MAX(d, 8); size_t i; assert(ctx->nbTrainSamples >= 5); assert(ctx->nbTrainSamples <= ctx->nbSamples); for (i = 0; i < ctx->nbTrainSamples; i++) { size_t start = ctx->offsets[i]; /* start of current dmer */ size_t const currSampleEnd = ctx->offsets[i+1]; while (start + readLength <= currSampleEnd) { const size_t dmerIndex = FASTCOVER_hashPtrToIndex(ctx->samples + start, f, d); freqs[dmerIndex]++; start = start + skip + 1; } } } /** * Prepare a context for dictionary building. * The context is only dependent on the parameter `d` and can be used multiple * times. * Returns 0 on success or error code on error. * The context must be destroyed with `FASTCOVER_ctx_destroy()`. */ static size_t FASTCOVER_ctx_init(FASTCOVER_ctx_t* ctx, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples, unsigned d, double splitPoint, unsigned f, FASTCOVER_accel_t accelParams) { const BYTE* const samples = (const BYTE*)samplesBuffer; const size_t totalSamplesSize = COVER_sum(samplesSizes, nbSamples); /* Split samples into testing and training sets */ const unsigned nbTrainSamples = splitPoint < 1.0 ? (unsigned)((double)nbSamples * splitPoint) : nbSamples; const unsigned nbTestSamples = splitPoint < 1.0 ? nbSamples - nbTrainSamples : nbSamples; const size_t trainingSamplesSize = splitPoint < 1.0 ? COVER_sum(samplesSizes, nbTrainSamples) : totalSamplesSize; const size_t testSamplesSize = splitPoint < 1.0 ? COVER_sum(samplesSizes + nbTrainSamples, nbTestSamples) : totalSamplesSize; /* Checks */ if (totalSamplesSize < MAX(d, sizeof(U64)) || totalSamplesSize >= (size_t)FASTCOVER_MAX_SAMPLES_SIZE) { DISPLAYLEVEL(1, "Total samples size is too large (%u MB), maximum size is %u MB\n", (unsigned)(totalSamplesSize >> 20), (FASTCOVER_MAX_SAMPLES_SIZE >> 20)); return ERROR(srcSize_wrong); } /* Check if there are at least 5 training samples */ if (nbTrainSamples < 5) { DISPLAYLEVEL(1, "Total number of training samples is %u and is invalid\n", nbTrainSamples); return ERROR(srcSize_wrong); } /* Check if there's testing sample */ if (nbTestSamples < 1) { DISPLAYLEVEL(1, "Total number of testing samples is %u and is invalid.\n", nbTestSamples); return ERROR(srcSize_wrong); } /* Zero the context */ memset(ctx, 0, sizeof(*ctx)); DISPLAYLEVEL(2, "Training on %u samples of total size %u\n", nbTrainSamples, (unsigned)trainingSamplesSize); DISPLAYLEVEL(2, "Testing on %u samples of total size %u\n", nbTestSamples, (unsigned)testSamplesSize); ctx->samples = samples; ctx->samplesSizes = samplesSizes; ctx->nbSamples = nbSamples; ctx->nbTrainSamples = nbTrainSamples; ctx->nbTestSamples = nbTestSamples; ctx->nbDmers = trainingSamplesSize - MAX(d, sizeof(U64)) + 1; ctx->d = d; ctx->f = f; ctx->accelParams = accelParams; /* The offsets of each file */ ctx->offsets = (size_t*)calloc((nbSamples + 1), sizeof(size_t)); if (ctx->offsets == NULL) { DISPLAYLEVEL(1, "Failed to allocate scratch buffers \n"); FASTCOVER_ctx_destroy(ctx); return ERROR(memory_allocation); } /* Fill offsets from the samplesSizes */ { U32 i; ctx->offsets[0] = 0; assert(nbSamples >= 5); for (i = 1; i <= nbSamples; ++i) { ctx->offsets[i] = ctx->offsets[i - 1] + samplesSizes[i - 1]; } } /* Initialize frequency array of size 2^f */ ctx->freqs = (U32*)calloc(((U64)1 << f), sizeof(U32)); if (ctx->freqs == NULL) { DISPLAYLEVEL(1, "Failed to allocate frequency table \n"); FASTCOVER_ctx_destroy(ctx); return ERROR(memory_allocation); } DISPLAYLEVEL(2, "Computing frequencies\n"); FASTCOVER_computeFrequency(ctx->freqs, ctx); return 0; } /** * Given the prepared context build the dictionary. */ static size_t FASTCOVER_buildDictionary(const FASTCOVER_ctx_t* ctx, U32* freqs, void* dictBuffer, size_t dictBufferCapacity, ZDICT_cover_params_t parameters, U16* segmentFreqs) { BYTE *const dict = (BYTE *)dictBuffer; size_t tail = dictBufferCapacity; /* Divide the data into epochs. We will select one segment from each epoch. */ const COVER_epoch_info_t epochs = COVER_computeEpochs( (U32)dictBufferCapacity, (U32)ctx->nbDmers, parameters.k, 1); const size_t maxZeroScoreRun = 10; size_t zeroScoreRun = 0; size_t epoch; DISPLAYLEVEL(2, "Breaking content into %u epochs of size %u\n", (U32)epochs.num, (U32)epochs.size); /* Loop through the epochs until there are no more segments or the dictionary * is full. */ for (epoch = 0; tail > 0; epoch = (epoch + 1) % epochs.num) { const U32 epochBegin = (U32)(epoch * epochs.size); const U32 epochEnd = epochBegin + epochs.size; size_t segmentSize; /* Select a segment */ COVER_segment_t segment = FASTCOVER_selectSegment( ctx, freqs, epochBegin, epochEnd, parameters, segmentFreqs); /* If the segment covers no dmers, then we are out of content. * There may be new content in other epochs, for continue for some time. */ if (segment.score == 0) { if (++zeroScoreRun >= maxZeroScoreRun) { break; } continue; } zeroScoreRun = 0; /* Trim the segment if necessary and if it is too small then we are done */ segmentSize = MIN(segment.end - segment.begin + parameters.d - 1, tail); if (segmentSize < parameters.d) { break; } /* We fill the dictionary from the back to allow the best segments to be * referenced with the smallest offsets. */ tail -= segmentSize; memcpy(dict + tail, ctx->samples + segment.begin, segmentSize); DISPLAYUPDATE( 2, "\r%u%% ", (unsigned)(((dictBufferCapacity - tail) * 100) / dictBufferCapacity)); } DISPLAYLEVEL(2, "\r%79s\r", ""); return tail; } /** * Parameters for FASTCOVER_tryParameters(). */ typedef struct FASTCOVER_tryParameters_data_s { const FASTCOVER_ctx_t* ctx; COVER_best_t* best; size_t dictBufferCapacity; ZDICT_cover_params_t parameters; } FASTCOVER_tryParameters_data_t; /** * Tries a set of parameters and updates the COVER_best_t with the results. * This function is thread safe if zstd is compiled with multithreaded support. * It takes its parameters as an *OWNING* opaque pointer to support threading. */ static void FASTCOVER_tryParameters(void* opaque) { /* Save parameters as local variables */ FASTCOVER_tryParameters_data_t *const data = (FASTCOVER_tryParameters_data_t*)opaque; const FASTCOVER_ctx_t *const ctx = data->ctx; const ZDICT_cover_params_t parameters = data->parameters; size_t dictBufferCapacity = data->dictBufferCapacity; size_t totalCompressedSize = ERROR(GENERIC); /* Initialize array to keep track of frequency of dmer within activeSegment */ U16* segmentFreqs = (U16*)calloc(((U64)1 << ctx->f), sizeof(U16)); /* Allocate space for hash table, dict, and freqs */ BYTE *const dict = (BYTE*)malloc(dictBufferCapacity); COVER_dictSelection_t selection = COVER_dictSelectionError(ERROR(GENERIC)); U32* freqs = (U32*) malloc(((U64)1 << ctx->f) * sizeof(U32)); if (!segmentFreqs || !dict || !freqs) { DISPLAYLEVEL(1, "Failed to allocate buffers: out of memory\n"); goto _cleanup; } /* Copy the frequencies because we need to modify them */ memcpy(freqs, ctx->freqs, ((U64)1 << ctx->f) * sizeof(U32)); /* Build the dictionary */ { const size_t tail = FASTCOVER_buildDictionary(ctx, freqs, dict, dictBufferCapacity, parameters, segmentFreqs); const unsigned nbFinalizeSamples = (unsigned)(ctx->nbTrainSamples * ctx->accelParams.finalize / 100); selection = COVER_selectDict(dict + tail, dictBufferCapacity, dictBufferCapacity - tail, ctx->samples, ctx->samplesSizes, nbFinalizeSamples, ctx->nbTrainSamples, ctx->nbSamples, parameters, ctx->offsets, totalCompressedSize); if (COVER_dictSelectionIsError(selection)) { DISPLAYLEVEL(1, "Failed to select dictionary\n"); goto _cleanup; } } _cleanup: free(dict); COVER_best_finish(data->best, parameters, selection); free(data); free(segmentFreqs); COVER_dictSelectionFree(selection); free(freqs); } static void FASTCOVER_convertToCoverParams(ZDICT_fastCover_params_t fastCoverParams, ZDICT_cover_params_t* coverParams) { coverParams->k = fastCoverParams.k; coverParams->d = fastCoverParams.d; coverParams->steps = fastCoverParams.steps; coverParams->nbThreads = fastCoverParams.nbThreads; coverParams->splitPoint = fastCoverParams.splitPoint; coverParams->zParams = fastCoverParams.zParams; coverParams->shrinkDict = fastCoverParams.shrinkDict; } static void FASTCOVER_convertToFastCoverParams(ZDICT_cover_params_t coverParams, ZDICT_fastCover_params_t* fastCoverParams, unsigned f, unsigned accel) { fastCoverParams->k = coverParams.k; fastCoverParams->d = coverParams.d; fastCoverParams->steps = coverParams.steps; fastCoverParams->nbThreads = coverParams.nbThreads; fastCoverParams->splitPoint = coverParams.splitPoint; fastCoverParams->f = f; fastCoverParams->accel = accel; fastCoverParams->zParams = coverParams.zParams; fastCoverParams->shrinkDict = coverParams.shrinkDict; } ZDICTLIB_API size_t ZDICT_trainFromBuffer_fastCover(void* dictBuffer, size_t dictBufferCapacity, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples, ZDICT_fastCover_params_t parameters) { BYTE* const dict = (BYTE*)dictBuffer; FASTCOVER_ctx_t ctx; ZDICT_cover_params_t coverParams; FASTCOVER_accel_t accelParams; /* Initialize global data */ g_displayLevel = (int)parameters.zParams.notificationLevel; /* Assign splitPoint and f if not provided */ parameters.splitPoint = 1.0; parameters.f = parameters.f == 0 ? DEFAULT_F : parameters.f; parameters.accel = parameters.accel == 0 ? DEFAULT_ACCEL : parameters.accel; /* Convert to cover parameter */ memset(&coverParams, 0 , sizeof(coverParams)); FASTCOVER_convertToCoverParams(parameters, &coverParams); /* Checks */ if (!FASTCOVER_checkParameters(coverParams, dictBufferCapacity, parameters.f, parameters.accel)) { DISPLAYLEVEL(1, "FASTCOVER parameters incorrect\n"); return ERROR(parameter_outOfBound); } if (nbSamples == 0) { DISPLAYLEVEL(1, "FASTCOVER must have at least one input file\n"); return ERROR(srcSize_wrong); } if (dictBufferCapacity < ZDICT_DICTSIZE_MIN) { DISPLAYLEVEL(1, "dictBufferCapacity must be at least %u\n", ZDICT_DICTSIZE_MIN); return ERROR(dstSize_tooSmall); } /* Assign corresponding FASTCOVER_accel_t to accelParams*/ accelParams = FASTCOVER_defaultAccelParameters[parameters.accel]; /* Initialize context */ { size_t const initVal = FASTCOVER_ctx_init(&ctx, samplesBuffer, samplesSizes, nbSamples, coverParams.d, parameters.splitPoint, parameters.f, accelParams); if (ZSTD_isError(initVal)) { DISPLAYLEVEL(1, "Failed to initialize context\n"); return initVal; } } COVER_warnOnSmallCorpus(dictBufferCapacity, ctx.nbDmers, g_displayLevel); /* Build the dictionary */ DISPLAYLEVEL(2, "Building dictionary\n"); { /* Initialize array to keep track of frequency of dmer within activeSegment */ U16* segmentFreqs = (U16 *)calloc(((U64)1 << parameters.f), sizeof(U16)); const size_t tail = FASTCOVER_buildDictionary(&ctx, ctx.freqs, dictBuffer, dictBufferCapacity, coverParams, segmentFreqs); const unsigned nbFinalizeSamples = (unsigned)(ctx.nbTrainSamples * ctx.accelParams.finalize / 100); const size_t dictionarySize = ZDICT_finalizeDictionary( dict, dictBufferCapacity, dict + tail, dictBufferCapacity - tail, samplesBuffer, samplesSizes, nbFinalizeSamples, coverParams.zParams); if (!ZSTD_isError(dictionarySize)) { DISPLAYLEVEL(2, "Constructed dictionary of size %u\n", (unsigned)dictionarySize); } FASTCOVER_ctx_destroy(&ctx); free(segmentFreqs); return dictionarySize; } } ZDICTLIB_API size_t ZDICT_optimizeTrainFromBuffer_fastCover( void* dictBuffer, size_t dictBufferCapacity, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples, ZDICT_fastCover_params_t* parameters) { ZDICT_cover_params_t coverParams; FASTCOVER_accel_t accelParams; /* constants */ const unsigned nbThreads = parameters->nbThreads; const double splitPoint = parameters->splitPoint <= 0.0 ? FASTCOVER_DEFAULT_SPLITPOINT : parameters->splitPoint; const unsigned kMinD = parameters->d == 0 ? 6 : parameters->d; const unsigned kMaxD = parameters->d == 0 ? 8 : parameters->d; const unsigned kMinK = parameters->k == 0 ? 50 : parameters->k; const unsigned kMaxK = parameters->k == 0 ? 2000 : parameters->k; const unsigned kSteps = parameters->steps == 0 ? 40 : parameters->steps; const unsigned kStepSize = MAX((kMaxK - kMinK) / kSteps, 1); const unsigned kIterations = (1 + (kMaxD - kMinD) / 2) * (1 + (kMaxK - kMinK) / kStepSize); const unsigned f = parameters->f == 0 ? DEFAULT_F : parameters->f; const unsigned accel = parameters->accel == 0 ? DEFAULT_ACCEL : parameters->accel; const unsigned shrinkDict = 0; /* Local variables */ const int displayLevel = (int)parameters->zParams.notificationLevel; unsigned iteration = 1; unsigned d; unsigned k; COVER_best_t best; POOL_ctx *pool = NULL; int warned = 0; /* Checks */ if (splitPoint <= 0 || splitPoint > 1) { LOCALDISPLAYLEVEL(displayLevel, 1, "Incorrect splitPoint\n"); return ERROR(parameter_outOfBound); } if (accel == 0 || accel > FASTCOVER_MAX_ACCEL) { LOCALDISPLAYLEVEL(displayLevel, 1, "Incorrect accel\n"); return ERROR(parameter_outOfBound); } if (kMinK < kMaxD || kMaxK < kMinK) { LOCALDISPLAYLEVEL(displayLevel, 1, "Incorrect k\n"); return ERROR(parameter_outOfBound); } if (nbSamples == 0) { LOCALDISPLAYLEVEL(displayLevel, 1, "FASTCOVER must have at least one input file\n"); return ERROR(srcSize_wrong); } if (dictBufferCapacity < ZDICT_DICTSIZE_MIN) { LOCALDISPLAYLEVEL(displayLevel, 1, "dictBufferCapacity must be at least %u\n", ZDICT_DICTSIZE_MIN); return ERROR(dstSize_tooSmall); } if (nbThreads > 1) { pool = POOL_create(nbThreads, 1); if (!pool) { return ERROR(memory_allocation); } } /* Initialization */ COVER_best_init(&best); memset(&coverParams, 0 , sizeof(coverParams)); FASTCOVER_convertToCoverParams(*parameters, &coverParams); accelParams = FASTCOVER_defaultAccelParameters[accel]; /* Turn down global display level to clean up display at level 2 and below */ g_displayLevel = displayLevel == 0 ? 0 : displayLevel - 1; /* Loop through d first because each new value needs a new context */ LOCALDISPLAYLEVEL(displayLevel, 2, "Trying %u different sets of parameters\n", kIterations); for (d = kMinD; d <= kMaxD; d += 2) { /* Initialize the context for this value of d */ FASTCOVER_ctx_t ctx; LOCALDISPLAYLEVEL(displayLevel, 3, "d=%u\n", d); { size_t const initVal = FASTCOVER_ctx_init(&ctx, samplesBuffer, samplesSizes, nbSamples, d, splitPoint, f, accelParams); if (ZSTD_isError(initVal)) { LOCALDISPLAYLEVEL(displayLevel, 1, "Failed to initialize context\n"); COVER_best_destroy(&best); POOL_free(pool); return initVal; } } if (!warned) { COVER_warnOnSmallCorpus(dictBufferCapacity, ctx.nbDmers, displayLevel); warned = 1; } /* Loop through k reusing the same context */ for (k = kMinK; k <= kMaxK; k += kStepSize) { /* Prepare the arguments */ FASTCOVER_tryParameters_data_t *data = (FASTCOVER_tryParameters_data_t *)malloc( sizeof(FASTCOVER_tryParameters_data_t)); LOCALDISPLAYLEVEL(displayLevel, 3, "k=%u\n", k); if (!data) { LOCALDISPLAYLEVEL(displayLevel, 1, "Failed to allocate parameters\n"); COVER_best_destroy(&best); FASTCOVER_ctx_destroy(&ctx); POOL_free(pool); return ERROR(memory_allocation); } data->ctx = &ctx; data->best = &best; data->dictBufferCapacity = dictBufferCapacity; data->parameters = coverParams; data->parameters.k = k; data->parameters.d = d; data->parameters.splitPoint = splitPoint; data->parameters.steps = kSteps; data->parameters.shrinkDict = shrinkDict; data->parameters.zParams.notificationLevel = (unsigned)g_displayLevel; /* Check the parameters */ if (!FASTCOVER_checkParameters(data->parameters, dictBufferCapacity, data->ctx->f, accel)) { DISPLAYLEVEL(1, "FASTCOVER parameters incorrect\n"); free(data); continue; } /* Call the function and pass ownership of data to it */ COVER_best_start(&best); if (pool) { POOL_add(pool, &FASTCOVER_tryParameters, data); } else { FASTCOVER_tryParameters(data); } /* Print status */ LOCALDISPLAYUPDATE(displayLevel, 2, "\r%u%% ", (unsigned)((iteration * 100) / kIterations)); ++iteration; } COVER_best_wait(&best); FASTCOVER_ctx_destroy(&ctx); } LOCALDISPLAYLEVEL(displayLevel, 2, "\r%79s\r", ""); /* Fill the output buffer and parameters with output of the best parameters */ { const size_t dictSize = best.dictSize; if (ZSTD_isError(best.compressedSize)) { const size_t compressedSize = best.compressedSize; COVER_best_destroy(&best); POOL_free(pool); return compressedSize; } FASTCOVER_convertToFastCoverParams(best.parameters, parameters, f, accel); memcpy(dictBuffer, best.dict, dictSize); COVER_best_destroy(&best); POOL_free(pool); return dictSize; } } zmat-0.9.9/src/blosc2/internal-complibs/zstd/dictBuilder/cover.c0000644000175200007730000012362214515254731025042 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* ***************************************************************************** * Constructs a dictionary using a heuristic based on the following paper: * * Liao, Petri, Moffat, Wirth * Effective Construction of Relative Lempel-Ziv Dictionaries * Published in WWW 2016. * * Adapted from code originally written by @ot (Giuseppe Ottaviano). ******************************************************************************/ /*-************************************* * Dependencies ***************************************/ #include /* fprintf */ #include /* malloc, free, qsort */ #include /* memset */ #include /* clock */ #ifndef ZDICT_STATIC_LINKING_ONLY # define ZDICT_STATIC_LINKING_ONLY #endif #include "../common/mem.h" /* read */ #include "../common/pool.h" #include "../common/threading.h" #include "../common/zstd_internal.h" /* includes zstd.h */ #include "../common/bits.h" /* ZSTD_highbit32 */ #include "../zdict.h" #include "cover.h" /*-************************************* * Constants ***************************************/ /** * There are 32bit indexes used to ref samples, so limit samples size to 4GB * on 64bit builds. * For 32bit builds we choose 1 GB. * Most 32bit platforms have 2GB user-mode addressable space and we allocate a large * contiguous buffer, so 1GB is already a high limit. */ #define COVER_MAX_SAMPLES_SIZE (sizeof(size_t) == 8 ? ((unsigned)-1) : ((unsigned)1 GB)) #define COVER_DEFAULT_SPLITPOINT 1.0 /*-************************************* * Console display ***************************************/ #ifndef LOCALDISPLAYLEVEL static int g_displayLevel = 0; #endif #undef DISPLAY #define DISPLAY(...) \ { \ fprintf(stderr, __VA_ARGS__); \ fflush(stderr); \ } #undef LOCALDISPLAYLEVEL #define LOCALDISPLAYLEVEL(displayLevel, l, ...) \ if (displayLevel >= l) { \ DISPLAY(__VA_ARGS__); \ } /* 0 : no display; 1: errors; 2: default; 3: details; 4: debug */ #undef DISPLAYLEVEL #define DISPLAYLEVEL(l, ...) LOCALDISPLAYLEVEL(g_displayLevel, l, __VA_ARGS__) #ifndef LOCALDISPLAYUPDATE static const clock_t g_refreshRate = CLOCKS_PER_SEC * 15 / 100; static clock_t g_time = 0; #endif #undef LOCALDISPLAYUPDATE #define LOCALDISPLAYUPDATE(displayLevel, l, ...) \ if (displayLevel >= l) { \ if ((clock() - g_time > g_refreshRate) || (displayLevel >= 4)) { \ g_time = clock(); \ DISPLAY(__VA_ARGS__); \ } \ } #undef DISPLAYUPDATE #define DISPLAYUPDATE(l, ...) LOCALDISPLAYUPDATE(g_displayLevel, l, __VA_ARGS__) /*-************************************* * Hash table *************************************** * A small specialized hash map for storing activeDmers. * The map does not resize, so if it becomes full it will loop forever. * Thus, the map must be large enough to store every value. * The map implements linear probing and keeps its load less than 0.5. */ #define MAP_EMPTY_VALUE ((U32)-1) typedef struct COVER_map_pair_t_s { U32 key; U32 value; } COVER_map_pair_t; typedef struct COVER_map_s { COVER_map_pair_t *data; U32 sizeLog; U32 size; U32 sizeMask; } COVER_map_t; /** * Clear the map. */ static void COVER_map_clear(COVER_map_t *map) { memset(map->data, MAP_EMPTY_VALUE, map->size * sizeof(COVER_map_pair_t)); } /** * Initializes a map of the given size. * Returns 1 on success and 0 on failure. * The map must be destroyed with COVER_map_destroy(). * The map is only guaranteed to be large enough to hold size elements. */ static int COVER_map_init(COVER_map_t *map, U32 size) { map->sizeLog = ZSTD_highbit32(size) + 2; map->size = (U32)1 << map->sizeLog; map->sizeMask = map->size - 1; map->data = (COVER_map_pair_t *)malloc(map->size * sizeof(COVER_map_pair_t)); if (!map->data) { map->sizeLog = 0; map->size = 0; return 0; } COVER_map_clear(map); return 1; } /** * Internal hash function */ static const U32 COVER_prime4bytes = 2654435761U; static U32 COVER_map_hash(COVER_map_t *map, U32 key) { return (key * COVER_prime4bytes) >> (32 - map->sizeLog); } /** * Helper function that returns the index that a key should be placed into. */ static U32 COVER_map_index(COVER_map_t *map, U32 key) { const U32 hash = COVER_map_hash(map, key); U32 i; for (i = hash;; i = (i + 1) & map->sizeMask) { COVER_map_pair_t *pos = &map->data[i]; if (pos->value == MAP_EMPTY_VALUE) { return i; } if (pos->key == key) { return i; } } } /** * Returns the pointer to the value for key. * If key is not in the map, it is inserted and the value is set to 0. * The map must not be full. */ static U32 *COVER_map_at(COVER_map_t *map, U32 key) { COVER_map_pair_t *pos = &map->data[COVER_map_index(map, key)]; if (pos->value == MAP_EMPTY_VALUE) { pos->key = key; pos->value = 0; } return &pos->value; } /** * Deletes key from the map if present. */ static void COVER_map_remove(COVER_map_t *map, U32 key) { U32 i = COVER_map_index(map, key); COVER_map_pair_t *del = &map->data[i]; U32 shift = 1; if (del->value == MAP_EMPTY_VALUE) { return; } for (i = (i + 1) & map->sizeMask;; i = (i + 1) & map->sizeMask) { COVER_map_pair_t *const pos = &map->data[i]; /* If the position is empty we are done */ if (pos->value == MAP_EMPTY_VALUE) { del->value = MAP_EMPTY_VALUE; return; } /* If pos can be moved to del do so */ if (((i - COVER_map_hash(map, pos->key)) & map->sizeMask) >= shift) { del->key = pos->key; del->value = pos->value; del = pos; shift = 1; } else { ++shift; } } } /** * Destroys a map that is inited with COVER_map_init(). */ static void COVER_map_destroy(COVER_map_t *map) { if (map->data) { free(map->data); } map->data = NULL; map->size = 0; } /*-************************************* * Context ***************************************/ typedef struct { const BYTE *samples; size_t *offsets; const size_t *samplesSizes; size_t nbSamples; size_t nbTrainSamples; size_t nbTestSamples; U32 *suffix; size_t suffixSize; U32 *freqs; U32 *dmerAt; unsigned d; } COVER_ctx_t; /* We need a global context for qsort... */ static COVER_ctx_t *g_coverCtx = NULL; /*-************************************* * Helper functions ***************************************/ /** * Returns the sum of the sample sizes. */ size_t COVER_sum(const size_t *samplesSizes, unsigned nbSamples) { size_t sum = 0; unsigned i; for (i = 0; i < nbSamples; ++i) { sum += samplesSizes[i]; } return sum; } /** * Returns -1 if the dmer at lp is less than the dmer at rp. * Return 0 if the dmers at lp and rp are equal. * Returns 1 if the dmer at lp is greater than the dmer at rp. */ static int COVER_cmp(COVER_ctx_t *ctx, const void *lp, const void *rp) { U32 const lhs = *(U32 const *)lp; U32 const rhs = *(U32 const *)rp; return memcmp(ctx->samples + lhs, ctx->samples + rhs, ctx->d); } /** * Faster version for d <= 8. */ static int COVER_cmp8(COVER_ctx_t *ctx, const void *lp, const void *rp) { U64 const mask = (ctx->d == 8) ? (U64)-1 : (((U64)1 << (8 * ctx->d)) - 1); U64 const lhs = MEM_readLE64(ctx->samples + *(U32 const *)lp) & mask; U64 const rhs = MEM_readLE64(ctx->samples + *(U32 const *)rp) & mask; if (lhs < rhs) { return -1; } return (lhs > rhs); } /** * Same as COVER_cmp() except ties are broken by pointer value * NOTE: g_coverCtx must be set to call this function. A global is required because * qsort doesn't take an opaque pointer. */ static int WIN_CDECL COVER_strict_cmp(const void *lp, const void *rp) { int result = COVER_cmp(g_coverCtx, lp, rp); if (result == 0) { result = lp < rp ? -1 : 1; } return result; } /** * Faster version for d <= 8. */ static int WIN_CDECL COVER_strict_cmp8(const void *lp, const void *rp) { int result = COVER_cmp8(g_coverCtx, lp, rp); if (result == 0) { result = lp < rp ? -1 : 1; } return result; } /** * Returns the first pointer in [first, last) whose element does not compare * less than value. If no such element exists it returns last. */ static const size_t *COVER_lower_bound(const size_t *first, const size_t *last, size_t value) { size_t count = last - first; while (count != 0) { size_t step = count / 2; const size_t *ptr = first; ptr += step; if (*ptr < value) { first = ++ptr; count -= step + 1; } else { count = step; } } return first; } /** * Generic groupBy function. * Groups an array sorted by cmp into groups with equivalent values. * Calls grp for each group. */ static void COVER_groupBy(const void *data, size_t count, size_t size, COVER_ctx_t *ctx, int (*cmp)(COVER_ctx_t *, const void *, const void *), void (*grp)(COVER_ctx_t *, const void *, const void *)) { const BYTE *ptr = (const BYTE *)data; size_t num = 0; while (num < count) { const BYTE *grpEnd = ptr + size; ++num; while (num < count && cmp(ctx, ptr, grpEnd) == 0) { grpEnd += size; ++num; } grp(ctx, ptr, grpEnd); ptr = grpEnd; } } /*-************************************* * Cover functions ***************************************/ /** * Called on each group of positions with the same dmer. * Counts the frequency of each dmer and saves it in the suffix array. * Fills `ctx->dmerAt`. */ static void COVER_group(COVER_ctx_t *ctx, const void *group, const void *groupEnd) { /* The group consists of all the positions with the same first d bytes. */ const U32 *grpPtr = (const U32 *)group; const U32 *grpEnd = (const U32 *)groupEnd; /* The dmerId is how we will reference this dmer. * This allows us to map the whole dmer space to a much smaller space, the * size of the suffix array. */ const U32 dmerId = (U32)(grpPtr - ctx->suffix); /* Count the number of samples this dmer shows up in */ U32 freq = 0; /* Details */ const size_t *curOffsetPtr = ctx->offsets; const size_t *offsetsEnd = ctx->offsets + ctx->nbSamples; /* Once *grpPtr >= curSampleEnd this occurrence of the dmer is in a * different sample than the last. */ size_t curSampleEnd = ctx->offsets[0]; for (; grpPtr != grpEnd; ++grpPtr) { /* Save the dmerId for this position so we can get back to it. */ ctx->dmerAt[*grpPtr] = dmerId; /* Dictionaries only help for the first reference to the dmer. * After that zstd can reference the match from the previous reference. * So only count each dmer once for each sample it is in. */ if (*grpPtr < curSampleEnd) { continue; } freq += 1; /* Binary search to find the end of the sample *grpPtr is in. * In the common case that grpPtr + 1 == grpEnd we can skip the binary * search because the loop is over. */ if (grpPtr + 1 != grpEnd) { const size_t *sampleEndPtr = COVER_lower_bound(curOffsetPtr, offsetsEnd, *grpPtr); curSampleEnd = *sampleEndPtr; curOffsetPtr = sampleEndPtr + 1; } } /* At this point we are never going to look at this segment of the suffix * array again. We take advantage of this fact to save memory. * We store the frequency of the dmer in the first position of the group, * which is dmerId. */ ctx->suffix[dmerId] = freq; } /** * Selects the best segment in an epoch. * Segments of are scored according to the function: * * Let F(d) be the frequency of dmer d. * Let S_i be the dmer at position i of segment S which has length k. * * Score(S) = F(S_1) + F(S_2) + ... + F(S_{k-d+1}) * * Once the dmer d is in the dictionary we set F(d) = 0. */ static COVER_segment_t COVER_selectSegment(const COVER_ctx_t *ctx, U32 *freqs, COVER_map_t *activeDmers, U32 begin, U32 end, ZDICT_cover_params_t parameters) { /* Constants */ const U32 k = parameters.k; const U32 d = parameters.d; const U32 dmersInK = k - d + 1; /* Try each segment (activeSegment) and save the best (bestSegment) */ COVER_segment_t bestSegment = {0, 0, 0}; COVER_segment_t activeSegment; /* Reset the activeDmers in the segment */ COVER_map_clear(activeDmers); /* The activeSegment starts at the beginning of the epoch. */ activeSegment.begin = begin; activeSegment.end = begin; activeSegment.score = 0; /* Slide the activeSegment through the whole epoch. * Save the best segment in bestSegment. */ while (activeSegment.end < end) { /* The dmerId for the dmer at the next position */ U32 newDmer = ctx->dmerAt[activeSegment.end]; /* The entry in activeDmers for this dmerId */ U32 *newDmerOcc = COVER_map_at(activeDmers, newDmer); /* If the dmer isn't already present in the segment add its score. */ if (*newDmerOcc == 0) { /* The paper suggest using the L-0.5 norm, but experiments show that it * doesn't help. */ activeSegment.score += freqs[newDmer]; } /* Add the dmer to the segment */ activeSegment.end += 1; *newDmerOcc += 1; /* If the window is now too large, drop the first position */ if (activeSegment.end - activeSegment.begin == dmersInK + 1) { U32 delDmer = ctx->dmerAt[activeSegment.begin]; U32 *delDmerOcc = COVER_map_at(activeDmers, delDmer); activeSegment.begin += 1; *delDmerOcc -= 1; /* If this is the last occurrence of the dmer, subtract its score */ if (*delDmerOcc == 0) { COVER_map_remove(activeDmers, delDmer); activeSegment.score -= freqs[delDmer]; } } /* If this segment is the best so far save it */ if (activeSegment.score > bestSegment.score) { bestSegment = activeSegment; } } { /* Trim off the zero frequency head and tail from the segment. */ U32 newBegin = bestSegment.end; U32 newEnd = bestSegment.begin; U32 pos; for (pos = bestSegment.begin; pos != bestSegment.end; ++pos) { U32 freq = freqs[ctx->dmerAt[pos]]; if (freq != 0) { newBegin = MIN(newBegin, pos); newEnd = pos + 1; } } bestSegment.begin = newBegin; bestSegment.end = newEnd; } { /* Zero out the frequency of each dmer covered by the chosen segment. */ U32 pos; for (pos = bestSegment.begin; pos != bestSegment.end; ++pos) { freqs[ctx->dmerAt[pos]] = 0; } } return bestSegment; } /** * Check the validity of the parameters. * Returns non-zero if the parameters are valid and 0 otherwise. */ static int COVER_checkParameters(ZDICT_cover_params_t parameters, size_t maxDictSize) { /* k and d are required parameters */ if (parameters.d == 0 || parameters.k == 0) { return 0; } /* k <= maxDictSize */ if (parameters.k > maxDictSize) { return 0; } /* d <= k */ if (parameters.d > parameters.k) { return 0; } /* 0 < splitPoint <= 1 */ if (parameters.splitPoint <= 0 || parameters.splitPoint > 1){ return 0; } return 1; } /** * Clean up a context initialized with `COVER_ctx_init()`. */ static void COVER_ctx_destroy(COVER_ctx_t *ctx) { if (!ctx) { return; } if (ctx->suffix) { free(ctx->suffix); ctx->suffix = NULL; } if (ctx->freqs) { free(ctx->freqs); ctx->freqs = NULL; } if (ctx->dmerAt) { free(ctx->dmerAt); ctx->dmerAt = NULL; } if (ctx->offsets) { free(ctx->offsets); ctx->offsets = NULL; } } /** * Prepare a context for dictionary building. * The context is only dependent on the parameter `d` and can be used multiple * times. * Returns 0 on success or error code on error. * The context must be destroyed with `COVER_ctx_destroy()`. */ static size_t COVER_ctx_init(COVER_ctx_t *ctx, const void *samplesBuffer, const size_t *samplesSizes, unsigned nbSamples, unsigned d, double splitPoint) { const BYTE *const samples = (const BYTE *)samplesBuffer; const size_t totalSamplesSize = COVER_sum(samplesSizes, nbSamples); /* Split samples into testing and training sets */ const unsigned nbTrainSamples = splitPoint < 1.0 ? (unsigned)((double)nbSamples * splitPoint) : nbSamples; const unsigned nbTestSamples = splitPoint < 1.0 ? nbSamples - nbTrainSamples : nbSamples; const size_t trainingSamplesSize = splitPoint < 1.0 ? COVER_sum(samplesSizes, nbTrainSamples) : totalSamplesSize; const size_t testSamplesSize = splitPoint < 1.0 ? COVER_sum(samplesSizes + nbTrainSamples, nbTestSamples) : totalSamplesSize; /* Checks */ if (totalSamplesSize < MAX(d, sizeof(U64)) || totalSamplesSize >= (size_t)COVER_MAX_SAMPLES_SIZE) { DISPLAYLEVEL(1, "Total samples size is too large (%u MB), maximum size is %u MB\n", (unsigned)(totalSamplesSize>>20), (COVER_MAX_SAMPLES_SIZE >> 20)); return ERROR(srcSize_wrong); } /* Check if there are at least 5 training samples */ if (nbTrainSamples < 5) { DISPLAYLEVEL(1, "Total number of training samples is %u and is invalid.", nbTrainSamples); return ERROR(srcSize_wrong); } /* Check if there's testing sample */ if (nbTestSamples < 1) { DISPLAYLEVEL(1, "Total number of testing samples is %u and is invalid.", nbTestSamples); return ERROR(srcSize_wrong); } /* Zero the context */ memset(ctx, 0, sizeof(*ctx)); DISPLAYLEVEL(2, "Training on %u samples of total size %u\n", nbTrainSamples, (unsigned)trainingSamplesSize); DISPLAYLEVEL(2, "Testing on %u samples of total size %u\n", nbTestSamples, (unsigned)testSamplesSize); ctx->samples = samples; ctx->samplesSizes = samplesSizes; ctx->nbSamples = nbSamples; ctx->nbTrainSamples = nbTrainSamples; ctx->nbTestSamples = nbTestSamples; /* Partial suffix array */ ctx->suffixSize = trainingSamplesSize - MAX(d, sizeof(U64)) + 1; ctx->suffix = (U32 *)malloc(ctx->suffixSize * sizeof(U32)); /* Maps index to the dmerID */ ctx->dmerAt = (U32 *)malloc(ctx->suffixSize * sizeof(U32)); /* The offsets of each file */ ctx->offsets = (size_t *)malloc((nbSamples + 1) * sizeof(size_t)); if (!ctx->suffix || !ctx->dmerAt || !ctx->offsets) { DISPLAYLEVEL(1, "Failed to allocate scratch buffers\n"); COVER_ctx_destroy(ctx); return ERROR(memory_allocation); } ctx->freqs = NULL; ctx->d = d; /* Fill offsets from the samplesSizes */ { U32 i; ctx->offsets[0] = 0; for (i = 1; i <= nbSamples; ++i) { ctx->offsets[i] = ctx->offsets[i - 1] + samplesSizes[i - 1]; } } DISPLAYLEVEL(2, "Constructing partial suffix array\n"); { /* suffix is a partial suffix array. * It only sorts suffixes by their first parameters.d bytes. * The sort is stable, so each dmer group is sorted by position in input. */ U32 i; for (i = 0; i < ctx->suffixSize; ++i) { ctx->suffix[i] = i; } /* qsort doesn't take an opaque pointer, so pass as a global. * On OpenBSD qsort() is not guaranteed to be stable, their mergesort() is. */ g_coverCtx = ctx; #if defined(__OpenBSD__) mergesort(ctx->suffix, ctx->suffixSize, sizeof(U32), (ctx->d <= 8 ? &COVER_strict_cmp8 : &COVER_strict_cmp)); #else qsort(ctx->suffix, ctx->suffixSize, sizeof(U32), (ctx->d <= 8 ? &COVER_strict_cmp8 : &COVER_strict_cmp)); #endif } DISPLAYLEVEL(2, "Computing frequencies\n"); /* For each dmer group (group of positions with the same first d bytes): * 1. For each position we set dmerAt[position] = dmerID. The dmerID is * (groupBeginPtr - suffix). This allows us to go from position to * dmerID so we can look up values in freq. * 2. We calculate how many samples the dmer occurs in and save it in * freqs[dmerId]. */ COVER_groupBy(ctx->suffix, ctx->suffixSize, sizeof(U32), ctx, (ctx->d <= 8 ? &COVER_cmp8 : &COVER_cmp), &COVER_group); ctx->freqs = ctx->suffix; ctx->suffix = NULL; return 0; } void COVER_warnOnSmallCorpus(size_t maxDictSize, size_t nbDmers, int displayLevel) { const double ratio = (double)nbDmers / (double)maxDictSize; if (ratio >= 10) { return; } LOCALDISPLAYLEVEL(displayLevel, 1, "WARNING: The maximum dictionary size %u is too large " "compared to the source size %u! " "size(source)/size(dictionary) = %f, but it should be >= " "10! This may lead to a subpar dictionary! We recommend " "training on sources at least 10x, and preferably 100x " "the size of the dictionary! \n", (U32)maxDictSize, (U32)nbDmers, ratio); } COVER_epoch_info_t COVER_computeEpochs(U32 maxDictSize, U32 nbDmers, U32 k, U32 passes) { const U32 minEpochSize = k * 10; COVER_epoch_info_t epochs; epochs.num = MAX(1, maxDictSize / k / passes); epochs.size = nbDmers / epochs.num; if (epochs.size >= minEpochSize) { assert(epochs.size * epochs.num <= nbDmers); return epochs; } epochs.size = MIN(minEpochSize, nbDmers); epochs.num = nbDmers / epochs.size; assert(epochs.size * epochs.num <= nbDmers); return epochs; } /** * Given the prepared context build the dictionary. */ static size_t COVER_buildDictionary(const COVER_ctx_t *ctx, U32 *freqs, COVER_map_t *activeDmers, void *dictBuffer, size_t dictBufferCapacity, ZDICT_cover_params_t parameters) { BYTE *const dict = (BYTE *)dictBuffer; size_t tail = dictBufferCapacity; /* Divide the data into epochs. We will select one segment from each epoch. */ const COVER_epoch_info_t epochs = COVER_computeEpochs( (U32)dictBufferCapacity, (U32)ctx->suffixSize, parameters.k, 4); const size_t maxZeroScoreRun = MAX(10, MIN(100, epochs.num >> 3)); size_t zeroScoreRun = 0; size_t epoch; DISPLAYLEVEL(2, "Breaking content into %u epochs of size %u\n", (U32)epochs.num, (U32)epochs.size); /* Loop through the epochs until there are no more segments or the dictionary * is full. */ for (epoch = 0; tail > 0; epoch = (epoch + 1) % epochs.num) { const U32 epochBegin = (U32)(epoch * epochs.size); const U32 epochEnd = epochBegin + epochs.size; size_t segmentSize; /* Select a segment */ COVER_segment_t segment = COVER_selectSegment( ctx, freqs, activeDmers, epochBegin, epochEnd, parameters); /* If the segment covers no dmers, then we are out of content. * There may be new content in other epochs, for continue for some time. */ if (segment.score == 0) { if (++zeroScoreRun >= maxZeroScoreRun) { break; } continue; } zeroScoreRun = 0; /* Trim the segment if necessary and if it is too small then we are done */ segmentSize = MIN(segment.end - segment.begin + parameters.d - 1, tail); if (segmentSize < parameters.d) { break; } /* We fill the dictionary from the back to allow the best segments to be * referenced with the smallest offsets. */ tail -= segmentSize; memcpy(dict + tail, ctx->samples + segment.begin, segmentSize); DISPLAYUPDATE( 2, "\r%u%% ", (unsigned)(((dictBufferCapacity - tail) * 100) / dictBufferCapacity)); } DISPLAYLEVEL(2, "\r%79s\r", ""); return tail; } ZDICTLIB_API size_t ZDICT_trainFromBuffer_cover( void *dictBuffer, size_t dictBufferCapacity, const void *samplesBuffer, const size_t *samplesSizes, unsigned nbSamples, ZDICT_cover_params_t parameters) { BYTE* const dict = (BYTE*)dictBuffer; COVER_ctx_t ctx; COVER_map_t activeDmers; parameters.splitPoint = 1.0; /* Initialize global data */ g_displayLevel = (int)parameters.zParams.notificationLevel; /* Checks */ if (!COVER_checkParameters(parameters, dictBufferCapacity)) { DISPLAYLEVEL(1, "Cover parameters incorrect\n"); return ERROR(parameter_outOfBound); } if (nbSamples == 0) { DISPLAYLEVEL(1, "Cover must have at least one input file\n"); return ERROR(srcSize_wrong); } if (dictBufferCapacity < ZDICT_DICTSIZE_MIN) { DISPLAYLEVEL(1, "dictBufferCapacity must be at least %u\n", ZDICT_DICTSIZE_MIN); return ERROR(dstSize_tooSmall); } /* Initialize context and activeDmers */ { size_t const initVal = COVER_ctx_init(&ctx, samplesBuffer, samplesSizes, nbSamples, parameters.d, parameters.splitPoint); if (ZSTD_isError(initVal)) { return initVal; } } COVER_warnOnSmallCorpus(dictBufferCapacity, ctx.suffixSize, g_displayLevel); if (!COVER_map_init(&activeDmers, parameters.k - parameters.d + 1)) { DISPLAYLEVEL(1, "Failed to allocate dmer map: out of memory\n"); COVER_ctx_destroy(&ctx); return ERROR(memory_allocation); } DISPLAYLEVEL(2, "Building dictionary\n"); { const size_t tail = COVER_buildDictionary(&ctx, ctx.freqs, &activeDmers, dictBuffer, dictBufferCapacity, parameters); const size_t dictionarySize = ZDICT_finalizeDictionary( dict, dictBufferCapacity, dict + tail, dictBufferCapacity - tail, samplesBuffer, samplesSizes, nbSamples, parameters.zParams); if (!ZSTD_isError(dictionarySize)) { DISPLAYLEVEL(2, "Constructed dictionary of size %u\n", (unsigned)dictionarySize); } COVER_ctx_destroy(&ctx); COVER_map_destroy(&activeDmers); return dictionarySize; } } size_t COVER_checkTotalCompressedSize(const ZDICT_cover_params_t parameters, const size_t *samplesSizes, const BYTE *samples, size_t *offsets, size_t nbTrainSamples, size_t nbSamples, BYTE *const dict, size_t dictBufferCapacity) { size_t totalCompressedSize = ERROR(GENERIC); /* Pointers */ ZSTD_CCtx *cctx; ZSTD_CDict *cdict; void *dst; /* Local variables */ size_t dstCapacity; size_t i; /* Allocate dst with enough space to compress the maximum sized sample */ { size_t maxSampleSize = 0; i = parameters.splitPoint < 1.0 ? nbTrainSamples : 0; for (; i < nbSamples; ++i) { maxSampleSize = MAX(samplesSizes[i], maxSampleSize); } dstCapacity = ZSTD_compressBound(maxSampleSize); dst = malloc(dstCapacity); } /* Create the cctx and cdict */ cctx = ZSTD_createCCtx(); cdict = ZSTD_createCDict(dict, dictBufferCapacity, parameters.zParams.compressionLevel); if (!dst || !cctx || !cdict) { goto _compressCleanup; } /* Compress each sample and sum their sizes (or error) */ totalCompressedSize = dictBufferCapacity; i = parameters.splitPoint < 1.0 ? nbTrainSamples : 0; for (; i < nbSamples; ++i) { const size_t size = ZSTD_compress_usingCDict( cctx, dst, dstCapacity, samples + offsets[i], samplesSizes[i], cdict); if (ZSTD_isError(size)) { totalCompressedSize = size; goto _compressCleanup; } totalCompressedSize += size; } _compressCleanup: ZSTD_freeCCtx(cctx); ZSTD_freeCDict(cdict); if (dst) { free(dst); } return totalCompressedSize; } /** * Initialize the `COVER_best_t`. */ void COVER_best_init(COVER_best_t *best) { if (best==NULL) return; /* compatible with init on NULL */ (void)ZSTD_pthread_mutex_init(&best->mutex, NULL); (void)ZSTD_pthread_cond_init(&best->cond, NULL); best->liveJobs = 0; best->dict = NULL; best->dictSize = 0; best->compressedSize = (size_t)-1; memset(&best->parameters, 0, sizeof(best->parameters)); } /** * Wait until liveJobs == 0. */ void COVER_best_wait(COVER_best_t *best) { if (!best) { return; } ZSTD_pthread_mutex_lock(&best->mutex); while (best->liveJobs != 0) { ZSTD_pthread_cond_wait(&best->cond, &best->mutex); } ZSTD_pthread_mutex_unlock(&best->mutex); } /** * Call COVER_best_wait() and then destroy the COVER_best_t. */ void COVER_best_destroy(COVER_best_t *best) { if (!best) { return; } COVER_best_wait(best); if (best->dict) { free(best->dict); } ZSTD_pthread_mutex_destroy(&best->mutex); ZSTD_pthread_cond_destroy(&best->cond); } /** * Called when a thread is about to be launched. * Increments liveJobs. */ void COVER_best_start(COVER_best_t *best) { if (!best) { return; } ZSTD_pthread_mutex_lock(&best->mutex); ++best->liveJobs; ZSTD_pthread_mutex_unlock(&best->mutex); } /** * Called when a thread finishes executing, both on error or success. * Decrements liveJobs and signals any waiting threads if liveJobs == 0. * If this dictionary is the best so far save it and its parameters. */ void COVER_best_finish(COVER_best_t *best, ZDICT_cover_params_t parameters, COVER_dictSelection_t selection) { void* dict = selection.dictContent; size_t compressedSize = selection.totalCompressedSize; size_t dictSize = selection.dictSize; if (!best) { return; } { size_t liveJobs; ZSTD_pthread_mutex_lock(&best->mutex); --best->liveJobs; liveJobs = best->liveJobs; /* If the new dictionary is better */ if (compressedSize < best->compressedSize) { /* Allocate space if necessary */ if (!best->dict || best->dictSize < dictSize) { if (best->dict) { free(best->dict); } best->dict = malloc(dictSize); if (!best->dict) { best->compressedSize = ERROR(GENERIC); best->dictSize = 0; ZSTD_pthread_cond_signal(&best->cond); ZSTD_pthread_mutex_unlock(&best->mutex); return; } } /* Save the dictionary, parameters, and size */ if (dict) { memcpy(best->dict, dict, dictSize); best->dictSize = dictSize; best->parameters = parameters; best->compressedSize = compressedSize; } } if (liveJobs == 0) { ZSTD_pthread_cond_broadcast(&best->cond); } ZSTD_pthread_mutex_unlock(&best->mutex); } } static COVER_dictSelection_t setDictSelection(BYTE* buf, size_t s, size_t csz) { COVER_dictSelection_t ds; ds.dictContent = buf; ds.dictSize = s; ds.totalCompressedSize = csz; return ds; } COVER_dictSelection_t COVER_dictSelectionError(size_t error) { return setDictSelection(NULL, 0, error); } unsigned COVER_dictSelectionIsError(COVER_dictSelection_t selection) { return (ZSTD_isError(selection.totalCompressedSize) || !selection.dictContent); } void COVER_dictSelectionFree(COVER_dictSelection_t selection){ free(selection.dictContent); } COVER_dictSelection_t COVER_selectDict(BYTE* customDictContent, size_t dictBufferCapacity, size_t dictContentSize, const BYTE* samplesBuffer, const size_t* samplesSizes, unsigned nbFinalizeSamples, size_t nbCheckSamples, size_t nbSamples, ZDICT_cover_params_t params, size_t* offsets, size_t totalCompressedSize) { size_t largestDict = 0; size_t largestCompressed = 0; BYTE* customDictContentEnd = customDictContent + dictContentSize; BYTE * largestDictbuffer = (BYTE *)malloc(dictBufferCapacity); BYTE * candidateDictBuffer = (BYTE *)malloc(dictBufferCapacity); double regressionTolerance = ((double)params.shrinkDictMaxRegression / 100.0) + 1.00; if (!largestDictbuffer || !candidateDictBuffer) { free(largestDictbuffer); free(candidateDictBuffer); return COVER_dictSelectionError(dictContentSize); } /* Initial dictionary size and compressed size */ memcpy(largestDictbuffer, customDictContent, dictContentSize); dictContentSize = ZDICT_finalizeDictionary( largestDictbuffer, dictBufferCapacity, customDictContent, dictContentSize, samplesBuffer, samplesSizes, nbFinalizeSamples, params.zParams); if (ZDICT_isError(dictContentSize)) { free(largestDictbuffer); free(candidateDictBuffer); return COVER_dictSelectionError(dictContentSize); } totalCompressedSize = COVER_checkTotalCompressedSize(params, samplesSizes, samplesBuffer, offsets, nbCheckSamples, nbSamples, largestDictbuffer, dictContentSize); if (ZSTD_isError(totalCompressedSize)) { free(largestDictbuffer); free(candidateDictBuffer); return COVER_dictSelectionError(totalCompressedSize); } if (params.shrinkDict == 0) { free(candidateDictBuffer); return setDictSelection(largestDictbuffer, dictContentSize, totalCompressedSize); } largestDict = dictContentSize; largestCompressed = totalCompressedSize; dictContentSize = ZDICT_DICTSIZE_MIN; /* Largest dict is initially at least ZDICT_DICTSIZE_MIN */ while (dictContentSize < largestDict) { memcpy(candidateDictBuffer, largestDictbuffer, largestDict); dictContentSize = ZDICT_finalizeDictionary( candidateDictBuffer, dictBufferCapacity, customDictContentEnd - dictContentSize, dictContentSize, samplesBuffer, samplesSizes, nbFinalizeSamples, params.zParams); if (ZDICT_isError(dictContentSize)) { free(largestDictbuffer); free(candidateDictBuffer); return COVER_dictSelectionError(dictContentSize); } totalCompressedSize = COVER_checkTotalCompressedSize(params, samplesSizes, samplesBuffer, offsets, nbCheckSamples, nbSamples, candidateDictBuffer, dictContentSize); if (ZSTD_isError(totalCompressedSize)) { free(largestDictbuffer); free(candidateDictBuffer); return COVER_dictSelectionError(totalCompressedSize); } if ((double)totalCompressedSize <= (double)largestCompressed * regressionTolerance) { free(largestDictbuffer); return setDictSelection( candidateDictBuffer, dictContentSize, totalCompressedSize ); } dictContentSize *= 2; } dictContentSize = largestDict; totalCompressedSize = largestCompressed; free(candidateDictBuffer); return setDictSelection( largestDictbuffer, dictContentSize, totalCompressedSize ); } /** * Parameters for COVER_tryParameters(). */ typedef struct COVER_tryParameters_data_s { const COVER_ctx_t *ctx; COVER_best_t *best; size_t dictBufferCapacity; ZDICT_cover_params_t parameters; } COVER_tryParameters_data_t; /** * Tries a set of parameters and updates the COVER_best_t with the results. * This function is thread safe if zstd is compiled with multithreaded support. * It takes its parameters as an *OWNING* opaque pointer to support threading. */ static void COVER_tryParameters(void *opaque) { /* Save parameters as local variables */ COVER_tryParameters_data_t *const data = (COVER_tryParameters_data_t*)opaque; const COVER_ctx_t *const ctx = data->ctx; const ZDICT_cover_params_t parameters = data->parameters; size_t dictBufferCapacity = data->dictBufferCapacity; size_t totalCompressedSize = ERROR(GENERIC); /* Allocate space for hash table, dict, and freqs */ COVER_map_t activeDmers; BYTE* const dict = (BYTE*)malloc(dictBufferCapacity); COVER_dictSelection_t selection = COVER_dictSelectionError(ERROR(GENERIC)); U32* const freqs = (U32*)malloc(ctx->suffixSize * sizeof(U32)); if (!COVER_map_init(&activeDmers, parameters.k - parameters.d + 1)) { DISPLAYLEVEL(1, "Failed to allocate dmer map: out of memory\n"); goto _cleanup; } if (!dict || !freqs) { DISPLAYLEVEL(1, "Failed to allocate buffers: out of memory\n"); goto _cleanup; } /* Copy the frequencies because we need to modify them */ memcpy(freqs, ctx->freqs, ctx->suffixSize * sizeof(U32)); /* Build the dictionary */ { const size_t tail = COVER_buildDictionary(ctx, freqs, &activeDmers, dict, dictBufferCapacity, parameters); selection = COVER_selectDict(dict + tail, dictBufferCapacity, dictBufferCapacity - tail, ctx->samples, ctx->samplesSizes, (unsigned)ctx->nbTrainSamples, ctx->nbTrainSamples, ctx->nbSamples, parameters, ctx->offsets, totalCompressedSize); if (COVER_dictSelectionIsError(selection)) { DISPLAYLEVEL(1, "Failed to select dictionary\n"); goto _cleanup; } } _cleanup: free(dict); COVER_best_finish(data->best, parameters, selection); free(data); COVER_map_destroy(&activeDmers); COVER_dictSelectionFree(selection); free(freqs); } ZDICTLIB_API size_t ZDICT_optimizeTrainFromBuffer_cover( void* dictBuffer, size_t dictBufferCapacity, const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples, ZDICT_cover_params_t* parameters) { /* constants */ const unsigned nbThreads = parameters->nbThreads; const double splitPoint = parameters->splitPoint <= 0.0 ? COVER_DEFAULT_SPLITPOINT : parameters->splitPoint; const unsigned kMinD = parameters->d == 0 ? 6 : parameters->d; const unsigned kMaxD = parameters->d == 0 ? 8 : parameters->d; const unsigned kMinK = parameters->k == 0 ? 50 : parameters->k; const unsigned kMaxK = parameters->k == 0 ? 2000 : parameters->k; const unsigned kSteps = parameters->steps == 0 ? 40 : parameters->steps; const unsigned kStepSize = MAX((kMaxK - kMinK) / kSteps, 1); const unsigned kIterations = (1 + (kMaxD - kMinD) / 2) * (1 + (kMaxK - kMinK) / kStepSize); const unsigned shrinkDict = 0; /* Local variables */ const int displayLevel = parameters->zParams.notificationLevel; unsigned iteration = 1; unsigned d; unsigned k; COVER_best_t best; POOL_ctx *pool = NULL; int warned = 0; /* Checks */ if (splitPoint <= 0 || splitPoint > 1) { LOCALDISPLAYLEVEL(displayLevel, 1, "Incorrect parameters\n"); return ERROR(parameter_outOfBound); } if (kMinK < kMaxD || kMaxK < kMinK) { LOCALDISPLAYLEVEL(displayLevel, 1, "Incorrect parameters\n"); return ERROR(parameter_outOfBound); } if (nbSamples == 0) { DISPLAYLEVEL(1, "Cover must have at least one input file\n"); return ERROR(srcSize_wrong); } if (dictBufferCapacity < ZDICT_DICTSIZE_MIN) { DISPLAYLEVEL(1, "dictBufferCapacity must be at least %u\n", ZDICT_DICTSIZE_MIN); return ERROR(dstSize_tooSmall); } if (nbThreads > 1) { pool = POOL_create(nbThreads, 1); if (!pool) { return ERROR(memory_allocation); } } /* Initialization */ COVER_best_init(&best); /* Turn down global display level to clean up display at level 2 and below */ g_displayLevel = displayLevel == 0 ? 0 : displayLevel - 1; /* Loop through d first because each new value needs a new context */ LOCALDISPLAYLEVEL(displayLevel, 2, "Trying %u different sets of parameters\n", kIterations); for (d = kMinD; d <= kMaxD; d += 2) { /* Initialize the context for this value of d */ COVER_ctx_t ctx; LOCALDISPLAYLEVEL(displayLevel, 3, "d=%u\n", d); { const size_t initVal = COVER_ctx_init(&ctx, samplesBuffer, samplesSizes, nbSamples, d, splitPoint); if (ZSTD_isError(initVal)) { LOCALDISPLAYLEVEL(displayLevel, 1, "Failed to initialize context\n"); COVER_best_destroy(&best); POOL_free(pool); return initVal; } } if (!warned) { COVER_warnOnSmallCorpus(dictBufferCapacity, ctx.suffixSize, displayLevel); warned = 1; } /* Loop through k reusing the same context */ for (k = kMinK; k <= kMaxK; k += kStepSize) { /* Prepare the arguments */ COVER_tryParameters_data_t *data = (COVER_tryParameters_data_t *)malloc( sizeof(COVER_tryParameters_data_t)); LOCALDISPLAYLEVEL(displayLevel, 3, "k=%u\n", k); if (!data) { LOCALDISPLAYLEVEL(displayLevel, 1, "Failed to allocate parameters\n"); COVER_best_destroy(&best); COVER_ctx_destroy(&ctx); POOL_free(pool); return ERROR(memory_allocation); } data->ctx = &ctx; data->best = &best; data->dictBufferCapacity = dictBufferCapacity; data->parameters = *parameters; data->parameters.k = k; data->parameters.d = d; data->parameters.splitPoint = splitPoint; data->parameters.steps = kSteps; data->parameters.shrinkDict = shrinkDict; data->parameters.zParams.notificationLevel = g_displayLevel; /* Check the parameters */ if (!COVER_checkParameters(data->parameters, dictBufferCapacity)) { DISPLAYLEVEL(1, "Cover parameters incorrect\n"); free(data); continue; } /* Call the function and pass ownership of data to it */ COVER_best_start(&best); if (pool) { POOL_add(pool, &COVER_tryParameters, data); } else { COVER_tryParameters(data); } /* Print status */ LOCALDISPLAYUPDATE(displayLevel, 2, "\r%u%% ", (unsigned)((iteration * 100) / kIterations)); ++iteration; } COVER_best_wait(&best); COVER_ctx_destroy(&ctx); } LOCALDISPLAYLEVEL(displayLevel, 2, "\r%79s\r", ""); /* Fill the output buffer and parameters with output of the best parameters */ { const size_t dictSize = best.dictSize; if (ZSTD_isError(best.compressedSize)) { const size_t compressedSize = best.compressedSize; COVER_best_destroy(&best); POOL_free(pool); return compressedSize; } *parameters = best.parameters; memcpy(dictBuffer, best.dict, dictSize); COVER_best_destroy(&best); POOL_free(pool); return dictSize; } } zmat-0.9.9/src/blosc2/internal-complibs/zstd/dictBuilder/divsufsort.c0000644000175200007730000015257114515254731026141 0ustar rlaboissrlaboiss/* * divsufsort.c for libdivsufsort-lite * Copyright (c) 2003-2008 Yuta Mori All Rights Reserved. * * Permission is hereby granted, free of charge, to any person * obtaining a copy of this software and associated documentation * files (the "Software"), to deal in the Software without * restriction, including without limitation the rights to use, * copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the * Software is furnished to do so, subject to the following * conditions: * * The above copyright notice and this permission notice shall be * included in all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR * OTHER DEALINGS IN THE SOFTWARE. */ /*- Compiler specifics -*/ #ifdef __clang__ #pragma clang diagnostic ignored "-Wshorten-64-to-32" #endif #if defined(_MSC_VER) # pragma warning(disable : 4244) # pragma warning(disable : 4127) /* C4127 : Condition expression is constant */ #endif /*- Dependencies -*/ #include #include #include #include "divsufsort.h" /*- Constants -*/ #if defined(INLINE) # undef INLINE #endif #if !defined(INLINE) # define INLINE __inline #endif #if defined(ALPHABET_SIZE) && (ALPHABET_SIZE < 1) # undef ALPHABET_SIZE #endif #if !defined(ALPHABET_SIZE) # define ALPHABET_SIZE (256) #endif #define BUCKET_A_SIZE (ALPHABET_SIZE) #define BUCKET_B_SIZE (ALPHABET_SIZE * ALPHABET_SIZE) #if defined(SS_INSERTIONSORT_THRESHOLD) # if SS_INSERTIONSORT_THRESHOLD < 1 # undef SS_INSERTIONSORT_THRESHOLD # define SS_INSERTIONSORT_THRESHOLD (1) # endif #else # define SS_INSERTIONSORT_THRESHOLD (8) #endif #if defined(SS_BLOCKSIZE) # if SS_BLOCKSIZE < 0 # undef SS_BLOCKSIZE # define SS_BLOCKSIZE (0) # elif 32768 <= SS_BLOCKSIZE # undef SS_BLOCKSIZE # define SS_BLOCKSIZE (32767) # endif #else # define SS_BLOCKSIZE (1024) #endif /* minstacksize = log(SS_BLOCKSIZE) / log(3) * 2 */ #if SS_BLOCKSIZE == 0 # define SS_MISORT_STACKSIZE (96) #elif SS_BLOCKSIZE <= 4096 # define SS_MISORT_STACKSIZE (16) #else # define SS_MISORT_STACKSIZE (24) #endif #define SS_SMERGE_STACKSIZE (32) #define TR_INSERTIONSORT_THRESHOLD (8) #define TR_STACKSIZE (64) /*- Macros -*/ #ifndef SWAP # define SWAP(_a, _b) do { t = (_a); (_a) = (_b); (_b) = t; } while(0) #endif /* SWAP */ #ifndef MIN # define MIN(_a, _b) (((_a) < (_b)) ? (_a) : (_b)) #endif /* MIN */ #ifndef MAX # define MAX(_a, _b) (((_a) > (_b)) ? (_a) : (_b)) #endif /* MAX */ #define STACK_PUSH(_a, _b, _c, _d)\ do {\ assert(ssize < STACK_SIZE);\ stack[ssize].a = (_a), stack[ssize].b = (_b),\ stack[ssize].c = (_c), stack[ssize++].d = (_d);\ } while(0) #define STACK_PUSH5(_a, _b, _c, _d, _e)\ do {\ assert(ssize < STACK_SIZE);\ stack[ssize].a = (_a), stack[ssize].b = (_b),\ stack[ssize].c = (_c), stack[ssize].d = (_d), stack[ssize++].e = (_e);\ } while(0) #define STACK_POP(_a, _b, _c, _d)\ do {\ assert(0 <= ssize);\ if(ssize == 0) { return; }\ (_a) = stack[--ssize].a, (_b) = stack[ssize].b,\ (_c) = stack[ssize].c, (_d) = stack[ssize].d;\ } while(0) #define STACK_POP5(_a, _b, _c, _d, _e)\ do {\ assert(0 <= ssize);\ if(ssize == 0) { return; }\ (_a) = stack[--ssize].a, (_b) = stack[ssize].b,\ (_c) = stack[ssize].c, (_d) = stack[ssize].d, (_e) = stack[ssize].e;\ } while(0) #define BUCKET_A(_c0) bucket_A[(_c0)] #if ALPHABET_SIZE == 256 #define BUCKET_B(_c0, _c1) (bucket_B[((_c1) << 8) | (_c0)]) #define BUCKET_BSTAR(_c0, _c1) (bucket_B[((_c0) << 8) | (_c1)]) #else #define BUCKET_B(_c0, _c1) (bucket_B[(_c1) * ALPHABET_SIZE + (_c0)]) #define BUCKET_BSTAR(_c0, _c1) (bucket_B[(_c0) * ALPHABET_SIZE + (_c1)]) #endif /*- Private Functions -*/ static const int lg_table[256]= { -1,0,1,1,2,2,2,2,3,3,3,3,3,3,3,3,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4, 5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5, 6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6, 6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7 }; #if (SS_BLOCKSIZE == 0) || (SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE) static INLINE int ss_ilg(int n) { #if SS_BLOCKSIZE == 0 return (n & 0xffff0000) ? ((n & 0xff000000) ? 24 + lg_table[(n >> 24) & 0xff] : 16 + lg_table[(n >> 16) & 0xff]) : ((n & 0x0000ff00) ? 8 + lg_table[(n >> 8) & 0xff] : 0 + lg_table[(n >> 0) & 0xff]); #elif SS_BLOCKSIZE < 256 return lg_table[n]; #else return (n & 0xff00) ? 8 + lg_table[(n >> 8) & 0xff] : 0 + lg_table[(n >> 0) & 0xff]; #endif } #endif /* (SS_BLOCKSIZE == 0) || (SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE) */ #if SS_BLOCKSIZE != 0 static const int sqq_table[256] = { 0, 16, 22, 27, 32, 35, 39, 42, 45, 48, 50, 53, 55, 57, 59, 61, 64, 65, 67, 69, 71, 73, 75, 76, 78, 80, 81, 83, 84, 86, 87, 89, 90, 91, 93, 94, 96, 97, 98, 99, 101, 102, 103, 104, 106, 107, 108, 109, 110, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 128, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 144, 145, 146, 147, 148, 149, 150, 150, 151, 152, 153, 154, 155, 155, 156, 157, 158, 159, 160, 160, 161, 162, 163, 163, 164, 165, 166, 167, 167, 168, 169, 170, 170, 171, 172, 173, 173, 174, 175, 176, 176, 177, 178, 178, 179, 180, 181, 181, 182, 183, 183, 184, 185, 185, 186, 187, 187, 188, 189, 189, 190, 191, 192, 192, 193, 193, 194, 195, 195, 196, 197, 197, 198, 199, 199, 200, 201, 201, 202, 203, 203, 204, 204, 205, 206, 206, 207, 208, 208, 209, 209, 210, 211, 211, 212, 212, 213, 214, 214, 215, 215, 216, 217, 217, 218, 218, 219, 219, 220, 221, 221, 222, 222, 223, 224, 224, 225, 225, 226, 226, 227, 227, 228, 229, 229, 230, 230, 231, 231, 232, 232, 233, 234, 234, 235, 235, 236, 236, 237, 237, 238, 238, 239, 240, 240, 241, 241, 242, 242, 243, 243, 244, 244, 245, 245, 246, 246, 247, 247, 248, 248, 249, 249, 250, 250, 251, 251, 252, 252, 253, 253, 254, 254, 255 }; static INLINE int ss_isqrt(int x) { int y, e; if(x >= (SS_BLOCKSIZE * SS_BLOCKSIZE)) { return SS_BLOCKSIZE; } e = (x & 0xffff0000) ? ((x & 0xff000000) ? 24 + lg_table[(x >> 24) & 0xff] : 16 + lg_table[(x >> 16) & 0xff]) : ((x & 0x0000ff00) ? 8 + lg_table[(x >> 8) & 0xff] : 0 + lg_table[(x >> 0) & 0xff]); if(e >= 16) { y = sqq_table[x >> ((e - 6) - (e & 1))] << ((e >> 1) - 7); if(e >= 24) { y = (y + 1 + x / y) >> 1; } y = (y + 1 + x / y) >> 1; } else if(e >= 8) { y = (sqq_table[x >> ((e - 6) - (e & 1))] >> (7 - (e >> 1))) + 1; } else { return sqq_table[x] >> 4; } return (x < (y * y)) ? y - 1 : y; } #endif /* SS_BLOCKSIZE != 0 */ /*---------------------------------------------------------------------------*/ /* Compares two suffixes. */ static INLINE int ss_compare(const unsigned char *T, const int *p1, const int *p2, int depth) { const unsigned char *U1, *U2, *U1n, *U2n; for(U1 = T + depth + *p1, U2 = T + depth + *p2, U1n = T + *(p1 + 1) + 2, U2n = T + *(p2 + 1) + 2; (U1 < U1n) && (U2 < U2n) && (*U1 == *U2); ++U1, ++U2) { } return U1 < U1n ? (U2 < U2n ? *U1 - *U2 : 1) : (U2 < U2n ? -1 : 0); } /*---------------------------------------------------------------------------*/ #if (SS_BLOCKSIZE != 1) && (SS_INSERTIONSORT_THRESHOLD != 1) /* Insertionsort for small size groups */ static void ss_insertionsort(const unsigned char *T, const int *PA, int *first, int *last, int depth) { int *i, *j; int t; int r; for(i = last - 2; first <= i; --i) { for(t = *i, j = i + 1; 0 < (r = ss_compare(T, PA + t, PA + *j, depth));) { do { *(j - 1) = *j; } while((++j < last) && (*j < 0)); if(last <= j) { break; } } if(r == 0) { *j = ~*j; } *(j - 1) = t; } } #endif /* (SS_BLOCKSIZE != 1) && (SS_INSERTIONSORT_THRESHOLD != 1) */ /*---------------------------------------------------------------------------*/ #if (SS_BLOCKSIZE == 0) || (SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE) static INLINE void ss_fixdown(const unsigned char *Td, const int *PA, int *SA, int i, int size) { int j, k; int v; int c, d, e; for(v = SA[i], c = Td[PA[v]]; (j = 2 * i + 1) < size; SA[i] = SA[k], i = k) { d = Td[PA[SA[k = j++]]]; if(d < (e = Td[PA[SA[j]]])) { k = j; d = e; } if(d <= c) { break; } } SA[i] = v; } /* Simple top-down heapsort. */ static void ss_heapsort(const unsigned char *Td, const int *PA, int *SA, int size) { int i, m; int t; m = size; if((size % 2) == 0) { m--; if(Td[PA[SA[m / 2]]] < Td[PA[SA[m]]]) { SWAP(SA[m], SA[m / 2]); } } for(i = m / 2 - 1; 0 <= i; --i) { ss_fixdown(Td, PA, SA, i, m); } if((size % 2) == 0) { SWAP(SA[0], SA[m]); ss_fixdown(Td, PA, SA, 0, m); } for(i = m - 1; 0 < i; --i) { t = SA[0], SA[0] = SA[i]; ss_fixdown(Td, PA, SA, 0, i); SA[i] = t; } } /*---------------------------------------------------------------------------*/ /* Returns the median of three elements. */ static INLINE int * ss_median3(const unsigned char *Td, const int *PA, int *v1, int *v2, int *v3) { int *t; if(Td[PA[*v1]] > Td[PA[*v2]]) { SWAP(v1, v2); } if(Td[PA[*v2]] > Td[PA[*v3]]) { if(Td[PA[*v1]] > Td[PA[*v3]]) { return v1; } else { return v3; } } return v2; } /* Returns the median of five elements. */ static INLINE int * ss_median5(const unsigned char *Td, const int *PA, int *v1, int *v2, int *v3, int *v4, int *v5) { int *t; if(Td[PA[*v2]] > Td[PA[*v3]]) { SWAP(v2, v3); } if(Td[PA[*v4]] > Td[PA[*v5]]) { SWAP(v4, v5); } if(Td[PA[*v2]] > Td[PA[*v4]]) { SWAP(v2, v4); SWAP(v3, v5); } if(Td[PA[*v1]] > Td[PA[*v3]]) { SWAP(v1, v3); } if(Td[PA[*v1]] > Td[PA[*v4]]) { SWAP(v1, v4); SWAP(v3, v5); } if(Td[PA[*v3]] > Td[PA[*v4]]) { return v4; } return v3; } /* Returns the pivot element. */ static INLINE int * ss_pivot(const unsigned char *Td, const int *PA, int *first, int *last) { int *middle; int t; t = last - first; middle = first + t / 2; if(t <= 512) { if(t <= 32) { return ss_median3(Td, PA, first, middle, last - 1); } else { t >>= 2; return ss_median5(Td, PA, first, first + t, middle, last - 1 - t, last - 1); } } t >>= 3; first = ss_median3(Td, PA, first, first + t, first + (t << 1)); middle = ss_median3(Td, PA, middle - t, middle, middle + t); last = ss_median3(Td, PA, last - 1 - (t << 1), last - 1 - t, last - 1); return ss_median3(Td, PA, first, middle, last); } /*---------------------------------------------------------------------------*/ /* Binary partition for substrings. */ static INLINE int * ss_partition(const int *PA, int *first, int *last, int depth) { int *a, *b; int t; for(a = first - 1, b = last;;) { for(; (++a < b) && ((PA[*a] + depth) >= (PA[*a + 1] + 1));) { *a = ~*a; } for(; (a < --b) && ((PA[*b] + depth) < (PA[*b + 1] + 1));) { } if(b <= a) { break; } t = ~*b; *b = *a; *a = t; } if(first < a) { *first = ~*first; } return a; } /* Multikey introsort for medium size groups. */ static void ss_mintrosort(const unsigned char *T, const int *PA, int *first, int *last, int depth) { #define STACK_SIZE SS_MISORT_STACKSIZE struct { int *a, *b, c; int d; } stack[STACK_SIZE]; const unsigned char *Td; int *a, *b, *c, *d, *e, *f; int s, t; int ssize; int limit; int v, x = 0; for(ssize = 0, limit = ss_ilg(last - first);;) { if((last - first) <= SS_INSERTIONSORT_THRESHOLD) { #if 1 < SS_INSERTIONSORT_THRESHOLD if(1 < (last - first)) { ss_insertionsort(T, PA, first, last, depth); } #endif STACK_POP(first, last, depth, limit); continue; } Td = T + depth; if(limit-- == 0) { ss_heapsort(Td, PA, first, last - first); } if(limit < 0) { for(a = first + 1, v = Td[PA[*first]]; a < last; ++a) { if((x = Td[PA[*a]]) != v) { if(1 < (a - first)) { break; } v = x; first = a; } } if(Td[PA[*first] - 1] < v) { first = ss_partition(PA, first, a, depth); } if((a - first) <= (last - a)) { if(1 < (a - first)) { STACK_PUSH(a, last, depth, -1); last = a, depth += 1, limit = ss_ilg(a - first); } else { first = a, limit = -1; } } else { if(1 < (last - a)) { STACK_PUSH(first, a, depth + 1, ss_ilg(a - first)); first = a, limit = -1; } else { last = a, depth += 1, limit = ss_ilg(a - first); } } continue; } /* choose pivot */ a = ss_pivot(Td, PA, first, last); v = Td[PA[*a]]; SWAP(*first, *a); /* partition */ for(b = first; (++b < last) && ((x = Td[PA[*b]]) == v);) { } if(((a = b) < last) && (x < v)) { for(; (++b < last) && ((x = Td[PA[*b]]) <= v);) { if(x == v) { SWAP(*b, *a); ++a; } } } for(c = last; (b < --c) && ((x = Td[PA[*c]]) == v);) { } if((b < (d = c)) && (x > v)) { for(; (b < --c) && ((x = Td[PA[*c]]) >= v);) { if(x == v) { SWAP(*c, *d); --d; } } } for(; b < c;) { SWAP(*b, *c); for(; (++b < c) && ((x = Td[PA[*b]]) <= v);) { if(x == v) { SWAP(*b, *a); ++a; } } for(; (b < --c) && ((x = Td[PA[*c]]) >= v);) { if(x == v) { SWAP(*c, *d); --d; } } } if(a <= d) { c = b - 1; if((s = a - first) > (t = b - a)) { s = t; } for(e = first, f = b - s; 0 < s; --s, ++e, ++f) { SWAP(*e, *f); } if((s = d - c) > (t = last - d - 1)) { s = t; } for(e = b, f = last - s; 0 < s; --s, ++e, ++f) { SWAP(*e, *f); } a = first + (b - a), c = last - (d - c); b = (v <= Td[PA[*a] - 1]) ? a : ss_partition(PA, a, c, depth); if((a - first) <= (last - c)) { if((last - c) <= (c - b)) { STACK_PUSH(b, c, depth + 1, ss_ilg(c - b)); STACK_PUSH(c, last, depth, limit); last = a; } else if((a - first) <= (c - b)) { STACK_PUSH(c, last, depth, limit); STACK_PUSH(b, c, depth + 1, ss_ilg(c - b)); last = a; } else { STACK_PUSH(c, last, depth, limit); STACK_PUSH(first, a, depth, limit); first = b, last = c, depth += 1, limit = ss_ilg(c - b); } } else { if((a - first) <= (c - b)) { STACK_PUSH(b, c, depth + 1, ss_ilg(c - b)); STACK_PUSH(first, a, depth, limit); first = c; } else if((last - c) <= (c - b)) { STACK_PUSH(first, a, depth, limit); STACK_PUSH(b, c, depth + 1, ss_ilg(c - b)); first = c; } else { STACK_PUSH(first, a, depth, limit); STACK_PUSH(c, last, depth, limit); first = b, last = c, depth += 1, limit = ss_ilg(c - b); } } } else { limit += 1; if(Td[PA[*first] - 1] < v) { first = ss_partition(PA, first, last, depth); limit = ss_ilg(last - first); } depth += 1; } } #undef STACK_SIZE } #endif /* (SS_BLOCKSIZE == 0) || (SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE) */ /*---------------------------------------------------------------------------*/ #if SS_BLOCKSIZE != 0 static INLINE void ss_blockswap(int *a, int *b, int n) { int t; for(; 0 < n; --n, ++a, ++b) { t = *a, *a = *b, *b = t; } } static INLINE void ss_rotate(int *first, int *middle, int *last) { int *a, *b, t; int l, r; l = middle - first, r = last - middle; for(; (0 < l) && (0 < r);) { if(l == r) { ss_blockswap(first, middle, l); break; } if(l < r) { a = last - 1, b = middle - 1; t = *a; do { *a-- = *b, *b-- = *a; if(b < first) { *a = t; last = a; if((r -= l + 1) <= l) { break; } a -= 1, b = middle - 1; t = *a; } } while(1); } else { a = first, b = middle; t = *a; do { *a++ = *b, *b++ = *a; if(last <= b) { *a = t; first = a + 1; if((l -= r + 1) <= r) { break; } a += 1, b = middle; t = *a; } } while(1); } } } /*---------------------------------------------------------------------------*/ static void ss_inplacemerge(const unsigned char *T, const int *PA, int *first, int *middle, int *last, int depth) { const int *p; int *a, *b; int len, half; int q, r; int x; for(;;) { if(*(last - 1) < 0) { x = 1; p = PA + ~*(last - 1); } else { x = 0; p = PA + *(last - 1); } for(a = first, len = middle - first, half = len >> 1, r = -1; 0 < len; len = half, half >>= 1) { b = a + half; q = ss_compare(T, PA + ((0 <= *b) ? *b : ~*b), p, depth); if(q < 0) { a = b + 1; half -= (len & 1) ^ 1; } else { r = q; } } if(a < middle) { if(r == 0) { *a = ~*a; } ss_rotate(a, middle, last); last -= middle - a; middle = a; if(first == middle) { break; } } --last; if(x != 0) { while(*--last < 0) { } } if(middle == last) { break; } } } /*---------------------------------------------------------------------------*/ /* Merge-forward with internal buffer. */ static void ss_mergeforward(const unsigned char *T, const int *PA, int *first, int *middle, int *last, int *buf, int depth) { int *a, *b, *c, *bufend; int t; int r; bufend = buf + (middle - first) - 1; ss_blockswap(buf, first, middle - first); for(t = *(a = first), b = buf, c = middle;;) { r = ss_compare(T, PA + *b, PA + *c, depth); if(r < 0) { do { *a++ = *b; if(bufend <= b) { *bufend = t; return; } *b++ = *a; } while(*b < 0); } else if(r > 0) { do { *a++ = *c, *c++ = *a; if(last <= c) { while(b < bufend) { *a++ = *b, *b++ = *a; } *a = *b, *b = t; return; } } while(*c < 0); } else { *c = ~*c; do { *a++ = *b; if(bufend <= b) { *bufend = t; return; } *b++ = *a; } while(*b < 0); do { *a++ = *c, *c++ = *a; if(last <= c) { while(b < bufend) { *a++ = *b, *b++ = *a; } *a = *b, *b = t; return; } } while(*c < 0); } } } /* Merge-backward with internal buffer. */ static void ss_mergebackward(const unsigned char *T, const int *PA, int *first, int *middle, int *last, int *buf, int depth) { const int *p1, *p2; int *a, *b, *c, *bufend; int t; int r; int x; bufend = buf + (last - middle) - 1; ss_blockswap(buf, middle, last - middle); x = 0; if(*bufend < 0) { p1 = PA + ~*bufend; x |= 1; } else { p1 = PA + *bufend; } if(*(middle - 1) < 0) { p2 = PA + ~*(middle - 1); x |= 2; } else { p2 = PA + *(middle - 1); } for(t = *(a = last - 1), b = bufend, c = middle - 1;;) { r = ss_compare(T, p1, p2, depth); if(0 < r) { if(x & 1) { do { *a-- = *b, *b-- = *a; } while(*b < 0); x ^= 1; } *a-- = *b; if(b <= buf) { *buf = t; break; } *b-- = *a; if(*b < 0) { p1 = PA + ~*b; x |= 1; } else { p1 = PA + *b; } } else if(r < 0) { if(x & 2) { do { *a-- = *c, *c-- = *a; } while(*c < 0); x ^= 2; } *a-- = *c, *c-- = *a; if(c < first) { while(buf < b) { *a-- = *b, *b-- = *a; } *a = *b, *b = t; break; } if(*c < 0) { p2 = PA + ~*c; x |= 2; } else { p2 = PA + *c; } } else { if(x & 1) { do { *a-- = *b, *b-- = *a; } while(*b < 0); x ^= 1; } *a-- = ~*b; if(b <= buf) { *buf = t; break; } *b-- = *a; if(x & 2) { do { *a-- = *c, *c-- = *a; } while(*c < 0); x ^= 2; } *a-- = *c, *c-- = *a; if(c < first) { while(buf < b) { *a-- = *b, *b-- = *a; } *a = *b, *b = t; break; } if(*b < 0) { p1 = PA + ~*b; x |= 1; } else { p1 = PA + *b; } if(*c < 0) { p2 = PA + ~*c; x |= 2; } else { p2 = PA + *c; } } } } /* D&C based merge. */ static void ss_swapmerge(const unsigned char *T, const int *PA, int *first, int *middle, int *last, int *buf, int bufsize, int depth) { #define STACK_SIZE SS_SMERGE_STACKSIZE #define GETIDX(a) ((0 <= (a)) ? (a) : (~(a))) #define MERGE_CHECK(a, b, c)\ do {\ if(((c) & 1) ||\ (((c) & 2) && (ss_compare(T, PA + GETIDX(*((a) - 1)), PA + *(a), depth) == 0))) {\ *(a) = ~*(a);\ }\ if(((c) & 4) && ((ss_compare(T, PA + GETIDX(*((b) - 1)), PA + *(b), depth) == 0))) {\ *(b) = ~*(b);\ }\ } while(0) struct { int *a, *b, *c; int d; } stack[STACK_SIZE]; int *l, *r, *lm, *rm; int m, len, half; int ssize; int check, next; for(check = 0, ssize = 0;;) { if((last - middle) <= bufsize) { if((first < middle) && (middle < last)) { ss_mergebackward(T, PA, first, middle, last, buf, depth); } MERGE_CHECK(first, last, check); STACK_POP(first, middle, last, check); continue; } if((middle - first) <= bufsize) { if(first < middle) { ss_mergeforward(T, PA, first, middle, last, buf, depth); } MERGE_CHECK(first, last, check); STACK_POP(first, middle, last, check); continue; } for(m = 0, len = MIN(middle - first, last - middle), half = len >> 1; 0 < len; len = half, half >>= 1) { if(ss_compare(T, PA + GETIDX(*(middle + m + half)), PA + GETIDX(*(middle - m - half - 1)), depth) < 0) { m += half + 1; half -= (len & 1) ^ 1; } } if(0 < m) { lm = middle - m, rm = middle + m; ss_blockswap(lm, middle, m); l = r = middle, next = 0; if(rm < last) { if(*rm < 0) { *rm = ~*rm; if(first < lm) { for(; *--l < 0;) { } next |= 4; } next |= 1; } else if(first < lm) { for(; *r < 0; ++r) { } next |= 2; } } if((l - first) <= (last - r)) { STACK_PUSH(r, rm, last, (next & 3) | (check & 4)); middle = lm, last = l, check = (check & 3) | (next & 4); } else { if((next & 2) && (r == middle)) { next ^= 6; } STACK_PUSH(first, lm, l, (check & 3) | (next & 4)); first = r, middle = rm, check = (next & 3) | (check & 4); } } else { if(ss_compare(T, PA + GETIDX(*(middle - 1)), PA + *middle, depth) == 0) { *middle = ~*middle; } MERGE_CHECK(first, last, check); STACK_POP(first, middle, last, check); } } #undef STACK_SIZE } #endif /* SS_BLOCKSIZE != 0 */ /*---------------------------------------------------------------------------*/ /* Substring sort */ static void sssort(const unsigned char *T, const int *PA, int *first, int *last, int *buf, int bufsize, int depth, int n, int lastsuffix) { int *a; #if SS_BLOCKSIZE != 0 int *b, *middle, *curbuf; int j, k, curbufsize, limit; #endif int i; if(lastsuffix != 0) { ++first; } #if SS_BLOCKSIZE == 0 ss_mintrosort(T, PA, first, last, depth); #else if((bufsize < SS_BLOCKSIZE) && (bufsize < (last - first)) && (bufsize < (limit = ss_isqrt(last - first)))) { if(SS_BLOCKSIZE < limit) { limit = SS_BLOCKSIZE; } buf = middle = last - limit, bufsize = limit; } else { middle = last, limit = 0; } for(a = first, i = 0; SS_BLOCKSIZE < (middle - a); a += SS_BLOCKSIZE, ++i) { #if SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE ss_mintrosort(T, PA, a, a + SS_BLOCKSIZE, depth); #elif 1 < SS_BLOCKSIZE ss_insertionsort(T, PA, a, a + SS_BLOCKSIZE, depth); #endif curbufsize = last - (a + SS_BLOCKSIZE); curbuf = a + SS_BLOCKSIZE; if(curbufsize <= bufsize) { curbufsize = bufsize, curbuf = buf; } for(b = a, k = SS_BLOCKSIZE, j = i; j & 1; b -= k, k <<= 1, j >>= 1) { ss_swapmerge(T, PA, b - k, b, b + k, curbuf, curbufsize, depth); } } #if SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE ss_mintrosort(T, PA, a, middle, depth); #elif 1 < SS_BLOCKSIZE ss_insertionsort(T, PA, a, middle, depth); #endif for(k = SS_BLOCKSIZE; i != 0; k <<= 1, i >>= 1) { if(i & 1) { ss_swapmerge(T, PA, a - k, a, middle, buf, bufsize, depth); a -= k; } } if(limit != 0) { #if SS_INSERTIONSORT_THRESHOLD < SS_BLOCKSIZE ss_mintrosort(T, PA, middle, last, depth); #elif 1 < SS_BLOCKSIZE ss_insertionsort(T, PA, middle, last, depth); #endif ss_inplacemerge(T, PA, first, middle, last, depth); } #endif if(lastsuffix != 0) { /* Insert last type B* suffix. */ int PAi[2]; PAi[0] = PA[*(first - 1)], PAi[1] = n - 2; for(a = first, i = *(first - 1); (a < last) && ((*a < 0) || (0 < ss_compare(T, &(PAi[0]), PA + *a, depth))); ++a) { *(a - 1) = *a; } *(a - 1) = i; } } /*---------------------------------------------------------------------------*/ static INLINE int tr_ilg(int n) { return (n & 0xffff0000) ? ((n & 0xff000000) ? 24 + lg_table[(n >> 24) & 0xff] : 16 + lg_table[(n >> 16) & 0xff]) : ((n & 0x0000ff00) ? 8 + lg_table[(n >> 8) & 0xff] : 0 + lg_table[(n >> 0) & 0xff]); } /*---------------------------------------------------------------------------*/ /* Simple insertionsort for small size groups. */ static void tr_insertionsort(const int *ISAd, int *first, int *last) { int *a, *b; int t, r; for(a = first + 1; a < last; ++a) { for(t = *a, b = a - 1; 0 > (r = ISAd[t] - ISAd[*b]);) { do { *(b + 1) = *b; } while((first <= --b) && (*b < 0)); if(b < first) { break; } } if(r == 0) { *b = ~*b; } *(b + 1) = t; } } /*---------------------------------------------------------------------------*/ static INLINE void tr_fixdown(const int *ISAd, int *SA, int i, int size) { int j, k; int v; int c, d, e; for(v = SA[i], c = ISAd[v]; (j = 2 * i + 1) < size; SA[i] = SA[k], i = k) { d = ISAd[SA[k = j++]]; if(d < (e = ISAd[SA[j]])) { k = j; d = e; } if(d <= c) { break; } } SA[i] = v; } /* Simple top-down heapsort. */ static void tr_heapsort(const int *ISAd, int *SA, int size) { int i, m; int t; m = size; if((size % 2) == 0) { m--; if(ISAd[SA[m / 2]] < ISAd[SA[m]]) { SWAP(SA[m], SA[m / 2]); } } for(i = m / 2 - 1; 0 <= i; --i) { tr_fixdown(ISAd, SA, i, m); } if((size % 2) == 0) { SWAP(SA[0], SA[m]); tr_fixdown(ISAd, SA, 0, m); } for(i = m - 1; 0 < i; --i) { t = SA[0], SA[0] = SA[i]; tr_fixdown(ISAd, SA, 0, i); SA[i] = t; } } /*---------------------------------------------------------------------------*/ /* Returns the median of three elements. */ static INLINE int * tr_median3(const int *ISAd, int *v1, int *v2, int *v3) { int *t; if(ISAd[*v1] > ISAd[*v2]) { SWAP(v1, v2); } if(ISAd[*v2] > ISAd[*v3]) { if(ISAd[*v1] > ISAd[*v3]) { return v1; } else { return v3; } } return v2; } /* Returns the median of five elements. */ static INLINE int * tr_median5(const int *ISAd, int *v1, int *v2, int *v3, int *v4, int *v5) { int *t; if(ISAd[*v2] > ISAd[*v3]) { SWAP(v2, v3); } if(ISAd[*v4] > ISAd[*v5]) { SWAP(v4, v5); } if(ISAd[*v2] > ISAd[*v4]) { SWAP(v2, v4); SWAP(v3, v5); } if(ISAd[*v1] > ISAd[*v3]) { SWAP(v1, v3); } if(ISAd[*v1] > ISAd[*v4]) { SWAP(v1, v4); SWAP(v3, v5); } if(ISAd[*v3] > ISAd[*v4]) { return v4; } return v3; } /* Returns the pivot element. */ static INLINE int * tr_pivot(const int *ISAd, int *first, int *last) { int *middle; int t; t = last - first; middle = first + t / 2; if(t <= 512) { if(t <= 32) { return tr_median3(ISAd, first, middle, last - 1); } else { t >>= 2; return tr_median5(ISAd, first, first + t, middle, last - 1 - t, last - 1); } } t >>= 3; first = tr_median3(ISAd, first, first + t, first + (t << 1)); middle = tr_median3(ISAd, middle - t, middle, middle + t); last = tr_median3(ISAd, last - 1 - (t << 1), last - 1 - t, last - 1); return tr_median3(ISAd, first, middle, last); } /*---------------------------------------------------------------------------*/ typedef struct _trbudget_t trbudget_t; struct _trbudget_t { int chance; int remain; int incval; int count; }; static INLINE void trbudget_init(trbudget_t *budget, int chance, int incval) { budget->chance = chance; budget->remain = budget->incval = incval; } static INLINE int trbudget_check(trbudget_t *budget, int size) { if(size <= budget->remain) { budget->remain -= size; return 1; } if(budget->chance == 0) { budget->count += size; return 0; } budget->remain += budget->incval - size; budget->chance -= 1; return 1; } /*---------------------------------------------------------------------------*/ static INLINE void tr_partition(const int *ISAd, int *first, int *middle, int *last, int **pa, int **pb, int v) { int *a, *b, *c, *d, *e, *f; int t, s; int x = 0; for(b = middle - 1; (++b < last) && ((x = ISAd[*b]) == v);) { } if(((a = b) < last) && (x < v)) { for(; (++b < last) && ((x = ISAd[*b]) <= v);) { if(x == v) { SWAP(*b, *a); ++a; } } } for(c = last; (b < --c) && ((x = ISAd[*c]) == v);) { } if((b < (d = c)) && (x > v)) { for(; (b < --c) && ((x = ISAd[*c]) >= v);) { if(x == v) { SWAP(*c, *d); --d; } } } for(; b < c;) { SWAP(*b, *c); for(; (++b < c) && ((x = ISAd[*b]) <= v);) { if(x == v) { SWAP(*b, *a); ++a; } } for(; (b < --c) && ((x = ISAd[*c]) >= v);) { if(x == v) { SWAP(*c, *d); --d; } } } if(a <= d) { c = b - 1; if((s = a - first) > (t = b - a)) { s = t; } for(e = first, f = b - s; 0 < s; --s, ++e, ++f) { SWAP(*e, *f); } if((s = d - c) > (t = last - d - 1)) { s = t; } for(e = b, f = last - s; 0 < s; --s, ++e, ++f) { SWAP(*e, *f); } first += (b - a), last -= (d - c); } *pa = first, *pb = last; } static void tr_copy(int *ISA, const int *SA, int *first, int *a, int *b, int *last, int depth) { /* sort suffixes of middle partition by using sorted order of suffixes of left and right partition. */ int *c, *d, *e; int s, v; v = b - SA - 1; for(c = first, d = a - 1; c <= d; ++c) { if((0 <= (s = *c - depth)) && (ISA[s] == v)) { *++d = s; ISA[s] = d - SA; } } for(c = last - 1, e = d + 1, d = b; e < d; --c) { if((0 <= (s = *c - depth)) && (ISA[s] == v)) { *--d = s; ISA[s] = d - SA; } } } static void tr_partialcopy(int *ISA, const int *SA, int *first, int *a, int *b, int *last, int depth) { int *c, *d, *e; int s, v; int rank, lastrank, newrank = -1; v = b - SA - 1; lastrank = -1; for(c = first, d = a - 1; c <= d; ++c) { if((0 <= (s = *c - depth)) && (ISA[s] == v)) { *++d = s; rank = ISA[s + depth]; if(lastrank != rank) { lastrank = rank; newrank = d - SA; } ISA[s] = newrank; } } lastrank = -1; for(e = d; first <= e; --e) { rank = ISA[*e]; if(lastrank != rank) { lastrank = rank; newrank = e - SA; } if(newrank != rank) { ISA[*e] = newrank; } } lastrank = -1; for(c = last - 1, e = d + 1, d = b; e < d; --c) { if((0 <= (s = *c - depth)) && (ISA[s] == v)) { *--d = s; rank = ISA[s + depth]; if(lastrank != rank) { lastrank = rank; newrank = d - SA; } ISA[s] = newrank; } } } static void tr_introsort(int *ISA, const int *ISAd, int *SA, int *first, int *last, trbudget_t *budget) { #define STACK_SIZE TR_STACKSIZE struct { const int *a; int *b, *c; int d, e; }stack[STACK_SIZE]; int *a, *b, *c; int t; int v, x = 0; int incr = ISAd - ISA; int limit, next; int ssize, trlink = -1; for(ssize = 0, limit = tr_ilg(last - first);;) { if(limit < 0) { if(limit == -1) { /* tandem repeat partition */ tr_partition(ISAd - incr, first, first, last, &a, &b, last - SA - 1); /* update ranks */ if(a < last) { for(c = first, v = a - SA - 1; c < a; ++c) { ISA[*c] = v; } } if(b < last) { for(c = a, v = b - SA - 1; c < b; ++c) { ISA[*c] = v; } } /* push */ if(1 < (b - a)) { STACK_PUSH5(NULL, a, b, 0, 0); STACK_PUSH5(ISAd - incr, first, last, -2, trlink); trlink = ssize - 2; } if((a - first) <= (last - b)) { if(1 < (a - first)) { STACK_PUSH5(ISAd, b, last, tr_ilg(last - b), trlink); last = a, limit = tr_ilg(a - first); } else if(1 < (last - b)) { first = b, limit = tr_ilg(last - b); } else { STACK_POP5(ISAd, first, last, limit, trlink); } } else { if(1 < (last - b)) { STACK_PUSH5(ISAd, first, a, tr_ilg(a - first), trlink); first = b, limit = tr_ilg(last - b); } else if(1 < (a - first)) { last = a, limit = tr_ilg(a - first); } else { STACK_POP5(ISAd, first, last, limit, trlink); } } } else if(limit == -2) { /* tandem repeat copy */ a = stack[--ssize].b, b = stack[ssize].c; if(stack[ssize].d == 0) { tr_copy(ISA, SA, first, a, b, last, ISAd - ISA); } else { if(0 <= trlink) { stack[trlink].d = -1; } tr_partialcopy(ISA, SA, first, a, b, last, ISAd - ISA); } STACK_POP5(ISAd, first, last, limit, trlink); } else { /* sorted partition */ if(0 <= *first) { a = first; do { ISA[*a] = a - SA; } while((++a < last) && (0 <= *a)); first = a; } if(first < last) { a = first; do { *a = ~*a; } while(*++a < 0); next = (ISA[*a] != ISAd[*a]) ? tr_ilg(a - first + 1) : -1; if(++a < last) { for(b = first, v = a - SA - 1; b < a; ++b) { ISA[*b] = v; } } /* push */ if(trbudget_check(budget, a - first)) { if((a - first) <= (last - a)) { STACK_PUSH5(ISAd, a, last, -3, trlink); ISAd += incr, last = a, limit = next; } else { if(1 < (last - a)) { STACK_PUSH5(ISAd + incr, first, a, next, trlink); first = a, limit = -3; } else { ISAd += incr, last = a, limit = next; } } } else { if(0 <= trlink) { stack[trlink].d = -1; } if(1 < (last - a)) { first = a, limit = -3; } else { STACK_POP5(ISAd, first, last, limit, trlink); } } } else { STACK_POP5(ISAd, first, last, limit, trlink); } } continue; } if((last - first) <= TR_INSERTIONSORT_THRESHOLD) { tr_insertionsort(ISAd, first, last); limit = -3; continue; } if(limit-- == 0) { tr_heapsort(ISAd, first, last - first); for(a = last - 1; first < a; a = b) { for(x = ISAd[*a], b = a - 1; (first <= b) && (ISAd[*b] == x); --b) { *b = ~*b; } } limit = -3; continue; } /* choose pivot */ a = tr_pivot(ISAd, first, last); SWAP(*first, *a); v = ISAd[*first]; /* partition */ tr_partition(ISAd, first, first + 1, last, &a, &b, v); if((last - first) != (b - a)) { next = (ISA[*a] != v) ? tr_ilg(b - a) : -1; /* update ranks */ for(c = first, v = a - SA - 1; c < a; ++c) { ISA[*c] = v; } if(b < last) { for(c = a, v = b - SA - 1; c < b; ++c) { ISA[*c] = v; } } /* push */ if((1 < (b - a)) && (trbudget_check(budget, b - a))) { if((a - first) <= (last - b)) { if((last - b) <= (b - a)) { if(1 < (a - first)) { STACK_PUSH5(ISAd + incr, a, b, next, trlink); STACK_PUSH5(ISAd, b, last, limit, trlink); last = a; } else if(1 < (last - b)) { STACK_PUSH5(ISAd + incr, a, b, next, trlink); first = b; } else { ISAd += incr, first = a, last = b, limit = next; } } else if((a - first) <= (b - a)) { if(1 < (a - first)) { STACK_PUSH5(ISAd, b, last, limit, trlink); STACK_PUSH5(ISAd + incr, a, b, next, trlink); last = a; } else { STACK_PUSH5(ISAd, b, last, limit, trlink); ISAd += incr, first = a, last = b, limit = next; } } else { STACK_PUSH5(ISAd, b, last, limit, trlink); STACK_PUSH5(ISAd, first, a, limit, trlink); ISAd += incr, first = a, last = b, limit = next; } } else { if((a - first) <= (b - a)) { if(1 < (last - b)) { STACK_PUSH5(ISAd + incr, a, b, next, trlink); STACK_PUSH5(ISAd, first, a, limit, trlink); first = b; } else if(1 < (a - first)) { STACK_PUSH5(ISAd + incr, a, b, next, trlink); last = a; } else { ISAd += incr, first = a, last = b, limit = next; } } else if((last - b) <= (b - a)) { if(1 < (last - b)) { STACK_PUSH5(ISAd, first, a, limit, trlink); STACK_PUSH5(ISAd + incr, a, b, next, trlink); first = b; } else { STACK_PUSH5(ISAd, first, a, limit, trlink); ISAd += incr, first = a, last = b, limit = next; } } else { STACK_PUSH5(ISAd, first, a, limit, trlink); STACK_PUSH5(ISAd, b, last, limit, trlink); ISAd += incr, first = a, last = b, limit = next; } } } else { if((1 < (b - a)) && (0 <= trlink)) { stack[trlink].d = -1; } if((a - first) <= (last - b)) { if(1 < (a - first)) { STACK_PUSH5(ISAd, b, last, limit, trlink); last = a; } else if(1 < (last - b)) { first = b; } else { STACK_POP5(ISAd, first, last, limit, trlink); } } else { if(1 < (last - b)) { STACK_PUSH5(ISAd, first, a, limit, trlink); first = b; } else if(1 < (a - first)) { last = a; } else { STACK_POP5(ISAd, first, last, limit, trlink); } } } } else { if(trbudget_check(budget, last - first)) { limit = tr_ilg(last - first), ISAd += incr; } else { if(0 <= trlink) { stack[trlink].d = -1; } STACK_POP5(ISAd, first, last, limit, trlink); } } } #undef STACK_SIZE } /*---------------------------------------------------------------------------*/ /* Tandem repeat sort */ static void trsort(int *ISA, int *SA, int n, int depth) { int *ISAd; int *first, *last; trbudget_t budget; int t, skip, unsorted; trbudget_init(&budget, tr_ilg(n) * 2 / 3, n); /* trbudget_init(&budget, tr_ilg(n) * 3 / 4, n); */ for(ISAd = ISA + depth; -n < *SA; ISAd += ISAd - ISA) { first = SA; skip = 0; unsorted = 0; do { if((t = *first) < 0) { first -= t; skip += t; } else { if(skip != 0) { *(first + skip) = skip; skip = 0; } last = SA + ISA[t] + 1; if(1 < (last - first)) { budget.count = 0; tr_introsort(ISA, ISAd, SA, first, last, &budget); if(budget.count != 0) { unsorted += budget.count; } else { skip = first - last; } } else if((last - first) == 1) { skip = -1; } first = last; } } while(first < (SA + n)); if(skip != 0) { *(first + skip) = skip; } if(unsorted == 0) { break; } } } /*---------------------------------------------------------------------------*/ /* Sorts suffixes of type B*. */ static int sort_typeBstar(const unsigned char *T, int *SA, int *bucket_A, int *bucket_B, int n, int openMP) { int *PAb, *ISAb, *buf; #ifdef LIBBSC_OPENMP int *curbuf; int l; #endif int i, j, k, t, m, bufsize; int c0, c1; #ifdef LIBBSC_OPENMP int d0, d1; #endif (void)openMP; /* Initialize bucket arrays. */ for(i = 0; i < BUCKET_A_SIZE; ++i) { bucket_A[i] = 0; } for(i = 0; i < BUCKET_B_SIZE; ++i) { bucket_B[i] = 0; } /* Count the number of occurrences of the first one or two characters of each type A, B and B* suffix. Moreover, store the beginning position of all type B* suffixes into the array SA. */ for(i = n - 1, m = n, c0 = T[n - 1]; 0 <= i;) { /* type A suffix. */ do { ++BUCKET_A(c1 = c0); } while((0 <= --i) && ((c0 = T[i]) >= c1)); if(0 <= i) { /* type B* suffix. */ ++BUCKET_BSTAR(c0, c1); SA[--m] = i; /* type B suffix. */ for(--i, c1 = c0; (0 <= i) && ((c0 = T[i]) <= c1); --i, c1 = c0) { ++BUCKET_B(c0, c1); } } } m = n - m; /* note: A type B* suffix is lexicographically smaller than a type B suffix that begins with the same first two characters. */ /* Calculate the index of start/end point of each bucket. */ for(c0 = 0, i = 0, j = 0; c0 < ALPHABET_SIZE; ++c0) { t = i + BUCKET_A(c0); BUCKET_A(c0) = i + j; /* start point */ i = t + BUCKET_B(c0, c0); for(c1 = c0 + 1; c1 < ALPHABET_SIZE; ++c1) { j += BUCKET_BSTAR(c0, c1); BUCKET_BSTAR(c0, c1) = j; /* end point */ i += BUCKET_B(c0, c1); } } if(0 < m) { /* Sort the type B* suffixes by their first two characters. */ PAb = SA + n - m; ISAb = SA + m; for(i = m - 2; 0 <= i; --i) { t = PAb[i], c0 = T[t], c1 = T[t + 1]; SA[--BUCKET_BSTAR(c0, c1)] = i; } t = PAb[m - 1], c0 = T[t], c1 = T[t + 1]; SA[--BUCKET_BSTAR(c0, c1)] = m - 1; /* Sort the type B* substrings using sssort. */ #ifdef LIBBSC_OPENMP if (openMP) { buf = SA + m; c0 = ALPHABET_SIZE - 2, c1 = ALPHABET_SIZE - 1, j = m; #pragma omp parallel default(shared) private(bufsize, curbuf, k, l, d0, d1) { bufsize = (n - (2 * m)) / omp_get_num_threads(); curbuf = buf + omp_get_thread_num() * bufsize; k = 0; for(;;) { #pragma omp critical(sssort_lock) { if(0 < (l = j)) { d0 = c0, d1 = c1; do { k = BUCKET_BSTAR(d0, d1); if(--d1 <= d0) { d1 = ALPHABET_SIZE - 1; if(--d0 < 0) { break; } } } while(((l - k) <= 1) && (0 < (l = k))); c0 = d0, c1 = d1, j = k; } } if(l == 0) { break; } sssort(T, PAb, SA + k, SA + l, curbuf, bufsize, 2, n, *(SA + k) == (m - 1)); } } } else { buf = SA + m, bufsize = n - (2 * m); for(c0 = ALPHABET_SIZE - 2, j = m; 0 < j; --c0) { for(c1 = ALPHABET_SIZE - 1; c0 < c1; j = i, --c1) { i = BUCKET_BSTAR(c0, c1); if(1 < (j - i)) { sssort(T, PAb, SA + i, SA + j, buf, bufsize, 2, n, *(SA + i) == (m - 1)); } } } } #else buf = SA + m, bufsize = n - (2 * m); for(c0 = ALPHABET_SIZE - 2, j = m; 0 < j; --c0) { for(c1 = ALPHABET_SIZE - 1; c0 < c1; j = i, --c1) { i = BUCKET_BSTAR(c0, c1); if(1 < (j - i)) { sssort(T, PAb, SA + i, SA + j, buf, bufsize, 2, n, *(SA + i) == (m - 1)); } } } #endif /* Compute ranks of type B* substrings. */ for(i = m - 1; 0 <= i; --i) { if(0 <= SA[i]) { j = i; do { ISAb[SA[i]] = i; } while((0 <= --i) && (0 <= SA[i])); SA[i + 1] = i - j; if(i <= 0) { break; } } j = i; do { ISAb[SA[i] = ~SA[i]] = j; } while(SA[--i] < 0); ISAb[SA[i]] = j; } /* Construct the inverse suffix array of type B* suffixes using trsort. */ trsort(ISAb, SA, m, 1); /* Set the sorted order of type B* suffixes. */ for(i = n - 1, j = m, c0 = T[n - 1]; 0 <= i;) { for(--i, c1 = c0; (0 <= i) && ((c0 = T[i]) >= c1); --i, c1 = c0) { } if(0 <= i) { t = i; for(--i, c1 = c0; (0 <= i) && ((c0 = T[i]) <= c1); --i, c1 = c0) { } SA[ISAb[--j]] = ((t == 0) || (1 < (t - i))) ? t : ~t; } } /* Calculate the index of start/end point of each bucket. */ BUCKET_B(ALPHABET_SIZE - 1, ALPHABET_SIZE - 1) = n; /* end point */ for(c0 = ALPHABET_SIZE - 2, k = m - 1; 0 <= c0; --c0) { i = BUCKET_A(c0 + 1) - 1; for(c1 = ALPHABET_SIZE - 1; c0 < c1; --c1) { t = i - BUCKET_B(c0, c1); BUCKET_B(c0, c1) = i; /* end point */ /* Move all type B* suffixes to the correct position. */ for(i = t, j = BUCKET_BSTAR(c0, c1); j <= k; --i, --k) { SA[i] = SA[k]; } } BUCKET_BSTAR(c0, c0 + 1) = i - BUCKET_B(c0, c0) + 1; /* start point */ BUCKET_B(c0, c0) = i; /* end point */ } } return m; } /* Constructs the suffix array by using the sorted order of type B* suffixes. */ static void construct_SA(const unsigned char *T, int *SA, int *bucket_A, int *bucket_B, int n, int m) { int *i, *j, *k; int s; int c0, c1, c2; if(0 < m) { /* Construct the sorted order of type B suffixes by using the sorted order of type B* suffixes. */ for(c1 = ALPHABET_SIZE - 2; 0 <= c1; --c1) { /* Scan the suffix array from right to left. */ for(i = SA + BUCKET_BSTAR(c1, c1 + 1), j = SA + BUCKET_A(c1 + 1) - 1, k = NULL, c2 = -1; i <= j; --j) { if(0 < (s = *j)) { assert(T[s] == c1); assert(((s + 1) < n) && (T[s] <= T[s + 1])); assert(T[s - 1] <= T[s]); *j = ~s; c0 = T[--s]; if((0 < s) && (T[s - 1] > c0)) { s = ~s; } if(c0 != c2) { if(0 <= c2) { BUCKET_B(c2, c1) = k - SA; } k = SA + BUCKET_B(c2 = c0, c1); } assert(k < j); assert(k != NULL); *k-- = s; } else { assert(((s == 0) && (T[s] == c1)) || (s < 0)); *j = ~s; } } } } /* Construct the suffix array by using the sorted order of type B suffixes. */ k = SA + BUCKET_A(c2 = T[n - 1]); *k++ = (T[n - 2] < c2) ? ~(n - 1) : (n - 1); /* Scan the suffix array from left to right. */ for(i = SA, j = SA + n; i < j; ++i) { if(0 < (s = *i)) { assert(T[s - 1] >= T[s]); c0 = T[--s]; if((s == 0) || (T[s - 1] < c0)) { s = ~s; } if(c0 != c2) { BUCKET_A(c2) = k - SA; k = SA + BUCKET_A(c2 = c0); } assert(i < k); *k++ = s; } else { assert(s < 0); *i = ~s; } } } /* Constructs the burrows-wheeler transformed string directly by using the sorted order of type B* suffixes. */ static int construct_BWT(const unsigned char *T, int *SA, int *bucket_A, int *bucket_B, int n, int m) { int *i, *j, *k, *orig; int s; int c0, c1, c2; if(0 < m) { /* Construct the sorted order of type B suffixes by using the sorted order of type B* suffixes. */ for(c1 = ALPHABET_SIZE - 2; 0 <= c1; --c1) { /* Scan the suffix array from right to left. */ for(i = SA + BUCKET_BSTAR(c1, c1 + 1), j = SA + BUCKET_A(c1 + 1) - 1, k = NULL, c2 = -1; i <= j; --j) { if(0 < (s = *j)) { assert(T[s] == c1); assert(((s + 1) < n) && (T[s] <= T[s + 1])); assert(T[s - 1] <= T[s]); c0 = T[--s]; *j = ~((int)c0); if((0 < s) && (T[s - 1] > c0)) { s = ~s; } if(c0 != c2) { if(0 <= c2) { BUCKET_B(c2, c1) = k - SA; } k = SA + BUCKET_B(c2 = c0, c1); } assert(k < j); assert(k != NULL); *k-- = s; } else if(s != 0) { *j = ~s; #ifndef NDEBUG } else { assert(T[s] == c1); #endif } } } } /* Construct the BWTed string by using the sorted order of type B suffixes. */ k = SA + BUCKET_A(c2 = T[n - 1]); *k++ = (T[n - 2] < c2) ? ~((int)T[n - 2]) : (n - 1); /* Scan the suffix array from left to right. */ for(i = SA, j = SA + n, orig = SA; i < j; ++i) { if(0 < (s = *i)) { assert(T[s - 1] >= T[s]); c0 = T[--s]; *i = c0; if((0 < s) && (T[s - 1] < c0)) { s = ~((int)T[s - 1]); } if(c0 != c2) { BUCKET_A(c2) = k - SA; k = SA + BUCKET_A(c2 = c0); } assert(i < k); *k++ = s; } else if(s != 0) { *i = ~s; } else { orig = i; } } return orig - SA; } /* Constructs the burrows-wheeler transformed string directly by using the sorted order of type B* suffixes. */ static int construct_BWT_indexes(const unsigned char *T, int *SA, int *bucket_A, int *bucket_B, int n, int m, unsigned char * num_indexes, int * indexes) { int *i, *j, *k, *orig; int s; int c0, c1, c2; int mod = n / 8; { mod |= mod >> 1; mod |= mod >> 2; mod |= mod >> 4; mod |= mod >> 8; mod |= mod >> 16; mod >>= 1; *num_indexes = (unsigned char)((n - 1) / (mod + 1)); } if(0 < m) { /* Construct the sorted order of type B suffixes by using the sorted order of type B* suffixes. */ for(c1 = ALPHABET_SIZE - 2; 0 <= c1; --c1) { /* Scan the suffix array from right to left. */ for(i = SA + BUCKET_BSTAR(c1, c1 + 1), j = SA + BUCKET_A(c1 + 1) - 1, k = NULL, c2 = -1; i <= j; --j) { if(0 < (s = *j)) { assert(T[s] == c1); assert(((s + 1) < n) && (T[s] <= T[s + 1])); assert(T[s - 1] <= T[s]); if ((s & mod) == 0) indexes[s / (mod + 1) - 1] = j - SA; c0 = T[--s]; *j = ~((int)c0); if((0 < s) && (T[s - 1] > c0)) { s = ~s; } if(c0 != c2) { if(0 <= c2) { BUCKET_B(c2, c1) = k - SA; } k = SA + BUCKET_B(c2 = c0, c1); } assert(k < j); assert(k != NULL); *k-- = s; } else if(s != 0) { *j = ~s; #ifndef NDEBUG } else { assert(T[s] == c1); #endif } } } } /* Construct the BWTed string by using the sorted order of type B suffixes. */ k = SA + BUCKET_A(c2 = T[n - 1]); if (T[n - 2] < c2) { if (((n - 1) & mod) == 0) indexes[(n - 1) / (mod + 1) - 1] = k - SA; *k++ = ~((int)T[n - 2]); } else { *k++ = n - 1; } /* Scan the suffix array from left to right. */ for(i = SA, j = SA + n, orig = SA; i < j; ++i) { if(0 < (s = *i)) { assert(T[s - 1] >= T[s]); if ((s & mod) == 0) indexes[s / (mod + 1) - 1] = i - SA; c0 = T[--s]; *i = c0; if(c0 != c2) { BUCKET_A(c2) = k - SA; k = SA + BUCKET_A(c2 = c0); } assert(i < k); if((0 < s) && (T[s - 1] < c0)) { if ((s & mod) == 0) indexes[s / (mod + 1) - 1] = k - SA; *k++ = ~((int)T[s - 1]); } else *k++ = s; } else if(s != 0) { *i = ~s; } else { orig = i; } } return orig - SA; } /*---------------------------------------------------------------------------*/ /*- Function -*/ int divsufsort(const unsigned char *T, int *SA, int n, int openMP) { int *bucket_A, *bucket_B; int m; int err = 0; /* Check arguments. */ if((T == NULL) || (SA == NULL) || (n < 0)) { return -1; } else if(n == 0) { return 0; } else if(n == 1) { SA[0] = 0; return 0; } else if(n == 2) { m = (T[0] < T[1]); SA[m ^ 1] = 0, SA[m] = 1; return 0; } bucket_A = (int *)malloc(BUCKET_A_SIZE * sizeof(int)); bucket_B = (int *)malloc(BUCKET_B_SIZE * sizeof(int)); /* Suffixsort. */ if((bucket_A != NULL) && (bucket_B != NULL)) { m = sort_typeBstar(T, SA, bucket_A, bucket_B, n, openMP); construct_SA(T, SA, bucket_A, bucket_B, n, m); } else { err = -2; } free(bucket_B); free(bucket_A); return err; } int divbwt(const unsigned char *T, unsigned char *U, int *A, int n, unsigned char * num_indexes, int * indexes, int openMP) { int *B; int *bucket_A, *bucket_B; int m, pidx, i; /* Check arguments. */ if((T == NULL) || (U == NULL) || (n < 0)) { return -1; } else if(n <= 1) { if(n == 1) { U[0] = T[0]; } return n; } if((B = A) == NULL) { B = (int *)malloc((size_t)(n + 1) * sizeof(int)); } bucket_A = (int *)malloc(BUCKET_A_SIZE * sizeof(int)); bucket_B = (int *)malloc(BUCKET_B_SIZE * sizeof(int)); /* Burrows-Wheeler Transform. */ if((B != NULL) && (bucket_A != NULL) && (bucket_B != NULL)) { m = sort_typeBstar(T, B, bucket_A, bucket_B, n, openMP); if (num_indexes == NULL || indexes == NULL) { pidx = construct_BWT(T, B, bucket_A, bucket_B, n, m); } else { pidx = construct_BWT_indexes(T, B, bucket_A, bucket_B, n, m, num_indexes, indexes); } /* Copy to output string. */ U[0] = T[n - 1]; for(i = 0; i < pidx; ++i) { U[i + 1] = (unsigned char)B[i]; } for(i += 1; i < n; ++i) { U[i] = (unsigned char)B[i]; } pidx += 1; } else { pidx = -2; } free(bucket_B); free(bucket_A); if(A == NULL) { free(B); } return pidx; } zmat-0.9.9/src/blosc2/internal-complibs/zstd/dictBuilder/cover.h0000644000175200007730000001164214515254731025045 0ustar rlaboissrlaboiss/* * Copyright (c) Meta Platforms, Inc. and affiliates. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ #ifndef ZDICT_STATIC_LINKING_ONLY # define ZDICT_STATIC_LINKING_ONLY #endif #include /* fprintf */ #include /* malloc, free, qsort */ #include /* memset */ #include /* clock */ #include "../common/mem.h" /* read */ #include "../common/pool.h" #include "../common/threading.h" #include "../common/zstd_internal.h" /* includes zstd.h */ #include "../zdict.h" /** * COVER_best_t is used for two purposes: * 1. Synchronizing threads. * 2. Saving the best parameters and dictionary. * * All of the methods except COVER_best_init() are thread safe if zstd is * compiled with multithreaded support. */ typedef struct COVER_best_s { ZSTD_pthread_mutex_t mutex; ZSTD_pthread_cond_t cond; size_t liveJobs; void *dict; size_t dictSize; ZDICT_cover_params_t parameters; size_t compressedSize; } COVER_best_t; /** * A segment is a range in the source as well as the score of the segment. */ typedef struct { U32 begin; U32 end; U32 score; } COVER_segment_t; /** *Number of epochs and size of each epoch. */ typedef struct { U32 num; U32 size; } COVER_epoch_info_t; /** * Struct used for the dictionary selection function. */ typedef struct COVER_dictSelection { BYTE* dictContent; size_t dictSize; size_t totalCompressedSize; } COVER_dictSelection_t; /** * Computes the number of epochs and the size of each epoch. * We will make sure that each epoch gets at least 10 * k bytes. * * The COVER algorithms divide the data up into epochs of equal size and * select one segment from each epoch. * * @param maxDictSize The maximum allowed dictionary size. * @param nbDmers The number of dmers we are training on. * @param k The parameter k (segment size). * @param passes The target number of passes over the dmer corpus. * More passes means a better dictionary. */ COVER_epoch_info_t COVER_computeEpochs(U32 maxDictSize, U32 nbDmers, U32 k, U32 passes); /** * Warns the user when their corpus is too small. */ void COVER_warnOnSmallCorpus(size_t maxDictSize, size_t nbDmers, int displayLevel); /** * Checks total compressed size of a dictionary */ size_t COVER_checkTotalCompressedSize(const ZDICT_cover_params_t parameters, const size_t *samplesSizes, const BYTE *samples, size_t *offsets, size_t nbTrainSamples, size_t nbSamples, BYTE *const dict, size_t dictBufferCapacity); /** * Returns the sum of the sample sizes. */ size_t COVER_sum(const size_t *samplesSizes, unsigned nbSamples) ; /** * Initialize the `COVER_best_t`. */ void COVER_best_init(COVER_best_t *best); /** * Wait until liveJobs == 0. */ void COVER_best_wait(COVER_best_t *best); /** * Call COVER_best_wait() and then destroy the COVER_best_t. */ void COVER_best_destroy(COVER_best_t *best); /** * Called when a thread is about to be launched. * Increments liveJobs. */ void COVER_best_start(COVER_best_t *best); /** * Called when a thread finishes executing, both on error or success. * Decrements liveJobs and signals any waiting threads if liveJobs == 0. * If this dictionary is the best so far save it and its parameters. */ void COVER_best_finish(COVER_best_t *best, ZDICT_cover_params_t parameters, COVER_dictSelection_t selection); /** * Error function for COVER_selectDict function. Checks if the return * value is an error. */ unsigned COVER_dictSelectionIsError(COVER_dictSelection_t selection); /** * Error function for COVER_selectDict function. Returns a struct where * return.totalCompressedSize is a ZSTD error. */ COVER_dictSelection_t COVER_dictSelectionError(size_t error); /** * Always call after selectDict is called to free up used memory from * newly created dictionary. */ void COVER_dictSelectionFree(COVER_dictSelection_t selection); /** * Called to finalize the dictionary and select one based on whether or not * the shrink-dict flag was enabled. If enabled the dictionary used is the * smallest dictionary within a specified regression of the compressed size * from the largest dictionary. */ COVER_dictSelection_t COVER_selectDict(BYTE* customDictContent, size_t dictBufferCapacity, size_t dictContentSize, const BYTE* samplesBuffer, const size_t* samplesSizes, unsigned nbFinalizeSamples, size_t nbCheckSamples, size_t nbSamples, ZDICT_cover_params_t params, size_t* offsets, size_t totalCompressedSize); zmat-0.9.9/src/blosc2/include/0000755000175200007730000000000014515254731016335 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/include/blosc2/0000755000175200007730000000000014515254731017521 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/include/blosc2/filters-registry.h0000644000175200007730000000100714515254731023206 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ enum { BLOSC_FILTER_NDCELL = 32, BLOSC_FILTER_NDMEAN = 33, BLOSC_FILTER_BYTEDELTA = 34, }; void register_filters(void); zmat-0.9.9/src/blosc2/include/blosc2/blosc2-export.h0000644000175200007730000000325614515254731022403 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef BLOSC_EXPORT_H #define BLOSC_EXPORT_H /* Macros for specifying exported symbols. BLOSC_EXPORT is used to decorate symbols that should be exported by the blosc shared library. BLOSC_NO_EXPORT is used to decorate symbols that should NOT be exported by the blosc shared library. */ #if defined(BLOSC_SHARED_LIBRARY) #if defined(_MSC_VER) #define BLOSC_EXPORT __declspec(dllexport) #elif (defined(__GNUC__) && __GNUC__ >= 4) || defined(__clang__) #if defined(_WIN32) || defined(__CYGWIN__) || defined(__MINGW32__) #define BLOSC_EXPORT __attribute__((dllexport)) #else #define BLOSC_EXPORT __attribute__((visibility("default"))) #endif /* defined(_WIN32) || defined(__CYGWIN__) */ #else #error Cannot determine how to define BLOSC_EXPORT for this compiler. #endif #else #define BLOSC_EXPORT #endif /* defined(BLOSC_SHARED_LIBRARY) */ #if defined(__GNUC__) || defined(__clang__) #define BLOSC_NO_EXPORT __attribute__((visibility("hidden"))) #else #define BLOSC_NO_EXPORT #endif /* defined(__GNUC__) || defined(__clang__) */ /* When testing, export everything to make it easier to implement tests. */ #if defined(BLOSC_TESTING) #undef BLOSC_NO_EXPORT #define BLOSC_NO_EXPORT BLOSC_EXPORT #endif /* defined(BLOSC_TESTING) */ #endif /* BLOSC_EXPORT_H */ zmat-0.9.9/src/blosc2/include/blosc2/blosc2-common.h0000644000175200007730000000545214515254731022352 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef SHUFFLE_COMMON_H #define SHUFFLE_COMMON_H #include "blosc2-export.h" #include // For shutting up stupid compiler warning about some 'unused' variables in GCC #ifdef __GNUC__ #define BLOSC_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) #define BLOSC_UNUSED_VAR __attribute__ ((unused)) #else #define BLOSC_UNUSED_VAR #endif // __GNUC__ // For shutting up compiler warning about unused parameters #define BLOSC_UNUSED_PARAM(x) ((void)(x)) /* Import standard integer type definitions */ #if defined(_WIN32) && !defined(__MINGW32__) /* stdint.h only available in VS2010 (VC++ 16.0) and newer */ #if defined(_MSC_VER) && _MSC_VER < 1600 #include "win32/stdint-windows.h" #else #include #endif /* Use inlined functions for supported systems */ #if defined(_MSC_VER) && !defined(__cplusplus) /* Visual Studio */ #define inline __inline /* Visual C is not C99, but supports some kind of inline */ #endif #else #include #endif /* _WIN32 */ /* Define the __SSE2__ symbol if compiling with Visual C++ and targeting the minimum architecture level supporting SSE2. Other compilers define this as expected and emit warnings when it is re-defined. */ #if !defined(__SSE2__) && defined(_MSC_VER) && \ (defined(_M_X64) || (defined(_M_IX86) && _M_IX86_FP >= 2)) #define __SSE2__ #endif /* * Detect if the architecture is fine with unaligned access. */ #if !defined(BLOSC_STRICT_ALIGN) #define BLOSC_STRICT_ALIGN #if defined(__i386__) || defined(__386) || defined (__amd64) /* GNU C, Sun Studio */ #undef BLOSC_STRICT_ALIGN #elif defined(__i486__) || defined(__i586__) || defined(__i686__) /* GNU C */ #undef BLOSC_STRICT_ALIGN #elif defined(_M_IX86) || defined(_M_X64) /* Intel, MSVC */ #undef BLOSC_STRICT_ALIGN #elif defined(__386) #undef BLOSC_STRICT_ALIGN #elif defined(_X86_) /* MinGW */ #undef BLOSC_STRICT_ALIGN #elif defined(__I86__) /* Digital Mars */ #undef BLOSC_STRICT_ALIGN /* Modern ARM systems (like ARM64) should support unaligned access quite efficiently. */ #elif defined(__ARM_FEATURE_UNALIGNED) /* ARM, GNU C */ #undef BLOSC_STRICT_ALIGN #elif defined(_ARCH_PPC) || defined(__PPC__) /* Modern PowerPC systems (like POWER8) should support unaligned access quite efficiently. */ #undef BLOSC_STRICT_ALIGN #endif #endif #if defined(__SSE2__) #include #endif #if defined(__AVX2__) #include #endif #endif /* SHUFFLE_COMMON_H */ zmat-0.9.9/src/blosc2/include/blosc2/tuners-registry.h0000644000175200007730000000143414515254731023062 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef BLOSC_BLOSC2_TUNERS_UTILS_H #define BLOSC_BLOSC2_TUNERS_UTILS_H #ifdef __cplusplus extern "C" { #endif enum { BLOSC_BTUNE = 32, }; void register_tuners(void); // For dynamically loaded tuners typedef struct { char *init; char *next_blocksize; char *next_cparams; char *update; char *free; } tuner_info; #ifdef __cplusplus } #endif #endif /* BLOSC_BLOSC2_TUNERS_UTILS_H */ zmat-0.9.9/src/blosc2/include/blosc2/codecs-registry.h0000644000175200007730000000107414515254731023002 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ enum { BLOSC_CODEC_NDLZ = 32, BLOSC_CODEC_ZFP_FIXED_ACCURACY = 33, BLOSC_CODEC_ZFP_FIXED_PRECISION = 34, BLOSC_CODEC_ZFP_FIXED_RATE = 35, }; void register_codecs(void); zmat-0.9.9/src/blosc2/include/blosc2/blosc2-stdio.h0000644000175200007730000000273614515254731022206 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef BLOSC_BLOSC2_STDIO_H #define BLOSC_BLOSC2_STDIO_H #include #include #include "blosc2-export.h" #if defined(_WIN32) && !defined(__MINGW32__) /* stdint.h only available in VS2010 (VC++ 16.0) and newer */ #if defined(_MSC_VER) && _MSC_VER < 1600 #include "win32/stdint-windows.h" #else #include #endif #else #include #endif #if defined(_MSC_VER) && (_MSC_VER >= 1400) #include #else #include #endif typedef struct { FILE *file; } blosc2_stdio_file; BLOSC_EXPORT void *blosc2_stdio_open(const char *urlpath, const char *mode, void* params); BLOSC_EXPORT int blosc2_stdio_close(void *stream); BLOSC_EXPORT int64_t blosc2_stdio_tell(void *stream); BLOSC_EXPORT int blosc2_stdio_seek(void *stream, int64_t offset, int whence); BLOSC_EXPORT int64_t blosc2_stdio_write(const void *ptr, int64_t size, int64_t nitems, void *stream); BLOSC_EXPORT int64_t blosc2_stdio_read(void *ptr, int64_t size, int64_t nitems, void *stream); BLOSC_EXPORT int blosc2_stdio_truncate(void *stream, int64_t size); #endif //BLOSC_BLOSC2_STDIO_H zmat-0.9.9/src/blosc2/include/blosc2.h0000644000175200007730000026341014515254731017700 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /********************************************************************* @file blosc2.h @brief Blosc2 header file. This file contains Blosc2 public API and the structures needed to use it. @author The Blosc Developers **********************************************************************/ #ifndef BLOSC2_H #define BLOSC2_H #include #include #include #include #include #ifdef __cplusplus extern "C" { #endif #include "blosc2/blosc2-export.h" #include "blosc2/blosc2-common.h" #include "blosc2/blosc2-stdio.h" #ifdef __cplusplus } #endif #if defined(_WIN32) && !defined(__MINGW32__) #include #include #include #define getpid _getpid #endif #ifdef __cplusplus extern "C" { #endif // For compatibility with the Blosc 1.x series #ifdef BLOSC1_COMPAT // Blosc2 symbols that should be accessible from Blosc 1.x API #define BLOSC_VERSION_MAJOR BLOSC2_VERSION_MAJOR #define BLOSC_VERSION_MINOR BLOSC2_VERSION_MINOR #define BLOSC_VERSION_RELEASE BLOSC2_VERSION_RELEASE #define BLOSC_VERSION_STRING BLOSC2_VERSION_STRING #define BLOSC_VERSION_DATE BLOSC2_VERSION_DATE #define BLOSC_MAX_OVERHEAD BLOSC2_MAX_OVERHEAD #define BLOSC_MAX_BUFFERSIZE BLOSC2_MAX_BUFFERSIZE // API that changed to blosc1_ prefix #define blosc_compress blosc1_compress #define blosc_decompress blosc1_decompress #define blosc_getitem blosc1_getitem #define blosc_get_compressor blosc1_get_compressor #define blosc_set_compressor blosc1_set_compressor #define blosc_cbuffer_sizes blosc1_cbuffer_sizes #define blosc_cbuffer_validate blosc1_cbuffer_validate #define blosc_cbuffer_metainfo blosc1_cbuffer_metainfo #define blosc_get_blocksize blosc1_get_blocksize #define blosc_set_blocksize blosc1_set_blocksize #define blosc_set_splitmode blosc1_set_splitmode // API that changed to blosc2_ prefix #define blosc_init blosc2_init #define blosc_destroy blosc2_destroy #define blosc_free_resources blosc2_free_resources #define blosc_get_nthreads blosc2_get_nthreads #define blosc_set_nthreads blosc2_set_nthreads #define blosc_compcode_to_compname blosc2_compcode_to_compname #define blosc_compname_to_compcode blosc2_compname_to_compcode #define blosc_list_compressors blosc2_list_compressors #define blosc_get_version_string blosc2_get_version_string #define blosc_get_complib_info blosc2_get_complib_info #define blosc_cbuffer_versions blosc2_cbuffer_versions #define blosc_cbuffer_complib blosc2_cbuffer_complib #endif /* Version numbers */ #define BLOSC2_VERSION_MAJOR 2 /* for major interface/format changes */ #define BLOSC2_VERSION_MINOR 8 /* for minor interface/format changes */ #define BLOSC2_VERSION_RELEASE 0 /* for tweaks, bug-fixes, or development */ #define BLOSC2_VERSION_STRING "2.8.0" /* string version. Sync with above! */ #define BLOSC2_VERSION_DATE "$Date:: 2023-03-24 #$" /* date version */ /* The maximum number of dimensions for Blosc2 NDim arrays */ #define BLOSC2_MAX_DIM 8 /* Tracing macros */ #define BLOSC_TRACE_ERROR(msg, ...) BLOSC_TRACE(error, msg, ##__VA_ARGS__) #define BLOSC_TRACE_WARNING(msg, ...) BLOSC_TRACE(warning, msg, ##__VA_ARGS__) #define BLOSC_TRACE(cat, msg, ...) \ do { \ const char *__e = getenv("BLOSC_TRACE"); \ if (!__e) { break; } \ fprintf(stderr, "[%s] - " msg " (%s:%d)\n", #cat, ##__VA_ARGS__, __FILE__, __LINE__); \ } while(0) #define BLOSC_ERROR_NULL(pointer, rc) \ do { \ if ((pointer) == NULL) { \ BLOSC_TRACE_ERROR("Pointer is null"); \ return rc; \ }\ } while (0) #define BLOSC_ERROR(rc) \ do { \ int rc_ = rc; \ if (rc_ < BLOSC2_ERROR_SUCCESS) { \ char *error_msg = print_error(rc_); \ BLOSC_TRACE_ERROR("%s", error_msg); \ return rc_; \ } \ } while (0) /* The VERSION_FORMAT symbols below should be just 1-byte long */ enum { /* Blosc format version, starting at 1 1 -> Blosc pre-1.0 2 -> Blosc 1.x stable series 3 -> Blosc 2-alpha.x series 4 -> Blosc 2.x beta.1 series 5 -> Blosc 2.x stable series */ BLOSC1_VERSION_FORMAT_PRE1 = 1, BLOSC1_VERSION_FORMAT = 2, BLOSC2_VERSION_FORMAT_ALPHA = 3, BLOSC2_VERSION_FORMAT_BETA1 = 4, BLOSC2_VERSION_FORMAT_STABLE = 5, BLOSC2_VERSION_FORMAT = BLOSC2_VERSION_FORMAT_STABLE, }; /* The FRAME_FORMAT_VERSION symbols below should be just 4-bit long */ enum { /* Blosc format version * 1 -> First version (introduced in beta.2) * 2 -> Second version (introduced in rc.1) * */ BLOSC2_VERSION_FRAME_FORMAT_BETA2 = 1, // for 2.0.0-beta2 and after BLOSC2_VERSION_FRAME_FORMAT_RC1 = 2, // for 2.0.0-rc1 and after BLOSC2_VERSION_FRAME_FORMAT = BLOSC2_VERSION_FRAME_FORMAT_RC1, }; //!< Struct for storing data from instrumentation of codecs // This can be flexible because it is typically used mainly for development typedef struct { float cratio; float cspeed; float filter_speed; //float memory; //float power; uint8_t flags[4]; } blosc2_instr; enum { #ifndef BLOSC_H BLOSC_MIN_HEADER_LENGTH = 16, //!< Minimum header length (Blosc1) #endif // BLOSC_H BLOSC_EXTENDED_HEADER_LENGTH = 32, //!< Extended header length (Blosc2, see README_HEADER) BLOSC2_MAX_OVERHEAD = BLOSC_EXTENDED_HEADER_LENGTH, //!< The maximum overhead during compression in bytes. This equals //!< to @ref BLOSC_EXTENDED_HEADER_LENGTH now, but can be higher in future //!< implementations. BLOSC2_MAX_BUFFERSIZE = (INT_MAX - BLOSC2_MAX_OVERHEAD), //!< Maximum source buffer size to be compressed #ifndef BLOSC_H BLOSC_MAX_TYPESIZE = UINT8_MAX, //!< Maximum typesize before considering source buffer as a stream of bytes. //!< Cannot be larger than 255. #endif // BLOSC_H BLOSC_MIN_BUFFERSIZE = 32, //!< Minimum buffer size to be compressed. }; enum { BLOSC2_DEFINED_FILTERS_START = 0, BLOSC2_DEFINED_FILTERS_STOP = 31, //!< Blosc-defined filters must be between 0 - 31. BLOSC2_GLOBAL_REGISTERED_FILTERS_START = 32, BLOSC2_GLOBAL_REGISTERED_FILTERS_STOP = 159, //!< Blosc-registered filters must be between 32 - 159. BLOSC2_GLOBAL_REGISTERED_FILTERS = 3, //!< Number of Blosc-registered filters at the moment. BLOSC2_USER_REGISTERED_FILTERS_START = 160, BLOSC2_USER_REGISTERED_FILTERS_STOP = 255, //!< User-defined filters must be between 128 - 255. BLOSC2_MAX_FILTERS = 6, //!< Maximum number of filters in the filter pipeline BLOSC2_MAX_UDFILTERS = 16, //!< Maximum number of filters that a user can register. }; /** * @brief Codes for filters. * * @sa #blosc1_compress */ enum { #ifndef BLOSC_H BLOSC_NOSHUFFLE = 0, //!< No shuffle (for compatibility with Blosc1). BLOSC_NOFILTER = 0, //!< No filter. BLOSC_SHUFFLE = 1, //!< Byte-wise shuffle. BLOSC_BITSHUFFLE = 2, //!< Bit-wise shuffle. #endif // BLOSC_H BLOSC_DELTA = 3, //!< Delta filter. BLOSC_TRUNC_PREC = 4, //!< Truncate mantissa precision; positive values in cparams.filters_meta will keep bits; negative values will reduce bits. BLOSC_LAST_FILTER = 5, //!< sentinel BLOSC_LAST_REGISTERED_FILTER = BLOSC2_GLOBAL_REGISTERED_FILTERS_START + BLOSC2_GLOBAL_REGISTERED_FILTERS - 1, //!< Determine the last registered filter. It is used to check if a filter is registered or not. }; /** * @brief Codes for internal flags (see blosc1_cbuffer_metainfo) */ enum { #ifndef BLOSC_H BLOSC_DOSHUFFLE = 0x1, //!< byte-wise shuffle BLOSC_MEMCPYED = 0x2, //!< plain copy BLOSC_DOBITSHUFFLE = 0x4, //!< bit-wise shuffle #endif // BLOSC_H BLOSC_DODELTA = 0x8, //!< delta coding }; /** * @brief Codes for new internal flags in Blosc2 */ enum { BLOSC2_USEDICT = 0x1, //!< use dictionaries with codec BLOSC2_BIGENDIAN = 0x2, //!< data is in big-endian ordering BLOSC2_INSTR_CODEC = 0x80, //!< codec is instrumented (mainly for development) }; /** * @brief Values for different Blosc2 capabilities */ enum { BLOSC2_MAXDICTSIZE = 128 * 1024, //!< maximum size for compression dicts BLOSC2_MAXBLOCKSIZE = 536866816 //!< maximum size for blocks }; enum { BLOSC2_DEFINED_CODECS_START = 0, BLOSC2_DEFINED_CODECS_STOP = 31, //!< Blosc-defined codecs must be between 0 - 31. BLOSC2_GLOBAL_REGISTERED_CODECS_START = 32, BLOSC2_GLOBAL_REGISTERED_CODECS_STOP = 159, //!< Blosc-registered codecs must be between 31 - 159. BLOSC2_GLOBAL_REGISTERED_CODECS = 1, //!< Number of Blosc-registered codecs at the moment. BLOSC2_USER_REGISTERED_CODECS_START = 160, BLOSC2_USER_REGISTERED_CODECS_STOP = 255, //!< User-defined codecs must be between 160 - 255. }; /** * @brief Codes for the different compressors shipped with Blosc */ enum { #ifndef BLOSC_H BLOSC_BLOSCLZ = 0, BLOSC_LZ4 = 1, BLOSC_LZ4HC = 2, BLOSC_ZLIB = 4, BLOSC_ZSTD = 5, #endif // BLOSC_H BLOSC_LAST_CODEC = 6, //!< Determine the last codec defined by Blosc. BLOSC_LAST_REGISTERED_CODEC = BLOSC2_GLOBAL_REGISTERED_CODECS_START + BLOSC2_GLOBAL_REGISTERED_CODECS - 1, //!< Determine the last registered codec. It is used to check if a codec is registered or not. }; // Names for the different compressors shipped with Blosc #ifndef BLOSC_H #define BLOSC_BLOSCLZ_COMPNAME "blosclz" #define BLOSC_LZ4_COMPNAME "lz4" #define BLOSC_LZ4HC_COMPNAME "lz4hc" #define BLOSC_ZLIB_COMPNAME "zlib" #define BLOSC_ZSTD_COMPNAME "zstd" #endif // BLOSC_H /** * @brief Codes for compression libraries shipped with Blosc (code must be < 8) */ enum { #ifndef BLOSC_H BLOSC_BLOSCLZ_LIB = 0, BLOSC_LZ4_LIB = 1, BLOSC_ZLIB_LIB = 3, BLOSC_ZSTD_LIB = 4, #endif // BLOSC_H BLOSC_UDCODEC_LIB = 6, BLOSC_SCHUNK_LIB = 7, //!< compressor library in super-chunk header }; /** * @brief Names for the different compression libraries shipped with Blosc */ #ifndef BLOSC_H #define BLOSC_BLOSCLZ_LIBNAME "BloscLZ" #define BLOSC_LZ4_LIBNAME "LZ4" #define BLOSC_ZLIB_LIBNAME "Zlib" #define BLOSC_ZSTD_LIBNAME "Zstd" #endif // BLOSC_H /** * @brief The codes for compressor formats shipped with Blosc */ enum { #ifndef BLOSC_H BLOSC_BLOSCLZ_FORMAT = BLOSC_BLOSCLZ_LIB, BLOSC_LZ4_FORMAT = BLOSC_LZ4_LIB, //!< LZ4HC and LZ4 share the same format BLOSC_LZ4HC_FORMAT = BLOSC_LZ4_LIB, BLOSC_ZLIB_FORMAT = BLOSC_ZLIB_LIB, BLOSC_ZSTD_FORMAT = BLOSC_ZSTD_LIB, #endif // BLOSC_H BLOSC_UDCODEC_FORMAT = BLOSC_UDCODEC_LIB, }; /** * @brief The version formats for compressors shipped with Blosc. * All versions here starts at 1 */ enum { #ifndef BLOSC_H BLOSC_BLOSCLZ_VERSION_FORMAT = 1, BLOSC_LZ4_VERSION_FORMAT = 1, BLOSC_LZ4HC_VERSION_FORMAT = 1, /* LZ4HC and LZ4 share the same format */ BLOSC_ZLIB_VERSION_FORMAT = 1, BLOSC_ZSTD_VERSION_FORMAT = 1, #endif // BLOSC_H BLOSC_UDCODEC_VERSION_FORMAT = 1, }; /** * @brief Split mode for blocks. * NEVER and ALWAYS are for experimenting with compression ratio. * AUTO for nearly optimal behaviour (based on heuristics). * FORWARD_COMPAT provides best forward compatibility (default). */ #ifndef BLOSC_H enum { BLOSC_ALWAYS_SPLIT = 1, BLOSC_NEVER_SPLIT = 2, BLOSC_AUTO_SPLIT = 3, BLOSC_FORWARD_COMPAT_SPLIT = 4, }; #endif // BLOSC_H /** * @brief Offsets for fields in Blosc2 chunk header. */ enum { BLOSC2_CHUNK_VERSION = 0x0, //!< the version for the chunk format BLOSC2_CHUNK_VERSIONLZ = 0x1, //!< the version for the format of internal codec BLOSC2_CHUNK_FLAGS = 0x2, //!< flags and codec info BLOSC2_CHUNK_TYPESIZE = 0x3, //!< (uint8) the number of bytes of the atomic type BLOSC2_CHUNK_NBYTES = 0x4, //!< (int32) uncompressed size of the buffer (this header is not included) BLOSC2_CHUNK_BLOCKSIZE = 0x8, //!< (int32) size of internal blocks BLOSC2_CHUNK_CBYTES = 0xc, //!< (int32) compressed size of the buffer (including this header) BLOSC2_CHUNK_FILTER_CODES = 0x10, //!< the codecs for the filter pipeline (1 byte per code) BLOSC2_CHUNK_FILTER_META = 0x18, //!< meta info for the filter pipeline (1 byte per code) BLOSC2_CHUNK_BLOSC2_FLAGS = 0x1F, //!< flags specific for Blosc2 functionality }; /** * @brief Run lengths for special values for chunks/frames */ enum { BLOSC2_NO_SPECIAL = 0x0, //!< no special value BLOSC2_SPECIAL_ZERO = 0x1, //!< zero special value BLOSC2_SPECIAL_NAN = 0x2, //!< NaN special value BLOSC2_SPECIAL_VALUE = 0x3, //!< generic special value BLOSC2_SPECIAL_UNINIT = 0x4, //!< non initialized values BLOSC2_SPECIAL_LASTID = 0x4, //!< last valid ID for special value (update this adequately) BLOSC2_SPECIAL_MASK = 0x7 //!< special value mask (prev IDs cannot be larger than this) }; /** * @brief Error codes */ enum { BLOSC2_ERROR_SUCCESS = 0, //=0). In case the compressor * is not recognized, or there is not support for it in this build, * it returns a -1. */ BLOSC_EXPORT int blosc1_set_compressor(const char* compname); /** * @brief Select the delta coding filter to be used. * * @param dodelta A value >0 will activate the delta filter. * If 0, it will be de-activated * * This call should always succeed. */ BLOSC_EXPORT void blosc2_set_delta(int dodelta); /** * @brief Get the compressor name associated with the compressor code. * * @param compcode The code identifying the compressor * @param compname The pointer to a string where the compressor name will be put. * * @return The compressor code. If the compressor code is not recognized, * or there is not support for it in this build, -1 is returned. */ BLOSC_EXPORT int blosc2_compcode_to_compname(int compcode, const char** compname); /** * @brief Get the compressor code associated with the compressor name. * * @param compname The string containing the compressor name. * * @return The compressor code. If the compressor name is not recognized, * or there is not support for it in this build, -1 is returned instead. */ BLOSC_EXPORT int blosc2_compname_to_compcode(const char* compname); /** * @brief Get a list of compressors supported in the current build. * * @return The comma separated string with the list of compressor names * supported. * * This function does not leak, so you should not free() the returned * list. * * This function should always succeed. */ BLOSC_EXPORT const char* blosc2_list_compressors(void); /** * @brief Get the version of Blosc in string format. * * @return The string with the current Blosc version. * Useful for dynamic libraries. */ BLOSC_EXPORT const char* blosc2_get_version_string(void); /** * @brief Get info from compression libraries included in the current build. * * @param compname The compressor name that you want info from. * @param complib The pointer to a string where the * compression library name, if available, will be put. * @param version The pointer to a string where the * compression library version, if available, will be put. * * @warning You are in charge of the @p complib and @p version strings, * you should free() them so as to avoid leaks. * * @return The code for the compression library (>=0). If it is not supported, * this function returns -1. */ BLOSC_EXPORT int blosc2_get_complib_info(const char* compname, char** complib, char** version); /** * @brief Free possible memory temporaries and thread resources. Use this * when you are not going to use Blosc for a long while. * * @return A 0 if succeeds, in case of problems releasing the resources, * it returns a negative number. */ BLOSC_EXPORT int blosc2_free_resources(void); /** * @brief Get information about a compressed buffer, namely the number of * uncompressed bytes (@p nbytes) and compressed (@p cbytes). It also * returns the @p blocksize (which is used internally for doing the * compression by blocks). * * @param cbuffer The buffer of compressed data. * @param nbytes The pointer where the number of uncompressed bytes will be put. * @param cbytes The pointer where the number of compressed bytes will be put. * @param blocksize The pointer where the block size will be put. * * You only need to pass the first BLOSC_MIN_HEADER_LENGTH bytes of a * compressed buffer for this call to work. * * This function should always succeed. */ BLOSC_EXPORT void blosc1_cbuffer_sizes(const void* cbuffer, size_t* nbytes, size_t* cbytes, size_t* blocksize); /** * @brief Get information about a compressed buffer, namely the number of * uncompressed bytes (@p nbytes) and compressed (@p cbytes). It also * returns the @p blocksize (which is used internally for doing the * compression by blocks). * * @param cbuffer The buffer of compressed data. * @param nbytes The pointer where the number of uncompressed bytes will be put. * @param cbytes The pointer where the number of compressed bytes will be put. * @param blocksize The pointer where the block size will be put. * * @note: if any of the nbytes, cbytes or blocksize is NULL, it will not be returned. * * You only need to pass the first BLOSC_MIN_HEADER_LENGTH bytes of a * compressed buffer for this call to work. * * @return On failure, returns negative value. */ BLOSC_EXPORT int blosc2_cbuffer_sizes(const void* cbuffer, int32_t* nbytes, int32_t* cbytes, int32_t* blocksize); /** * @brief Checks that the compressed buffer starting at @p cbuffer of length @p cbytes * may contain valid blosc compressed data, and that it is safe to call * blosc1_decompress/blosc1_getitem. * On success, returns 0 and sets @p nbytes to the size of the uncompressed data. * This does not guarantee that the decompression function won't return an error, * but does guarantee that it is safe to attempt decompression. * * @param cbuffer The buffer of compressed data. * @param cbytes The number of compressed bytes. * @param nbytes The pointer where the number of uncompressed bytes will be put. * * @return On failure, returns negative value. */ BLOSC_EXPORT int blosc1_cbuffer_validate(const void* cbuffer, size_t cbytes, size_t* nbytes); /** * @brief Get information about a compressed buffer, namely the type size * (@p typesize), as well as some internal @p flags. * * @param cbuffer The buffer of compressed data. * @param typesize The pointer where the type size will be put. * @param flags The pointer of the integer where the additional info is encoded. * The @p flags is a set of bits, where the currently used ones are: * * bit 0: whether the shuffle filter has been applied or not * * bit 1: whether the internal buffer is a pure memcpy or not * * bit 2: whether the bitshuffle filter has been applied or not * * bit 3: whether the delta coding filter has been applied or not * * You can use the @p BLOSC_DOSHUFFLE, @p BLOSC_DOBITSHUFFLE, @p BLOSC_DODELTA * and @p BLOSC_MEMCPYED symbols for extracting the interesting bits * (e.g. @p flags & @p BLOSC_DOSHUFFLE says whether the buffer is byte-shuffled * or not). * * This function should always succeed. */ BLOSC_EXPORT void blosc1_cbuffer_metainfo(const void* cbuffer, size_t* typesize, int* flags); /** * @brief Get information about a compressed buffer, namely the internal * Blosc format version (@p version) and the format for the internal * Lempel-Ziv compressor used (@p versionlz). * * @param cbuffer The buffer of compressed data. * @param version The pointer where the Blosc format version will be put. * @param versionlz The pointer where the Lempel-Ziv version will be put. * * This function should always succeed. */ BLOSC_EXPORT void blosc2_cbuffer_versions(const void* cbuffer, int* version, int* versionlz); /** * @brief Get the compressor library/format used in a compressed buffer. * * @param cbuffer The buffer of compressed data. * * @return The string identifying the compressor library/format used. * * This function should always succeed. */ BLOSC_EXPORT const char* blosc2_cbuffer_complib(const void* cbuffer); /********************************************************************* Structures and functions related with user-defined input/output. *********************************************************************/ enum { BLOSC2_IO_FILESYSTEM = 0, BLOSC_IO_LAST_BLOSC_DEFINED = 1, // sentinel BLOSC_IO_LAST_REGISTERED = 32, // sentinel }; enum { BLOSC2_IO_BLOSC_DEFINED = 32, BLOSC2_IO_REGISTERED = 160, BLOSC2_IO_USER_DEFINED = 256 }; typedef void* (*blosc2_open_cb)(const char *urlpath, const char *mode, void *params); typedef int (*blosc2_close_cb)(void *stream); typedef int64_t (*blosc2_tell_cb)(void *stream); typedef int (*blosc2_seek_cb)(void *stream, int64_t offset, int whence); typedef int64_t (*blosc2_write_cb)(const void *ptr, int64_t size, int64_t nitems, void *stream); typedef int64_t (*blosc2_read_cb)(void *ptr, int64_t size, int64_t nitems, void *stream); typedef int (*blosc2_truncate_cb)(void *stream, int64_t size); /* * Input/Output callbacks. */ typedef struct { uint8_t id; //!< The IO identifier. blosc2_open_cb open; //!< The IO open callback. blosc2_close_cb close; //!< The IO close callback. blosc2_tell_cb tell; //!< The IO tell callback. blosc2_seek_cb seek; //!< The IO seek callback. blosc2_write_cb write; //!< The IO write callback. blosc2_read_cb read; //!< The IO read callback. blosc2_truncate_cb truncate; //!< The IO truncate callback. } blosc2_io_cb; /* * Input/Output parameters. */ typedef struct { uint8_t id; //!< The IO identifier. void *params; //!< The IO parameters. } blosc2_io; static blosc2_io_cb BLOSC2_IO_CB_DEFAULTS = {0}; static const blosc2_io BLOSC2_IO_DEFAULTS = { .id = BLOSC2_IO_FILESYSTEM, .params = NULL, }; /** * @brief Register a user-defined input/output callbacks in Blosc. * * @param io The callbacks API to register. * * @return 0 if succeeds. Else a negative code is returned. */ BLOSC_EXPORT int blosc2_register_io_cb(const blosc2_io_cb *io); BLOSC_EXPORT blosc2_io_cb *blosc2_get_io_cb(uint8_t id); /********************************************************************* Structures and functions related with contexts. *********************************************************************/ typedef struct blosc2_context_s blosc2_context; /* opaque type */ typedef struct { void (*btune_init)(void * config, blosc2_context* cctx, blosc2_context* dctx); //!< Initialize BTune. void (*btune_next_blocksize)(blosc2_context * context); //!< Only compute the next blocksize. Only it is executed if BTune is not initialized. void (*btune_next_cparams)(blosc2_context * context); //!< Compute the next cparams. Only is executed if BTune is initialized. void (*btune_update)(blosc2_context * context, double ctime); //!< Update the BTune parameters. void (*btune_free)(blosc2_context * context); //!< Free the BTune. void *btune_config; //!> BTune configuration. }blosc2_btune; /** * @brief The parameters for a prefilter function. * */ typedef struct { void *user_data; // user-provided info (optional) const uint8_t *input; // the input buffer uint8_t *output; // the output buffer int32_t output_size; // the output size (in bytes) int32_t output_typesize; // the output typesize int32_t output_offset; // offset to reach the start of the output buffer int64_t nchunk; // the current nchunk in associated schunk (if exists; if not -1) int32_t nblock; // the current nblock in associated chunk int32_t tid; // thread id uint8_t *ttmp; // a temporary that is able to hold several blocks for the output and is private for each thread size_t ttmp_nbytes; // the size of the temporary in bytes blosc2_context *ctx; // the compression context } blosc2_prefilter_params; /** * @brief The parameters for a postfilter function. * */ typedef struct { void *user_data; // user-provided info (optional) const uint8_t *input; // the input buffer uint8_t *output; // the output buffer int32_t size; // the input size (in bytes) int32_t typesize; // the input typesize int32_t offset; // offset to reach the start of the input buffer int64_t nchunk; // the current nchunk in associated schunk (if exists; if not -1) int32_t nblock; // the current nblock in associated chunk int32_t tid; // thread id uint8_t *ttmp; // a temporary that is able to hold several blocks for the output and is private for each thread size_t ttmp_nbytes; // the size of the temporary in bytes blosc2_context *ctx; // the decompression context } blosc2_postfilter_params; /** * @brief The type of the prefilter function. * * If the function call is successful, the return value should be 0; else, a negative value. */ typedef int (*blosc2_prefilter_fn)(blosc2_prefilter_params* params); /** * @brief The type of the postfilter function. * * If the function call is successful, the return value should be 0; else, a negative value. */ typedef int (*blosc2_postfilter_fn)(blosc2_postfilter_params* params); /** * @brief The parameters for creating a context for compression purposes. * * In parenthesis it is shown the default value used internally when a 0 * (zero) in the fields of the struct is passed to a function. */ typedef struct { uint8_t compcode; //!< The compressor codec. uint8_t compcode_meta; //!< The metadata for the compressor codec. uint8_t clevel; //!< The compression level (5). int use_dict; //!< Use dicts or not when compressing (only for ZSTD). int32_t typesize; //!< The type size (8). int16_t nthreads; //!< The number of threads to use internally (1). int32_t blocksize; //!< The requested size of the compressed blocks (0 means automatic). int32_t splitmode; //!< Whether the blocks should be split or not. void* schunk; //!< The associated schunk, if any (NULL). uint8_t filters[BLOSC2_MAX_FILTERS]; //!< The (sequence of) filters. uint8_t filters_meta[BLOSC2_MAX_FILTERS]; //!< The metadata for filters. blosc2_prefilter_fn prefilter; //!< The prefilter function. blosc2_prefilter_params *preparams; //!< The prefilter parameters. blosc2_btune *udbtune; //!< The user-defined BTune parameters. bool instr_codec; //!< Whether the codec is instrumented or not void *codec_params; //!< User defined parameters for the codec void *filter_params[BLOSC2_MAX_FILTERS]; //!< User defined parameters for the filters } blosc2_cparams; /** * @brief Default struct for compression params meant for user initialization. */ static const blosc2_cparams BLOSC2_CPARAMS_DEFAULTS = { BLOSC_BLOSCLZ, 0, 5, 0, 8, 1, 0, BLOSC_FORWARD_COMPAT_SPLIT, NULL, {0, 0, 0, 0, 0, BLOSC_SHUFFLE}, {0, 0, 0, 0, 0, 0}, NULL, NULL, NULL, 0, NULL, {NULL, NULL, NULL, NULL, NULL, NULL} }; /** @brief The parameters for creating a context for decompression purposes. In parenthesis it is shown the default value used internally when a 0 (zero) in the fields of the struct is passed to a function. */ typedef struct { int16_t nthreads; //!< The number of threads to use internally (1). void* schunk; //!< The associated schunk, if any (NULL). blosc2_postfilter_fn postfilter; //!< The postfilter function. blosc2_postfilter_params *postparams; //!< The postfilter parameters. } blosc2_dparams; /** * @brief Default struct for decompression params meant for user initialization. */ static const blosc2_dparams BLOSC2_DPARAMS_DEFAULTS = {1, NULL, NULL, NULL}; /** * @brief Create a context for @a *_ctx() compression functions. * * @param cparams The blosc2_cparams struct with the compression parameters. * * @return A pointer to the new context. NULL is returned if this fails. * * @note This support the same environment variables than #blosc2_compress * for overriding the programmatic compression values. * * @sa #blosc2_compress */ BLOSC_EXPORT blosc2_context* blosc2_create_cctx(blosc2_cparams cparams); /** * @brief Create a context for *_ctx() decompression functions. * * @param dparams The blosc2_dparams struct with the decompression parameters. * * @return A pointer to the new context. NULL is returned if this fails. * * @note This support the same environment variables than #blosc2_decompress * for overriding the programmatic decompression values. * * @sa #blosc2_decompress * */ BLOSC_EXPORT blosc2_context* blosc2_create_dctx(blosc2_dparams dparams); /** * @brief Free the resources associated with a context. * * @param context The context to free. * * This function should always succeed and is valid for contexts meant for * both compression and decompression. */ BLOSC_EXPORT void blosc2_free_ctx(blosc2_context* context); /** * @brief Create a @p cparams associated to a context. * * @param ctx The context from where to extract the compression parameters. * @param cparams The pointer where the compression params will be stored. * * @return 0 if succeeds. Else a negative code is returned. */ BLOSC_EXPORT int blosc2_ctx_get_cparams(blosc2_context *ctx, blosc2_cparams *cparams); /** * @brief Create a @p dparams associated to a context. * * @param ctx The context from where to extract the decompression parameters. * @param dparams The pointer where the decompression params will be stored. * * @return 0 if succeeds. Else a negative code is returned. */ BLOSC_EXPORT int blosc2_ctx_get_dparams(blosc2_context *ctx, blosc2_dparams *dparams); /** * @brief Set a maskout so as to avoid decompressing specified blocks. * * @param ctx The decompression context to update. * * @param maskout The boolean mask for the blocks where decompression * is to be avoided. * * @remark The maskout is valid for contexts *only* meant for decompressing * a chunk via #blosc2_decompress_ctx. Once a call to #blosc2_decompress_ctx * is done, this mask is reset so that next call to #blosc2_decompress_ctx * will decompress the whole chunk. * * @param nblocks The number of blocks in maskout above. * * @return If success, a 0 is returned. An error is signaled with a negative int. * */ BLOSC_EXPORT int blosc2_set_maskout(blosc2_context *ctx, bool *maskout, int nblocks); /** * @brief Compress a block of data in the @p src buffer and returns the size of * compressed block. * * @remark Compression is memory safe and guaranteed not to write @p dest * more than what is specified in @p destsize. * There is not a minimum for @p src buffer size @p nbytes. * * @warning The @p src buffer and the @p dest buffer can not overlap. * * @param clevel The desired compression level and must be a number * between 0 (no compression) and 9 (maximum compression). * @param doshuffle Specifies whether the shuffle compression preconditioner * should be applied or not. #BLOSC_NOFILTER means not applying filters, * #BLOSC_SHUFFLE means applying shuffle at a byte level and * #BLOSC_BITSHUFFLE at a bit level (slower but *may* achieve better * compression). * @param typesize Is the number of bytes for the atomic type in binary * @p src buffer. This is mainly useful for the shuffle preconditioner. * For implementation reasons, only a 1 < typesize < 256 will allow the * shuffle filter to work. When typesize is not in this range, shuffle * will be silently disabled. * @param src The buffer containing the data to compress. * @param srcsize The number of bytes to compress in the @p src buffer. * @param dest The buffer where the compressed data will be put, * must have at least the size of @p destsize. * @param destsize The size of the dest buffer. Blosc * guarantees that if you set @p destsize to, at least, * (@p nbytes + #BLOSC2_MAX_OVERHEAD), the compression will always succeed. * * @return The number of bytes compressed. * If @p src buffer cannot be compressed into @p destsize, the return * value is zero and you should discard the contents of the @p dest * buffer. A negative return value means that an internal error happened. This * should never happen. If you see this, please report it back * together with the buffer data causing this and compression settings. */ /* * Environment variables * _____________________ * * *blosc2_compress()* honors different environment variables to control * internal parameters without the need of doing that programmatically. * Here are the ones supported: * * **BLOSC_CLEVEL=(INTEGER)**: This will overwrite the @p clevel parameter * before the compression process starts. * * **BLOSC_SHUFFLE=[NOSHUFFLE | SHUFFLE | BITSHUFFLE]**: This will * overwrite the *doshuffle* parameter before the compression process * starts. * * **BLOSC_DELTA=(1|0)**: This will call *blosc2_set_delta()^* before the * compression process starts. * * **BLOSC_TYPESIZE=(INTEGER)**: This will overwrite the *typesize* * parameter before the compression process starts. * * **BLOSC_COMPRESSOR=[BLOSCLZ | LZ4 | LZ4HC | ZLIB | ZSTD]**: * This will call #blosc_set_compressor before the compression process starts. * * **BLOSC_NTHREADS=(INTEGER)**: This will call * #blosc_set_nthreads before the compression process * starts. * * **BLOSC_SPLITMODE=(ALWAYS | NEVER | AUTO | FORWARD_COMPAT)**: * This will call #blosc1_set_splitmode() before the compression process starts. * * **BLOSC_BLOCKSIZE=(INTEGER)**: This will call * #blosc_set_blocksize before the compression process starts. * *NOTE:* The blocksize is a critical parameter with * important restrictions in the allowed values, so use this with care. * * **BLOSC_NOLOCK=(ANY VALUE)**: This will call #blosc2_compress_ctx under * the hood, with the *compressor*, *blocksize* and * *numinternalthreads* parameters set to the same as the last calls to * #blosc1_set_compressor, #blosc1_set_blocksize and * #blosc2_set_nthreads. *BLOSC_CLEVEL*, *BLOSC_SHUFFLE*, *BLOSC_DELTA* and * *BLOSC_TYPESIZE* environment vars will also be honored. * */ BLOSC_EXPORT int blosc2_compress(int clevel, int doshuffle, int32_t typesize, const void* src, int32_t srcsize, void* dest, int32_t destsize); /** * @brief Decompress a block of compressed data in @p src, put the result in * @p dest and returns the size of the decompressed block. * * @warning The @p src buffer and the @p dest buffer can not overlap. * * @remark Decompression is memory safe and guaranteed not to write the @p dest * buffer more than what is specified in @p destsize. * * @remark In case you want to keep under control the number of bytes read from * source, you can call #blosc1_cbuffer_sizes first to check whether the * @p nbytes (i.e. the number of bytes to be read from @p src buffer by this * function) in the compressed buffer is ok with you. * * @param src The buffer to be decompressed. * @param srcsize The size of the buffer to be decompressed. * @param dest The buffer where the decompressed data will be put. * @param destsize The size of the @p dest buffer. * * @return The number of bytes decompressed. * If an error occurs, e.g. the compressed data is corrupted or the * output buffer is not large enough, then a negative value * will be returned instead. */ /* * Environment variables * _____________________ * * *blosc1_decompress* honors different environment variables to control * internal parameters without the need of doing that programmatically. * Here are the ones supported: * * **BLOSC_NTHREADS=(INTEGER)**: This will call * *blosc_set_nthreads(BLOSC_NTHREADS)* before the proper decompression * process starts. * * **BLOSC_NOLOCK=(ANY VALUE)**: This will call *blosc2_decompress_ctx* * under the hood, with the *numinternalthreads* parameter set to the * same value as the last call to *blosc2_set_nthreads*. * */ BLOSC_EXPORT int blosc2_decompress(const void* src, int32_t srcsize, void* dest, int32_t destsize); /** * @brief Context interface to Blosc compression. This does not require a call * to #blosc2_init and can be called from multithreaded applications * without the global lock being used, so allowing Blosc be executed * simultaneously in those scenarios. * * @param context A blosc2_context struct with the different compression params. * @param src The buffer containing the data to be compressed. * @param srcsize The number of bytes to be compressed from the @p src buffer. * @param dest The buffer where the compressed data will be put. * @param destsize The size in bytes of the @p dest buffer. * * @return The number of bytes compressed. * If @p src buffer cannot be compressed into @p destsize, the return * value is zero and you should discard the contents of the @p dest * buffer. A negative return value means that an internal error happened. * It could happen that context is not meant for compression (which is stated in stderr). * Otherwise, please report it back together with the buffer data causing this * and compression settings. */ BLOSC_EXPORT int blosc2_compress_ctx( blosc2_context* context, const void* src, int32_t srcsize, void* dest, int32_t destsize); /** * @brief Context interface to Blosc decompression. This does not require a * call to #blosc2_init and can be called from multithreaded * applications without the global lock being used, so allowing Blosc * be executed simultaneously in those scenarios. * * @param context The blosc2_context struct with the different compression params. * @param src The buffer of compressed data. * @param srcsize The length of buffer of compressed data. * @param dest The buffer where the decompressed data will be put. * @param destsize The size in bytes of the @p dest buffer. * * @warning The @p src buffer and the @p dest buffer can not overlap. * * @remark Decompression is memory safe and guaranteed not to write the @p dest * buffer more than what is specified in @p destsize. * * @remark In case you want to keep under control the number of bytes read from * source, you can call #blosc1_cbuffer_sizes first to check the @p nbytes * (i.e. the number of bytes to be read from @p src buffer by this function) * in the compressed buffer. * * @remark If #blosc2_set_maskout is called prior to this function, its * @p block_maskout parameter will be honored for just *one single* shot; * i.e. the maskout in context will be automatically reset to NULL, so * mask won't be used next time (unless #blosc2_set_maskout is called again). * * @return The number of bytes decompressed (i.e. the maskout blocks are not * counted). If an error occurs, e.g. the compressed data is corrupted, * @p destsize is not large enough or context is not meant for decompression, * then a negative value will be returned instead. */ BLOSC_EXPORT int blosc2_decompress_ctx(blosc2_context* context, const void* src, int32_t srcsize, void* dest, int32_t destsize); /** * @brief Create a chunk made of zeros. * * @param cparams The compression parameters. * @param nbytes The size (in bytes) of the chunk. * @param dest The buffer where the data chunk will be put. * @param destsize The size (in bytes) of the @p dest buffer; * must be BLOSC_EXTENDED_HEADER_LENGTH at least. * * @return The number of bytes compressed (BLOSC_EXTENDED_HEADER_LENGTH). * If negative, there has been an error and @p dest is unusable. * */ BLOSC_EXPORT int blosc2_chunk_zeros(blosc2_cparams cparams, int32_t nbytes, void* dest, int32_t destsize); /** * @brief Create a chunk made of nans. * * @param cparams The compression parameters; * only 4 bytes (float) and 8 bytes (double) are supported. * @param nbytes The size (in bytes) of the chunk. * @param dest The buffer where the data chunk will be put. * @param destsize The size (in bytes) of the @p dest buffer; * must be BLOSC_EXTENDED_HEADER_LENGTH at least. * * @note Whether the NaNs are floats or doubles will be given by the typesize. * * @return The number of bytes compressed (BLOSC_EXTENDED_HEADER_LENGTH). * If negative, there has been an error and @p dest is unusable. * */ BLOSC_EXPORT int blosc2_chunk_nans(blosc2_cparams cparams, int32_t nbytes, void* dest, int32_t destsize); /** * @brief Create a chunk made of repeated values. * * @param cparams The compression parameters. * @param nbytes The size (in bytes) of the chunk. * @param dest The buffer where the data chunk will be put. * @param destsize The size (in bytes) of the @p dest buffer. * @param repeatval A pointer to the repeated value (little endian). * The size of the value is given by @p cparams.typesize param. * * @return The number of bytes compressed (BLOSC_EXTENDED_HEADER_LENGTH + typesize). * If negative, there has been an error and @p dest is unusable. * */ BLOSC_EXPORT int blosc2_chunk_repeatval(blosc2_cparams cparams, int32_t nbytes, void* dest, int32_t destsize, const void* repeatval); /** * @brief Create a chunk made of uninitialized values. * * @param cparams The compression parameters. * @param nbytes The size (in bytes) of the chunk. * @param dest The buffer where the data chunk will be put. * @param destsize The size (in bytes) of the @p dest buffer; * must be BLOSC_EXTENDED_HEADER_LENGTH at least. * * @return The number of bytes compressed (BLOSC_EXTENDED_HEADER_LENGTH). * If negative, there has been an error and @p dest is unusable. * */ BLOSC_EXPORT int blosc2_chunk_uninit(blosc2_cparams cparams, int32_t nbytes, void* dest, int32_t destsize); /** * @brief Context interface counterpart for #blosc1_getitem. * * @param context Context pointer. * @param src The compressed buffer from data will be decompressed. * @param srcsize Compressed buffer length. * @param start The position of the first item (of @p typesize size) from where data * will be retrieved. * @param nitems The number of items (of @p typesize size) that will be retrieved. * @param dest The buffer where the decompressed data retrieved will be put. * @param destsize Output buffer length. * * @return The number of bytes copied to @p dest or a negative value if * some error happens. */ BLOSC_EXPORT int blosc2_getitem_ctx(blosc2_context* context, const void* src, int32_t srcsize, int start, int nitems, void* dest, int32_t destsize); /********************************************************************* Super-chunk related structures and functions. *********************************************************************/ #define BLOSC2_MAX_METALAYERS 16 #define BLOSC2_METALAYER_NAME_MAXLEN 31 // Allow for a reasonable number of vl metalayers // max is 64 * 1024 due to msgpack map 16 in frame // mem usage 8 * 1024 entries for blosc2_schunk.vlmetalayers[] is 64 KB #define BLOSC2_MAX_VLMETALAYERS (8 * 1024) #define BLOSC2_VLMETALAYERS_NAME_MAXLEN BLOSC2_METALAYER_NAME_MAXLEN /** * @brief This struct is meant for holding storage parameters for a * for a blosc2 container, allowing to specify, for example, how to interpret * the contents included in the schunk. */ typedef struct { bool contiguous; //!< Whether the chunks are contiguous or sparse. char* urlpath; //!< The path for persistent storage. If NULL, that means in-memory. blosc2_cparams* cparams; //!< The compression params when creating a schunk. //!< If NULL, sensible defaults are used depending on the context. blosc2_dparams* dparams; //!< The decompression params when creating a schunk. //!< If NULL, sensible defaults are used depending on the context. blosc2_io *io; //!< Input/output backend. } blosc2_storage; /** * @brief Default struct for #blosc2_storage meant for user initialization. */ static const blosc2_storage BLOSC2_STORAGE_DEFAULTS = {false, NULL, NULL, NULL, NULL}; typedef struct blosc2_frame_s blosc2_frame; /* opaque type */ /** * @brief This struct is meant to store metadata information inside * a #blosc2_schunk, allowing to specify, for example, how to interpret * the contents included in the schunk. */ typedef struct blosc2_metalayer { char* name; //!< The metalayer identifier for Blosc client (e.g. Blosc2 NDim). uint8_t* content; //!< The serialized (msgpack preferably) content of the metalayer. int32_t content_len; //!< The length in bytes of the content. } blosc2_metalayer; /** * @brief This struct is the standard container for Blosc 2 compressed data. * * This is essentially a container for Blosc 1 chunks of compressed data, * and it allows to overcome the 32-bit limitation in Blosc 1. Optionally, * a #blosc2_frame can be attached so as to store the compressed chunks contiguously. */ typedef struct blosc2_schunk { uint8_t version; uint8_t compcode; //!< The default compressor. Each chunk can override this. uint8_t compcode_meta; //!< The default compressor metadata. Each chunk can override this. uint8_t clevel; //!< The compression level and other compress params. uint8_t splitmode; //!< The split mode. int32_t typesize; //!< The type size. int32_t blocksize; //!< The requested size of the compressed blocks (0; meaning automatic). int32_t chunksize; //!< Size of each chunk. 0 if not a fixed chunksize. uint8_t filters[BLOSC2_MAX_FILTERS]; //!< The (sequence of) filters. 8-bit per filter. uint8_t filters_meta[BLOSC2_MAX_FILTERS]; //!< Metadata for filters. 8-bit per meta-slot. int64_t nchunks; //!< Number of chunks in super-chunk. int64_t current_nchunk; //!< The current chunk that is being accessed int64_t nbytes; //!< The data size (uncompressed). int64_t cbytes; //!< The data size + chunks header size (compressed). uint8_t** data; //!< Pointer to chunk data pointers buffer. size_t data_len; //!< Length of the chunk data pointers buffer. blosc2_storage* storage; //!< Pointer to storage info. blosc2_frame* frame; //!< Pointer to frame used as store for chunks. //! BLOSC2_METALAYER_NAME_MAXLEN) { BLOSC_TRACE_ERROR("Metalayers cannot be larger than %d chars.", BLOSC2_METALAYER_NAME_MAXLEN); return BLOSC2_ERROR_INVALID_PARAM; } if (schunk == NULL) { BLOSC_TRACE_ERROR("Schunk must not be NUll."); return BLOSC2_ERROR_INVALID_PARAM; } for (int nmetalayer = 0; nmetalayer < schunk->nmetalayers; nmetalayer++) { if (strcmp(name, schunk->metalayers[nmetalayer]->name) == 0) { return nmetalayer; } } return BLOSC2_ERROR_NOT_FOUND; } /** * @brief Add content into a new metalayer. * * @param schunk The super-chunk to which the metalayer should be added. * @param name The name of the metalayer. * @param content The content of the metalayer. * @param content_len The length of the content. * * @return If successful, the index of the new metalayer. Else, return a negative value. */ BLOSC_EXPORT int blosc2_meta_add(blosc2_schunk *schunk, const char *name, uint8_t *content, int32_t content_len); /** * @brief Update the content of an existing metalayer. * * @param schunk The frame containing the metalayer. * @param name The name of the metalayer to be updated. * @param content The new content of the metalayer. * @param content_len The length of the content. * * @note Contrarily to #blosc2_meta_add the updates to metalayers * are automatically serialized into a possible attached frame. * * @return If successful, the index of the metalayer. Else, return a negative value. */ BLOSC_EXPORT int blosc2_meta_update(blosc2_schunk *schunk, const char *name, uint8_t *content, int32_t content_len); static inline void swap_store(void *dest, const void *pa, int size) { uint8_t *pa_ = (uint8_t *) pa; uint8_t *pa2_ = (uint8_t*)malloc((size_t) size); int i = 1; /* for big/little endian detection */ char *p = (char *) &i; if (p[0] == 1) { /* little endian */ switch (size) { case 8: pa2_[0] = pa_[7]; pa2_[1] = pa_[6]; pa2_[2] = pa_[5]; pa2_[3] = pa_[4]; pa2_[4] = pa_[3]; pa2_[5] = pa_[2]; pa2_[6] = pa_[1]; pa2_[7] = pa_[0]; break; case 4: pa2_[0] = pa_[3]; pa2_[1] = pa_[2]; pa2_[2] = pa_[1]; pa2_[3] = pa_[0]; break; case 2: pa2_[0] = pa_[1]; pa2_[1] = pa_[0]; break; case 1: pa2_[0] = pa_[0]; break; default: fprintf(stderr, "Unhandled nitems: %d\n", size); } } memcpy(dest, pa2_, size); free(pa2_); } /** * @brief Get the content out of a metalayer. * * @param schunk The frame containing the metalayer. * @param name The name of the metalayer. * @param content The pointer where the content will be put. * @param content_len The length of the content. * * @warning The @p **content receives a malloc'ed copy of the content. * The user is responsible of freeing it. * * @note This function is inlined and available even when not linking with libblosc2. * * @return If successful, the index of the new metalayer. Else, return a negative value. */ static inline int blosc2_meta_get(blosc2_schunk *schunk, const char *name, uint8_t **content, int32_t *content_len) { int nmetalayer = blosc2_meta_exists(schunk, name); if (nmetalayer < 0) { BLOSC_TRACE_WARNING("Metalayer \"%s\" not found.", name); return nmetalayer; } *content_len = schunk->metalayers[nmetalayer]->content_len; *content = (uint8_t*)malloc((size_t)*content_len); memcpy(*content, schunk->metalayers[nmetalayer]->content, (size_t)*content_len); return nmetalayer; } /********************************************************************* Variable-length metalayers functions. *********************************************************************/ /** * @brief Find whether the schunk has a variable-length metalayer or not. * * @param schunk The super-chunk from which the variable-length metalayer will be checked. * @param name The name of the variable-length metalayer to be checked. * * @return If successful, return the index of the variable-length metalayer. Else, return a negative value. */ BLOSC_EXPORT int blosc2_vlmeta_exists(blosc2_schunk *schunk, const char *name); /** * @brief Add content into a new variable-length metalayer. * * @param schunk The super-chunk to which the variable-length metalayer should be added. * @param name The name of the variable-length metalayer. * @param content The content to be added. * @param content_len The length of the content. * @param cparams The parameters for compressing the variable-length metalayer content. If NULL, * the `BLOSC2_CPARAMS_DEFAULTS` will be used. * * @return If successful, the index of the new variable-length metalayer. Else, return a negative value. */ BLOSC_EXPORT int blosc2_vlmeta_add(blosc2_schunk *schunk, const char *name, uint8_t *content, int32_t content_len, blosc2_cparams *cparams); /** * @brief Update the content of an existing variable-length metalayer. * * @param schunk The super-chunk containing the variable-length metalayer. * @param name The name of the variable-length metalayer to be updated. * @param content The new content of the variable-length metalayer. * @param content_len The length of the content. * @param cparams The parameters for compressing the variable-length metalayer content. If NULL, * the `BLOSC2_CPARAMS_DEFAULTS` will be used. * * @return If successful, the index of the variable-length metalayer. Else, return a negative value. */ BLOSC_EXPORT int blosc2_vlmeta_update(blosc2_schunk *schunk, const char *name, uint8_t *content, int32_t content_len, blosc2_cparams *cparams); /** * @brief Get the content out of a variable-length metalayer. * * @param schunk The super-chunk containing the variable-length metalayer. * @param name The name of the variable-length metalayer. * @param content The pointer where the content will be put. * @param content_len The pointer where the length of the content will be put. * * @warning The @p **content receives a malloc'ed copy of the content. * The user is responsible of freeing it. * * @return If successful, the index of the new variable-length metalayer. Else, return a negative value. */ BLOSC_EXPORT int blosc2_vlmeta_get(blosc2_schunk *schunk, const char *name, uint8_t **content, int32_t *content_len); /** * @brief Delete the variable-length metalayer from the super-chunk. * * @param schunk The super-chunk containing the variable-length metalayer. * @param name The name of the variable-length metalayer. * * @return If successful, the number of the variable-length metalayers in the super-chunk. Else, return a negative value. */ BLOSC_EXPORT int blosc2_vlmeta_delete(blosc2_schunk *schunk, const char *name); /** * @brief Get a list of all the variable-length metalayer names. * * @param schunk The super-chunk containing the variable-length metalayers. * @param names The pointer to a char** to store the name pointers. This should * be of size *schunk->nvlmetalayers * sizeof(char*). * * @return The number of the variable-length metalayers in the super-chunk. * This cannot fail unless the user does not pass a @p names which is large enough to * keep pointers to all names, in which case funny things (seg faults and such) will happen. */ BLOSC_EXPORT int blosc2_vlmeta_get_names(blosc2_schunk *schunk, char **names); /********************************************************************* Time measurement utilities. *********************************************************************/ #if defined(_WIN32) /* For QueryPerformanceCounter(), etc. */ #include #elif defined(__MACH__) #include #include #include #elif defined(__unix__) #if defined(__linux__) #include #else #include #endif #else #error Unable to detect platform. #endif /* The type of timestamp used on this system. */ #if defined(_WIN32) #define blosc_timestamp_t LARGE_INTEGER #else #define blosc_timestamp_t struct timespec #endif /* * Set a timestamp. */ BLOSC_EXPORT void blosc_set_timestamp(blosc_timestamp_t* timestamp); /* * Return the nanoseconds between 2 timestamps. */ BLOSC_EXPORT double blosc_elapsed_nsecs(blosc_timestamp_t start_time, blosc_timestamp_t end_time); /* * Return the seconds between 2 timestamps. */ BLOSC_EXPORT double blosc_elapsed_secs(blosc_timestamp_t start_time, blosc_timestamp_t end_time); /********************************************************************* Low-level functions follows. Use them only if you are an expert! *********************************************************************/ /** * @brief Get the internal blocksize to be used during compression. 0 means * that an automatic blocksize is computed internally. * * @return The size in bytes of the internal block size. */ BLOSC_EXPORT int blosc1_get_blocksize(void); /** * @brief Force the use of a specific blocksize. If 0, an automatic * blocksize will be used (the default). * * @warning The blocksize is a critical parameter with important * restrictions in the allowed values, so use this with care. */ BLOSC_EXPORT void blosc1_set_blocksize(size_t blocksize); /** * @brief Set the split mode. * @param splitmode It can take the next values: * BLOSC_FORWARD_COMPAT_SPLIT * BLOSC_AUTO_SPLIT * BLOSC_NEVER_SPLIT * BLOSC_ALWAYS_SPLIT * * BLOSC_FORWARD_COMPAT offers reasonably forward compatibility, * BLOSC_AUTO_SPLIT is for nearly optimal results (based on heuristics), * BLOSC_NEVER_SPLIT and BLOSC_ALWAYS_SPLIT are for the user experimenting * when trying to get best compression ratios and/or speed. * * If not called, the default mode is BLOSC_FORWARD_COMPAT_SPLIT. * * This function should always succeed. */ BLOSC_EXPORT void blosc1_set_splitmode(int splitmode); /** * @brief Get the offsets of a frame in a super-chunk. * * @param schunk The super-chunk containing the frame. * * @return If successful, return a pointer to a buffer of the decompressed offsets. * The number of offsets is equal to schunk->nchunks; the user is * responsible to free this buffer. Else, return a NULL value. */ BLOSC_EXPORT int64_t* blosc2_frame_get_offsets(blosc2_schunk *schunk); /********************************************************************* Structures and functions related with compression codecs. *********************************************************************/ typedef int (* blosc2_codec_encoder_cb) (const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_cparams *cparams, const void* chunk); typedef int (* blosc2_codec_decoder_cb) (const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_dparams *dparams, const void* chunk); typedef struct { uint8_t compcode; //!< The codec identifier. char *compname; //!< The codec name. uint8_t complib; //!< The codec library format. uint8_t compver; //!< The codec version. blosc2_codec_encoder_cb encoder; //!< The codec encoder that is used during compression. blosc2_codec_decoder_cb decoder; //!< The codec decoder that is used during decompression. } blosc2_codec; /** * @brief Register locally a user-defined codec in Blosc. * * @param codec The codec to register. * * @return 0 if succeeds. Else a negative code is returned. */ BLOSC_EXPORT int blosc2_register_codec(blosc2_codec *codec); /********************************************************************* Structures and functions related with filters plugins. *********************************************************************/ typedef int (* blosc2_filter_forward_cb) (const uint8_t *, uint8_t *, int32_t, uint8_t, blosc2_cparams *, uint8_t); typedef int (* blosc2_filter_backward_cb) (const uint8_t *, uint8_t *, int32_t, uint8_t, blosc2_dparams *, uint8_t); /** * @brief The parameters for a user-defined filter. */ typedef struct { uint8_t id; //!< The filter identifier. blosc2_filter_forward_cb forward; //!< The filter function that is used during compression. blosc2_filter_backward_cb backward; //!< The filter function that is used during decompression. } blosc2_filter; /** * @brief Register locally a user-defined filter in Blosc. * * @param filter The filter to register. * * @return 0 if succeeds. Else a negative code is returned. */ BLOSC_EXPORT int blosc2_register_filter(blosc2_filter *filter); /********************************************************************* Directory utilities. *********************************************************************/ /* * @brief Remove a directory and its files. */ BLOSC_EXPORT int blosc2_remove_dir(const char *path); /* * @brief Remove a file or a directory given by path. */ BLOSC_EXPORT int blosc2_remove_urlpath(const char *path); /* * @brief Rename a file or a directory given by old_urlpath to new_path. */ BLOSC_EXPORT int blosc2_rename_urlpath(char* old_urlpath, char* new_path); /********************************************************************* Index utilities. *********************************************************************/ /* * @brief Convert a sequential index into a multidimensional index */ BLOSC_EXPORT void blosc2_unidim_to_multidim(uint8_t ndim, int64_t *shape, int64_t i, int64_t *index); /* * @brief Convert a multidimensional index into a sequential index */ BLOSC_EXPORT void blosc2_multidim_to_unidim(const int64_t *index, int8_t ndim, const int64_t *strides, int64_t *i); #ifdef __cplusplus } #endif #endif /* BLOSC2_H */ zmat-0.9.9/src/blosc2/include/b2nd.h0000644000175200007730000004631514515254731017344 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /** @file b2nd.h * @brief Blosc2 NDim header file. * * This file contains Blosc2 NDim public API and the structures needed to use it. * @author Blosc Development team */ #ifndef B2ND_B2ND_H_ #define B2ND_B2ND_H_ #include #include #include #ifdef __cplusplus extern "C" { #endif #include "blosc2/blosc2-export.h" /* The version for metalayer format; starts from 0 and it must not exceed 127 */ #define B2ND_METALAYER_VERSION 0 /* The maximum number of dimensions for b2nd arrays */ #define B2ND_MAX_DIM 8 /* The maximum number of metalayers for b2nd arrays */ #define B2ND_MAX_METALAYERS (BLOSC2_MAX_METALAYERS - 1) /* NumPy dtype format * https://numpy.org/doc/stable/reference/arrays.dtypes.html#arrays-dtypes-constructing */ #define DTYPE_NUMPY_FORMAT 0 /* The default data type */ #define B2ND_DEFAULT_DTYPE "|u1" /* The default data format */ #define B2ND_DEFAULT_DTYPE_FORMAT DTYPE_NUMPY_FORMAT /** * @brief An *optional* cache for a single block. * * When a chunk is needed, it is copied into this cache. In this way, if the same chunk is needed * again afterwards, it is not necessary to recover it because it is already in the cache. */ struct chunk_cache_s { uint8_t *data; //!< The chunk data. int64_t nchunk; //!< The chunk number in cache. If @p nchunk equals to -1, it means that the cache is empty. }; /** * @brief General parameters needed for the creation of a b2nd array. */ typedef struct b2nd_context_s b2nd_context_t; /* opaque type */ /** * @brief A multidimensional array of data that can be compressed. */ typedef struct { blosc2_schunk *sc; //!< Pointer to a Blosc super-chunk int64_t shape[B2ND_MAX_DIM]; //!< Shape of original data. int32_t chunkshape[B2ND_MAX_DIM]; //!< Shape of each chunk. int64_t extshape[B2ND_MAX_DIM]; //!< Shape of padded data. int32_t blockshape[B2ND_MAX_DIM]; //!< Shape of each block. int64_t extchunkshape[B2ND_MAX_DIM]; //!< Shape of padded chunk. int64_t nitems; //!< Number of items in original data. int32_t chunknitems; //!< Number of items in each chunk. int64_t extnitems; //!< Number of items in padded data. int32_t blocknitems; //!< Number of items in each block. int64_t extchunknitems; //!< Number of items in a padded chunk. int8_t ndim; //!< Data dimensions. struct chunk_cache_s chunk_cache; //!< A partition cache. int64_t item_array_strides[B2ND_MAX_DIM]; //!< Item - shape strides. int64_t item_chunk_strides[B2ND_MAX_DIM]; //!< Item - shape strides. int64_t item_extchunk_strides[B2ND_MAX_DIM]; //!< Item - shape strides. int64_t item_block_strides[B2ND_MAX_DIM]; //!< Item - shape strides. int64_t block_chunk_strides[B2ND_MAX_DIM]; //!< Item - shape strides. int64_t chunk_array_strides[B2ND_MAX_DIM]; //!< Item - shape strides. char *dtype; //!< Data type. Different formats can be supported (see dtype_format). int8_t dtype_format; //!< The format of the data type. Default is DTYPE_NUMPY_FORMAT. } b2nd_array_t; /** * @brief Create b2nd params. * * @param b2_storage The Blosc2 storage params. * @param ndim The dimensions. * @param shape The shape. * @param chunkshape The chunk shape. * @param blockshape The block shape. * @param dtype The data type expressed as a string version. * @param dtype_format The data type format; default is DTYPE_NUMPY_FORMAT. * @param metalayers The memory pointer to the list of the metalayers desired. * @param nmetalayers The number of metalayers. * * @return A pointer to the new b2nd params. NULL is returned if this fails. * * @note The pointer returned must be freed when not used anymore with #b2nd_free_ctx. * */ BLOSC_EXPORT b2nd_context_t * b2nd_create_ctx(const blosc2_storage *b2_storage, int8_t ndim, const int64_t *shape, const int32_t *chunkshape, const int32_t *blockshape, const char *dtype, int8_t dtype_format, const blosc2_metalayer *metalayers, int32_t nmetalayers); /** * @brief Free the resources associated with b2nd_context_t. * * @param ctx The b2nd context to free. * * @return An error code. * * @note This is safe in the sense that it will not free the schunk pointer in internal cparams. * */ BLOSC_EXPORT int b2nd_free_ctx(b2nd_context_t *ctx); /** * @brief Create an uninitialized array. * * @param ctx The b2nd context for the new array. * @param array The memory pointer where the array will be created. * * @return An error code. */ BLOSC_EXPORT int b2nd_uninit(b2nd_context_t *ctx, b2nd_array_t **array); /** * @brief Create an empty array. * * @param ctx The b2nd context for the new array. * @param array The memory pointer where the array will be created. * * @return An error code. */ BLOSC_EXPORT int b2nd_empty(b2nd_context_t *ctx, b2nd_array_t **array); /** * Create an array, with zero being used as the default value for * uninitialized portions of the array. * * @param ctx The b2nd context for the new array. * @param array The memory pointer where the array will be created. * * @return An error code. */ BLOSC_EXPORT int b2nd_zeros(b2nd_context_t *ctx, b2nd_array_t **array); /** * Create an array, with @p fill_value being used as the default value for * uninitialized portions of the array. * * @param ctx The b2nd context for the new array. * @param fill_value Default value for uninitialized portions of the array. * @param array The memory pointer where the array will be created. * * @return An error code. */ BLOSC_EXPORT int b2nd_full(b2nd_context_t *ctx, b2nd_array_t **array, const void *fill_value); /** * @brief Free an array. * * @param array The array. * * @return An error code. */ BLOSC_EXPORT int b2nd_free(b2nd_array_t *array); /** * @brief Create a b2nd array from a super-chunk. It can only be used if the array * is backed by a blosc super-chunk. * * @param schunk The blosc super-chunk where the b2nd array is stored. * @param array The memory pointer where the array will be created. * * @return An error code. */ BLOSC_EXPORT int b2nd_from_schunk(blosc2_schunk *schunk, b2nd_array_t **array); /** * Create a serialized super-chunk from a b2nd array. * * @param array The b2nd array to be serialized. * @param cframe The pointer of the buffer where the in-memory array will be copied. * @param cframe_len The length of the in-memory array buffer. * @param needs_free Whether the buffer should be freed or not. * * @return An error code */ BLOSC_EXPORT int b2nd_to_cframe(const b2nd_array_t *array, uint8_t **cframe, int64_t *cframe_len, bool *needs_free); /** * @brief Create a b2nd array from a serialized super-chunk. * * @param cframe The buffer of the in-memory array. * @param cframe_len The size (in bytes) of the in-memory array. * @param copy Whether b2nd should make a copy of the cframe data or not. The copy will be made to an internal sparse frame. * @param array The memory pointer where the array will be created. * * @return An error code. */ BLOSC_EXPORT int b2nd_from_cframe(uint8_t *cframe, int64_t cframe_len, bool copy, b2nd_array_t **array); /** * @brief Open a b2nd array from a file. * * @param urlpath The path of the b2nd array on disk. * @param array The memory pointer where the array info will be stored. * * @return An error code. */ BLOSC_EXPORT int b2nd_open(const char *urlpath, b2nd_array_t **array); /** * @brief Open a b2nd array from a file using an offset. * * @param urlpath The path of the b2nd array on disk. * @param array The memory pointer where the array info will be stored. * @param offset The offset in the file where the b2nd array frame starts. * * @return An error code. */ BLOSC_EXPORT int b2nd_open_offset(const char *urlpath, b2nd_array_t **array, int64_t offset); /** * @brief Save b2nd array into a specific urlpath. * * @param array The array to be saved. * @param urlpath The urlpath where the array will be stored. * * @return An error code. */ BLOSC_EXPORT int b2nd_save(const b2nd_array_t *array, char *urlpath); /** * @brief Create a b2nd array from a C buffer. * * @param ctx The b2nd context for the new array. * @param array The memory pointer where the array will be created. * @param buffer The buffer where source data is stored. * @param buffersize The size (in bytes) of the buffer. * * @return An error code. */ BLOSC_EXPORT int b2nd_from_cbuffer(b2nd_context_t *ctx, b2nd_array_t **array, const void *buffer, int64_t buffersize); /** * @brief Extract the data from a b2nd array into a C buffer. * * @param array The b2nd array. * @param buffer The buffer where the data will be stored. * @param buffersize Size (in bytes) of the buffer. * * @return An error code. */ BLOSC_EXPORT int b2nd_to_cbuffer(const b2nd_array_t *array, void *buffer, int64_t buffersize); /** * @brief Get a slice from an array and store it into a new array. * * @param ctx The b2nd context for the new array. * @param array The memory pointer where the array will be created. * @param src The array from which the slice will be extracted * @param start The coordinates where the slice will begin. * @param stop The coordinates where the slice will end. * * @return An error code. * * @note The ndim and shape from ctx will be overwritten by the src and stop-start respectively. * */ BLOSC_EXPORT int b2nd_get_slice(b2nd_context_t *ctx, b2nd_array_t **array, const b2nd_array_t *src, const int64_t *start, const int64_t *stop); /** * @brief Squeeze a b2nd array * * This function remove selected single-dimensional entries from the shape of a b2nd array. * * @param array The b2nd array. * @param index Indexes of the single-dimensional entries to remove. * * @return An error code */ BLOSC_EXPORT int b2nd_squeeze_index(b2nd_array_t *array, const bool *index); /** * @brief Squeeze a b2nd array * * This function remove single-dimensional entries from the shape of a b2nd array. * * @param array The b2nd array. * * @return An error code */ BLOSC_EXPORT int b2nd_squeeze(b2nd_array_t *array); /** * @brief Get a slice from an array and store it into a C buffer. * * @param array The array from which the slice will be extracted. * @param start The coordinates where the slice will begin. * @param stop The coordinates where the slice will end. * @param buffershape The shape of the buffer. * @param buffer The buffer where the data will be stored. * @param buffersize The size (in bytes) of the buffer. * * @return An error code. */ BLOSC_EXPORT int b2nd_get_slice_cbuffer(const b2nd_array_t *array, const int64_t *start, const int64_t *stop, void *buffer, const int64_t *buffershape, int64_t buffersize); /** * @brief Set a slice in a b2nd array using a C buffer. * * @param buffer The buffer where the slice data is. * @param buffersize The size (in bytes) of the buffer. * @param start The coordinates where the slice will begin. * @param stop The coordinates where the slice will end. * @param buffershape The shape of the buffer. * @param array The b2nd array where the slice will be set * * @return An error code. */ BLOSC_EXPORT int b2nd_set_slice_cbuffer(const void *buffer, const int64_t *buffershape, int64_t buffersize, const int64_t *start, const int64_t *stop, b2nd_array_t *array); /** * @brief Make a copy of the array data. The copy is done into a new b2nd array. * * @param ctx The b2nd context for the new array. * @param src The array from which data is copied. * @param array The memory pointer where the array will be created. * * @return An error code * * @note The ndim and shape in ctx will be overwritten by the src ctx. * */ BLOSC_EXPORT int b2nd_copy(b2nd_context_t *ctx, const b2nd_array_t *src, b2nd_array_t **array); /** * @brief Print metalayer parameters. * * @param array The array where the metalayer is stored. * * @return An error code */ BLOSC_EXPORT int b2nd_print_meta(const b2nd_array_t *array); /** * @brief Resize the shape of an array * * @param array The array to be resized. * @param new_shape The new shape from the array. * @param start The position in which the array will be extended or shrunk. * * @return An error code */ BLOSC_EXPORT int b2nd_resize(b2nd_array_t *array, const int64_t *new_shape, const int64_t *start); /** * @brief Insert given buffer in an array extending the given axis. * * @param array The array to insert the data in. * @param buffer The buffer data to be inserted. * @param buffersize The size (in bytes) of the buffer. * @param axis The axis that will be extended. * @param insert_start The position inside the axis to start inserting the data. * * @return An error code. */ BLOSC_EXPORT int b2nd_insert(b2nd_array_t *array, const void *buffer, int64_t buffersize, int8_t axis, int64_t insert_start); /** * Append a buffer at the end of a b2nd array. * * @param array The array to append the data in. * @param buffer The buffer data to be appended. * @param buffersize Size (in bytes) of the buffer. * @param axis The axis that will be extended to append the data. * * @return An error code. */ BLOSC_EXPORT int b2nd_append(b2nd_array_t *array, const void *buffer, int64_t buffersize, int8_t axis); /** * @brief Delete shrinking the given axis delete_len items. * * @param array The array to shrink. * @param axis The axis to shrink. * @param delete_start The start position from the axis to start deleting chunks. * @param delete_len The number of items to delete to the array->shape[axis]. * The newshape[axis] will be the old array->shape[axis] - delete_len * * @return An error code. * * @note See also b2nd_resize */ BLOSC_EXPORT int b2nd_delete(b2nd_array_t *array, int8_t axis, int64_t delete_start, int64_t delete_len); // Indexing section /** * @brief Get an element selection along each dimension of an array independently. * * @param array The array to get the data from. * @param selection The elements along each dimension. * @param selection_size The size of the selection along each dimension. * @param buffer The buffer for getting the data. * @param buffershape The shape of the buffer. * @param buffersize The buffer size (in bytes). * * @return An error code. * * @note See also b2nd_set_orthogonal_selection. */ BLOSC_EXPORT int b2nd_get_orthogonal_selection(const b2nd_array_t *array, int64_t **selection, int64_t *selection_size, void *buffer, int64_t *buffershape, int64_t buffersize); /** * @brief Set an element selection along each dimension of an array independently. * * @param array The array to set the data to. * @param selection The elements along each dimension. * @param selection_size The size of the selection along each dimension. * @param buffer The buffer with the data for setting. * @param buffershape The shape of the buffer. * @param buffersize The buffer size (in bytes). * * @return An error code. * * @note See also b2nd_get_orthogonal_selection. */ BLOSC_EXPORT int b2nd_set_orthogonal_selection(b2nd_array_t *array, int64_t **selection, int64_t *selection_size, const void *buffer, int64_t *buffershape, int64_t buffersize); /** * @brief Create the metainfo for the b2nd metalayer. * * @param ndim The number of dimensions in the array. * @param shape The shape of the array. * @param chunkshape The shape of the chunks in the array. * @param blockshape The shape of the blocks in the array. * @param dtype A string representation of the data type of the array. * @param dtype_format The format of the dtype representation. 0 means NumPy. * @param smeta The msgpack buffer (output). * * @return An error code. */ BLOSC_EXPORT int b2nd_serialize_meta(int8_t ndim, const int64_t *shape, const int32_t *chunkshape, const int32_t *blockshape, const char *dtype, int8_t dtype_format, uint8_t **smeta); /** * @brief Read the metainfo in the b2nd metalayer. * * @param smeta The msgpack buffer (input). * @param smeta_len The length of the smeta buffer (input). * @param ndim The number of dimensions in the array (output). * @param shape The shape of the array (output). * @param chunkshape The shape of the chunks in the array (output). * @param blockshape The shape of the blocks in the array (output). * @param dtype A string representation of the data type of the array (output). * @param dtype_format The format of the dtype representation (output). 0 means NumPy (the default). * * @note This function is inlined and available even when not linking with libblosc2. * * @return An error code. */ static inline int b2nd_deserialize_meta( const uint8_t *smeta, int32_t smeta_len, int8_t *ndim, int64_t *shape, int32_t *chunkshape, int32_t *blockshape, char **dtype, int8_t *dtype_format) { const uint8_t *pmeta = smeta; // Check that we have an array with 7 entries (version, ndim, shape, chunkshape, blockshape, dtype_format, dtype) pmeta += 1; // version entry // int8_t version = (int8_t)pmeta[0]; // positive fixnum (7-bit positive integer) commented to avoid warning pmeta += 1; // ndim entry *ndim = (int8_t) pmeta[0]; int8_t ndim_aux = *ndim; // positive fixnum (7-bit positive integer) pmeta += 1; // shape entry // Initialize to ones, as required by b2nd for (int i = 0; i < ndim_aux; i++) shape[i] = 1; pmeta += 1; for (int8_t i = 0; i < ndim_aux; i++) { pmeta += 1; swap_store(shape + i, pmeta, sizeof(int64_t)); pmeta += sizeof(int64_t); } // chunkshape entry // Initialize to ones, as required by b2nd for (int i = 0; i < ndim_aux; i++) chunkshape[i] = 1; pmeta += 1; for (int8_t i = 0; i < ndim_aux; i++) { pmeta += 1; swap_store(chunkshape + i, pmeta, sizeof(int32_t)); pmeta += sizeof(int32_t); } // blockshape entry // Initialize to ones, as required by b2nd for (int i = 0; i < ndim_aux; i++) blockshape[i] = 1; pmeta += 1; for (int8_t i = 0; i < ndim_aux; i++) { pmeta += 1; swap_store(blockshape + i, pmeta, sizeof(int32_t)); pmeta += sizeof(int32_t); } // dtype entry if (pmeta - smeta < smeta_len) { // dtype info is here *dtype_format = (int8_t) *(pmeta++); pmeta += 1; int dtype_len; swap_store(&dtype_len, pmeta, sizeof(int32_t)); pmeta += sizeof(int32_t); *dtype = (char*)malloc(dtype_len + 1); char* dtype_ = *dtype; memcpy(dtype_, (char*)pmeta, dtype_len); dtype_[dtype_len] = '\0'; pmeta += dtype_len; } else { // dtype is mandatory in b2nd metalayer, but this is mainly meant as // a fall-back for deprecated caterva headers *dtype = NULL; *dtype_format = 0; } int32_t slen = (int32_t) (pmeta - smeta); return (int)slen; } #ifdef __cplusplus } #endif #endif // B2ND_B2ND_H_ zmat-0.9.9/src/blosc2/Makefile0000644000175200007730000000070114515254731016350 0ustar rlaboissrlaboiss############################################################ # blosc2 makefile ############################################################ BLOSC_SRC=blosc ZSTD_SRC=internal-complibs/zstd LIBBLOSC2=$(BLOSC_SRC)/lib/libblosc2.a export all: lib zstd lib: -$(MAKE) -C $(BLOSC_SRC) lib zstd: -$(MAKE) -C $(ZSTD_SRC) libzstd.a clean: -$(MAKE) -C $(BLOSC_SRC) clean -$(MAKE) -C $(ZSTD_SRC) clean .PHONY: all lib zstd clean .DEFAULT_GOAL := all zmat-0.9.9/src/blosc2/THANKS.rst0000644000175200007730000000350114515254731016433 0ustar rlaboissrlaboissThanks ====== * Valentin Haenel did a terrific work implementing the support for the Snappy compression, fixing typos and improving docs and the plotting script. * Thibault North, with ideas from Oscar Villellas, contributed a way to call Blosc from different threads in a safe way. Christopher Speller introduced contexts so that a global lock is not necessary anymore. * The CMake support was initially contributed by Thibault North, and Antonio Valentino and Mark Wiebe made great enhancements to it. * Christopher Speller also introduced the two new '_ctx' calls to avoid the use of the blosc_init() and blosc_destroy(). * Jack Pappas contributed important portability enhancements, specially runtime and cross-platform detection of SSE2/AVX2 as well as high precision timers (HPET) for the benchmark program. * @littlezhou implemented the AVX2 version of shuffle routines. * Julian Taylor contributed a way to detect AVX2 in runtime and calling the appropriate routines only if the underlying hardware supports it. * Lucian Marc provided the support for ARM/NEON for the shuffle filter. * Jerome Kieffer contributed support for PowerPC/ALTIVEC for the shuffle/bitshuffle filter. * Alberto Sabater, for his great efforts on producing really nice Blosc2 docs, among other aspects. * Kiyo Masui for relicensing his bitshuffle project for allowing the inclusion of part of his code in Blosc. * Aleix Alcacer for his implementation of mutable super-chunks, multiple variable length metalayers and many other things. * Oscar Guiñón for the optimization of reading a (sparse) set of blocks of a chunk in parallel. * Nathan Moinvaziri for his outstanding work on the security side of the things via `fuzzer testing `_. * Marta Iborra for her implementation of sparse storage for persistent super-chunks. zmat-0.9.9/src/blosc2/blosc/0000755000175200007730000000000014603203254016004 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/blosc/shuffle-altivec.h0000644000175200007730000000204114515254731021243 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /* ALTIVEC-accelerated shuffle/unshuffle routines. */ #ifndef SHUFFLE_ALTIVEC_H #define SHUFFLE_ALTIVEC_H #include "blosc2/blosc2-common.h" #ifdef __cplusplus extern "C" { #endif /** ALTIVEC-accelerated shuffle routine. */ BLOSC_NO_EXPORT void shuffle_altivec(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest); /** ALTIVEC-accelerated unshuffle routine. */ BLOSC_NO_EXPORT void unshuffle_altivec(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest); #ifdef __cplusplus } #endif #endif /* SHUFFLE_ALTIVEC_H */ zmat-0.9.9/src/blosc2/blosc/config.h0000644000175200007730000000035514515254731017435 0ustar rlaboissrlaboiss#ifndef _CONFIGURATION_HEADER_GUARD_H_ #define _CONFIGURATION_HEADER_GUARD_H_ #define HAVE_ZLIB TRUE #define HAVE_ZLIB_NG TRUE #define HAVE_ZSTD TRUE /* #undef HAVE_IPP */ /* #undef BLOSC_DLL_EXPORT */ #define HAVE_PLUGINS TRUE #endif zmat-0.9.9/src/blosc2/blosc/cmake_install.cmake0000644000175200007730000000322614515254731021627 0ustar rlaboissrlaboiss# Install script for directory: /home/fangq/space/git/Project/github/zmat/src/blosc2/blosc # Set the install prefix if(NOT DEFINED CMAKE_INSTALL_PREFIX) set(CMAKE_INSTALL_PREFIX "/usr/local") endif() string(REGEX REPLACE "/$" "" CMAKE_INSTALL_PREFIX "${CMAKE_INSTALL_PREFIX}") # Set the install configuration name. if(NOT DEFINED CMAKE_INSTALL_CONFIG_NAME) if(BUILD_TYPE) string(REGEX REPLACE "^[^A-Za-z0-9_]+" "" CMAKE_INSTALL_CONFIG_NAME "${BUILD_TYPE}") else() set(CMAKE_INSTALL_CONFIG_NAME "") endif() message(STATUS "Install configuration: \"${CMAKE_INSTALL_CONFIG_NAME}\"") endif() # Set the component getting installed. if(NOT CMAKE_INSTALL_COMPONENT) if(COMPONENT) message(STATUS "Install component: \"${COMPONENT}\"") set(CMAKE_INSTALL_COMPONENT "${COMPONENT}") else() set(CMAKE_INSTALL_COMPONENT) endif() endif() # Install shared libraries without execute permission? if(NOT DEFINED CMAKE_INSTALL_SO_NO_EXE) set(CMAKE_INSTALL_SO_NO_EXE "1") endif() # Is this installation the result of a crosscompile? if(NOT DEFINED CMAKE_CROSSCOMPILING) set(CMAKE_CROSSCOMPILING "FALSE") endif() # Set default install directory permissions. if(NOT DEFINED CMAKE_OBJDUMP) set(CMAKE_OBJDUMP "/usr/bin/objdump") endif() if(CMAKE_INSTALL_COMPONENT) set(CMAKE_INSTALL_MANIFEST "install_manifest_${CMAKE_INSTALL_COMPONENT}.txt") else() set(CMAKE_INSTALL_MANIFEST "install_manifest.txt") endif() string(REPLACE ";" "\n" CMAKE_INSTALL_MANIFEST_CONTENT "${CMAKE_INSTALL_MANIFEST_FILES}") file(WRITE "/home/fangq/space/git/Project/github/zmat/src/blosc2/blosc/${CMAKE_INSTALL_MANIFEST}" "${CMAKE_INSTALL_MANIFEST_CONTENT}") zmat-0.9.9/src/blosc2/blosc/shuffle-sse2.h0000644000175200007730000000200314515254731020466 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /* SSE2-accelerated shuffle/unshuffle routines. */ #ifndef SHUFFLE_SSE2_H #define SHUFFLE_SSE2_H #include "blosc2/blosc2-common.h" #ifdef __cplusplus extern "C" { #endif /** SSE2-accelerated shuffle routine. */ BLOSC_NO_EXPORT void shuffle_sse2(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest); /** SSE2-accelerated unshuffle routine. */ BLOSC_NO_EXPORT void unshuffle_sse2(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest); #ifdef __cplusplus } #endif #endif /* SHUFFLE_SSE2_H */ zmat-0.9.9/src/blosc2/blosc/schunk.c0000644000175200007730000015533514515254731017467 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include #include #include #include #include "blosc2.h" #include "frame.h" #include "stune.h" #include #include "blosc-private.h" #if defined(_WIN32) #include #include #include #define mkdir(D, M) _mkdir(D) /* stdint.h only available in VS2010 (VC++ 16.0) and newer */ #if defined(_MSC_VER) && _MSC_VER < 1600 #include "win32/stdint-windows.h" #else #include #endif #endif /* _WIN32 */ /* If C11 is supported, use it's built-in aligned allocation. */ #if __STDC_VERSION__ >= 201112L #include #endif /* Get the cparams associated with a super-chunk */ int blosc2_schunk_get_cparams(blosc2_schunk *schunk, blosc2_cparams **cparams) { *cparams = calloc(sizeof(blosc2_cparams), 1); (*cparams)->schunk = schunk; for (int i = 0; i < BLOSC2_MAX_FILTERS; i++) { (*cparams)->filters[i] = schunk->filters[i]; (*cparams)->filters_meta[i] = schunk->filters_meta[i]; } (*cparams)->compcode = schunk->compcode; (*cparams)->compcode_meta = schunk->compcode_meta; (*cparams)->clevel = schunk->clevel; (*cparams)->typesize = schunk->typesize; (*cparams)->blocksize = schunk->blocksize; (*cparams)->splitmode = schunk->splitmode; if (schunk->cctx == NULL) { (*cparams)->nthreads = blosc2_get_nthreads(); } else { (*cparams)->nthreads = (int16_t)schunk->cctx->nthreads; } return 0; } /* Get the dparams associated with a super-chunk */ int blosc2_schunk_get_dparams(blosc2_schunk *schunk, blosc2_dparams **dparams) { *dparams = calloc(sizeof(blosc2_dparams), 1); (*dparams)->schunk = schunk; if (schunk->dctx == NULL) { (*dparams)->nthreads = blosc2_get_nthreads(); } else { (*dparams)->nthreads = schunk->dctx->nthreads; } return 0; } void update_schunk_properties(struct blosc2_schunk* schunk) { blosc2_cparams* cparams = schunk->storage->cparams; blosc2_dparams* dparams = schunk->storage->dparams; for (int i = 0; i < BLOSC2_MAX_FILTERS; i++) { schunk->filters[i] = cparams->filters[i]; schunk->filters_meta[i] = cparams->filters_meta[i]; } schunk->compcode = cparams->compcode; schunk->compcode_meta = cparams->compcode_meta; schunk->clevel = cparams->clevel; schunk->splitmode = cparams->splitmode; schunk->typesize = cparams->typesize; schunk->blocksize = cparams->blocksize; schunk->chunksize = -1; /* The compression context */ if (schunk->cctx != NULL) { blosc2_free_ctx(schunk->cctx); } cparams->schunk = schunk; schunk->cctx = blosc2_create_cctx(*cparams); /* The decompression context */ if (schunk->dctx != NULL) { blosc2_free_ctx(schunk->dctx); } dparams->schunk = schunk; schunk->dctx = blosc2_create_dctx(*dparams); } static bool file_exists (char *filename) { struct stat buffer; return (stat (filename, &buffer) == 0); } /* Create a new super-chunk */ blosc2_schunk* blosc2_schunk_new(blosc2_storage *storage) { blosc2_schunk* schunk = calloc(1, sizeof(blosc2_schunk)); schunk->version = 0; /* pre-first version */ // Get the storage with proper defaults schunk->storage = get_new_storage(storage, &BLOSC2_CPARAMS_DEFAULTS, &BLOSC2_DPARAMS_DEFAULTS, &BLOSC2_IO_DEFAULTS); // Update the (local variable) storage storage = schunk->storage; schunk->udbtune = malloc(sizeof(blosc2_btune)); if (schunk->storage->cparams->udbtune == NULL) { memcpy(schunk->udbtune, &BTUNE_DEFAULTS, sizeof(blosc2_btune)); } else { memcpy(schunk->udbtune, schunk->storage->cparams->udbtune, sizeof(blosc2_btune)); } schunk->storage->cparams->udbtune = schunk->udbtune; // ...and update internal properties update_schunk_properties(schunk); schunk->cctx->udbtune->btune_init(schunk->udbtune->btune_config, schunk->cctx, schunk->dctx); if (!storage->contiguous && storage->urlpath != NULL){ char* urlpath; char last_char = storage->urlpath[strlen(storage->urlpath) - 1]; urlpath = malloc(strlen(storage->urlpath) + 1); strcpy(urlpath, storage->urlpath); if (last_char == '\\' || last_char == '/') { urlpath[strlen(storage->urlpath) - 1] = '\0'; } // Create directory if (mkdir(urlpath, 0777) == -1) { BLOSC_TRACE_ERROR("Error during the creation of the directory, maybe it already exists."); return NULL; } // We want a sparse (directory) frame as storage blosc2_frame_s* frame = frame_new(urlpath); free(urlpath); frame->sframe = true; // Initialize frame (basically, encode the header) frame->schunk = schunk; int64_t frame_len = frame_from_schunk(schunk, frame); if (frame_len < 0) { BLOSC_TRACE_ERROR("Error during the conversion of schunk to frame."); return NULL; } schunk->frame = (blosc2_frame*)frame; } if (storage->contiguous){ // We want a contiguous frame as storage if (storage->urlpath != NULL) { if (file_exists(storage->urlpath)) { BLOSC_TRACE_ERROR("You are trying to overwrite an existing frame. Remove it first!"); return NULL; } } blosc2_frame_s* frame = frame_new(storage->urlpath); frame->sframe = false; // Initialize frame (basically, encode the header) frame->schunk = schunk; int64_t frame_len = frame_from_schunk(schunk, frame); if (frame_len < 0) { BLOSC_TRACE_ERROR("Error during the conversion of schunk to frame."); return NULL; } schunk->frame = (blosc2_frame*)frame; } return schunk; } /* Create a copy of a super-chunk */ blosc2_schunk* blosc2_schunk_copy(blosc2_schunk *schunk, blosc2_storage *storage) { if (schunk == NULL) { BLOSC_TRACE_ERROR("Can not copy a NULL `schunk`."); return NULL; } // Check if cparams are equals bool cparams_equal = true; blosc2_cparams cparams = {0}; if (storage->cparams == NULL) { // When cparams are not specified, just use the same of schunk cparams.typesize = schunk->cctx->typesize; cparams.clevel = schunk->cctx->clevel; cparams.compcode = schunk->cctx->compcode; cparams.compcode_meta = schunk->cctx->compcode_meta; cparams.splitmode = schunk->cctx->splitmode; cparams.use_dict = schunk->cctx->use_dict; cparams.blocksize = schunk->cctx->blocksize; memcpy(cparams.filters, schunk->cctx->filters, BLOSC2_MAX_FILTERS); memcpy(cparams.filters_meta, schunk->cctx->filters_meta, BLOSC2_MAX_FILTERS); storage->cparams = &cparams; } else { cparams = *storage->cparams; } if (cparams.blocksize == 0) { // TODO: blocksize should be read from schunk->blocksize // For this, it should be updated during the first append // (or change API to make this a property during schunk creation). cparams.blocksize = schunk->cctx->blocksize; } if (cparams.typesize != schunk->cctx->typesize || cparams.clevel != schunk->cctx->clevel || cparams.compcode != schunk->cctx->compcode || cparams.use_dict != schunk->cctx->use_dict || cparams.blocksize != schunk->cctx->blocksize || // In case of prefilters or postfilters, force their execution. schunk->cctx->prefilter != NULL || schunk->dctx->postfilter != NULL) { cparams_equal = false; } for (int i = 0; i < BLOSC2_MAX_FILTERS; ++i) { if (cparams.filters[i] != schunk->cctx->filters[i] || cparams.filters_meta[i] != schunk->cctx->filters_meta[i]) { cparams_equal = false; } } // Create new schunk blosc2_schunk *new_schunk = blosc2_schunk_new(storage); if (new_schunk == NULL) { BLOSC_TRACE_ERROR("Can not create a new schunk"); return NULL; } // Copy metalayers for (int nmeta = 0; nmeta < schunk->nmetalayers; ++nmeta) { blosc2_metalayer *meta = schunk->metalayers[nmeta]; if (blosc2_meta_add(new_schunk, meta->name, meta->content, meta->content_len) < 0) { BLOSC_TRACE_ERROR("Can not add %s `metalayer`.", meta->name); return NULL; } } // Copy chunks if (cparams_equal) { for (int nchunk = 0; nchunk < schunk->nchunks; ++nchunk) { uint8_t *chunk; bool needs_free; if (blosc2_schunk_get_chunk(schunk, nchunk, &chunk, &needs_free) < 0) { BLOSC_TRACE_ERROR("Can not get the `chunk` %d.", nchunk); return NULL; } if (blosc2_schunk_append_chunk(new_schunk, chunk, !needs_free) < 0) { BLOSC_TRACE_ERROR("Can not append the `chunk` into super-chunk."); return NULL; } } } else { int32_t chunksize = schunk->chunksize == -1 ? 0 : schunk->chunksize; uint8_t *buffer = malloc(chunksize); for (int nchunk = 0; nchunk < schunk->nchunks; ++nchunk) { if (blosc2_schunk_decompress_chunk(schunk, nchunk, buffer, schunk->chunksize) < 0) { BLOSC_TRACE_ERROR("Can not decompress the `chunk` %d.", nchunk); return NULL; } if (blosc2_schunk_append_buffer(new_schunk, buffer, schunk->chunksize) < 0) { BLOSC_TRACE_ERROR("Can not append the `buffer` into super-chunk."); return NULL; } } free(buffer); } // Copy vlmetalayers for (int nmeta = 0; nmeta < schunk->nvlmetalayers; ++nmeta) { uint8_t *content; int32_t content_len; char* name = schunk->vlmetalayers[nmeta]->name; if (blosc2_vlmeta_get(schunk, name, &content, &content_len) < 0) { BLOSC_TRACE_ERROR("Can not get %s `vlmetalayer`.", name); } if (blosc2_vlmeta_add(new_schunk, name, content, content_len, NULL) < 0) { BLOSC_TRACE_ERROR("Can not add %s `vlmetalayer`.", name); return NULL; } free(content); } return new_schunk; } /* Open an existing super-chunk that is on-disk (no copy is made). */ blosc2_schunk* blosc2_schunk_open_udio(const char* urlpath, const blosc2_io *udio) { if (urlpath == NULL) { BLOSC_TRACE_ERROR("You need to supply a urlpath."); return NULL; } blosc2_frame_s* frame = frame_from_file_offset(urlpath, udio, 0); if (frame == NULL) { return NULL; } blosc2_schunk* schunk = frame_to_schunk(frame, false, udio); // Set the storage with proper defaults size_t pathlen = strlen(urlpath); schunk->storage->urlpath = malloc(pathlen + 1); strcpy(schunk->storage->urlpath, urlpath); schunk->storage->contiguous = !frame->sframe; return schunk; } blosc2_schunk* blosc2_schunk_open(const char* urlpath) { return blosc2_schunk_open_udio(urlpath, &BLOSC2_IO_DEFAULTS); } BLOSC_EXPORT blosc2_schunk* blosc2_schunk_open_offset(const char* urlpath, int64_t offset) { if (urlpath == NULL) { BLOSC_TRACE_ERROR("You need to supply a urlpath."); return NULL; } blosc2_frame_s* frame = frame_from_file_offset(urlpath, &BLOSC2_IO_DEFAULTS, offset); if (frame == NULL) { return NULL; } blosc2_schunk* schunk = frame_to_schunk(frame, false, &BLOSC2_IO_DEFAULTS); // Set the storage with proper defaults size_t pathlen = strlen(urlpath); schunk->storage->urlpath = malloc(pathlen + 1); strcpy(schunk->storage->urlpath, urlpath); schunk->storage->contiguous = !frame->sframe; return schunk; } int64_t blosc2_schunk_to_buffer(blosc2_schunk* schunk, uint8_t** dest, bool* needs_free) { blosc2_frame_s* frame; int64_t cframe_len; // Initialize defaults in case of errors *dest = NULL; *needs_free = false; if ((schunk->storage->contiguous == true) && (schunk->storage->urlpath == NULL)) { frame = (blosc2_frame_s*)(schunk->frame); *dest = frame->cframe; cframe_len = frame->len; *needs_free = false; } else { // Copy to a contiguous storage blosc2_storage frame_storage = {.contiguous=true}; blosc2_schunk* schunk_copy = blosc2_schunk_copy(schunk, &frame_storage); if (schunk_copy == NULL) { BLOSC_TRACE_ERROR("Error during the conversion of schunk to buffer."); return BLOSC2_ERROR_SCHUNK_COPY; } frame = (blosc2_frame_s*)(schunk_copy->frame); *dest = frame->cframe; cframe_len = frame->len; *needs_free = true; frame->avoid_cframe_free = true; blosc2_schunk_free(schunk_copy); } return cframe_len; } /* Write an in-memory frame out to a file. */ int64_t frame_to_file(blosc2_frame_s* frame, const char* urlpath) { blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } void* fp = io_cb->open(urlpath, "wb", frame->schunk->storage->io); int64_t nitems = io_cb->write(frame->cframe, frame->len, 1, fp); io_cb->close(fp); return nitems * frame->len; } /* Append an in-memory frame to a file. */ int64_t append_frame_to_file(blosc2_frame_s* frame, const char* urlpath) { blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } void* fp = io_cb->open(urlpath, "ab", frame->schunk->storage->io); int64_t offset; # if (UNIX) offset = io_cb->tell(fp); # else io_cb->seek(fp, 0, SEEK_END); offset = io_cb->tell(fp); # endif io_cb->write(frame->cframe, frame->len, 1, fp); io_cb->close(fp); return offset; } /* Write super-chunk out to a file. */ int64_t blosc2_schunk_to_file(blosc2_schunk* schunk, const char* urlpath) { if (urlpath == NULL) { BLOSC_TRACE_ERROR("urlpath cannot be NULL"); return BLOSC2_ERROR_INVALID_PARAM; } // Accelerated path for in-memory frames if (schunk->storage->contiguous && schunk->storage->urlpath == NULL) { int64_t len = frame_to_file((blosc2_frame_s*)(schunk->frame), urlpath); if (len <= 0) { BLOSC_TRACE_ERROR("Error writing to file"); return len; } return len; } // Copy to a contiguous file blosc2_storage frame_storage = {.contiguous=true, .urlpath=(char*)urlpath}; blosc2_schunk* schunk_copy = blosc2_schunk_copy(schunk, &frame_storage); if (schunk_copy == NULL) { BLOSC_TRACE_ERROR("Error during the conversion of schunk to buffer."); return BLOSC2_ERROR_SCHUNK_COPY; } blosc2_frame_s* frame = (blosc2_frame_s*)(schunk_copy->frame); int64_t frame_len = frame->len; blosc2_schunk_free(schunk_copy); return frame_len; } /* Append a super-chunk to a file. */ int64_t blosc2_schunk_append_file(blosc2_schunk* schunk, const char* urlpath) { if (urlpath == NULL) { BLOSC_TRACE_ERROR("urlpath cannot be NULL"); return BLOSC2_ERROR_INVALID_PARAM; } // Accelerated path for in-memory frames if (schunk->storage->contiguous && schunk->storage->urlpath == NULL) { int64_t offset = append_frame_to_file((blosc2_frame_s*)(schunk->frame), urlpath); if (offset <= 0) { BLOSC_TRACE_ERROR("Error writing to file"); return offset; } return offset; } // Copy to a contiguous file blosc2_storage frame_storage = {.contiguous=true, .urlpath=NULL}; blosc2_schunk* schunk_copy = blosc2_schunk_copy(schunk, &frame_storage); if (schunk_copy == NULL) { BLOSC_TRACE_ERROR("Error during the conversion of schunk to buffer."); return BLOSC2_ERROR_SCHUNK_COPY; } blosc2_frame_s* frame = (blosc2_frame_s*)(schunk_copy->frame); int64_t offset = append_frame_to_file(frame, urlpath); blosc2_schunk_free(schunk_copy); return offset; } /* Free all memory from a super-chunk. */ int blosc2_schunk_free(blosc2_schunk *schunk) { if (schunk->data != NULL) { for (int i = 0; i < schunk->nchunks; i++) { free(schunk->data[i]); } free(schunk->data); } if (schunk->cctx != NULL) blosc2_free_ctx(schunk->cctx); if (schunk->dctx != NULL) blosc2_free_ctx(schunk->dctx); if (schunk->blockshape != NULL) free(schunk->blockshape); if (schunk->nmetalayers > 0) { for (int i = 0; i < schunk->nmetalayers; i++) { if (schunk->metalayers[i] != NULL) { if (schunk->metalayers[i]->name != NULL) free(schunk->metalayers[i]->name); if (schunk->metalayers[i]->content != NULL) free(schunk->metalayers[i]->content); free(schunk->metalayers[i]); } } schunk->nmetalayers = 0; } if (schunk->storage != NULL) { if (schunk->storage->urlpath != NULL) { free(schunk->storage->urlpath); } free(schunk->storage->cparams); free(schunk->storage->dparams); free(schunk->storage->io); free(schunk->storage); } if (schunk->frame != NULL) { frame_free((blosc2_frame_s *) schunk->frame); } if (schunk->nvlmetalayers > 0) { for (int i = 0; i < schunk->nvlmetalayers; ++i) { if (schunk->vlmetalayers[i] != NULL) { if (schunk->vlmetalayers[i]->name != NULL) free(schunk->vlmetalayers[i]->name); if (schunk->vlmetalayers[i]->content != NULL) free(schunk->vlmetalayers[i]->content); free(schunk->vlmetalayers[i]); } } } if (schunk->udbtune != NULL) { free(schunk->udbtune); } free(schunk); return 0; } /* Create a super-chunk out of a contiguous frame buffer */ blosc2_schunk* blosc2_schunk_from_buffer(uint8_t *cframe, int64_t len, bool copy) { blosc2_frame_s* frame = frame_from_cframe(cframe, len, false); if (frame == NULL) { return NULL; } // Check that the buffer actually comes from a cframe char *magic_number = (char *)cframe; magic_number += FRAME_HEADER_MAGIC; if (strcmp(magic_number, "b2frame\0") != 0) { return NULL; } blosc2_schunk* schunk = frame_to_schunk(frame, copy, &BLOSC2_IO_DEFAULTS); if (schunk && copy) { // Super-chunk has its own copy of frame frame_free(frame); } return schunk; } /* Create a super-chunk out of a contiguous frame buffer */ void blosc2_schunk_avoid_cframe_free(blosc2_schunk *schunk, bool avoid_cframe_free) { blosc2_frame_s* frame = (blosc2_frame_s*)schunk->frame; frame_avoid_cframe_free(frame, avoid_cframe_free); } /* Fill an empty frame with special values (fast path). */ int64_t blosc2_schunk_fill_special(blosc2_schunk* schunk, int64_t nitems, int special_value, int32_t chunksize) { if (nitems == 0) { return 0; } int32_t typesize = schunk->typesize; if ((nitems * typesize / chunksize) > INT_MAX) { BLOSC_TRACE_ERROR("nitems is too large. Try increasing the chunksize."); return BLOSC2_ERROR_SCHUNK_SPECIAL; } if ((schunk->nbytes > 0) || (schunk->cbytes > 0)) { BLOSC_TRACE_ERROR("Filling with special values only works on empty super-chunks"); return BLOSC2_ERROR_FRAME_SPECIAL; } // Compute the number of chunks and the length of the offsets chunk int32_t chunkitems = chunksize / typesize; int64_t nchunks = nitems / chunkitems; int32_t leftover_items = (int32_t)(nitems % chunkitems); if (schunk->frame == NULL) { // Build the special chunks int32_t leftover_size = leftover_items * typesize; void* chunk = malloc(BLOSC_EXTENDED_HEADER_LENGTH); void* chunk2 = malloc(BLOSC_EXTENDED_HEADER_LENGTH); blosc2_cparams* cparams; blosc2_schunk_get_cparams(schunk, &cparams); int csize, csize2; switch (special_value) { case BLOSC2_SPECIAL_ZERO: csize = blosc2_chunk_zeros(*cparams, chunksize, chunk, BLOSC_EXTENDED_HEADER_LENGTH); csize2 = blosc2_chunk_zeros(*cparams, leftover_size, chunk2, BLOSC_EXTENDED_HEADER_LENGTH); break; case BLOSC2_SPECIAL_UNINIT: csize = blosc2_chunk_uninit(*cparams, chunksize, chunk, BLOSC_EXTENDED_HEADER_LENGTH); csize2 = blosc2_chunk_uninit(*cparams, leftover_size, chunk2, BLOSC_EXTENDED_HEADER_LENGTH); break; case BLOSC2_SPECIAL_NAN: csize = blosc2_chunk_nans(*cparams, chunksize, chunk, BLOSC_EXTENDED_HEADER_LENGTH); csize2 = blosc2_chunk_nans(*cparams, leftover_size, chunk2, BLOSC_EXTENDED_HEADER_LENGTH); break; default: BLOSC_TRACE_ERROR("Only zeros, NaNs or non-initialized values are supported."); return BLOSC2_ERROR_SCHUNK_SPECIAL; } free(cparams); if (csize < 0 || csize2 < 0) { BLOSC_TRACE_ERROR("Error creating special chunks."); return BLOSC2_ERROR_SCHUNK_SPECIAL; } for (int nchunk = 0; nchunk < nchunks; nchunk++) { int64_t nchunk_ = blosc2_schunk_append_chunk(schunk, chunk, true); if (nchunk_ != nchunk + 1) { BLOSC_TRACE_ERROR("Error appending special chunks."); return BLOSC2_ERROR_SCHUNK_SPECIAL; } } if (leftover_items) { int64_t nchunk_ = blosc2_schunk_append_chunk(schunk, chunk2, true); if (nchunk_ != nchunks + 1) { BLOSC_TRACE_ERROR("Error appending last special chunk."); return BLOSC2_ERROR_SCHUNK_SPECIAL; } } free(chunk); free(chunk2); } else { /* Fill an empty frame with special values (fast path). */ blosc2_frame_s *frame = (blosc2_frame_s *) schunk->frame; /* Update counters (necessary for the frame_fill_special() logic) */ if (leftover_items) { nchunks += 1; } schunk->chunksize = chunksize; schunk->nchunks = nchunks; schunk->nbytes = nitems * typesize; int64_t frame_len = frame_fill_special(frame, nitems, special_value, chunksize, schunk); if (frame_len < 0) { BLOSC_TRACE_ERROR("Error creating special frame."); return frame_len; } } return schunk->nchunks; } /* Append an existing chunk into a super-chunk. */ int64_t blosc2_schunk_append_chunk(blosc2_schunk *schunk, uint8_t *chunk, bool copy) { int32_t chunk_nbytes; int32_t chunk_cbytes; int64_t nchunks = schunk->nchunks; int rc = blosc2_cbuffer_sizes(chunk, &chunk_nbytes, &chunk_cbytes, NULL); if (rc < 0) { return rc; } if (schunk->chunksize == -1) { schunk->chunksize = chunk_nbytes; // The super-chunk is initialized now } if (chunk_nbytes > schunk->chunksize) { BLOSC_TRACE_ERROR("Appending chunks that have different lengths in the same schunk " "is not supported yet: %d > %d.", chunk_nbytes, schunk->chunksize); return BLOSC2_ERROR_CHUNK_APPEND; } /* Update counters */ schunk->current_nchunk = nchunks; schunk->nchunks = nchunks + 1; schunk->nbytes += chunk_nbytes; if (schunk->frame == NULL) { schunk->cbytes += chunk_cbytes; } else { // A frame int special_value = (chunk[BLOSC2_CHUNK_BLOSC2_FLAGS] >> 4) & BLOSC2_SPECIAL_MASK; switch (special_value) { case BLOSC2_SPECIAL_ZERO: case BLOSC2_SPECIAL_NAN: case BLOSC2_SPECIAL_UNINIT: schunk->cbytes += 0; break; default: schunk->cbytes += chunk_cbytes; } } if (copy) { // Make a copy of the chunk uint8_t *chunk_copy = malloc(chunk_cbytes); memcpy(chunk_copy, chunk, chunk_cbytes); chunk = chunk_copy; } // Update super-chunk or frame blosc2_frame_s* frame = (blosc2_frame_s*)schunk->frame; if (frame == NULL) { // Check that we are not appending a small chunk after another small chunk if ((schunk->nchunks > 1) && (chunk_nbytes < schunk->chunksize)) { uint8_t* last_chunk = schunk->data[nchunks - 1]; int32_t last_nbytes; rc = blosc2_cbuffer_sizes(last_chunk, &last_nbytes, NULL, NULL); if (rc < 0) { return rc; } if ((last_nbytes < schunk->chunksize) && (chunk_nbytes < schunk->chunksize)) { BLOSC_TRACE_ERROR( "Appending two consecutive chunks with a chunksize smaller than the schunk chunksize " "is not allowed yet: %d != %d.", chunk_nbytes, schunk->chunksize); return BLOSC2_ERROR_CHUNK_APPEND; } } if (!copy && (chunk_cbytes < chunk_nbytes)) { // We still want to do a shrink of the chunk chunk = realloc(chunk, chunk_cbytes); } /* Make space for appending the copy of the chunk and do it */ if ((nchunks + 1) * sizeof(void *) > schunk->data_len) { // Extend the data pointer by one memory page (4k) schunk->data_len += 4096; // must be a multiple of sizeof(void*) schunk->data = realloc(schunk->data, schunk->data_len); } schunk->data[nchunks] = chunk; } else { if (frame_append_chunk(frame, chunk, schunk) == NULL) { BLOSC_TRACE_ERROR("Problems appending a chunk."); return BLOSC2_ERROR_CHUNK_APPEND; } } return schunk->nchunks; } /* Insert an existing @p chunk in a specified position on a super-chunk */ int64_t blosc2_schunk_insert_chunk(blosc2_schunk *schunk, int64_t nchunk, uint8_t *chunk, bool copy) { int32_t chunk_nbytes; int32_t chunk_cbytes; int64_t nchunks = schunk->nchunks; int rc = blosc2_cbuffer_sizes(chunk, &chunk_nbytes, &chunk_cbytes, NULL); if (rc < 0) { return rc; } if (schunk->chunksize == -1) { schunk->chunksize = chunk_nbytes; // The super-chunk is initialized now } if (chunk_nbytes > schunk->chunksize) { BLOSC_TRACE_ERROR("Inserting chunks that have different lengths in the same schunk " "is not supported yet: %d > %d.", chunk_nbytes, schunk->chunksize); return BLOSC2_ERROR_CHUNK_INSERT; } /* Update counters */ schunk->current_nchunk = nchunk; schunk->nchunks = nchunks + 1; schunk->nbytes += chunk_nbytes; if (schunk->frame == NULL) { schunk->cbytes += chunk_cbytes; } else { // A frame int special_value = (chunk[BLOSC2_CHUNK_BLOSC2_FLAGS] >> 4) & BLOSC2_SPECIAL_MASK; switch (special_value) { case BLOSC2_SPECIAL_ZERO: case BLOSC2_SPECIAL_NAN: case BLOSC2_SPECIAL_UNINIT: schunk->cbytes += 0; break; default: schunk->cbytes += chunk_cbytes; } } if (copy) { // Make a copy of the chunk uint8_t *chunk_copy = malloc(chunk_cbytes); memcpy(chunk_copy, chunk, chunk_cbytes); chunk = chunk_copy; } // Update super-chunk or frame blosc2_frame_s* frame = (blosc2_frame_s*)schunk->frame; if (frame == NULL) { // Check that we are not appending a small chunk after another small chunk if ((schunk->nchunks > 0) && (chunk_nbytes < schunk->chunksize)) { uint8_t* last_chunk = schunk->data[nchunks - 1]; int32_t last_nbytes; rc = blosc2_cbuffer_sizes(last_chunk, &last_nbytes, NULL, NULL); if (rc < 0) { return rc; } if ((last_nbytes < schunk->chunksize) && (chunk_nbytes < schunk->chunksize)) { BLOSC_TRACE_ERROR("Appending two consecutive chunks with a chunksize smaller " "than the schunk chunksize is not allowed yet: %d != %d", chunk_nbytes, schunk->chunksize); return BLOSC2_ERROR_CHUNK_APPEND; } } if (!copy && (chunk_cbytes < chunk_nbytes)) { // We still want to do a shrink of the chunk chunk = realloc(chunk, chunk_cbytes); } // Make space for appending the copy of the chunk and do it if ((nchunks + 1) * sizeof(void *) > schunk->data_len) { // Extend the data pointer by one memory page (4k) schunk->data_len += 4096; // must be a multiple of sizeof(void*) schunk->data = realloc(schunk->data, schunk->data_len); } // Reorder the offsets and insert the new chunk for (int64_t i = nchunks; i > nchunk; --i) { schunk->data[i] = schunk->data[i-1]; } schunk->data[nchunk] = chunk; } else { if (frame_insert_chunk(frame, nchunk, chunk, schunk) == NULL) { BLOSC_TRACE_ERROR("Problems inserting a chunk in a frame."); return BLOSC2_ERROR_CHUNK_INSERT; } } return schunk->nchunks; } int64_t blosc2_schunk_update_chunk(blosc2_schunk *schunk, int64_t nchunk, uint8_t *chunk, bool copy) { int32_t chunk_nbytes; int32_t chunk_cbytes; int rc = blosc2_cbuffer_sizes(chunk, &chunk_nbytes, &chunk_cbytes, NULL); if (rc < 0) { return rc; } if (schunk->chunksize == -1) { schunk->chunksize = chunk_nbytes; // The super-chunk is initialized now } if (schunk->chunksize != 0 && (chunk_nbytes > schunk->chunksize || (chunk_nbytes < schunk->chunksize && nchunk != schunk->nchunks - 1))) { BLOSC_TRACE_ERROR("Updating chunks that have different lengths in the same schunk " "is not supported yet (unless it's the last one and smaller):" " %d > %d.", chunk_nbytes, schunk->chunksize); return BLOSC2_ERROR_CHUNK_UPDATE; } bool needs_free; uint8_t *chunk_old; int err = blosc2_schunk_get_chunk(schunk, nchunk, &chunk_old, &needs_free); if (err < 0) { BLOSC_TRACE_ERROR("%" PRId64 " chunk can not be obtained from schunk.", nchunk); return -1; } int32_t chunk_nbytes_old = 0; int32_t chunk_cbytes_old = 0; schunk->current_nchunk = nchunk; if (chunk_old != 0) { rc = blosc2_cbuffer_sizes(chunk_old, &chunk_nbytes_old, &chunk_cbytes_old, NULL); if (rc < 0) { return rc; } if (chunk_cbytes_old == BLOSC2_MAX_OVERHEAD) { chunk_cbytes_old = 0; } } if (needs_free) { free(chunk_old); } if (copy) { // Make a copy of the chunk uint8_t *chunk_copy = malloc(chunk_cbytes); memcpy(chunk_copy, chunk, chunk_cbytes); chunk = chunk_copy; } blosc2_frame_s* frame = (blosc2_frame_s*)(schunk->frame); if (schunk->frame == NULL) { /* Update counters */ schunk->nbytes += chunk_nbytes; schunk->nbytes -= chunk_nbytes_old; schunk->cbytes += chunk_cbytes; schunk->cbytes -= chunk_cbytes_old; } else { // A frame int special_value = (chunk[BLOSC2_CHUNK_BLOSC2_FLAGS] >> 4) & BLOSC2_SPECIAL_MASK; switch (special_value) { case BLOSC2_SPECIAL_ZERO: case BLOSC2_SPECIAL_NAN: case BLOSC2_SPECIAL_UNINIT: schunk->nbytes += chunk_nbytes; schunk->nbytes -= chunk_nbytes_old; if (frame->sframe) { schunk->cbytes -= chunk_cbytes_old; } break; default: /* Update counters */ schunk->nbytes += chunk_nbytes; schunk->nbytes -= chunk_nbytes_old; schunk->cbytes += chunk_cbytes; if (frame->sframe) { schunk->cbytes -= chunk_cbytes_old; } else { if (chunk_cbytes_old >= chunk_cbytes) { schunk->cbytes -= chunk_cbytes; } } } } // Update super-chunk or frame if (schunk->frame == NULL) { if (!copy && (chunk_cbytes < chunk_nbytes)) { // We still want to do a shrink of the chunk chunk = realloc(chunk, chunk_cbytes); } // Free old chunk and add reference to new chunk if (schunk->data[nchunk] != 0) { free(schunk->data[nchunk]); } schunk->data[nchunk] = chunk; } else { if (frame_update_chunk(frame, nchunk, chunk, schunk) == NULL) { BLOSC_TRACE_ERROR("Problems updating a chunk in a frame."); return BLOSC2_ERROR_CHUNK_UPDATE; } } return schunk->nchunks; } int64_t blosc2_schunk_delete_chunk(blosc2_schunk *schunk, int64_t nchunk) { int rc; if (schunk->nchunks < nchunk) { BLOSC_TRACE_ERROR("The schunk has not enough chunks (%" PRId64 ")!", schunk->nchunks); } bool needs_free; uint8_t *chunk_old; int err = blosc2_schunk_get_chunk(schunk, nchunk, &chunk_old, &needs_free); if (err < 0) { BLOSC_TRACE_ERROR("%" PRId64 "chunk can not be obtained from schunk.", nchunk); return -1; } int32_t chunk_nbytes_old = 0; int32_t chunk_cbytes_old = 0; schunk->current_nchunk = nchunk; if (chunk_old != 0) { rc = blosc2_cbuffer_sizes(chunk_old, &chunk_nbytes_old, &chunk_cbytes_old, NULL); if (rc < 0) { return rc; } if (chunk_cbytes_old == BLOSC2_MAX_OVERHEAD) { chunk_cbytes_old = 0; } } if (needs_free) { free(chunk_old); } blosc2_frame_s* frame = (blosc2_frame_s*)(schunk->frame); schunk->nchunks -= 1; if (schunk->frame == NULL) { /* Update counters */ schunk->nbytes -= chunk_nbytes_old; schunk->cbytes -= chunk_cbytes_old; } else { // A frame schunk->nbytes -= chunk_nbytes_old; if (frame->sframe) { schunk->cbytes -= chunk_cbytes_old; } } // Update super-chunk or frame if (schunk->frame == NULL) { // Free old chunk if (schunk->data[nchunk] != 0) { free(schunk->data[nchunk]); } // Reorder the offsets and insert the new chunk for (int64_t i = nchunk; i < schunk->nchunks; i++) { schunk->data[i] = schunk->data[i + 1]; } schunk->data[schunk->nchunks] = NULL; } else { if (frame_delete_chunk(frame, nchunk, schunk) == NULL) { BLOSC_TRACE_ERROR("Problems deleting a chunk in a frame."); return BLOSC2_ERROR_CHUNK_UPDATE; } } return schunk->nchunks; } /* Append a data buffer to a super-chunk. */ int64_t blosc2_schunk_append_buffer(blosc2_schunk *schunk, void *src, int32_t nbytes) { uint8_t* chunk = malloc(nbytes + BLOSC2_MAX_OVERHEAD); schunk->current_nchunk = schunk->nchunks; /* Compress the src buffer using super-chunk context */ int cbytes = blosc2_compress_ctx(schunk->cctx, src, nbytes, chunk, nbytes + BLOSC2_MAX_OVERHEAD); if (cbytes < 0) { free(chunk); return cbytes; } // We don't need a copy of the chunk, as it will be shrunk if necessary int64_t nchunks = blosc2_schunk_append_chunk(schunk, chunk, false); if (nchunks < 0) { BLOSC_TRACE_ERROR("Error appending a buffer in super-chunk"); return nchunks; } return nchunks; } /* Decompress and return a chunk that is part of a super-chunk. */ int blosc2_schunk_decompress_chunk(blosc2_schunk *schunk, int64_t nchunk, void *dest, int32_t nbytes) { int32_t chunk_nbytes; int32_t chunk_cbytes; int chunksize; int rc; blosc2_frame_s* frame = (blosc2_frame_s*)schunk->frame; schunk->current_nchunk = nchunk; if (frame == NULL) { if (nchunk >= schunk->nchunks) { BLOSC_TRACE_ERROR("nchunk ('%" PRId64 "') exceeds the number of chunks " "('%" PRId64 "') in super-chunk.", nchunk, schunk->nchunks); return BLOSC2_ERROR_INVALID_PARAM; } uint8_t* src = schunk->data[nchunk]; if (src == 0) { return 0; } rc = blosc2_cbuffer_sizes(src, &chunk_nbytes, &chunk_cbytes, NULL); if (rc < 0) { return rc; } if (nbytes < chunk_nbytes) { BLOSC_TRACE_ERROR("Buffer size is too small for the decompressed buffer " "('%d' bytes, but '%d' are needed).", nbytes, chunk_nbytes); return BLOSC2_ERROR_INVALID_PARAM; } chunksize = blosc2_decompress_ctx(schunk->dctx, src, chunk_cbytes, dest, nbytes); if (chunksize < 0 || chunksize != chunk_nbytes) { BLOSC_TRACE_ERROR("Error in decompressing chunk."); if (chunksize < 0) return chunksize; return BLOSC2_ERROR_FAILURE; } } else { chunksize = frame_decompress_chunk(schunk->dctx, frame, nchunk, dest, nbytes); if (chunksize < 0) { return chunksize; } } return chunksize; } /* Return a compressed chunk that is part of a super-chunk in the `chunk` parameter. * If the super-chunk is backed by a frame that is disk-based, a buffer is allocated for the * (compressed) chunk, and hence a free is needed. You can check if the chunk requires a free * with the `needs_free` parameter. * If the chunk does not need a free, it means that a pointer to the location in the super-chunk * (or the backing in-memory frame) is returned in the `chunk` parameter. * * The size of the (compressed) chunk is returned. If some problem is detected, a negative code * is returned instead. */ int blosc2_schunk_get_chunk(blosc2_schunk *schunk, int64_t nchunk, uint8_t **chunk, bool *needs_free) { if (schunk->dctx->threads_started > 1) { pthread_mutex_lock(&schunk->dctx->nchunk_mutex); schunk->current_nchunk = nchunk; pthread_mutex_unlock(&schunk->dctx->nchunk_mutex); } else { schunk->current_nchunk = nchunk; } blosc2_frame_s* frame = (blosc2_frame_s*)schunk->frame; if (frame != NULL) { return frame_get_chunk(frame, nchunk, chunk, needs_free); } if (nchunk >= schunk->nchunks) { BLOSC_TRACE_ERROR("nchunk ('%" PRId64 "') exceeds the number of chunks " "('%" PRId64 "') in schunk.", nchunk, schunk->nchunks); return BLOSC2_ERROR_INVALID_PARAM; } *chunk = schunk->data[nchunk]; if (*chunk == 0) { *needs_free = 0; return 0; } *needs_free = false; int32_t chunk_cbytes; int rc = blosc2_cbuffer_sizes(*chunk, NULL, &chunk_cbytes, NULL); if (rc < 0) { return rc; } return (int)chunk_cbytes; } /* Return a compressed chunk that is part of a super-chunk in the `chunk` parameter. * If the super-chunk is backed by a frame that is disk-based, a buffer is allocated for the * (compressed) chunk, and hence a free is needed. You can check if the chunk requires a free * with the `needs_free` parameter. * If the chunk does not need a free, it means that a pointer to the location in the super-chunk * (or the backing in-memory frame) is returned in the `chunk` parameter. * * The size of the (compressed) chunk is returned. If some problem is detected, a negative code * is returned instead. */ int blosc2_schunk_get_lazychunk(blosc2_schunk *schunk, int64_t nchunk, uint8_t **chunk, bool *needs_free) { if (schunk->dctx->threads_started > 1) { pthread_mutex_lock(&schunk->dctx->nchunk_mutex); schunk->current_nchunk = nchunk; pthread_mutex_unlock(&schunk->dctx->nchunk_mutex); } else { schunk->current_nchunk = nchunk; } blosc2_frame_s* frame = (blosc2_frame_s*)schunk->frame; if (schunk->frame != NULL) { return frame_get_lazychunk(frame, nchunk, chunk, needs_free); } if (nchunk >= schunk->nchunks) { BLOSC_TRACE_ERROR("nchunk ('%" PRId64 "') exceeds the number of chunks " "('%" PRId64 "') in schunk.", nchunk, schunk->nchunks); return BLOSC2_ERROR_INVALID_PARAM; } *chunk = schunk->data[nchunk]; if (*chunk == 0) { *needs_free = 0; return 0; } *needs_free = false; int32_t chunk_cbytes; int rc = blosc2_cbuffer_sizes(*chunk, NULL, &chunk_cbytes, NULL); if (rc < 0) { return rc; } return (int)chunk_cbytes; } int blosc2_schunk_get_slice_buffer(blosc2_schunk *schunk, int64_t start, int64_t stop, void *buffer) { int64_t byte_start = start * schunk->typesize; int64_t byte_stop = stop * schunk->typesize; int64_t nchunk_start = byte_start / schunk->chunksize; int32_t chunk_start = (int32_t) (byte_start % schunk->chunksize); // 0 indexed int32_t chunk_stop; // 0 indexed if (byte_stop >= (nchunk_start + 1) * schunk->chunksize) { chunk_stop = schunk->chunksize; } else { chunk_stop = (int32_t) (byte_stop % schunk->chunksize); } uint8_t *dst_ptr = (uint8_t *) buffer; bool needs_free; uint8_t *chunk; int32_t cbytes; int64_t nchunk = nchunk_start; int64_t nbytes_read = 0; int32_t nbytes; int32_t chunksize = schunk->chunksize; while (nbytes_read < ((stop - start) * schunk->typesize)) { cbytes = blosc2_schunk_get_lazychunk(schunk, nchunk, &chunk, &needs_free); if (cbytes < 0) { BLOSC_TRACE_ERROR("Cannot get lazychunk ('%" PRId64 "').", nchunk); return BLOSC2_ERROR_FAILURE; } int32_t blocksize = sw32_(chunk + BLOSC2_CHUNK_BLOCKSIZE); int32_t nblock_start = (int32_t) (chunk_start / blocksize); int32_t nblock_stop = (int32_t) ((chunk_stop - 1) / blocksize); if (nchunk == (schunk->nchunks - 1) && schunk->nbytes % schunk->chunksize != 0) { chunksize = schunk->nbytes % schunk->chunksize; } int32_t nblocks = chunksize / blocksize; if (chunksize % blocksize != 0) { nblocks++; } if (chunk_start == 0 && chunk_stop == chunksize) { // Avoid memcpy nbytes = blosc2_decompress_ctx(schunk->dctx, chunk, cbytes, dst_ptr, chunksize); if (nbytes < 0) { BLOSC_TRACE_ERROR("Cannot decompress chunk ('%" PRId64 "').", nchunk); return BLOSC2_ERROR_FAILURE; } } else { // After extensive timing I have not been able to see lots of situations where // a maskout read is better than a getitem one. Disabling for now. // if (nblock_start != nblock_stop) { if (false) { uint8_t *data = malloc(chunksize); /* We have more than 1 block to read, so use a masked read */ bool *block_maskout = calloc(nblocks, 1); for (int32_t nblock = 0; nblock < nblocks; nblock++) { if ((nblock < nblock_start) || (nblock > nblock_stop)) { block_maskout[nblock] = true; } } if (blosc2_set_maskout(schunk->dctx, block_maskout, nblocks) < 0) { BLOSC_TRACE_ERROR("Cannot set maskout"); return BLOSC2_ERROR_FAILURE; } nbytes = blosc2_decompress_ctx(schunk->dctx, chunk, cbytes, data, chunksize); if (nbytes < 0) { BLOSC_TRACE_ERROR("Cannot decompress chunk ('%" PRId64 "').", nchunk); return BLOSC2_ERROR_FAILURE; } nbytes = chunk_stop - chunk_start; memcpy(dst_ptr, &data[chunk_start], nbytes); free(block_maskout); free(data); } else { /* Less than 1 block to read; use a getitem call */ nbytes = blosc2_getitem_ctx(schunk->dctx, chunk, cbytes, (int32_t) (chunk_start / schunk->typesize), (chunk_stop - chunk_start) / schunk->typesize, dst_ptr, chunksize); if (nbytes < 0) { BLOSC_TRACE_ERROR("Cannot get item from ('%" PRId64 "') chunk.", nchunk); return BLOSC2_ERROR_FAILURE; } } } dst_ptr += nbytes; nbytes_read += nbytes; nchunk++; if (needs_free) { free(chunk); } chunk_start = 0; if (byte_stop >= (nchunk + 1) * chunksize) { chunk_stop = chunksize; } else { chunk_stop = (int32_t)(byte_stop % chunksize); } } return BLOSC2_ERROR_SUCCESS; } int blosc2_schunk_set_slice_buffer(blosc2_schunk *schunk, int64_t start, int64_t stop, void *buffer) { int64_t byte_start = start * schunk->typesize; int64_t byte_stop = stop * schunk->typesize; int64_t nchunk_start = byte_start / schunk->chunksize; int32_t chunk_start = (int32_t) (byte_start % schunk->chunksize); int32_t chunk_stop; if (byte_stop >= (nchunk_start + 1) * schunk->chunksize) { chunk_stop = schunk->chunksize; } else { chunk_stop = (int32_t) (byte_stop % schunk->chunksize); } uint8_t *src_ptr = (uint8_t *) buffer; int64_t nchunk = nchunk_start; int64_t nbytes_written = 0; int32_t nbytes; uint8_t *data = malloc(schunk->chunksize); int64_t nchunks; int32_t chunksize = schunk->chunksize; while (nbytes_written < ((stop - start) * schunk->typesize)) { if (chunk_start == 0 && (chunk_stop == schunk->chunksize || chunk_stop == schunk->nbytes % schunk->chunksize)) { if (chunk_stop == schunk->nbytes % schunk->chunksize) { chunksize = chunk_stop; } uint8_t *chunk = malloc(chunksize + BLOSC2_MAX_OVERHEAD); if (blosc2_compress_ctx(schunk->cctx, src_ptr, chunksize, chunk, chunksize + BLOSC2_MAX_OVERHEAD) < 0) { BLOSC_TRACE_ERROR("Cannot compress data of chunk ('%" PRId64 "').", nchunk); return BLOSC2_ERROR_FAILURE; } nchunks = blosc2_schunk_update_chunk(schunk, nchunk, chunk, false); if (nchunks != schunk->nchunks) { BLOSC_TRACE_ERROR("Cannot update chunk ('%" PRId64 "').", nchunk); return BLOSC2_ERROR_CHUNK_UPDATE; } } else { nbytes = blosc2_schunk_decompress_chunk(schunk, nchunk, data, schunk->chunksize); if (nbytes < 0) { BLOSC_TRACE_ERROR("Cannot decompress chunk ('%" PRId64 "').", nchunk); return BLOSC2_ERROR_FAILURE; } memcpy(&data[chunk_start], src_ptr, chunk_stop - chunk_start); uint8_t *chunk = malloc(nbytes + BLOSC2_MAX_OVERHEAD); if (blosc2_compress_ctx(schunk->cctx, data, nbytes, chunk, nbytes + BLOSC2_MAX_OVERHEAD) < 0) { BLOSC_TRACE_ERROR("Cannot compress data of chunk ('%" PRId64 "').", nchunk); return BLOSC2_ERROR_FAILURE; } nchunks = blosc2_schunk_update_chunk(schunk, nchunk, chunk, false); if (nchunks != schunk->nchunks) { BLOSC_TRACE_ERROR("Cannot update chunk ('%" PRId64 "').", nchunk); return BLOSC2_ERROR_CHUNK_UPDATE; } } nchunk++; nbytes_written += chunk_stop - chunk_start; src_ptr += chunk_stop - chunk_start; chunk_start = 0; if (byte_stop >= (nchunk + 1) * schunk->chunksize) { chunk_stop = schunk->chunksize; } else { chunk_stop = (int32_t) (byte_stop % schunk->chunksize); } } free(data); return BLOSC2_ERROR_SUCCESS; } /* Reorder the chunk offsets of an existing super-chunk. */ int blosc2_schunk_reorder_offsets(blosc2_schunk *schunk, int64_t *offsets_order) { // Check that the offsets order are correct bool *index_check = (bool *) calloc(schunk->nchunks, sizeof(bool)); for (int i = 0; i < schunk->nchunks; ++i) { int64_t index = offsets_order[i]; if (index >= schunk->nchunks) { BLOSC_TRACE_ERROR("Index is bigger than the number of chunks."); free(index_check); return BLOSC2_ERROR_DATA; } if (index_check[index] == false) { index_check[index] = true; } else { BLOSC_TRACE_ERROR("Index is yet used."); free(index_check); return BLOSC2_ERROR_DATA; } } free(index_check); blosc2_frame_s* frame = (blosc2_frame_s*)schunk->frame; if (frame != NULL) { return frame_reorder_offsets(frame, offsets_order, schunk); } uint8_t **offsets = schunk->data; // Make a copy of the chunk offsets and reorder it uint8_t **offsets_copy = malloc(schunk->data_len); memcpy(offsets_copy, offsets, schunk->data_len); for (int i = 0; i < schunk->nchunks; ++i) { offsets[i] = offsets_copy[offsets_order[i]]; } free(offsets_copy); return 0; } // Get the length (in bytes) of the internal frame of the super-chunk int64_t blosc2_schunk_frame_len(blosc2_schunk* schunk) { int64_t len; blosc2_frame_s* frame_s = (blosc2_frame_s*)(schunk->frame); if (frame_s != NULL) { len = frame_s->len; } else { // No attached frame, but we can still come with an estimate len = (int64_t) (schunk->cbytes + schunk->nchunks * sizeof(int64_t)); } return len; } /** * @brief Flush metalayers content into a possible attached frame. * * @param schunk The super-chunk to which the flush should be applied. * * @return If successful, a 0 is returned. Else, return a negative value. */ // Initially, this was a public function, but as it is really meant to be used only // in the schunk_add_metalayer(), I decided to convert it into private and call it // implicitly instead of requiring the user to do so. The only drawback is that // each add operation requires a complete frame re-build, but as users should need // very few metalayers, this overhead should be negligible in practice. int metalayer_flush(blosc2_schunk* schunk) { int rc = BLOSC2_ERROR_SUCCESS; blosc2_frame_s* frame = (blosc2_frame_s*)schunk->frame; if (frame == NULL) { return rc; } rc = frame_update_header(frame, schunk, true); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to update metalayers into frame."); return rc; } rc = frame_update_trailer(frame, schunk); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to update trailer into frame."); return rc; } return rc; } /* Add content into a new metalayer. * * If successful, return the index of the new metalayer. Else, return a negative value. */ int blosc2_meta_add(blosc2_schunk *schunk, const char *name, uint8_t *content, int32_t content_len) { int nmetalayer = blosc2_meta_exists(schunk, name); if (nmetalayer >= 0) { BLOSC_TRACE_ERROR("Metalayer \"%s\" already exists.", name); return BLOSC2_ERROR_INVALID_PARAM; } // Add the metalayer blosc2_metalayer *metalayer = malloc(sizeof(blosc2_metalayer)); char* name_ = malloc(strlen(name) + 1); strcpy(name_, name); metalayer->name = name_; uint8_t* content_buf = malloc((size_t)content_len); memcpy(content_buf, content, content_len); metalayer->content = content_buf; metalayer->content_len = content_len; schunk->metalayers[schunk->nmetalayers] = metalayer; schunk->nmetalayers += 1; int rc = metalayer_flush(schunk); if (rc < 0) { return rc; } return schunk->nmetalayers - 1; } /* Update the content of an existing metalayer. * * If successful, return the index of the new metalayer. Else, return a negative value. */ int blosc2_meta_update(blosc2_schunk *schunk, const char *name, uint8_t *content, int32_t content_len) { int nmetalayer = blosc2_meta_exists(schunk, name); if (nmetalayer < 0) { BLOSC_TRACE_ERROR("Metalayer \"%s\" not found.", name); return nmetalayer; } blosc2_metalayer *metalayer = schunk->metalayers[nmetalayer]; if (content_len > metalayer->content_len) { BLOSC_TRACE_ERROR("`content_len` cannot exceed the existing size of %d bytes.", metalayer->content_len); return nmetalayer; } // Update the contents of the metalayer memcpy(metalayer->content, content, content_len); // Update the metalayers in frame (as size has not changed, we don't need to update the trailer) blosc2_frame_s* frame = (blosc2_frame_s*)schunk->frame; if (frame != NULL) { int rc = frame_update_header(frame, schunk, false); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to update meta info from frame."); return rc; } } return nmetalayer; } /* Find whether the schunk has a variable-length metalayer or not. * * If successful, return the index of the variable-length metalayer. Else, return a negative value. */ int blosc2_vlmeta_exists(blosc2_schunk *schunk, const char *name) { if (strlen(name) > BLOSC2_METALAYER_NAME_MAXLEN) { BLOSC_TRACE_ERROR("Variable-length metalayer names cannot be larger than %d chars.", BLOSC2_METALAYER_NAME_MAXLEN); return BLOSC2_ERROR_INVALID_PARAM; } for (int nvlmetalayer = 0; nvlmetalayer < schunk->nvlmetalayers; nvlmetalayer++) { if (strcmp(name, schunk->vlmetalayers[nvlmetalayer]->name) == 0) { return nvlmetalayer; } } return BLOSC2_ERROR_NOT_FOUND; } int vlmetalayer_flush(blosc2_schunk* schunk) { int rc = BLOSC2_ERROR_SUCCESS; blosc2_frame_s* frame = (blosc2_frame_s*)schunk->frame; if (frame == NULL) { return rc; } rc = frame_update_header(frame, schunk, false); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to update metalayers into frame."); return rc; } rc = frame_update_trailer(frame, schunk); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to update trailer into frame."); return rc; } return rc; } /* Add content into a new variable-length metalayer. * * If successful, return the index of the new variable-length metalayer. Else, return a negative value. */ int blosc2_vlmeta_add(blosc2_schunk *schunk, const char *name, uint8_t *content, int32_t content_len, blosc2_cparams *cparams) { int nvlmetalayer = blosc2_vlmeta_exists(schunk, name); if (nvlmetalayer >= 0) { BLOSC_TRACE_ERROR("Variable-length metalayer \"%s\" already exists.", name); return BLOSC2_ERROR_INVALID_PARAM; } // Add the vlmetalayer blosc2_metalayer *vlmetalayer = malloc(sizeof(blosc2_metalayer)); vlmetalayer->name = strdup(name); uint8_t* content_buf = malloc((size_t) content_len + BLOSC2_MAX_OVERHEAD); blosc2_context *cctx; if (cparams != NULL) { cctx = blosc2_create_cctx(*cparams); } else { cctx = blosc2_create_cctx(BLOSC2_CPARAMS_DEFAULTS); } int csize = blosc2_compress_ctx(cctx, content, content_len, content_buf, content_len + BLOSC2_MAX_OVERHEAD); if (csize < 0) { BLOSC_TRACE_ERROR("Can not compress the `%s` variable-length metalayer.", name); return csize; } blosc2_free_ctx(cctx); vlmetalayer->content = realloc(content_buf, csize); vlmetalayer->content_len = csize; schunk->vlmetalayers[schunk->nvlmetalayers] = vlmetalayer; schunk->nvlmetalayers += 1; // Propagate to frames int rc = vlmetalayer_flush(schunk); if (rc < 0) { BLOSC_TRACE_ERROR("Can not propagate de `%s` variable-length metalayer to a frame.", name); return rc; } return schunk->nvlmetalayers - 1; } int blosc2_vlmeta_get(blosc2_schunk *schunk, const char *name, uint8_t **content, int32_t *content_len) { int nvlmetalayer = blosc2_vlmeta_exists(schunk, name); if (nvlmetalayer < 0) { BLOSC_TRACE_ERROR("User metalayer \"%s\" not found.", name); return nvlmetalayer; } blosc2_metalayer *meta = schunk->vlmetalayers[nvlmetalayer]; int32_t nbytes, cbytes; blosc2_cbuffer_sizes(meta->content, &nbytes, &cbytes, NULL); if (cbytes != meta->content_len) { BLOSC_TRACE_ERROR("User metalayer \"%s\" is corrupted.", meta->name); return BLOSC2_ERROR_DATA; } *content_len = nbytes; *content = malloc((size_t) nbytes); blosc2_context *dctx = blosc2_create_dctx(*schunk->storage->dparams); int nbytes_ = blosc2_decompress_ctx(dctx, meta->content, meta->content_len, *content, nbytes); blosc2_free_ctx(dctx); if (nbytes_ != nbytes) { BLOSC_TRACE_ERROR("User metalayer \"%s\" is corrupted.", meta->name); return BLOSC2_ERROR_READ_BUFFER; } return nvlmetalayer; } int blosc2_vlmeta_update(blosc2_schunk *schunk, const char *name, uint8_t *content, int32_t content_len, blosc2_cparams *cparams) { int nvlmetalayer = blosc2_vlmeta_exists(schunk, name); if (nvlmetalayer < 0) { BLOSC_TRACE_ERROR("User vlmetalayer \"%s\" not found.", name); return nvlmetalayer; } blosc2_metalayer *vlmetalayer = schunk->vlmetalayers[nvlmetalayer]; free(vlmetalayer->content); uint8_t* content_buf = malloc((size_t) content_len + BLOSC2_MAX_OVERHEAD); blosc2_context *cctx; if (cparams != NULL) { cctx = blosc2_create_cctx(*cparams); } else { cctx = blosc2_create_cctx(BLOSC2_CPARAMS_DEFAULTS); } int csize = blosc2_compress_ctx(cctx, content, content_len, content_buf, content_len + BLOSC2_MAX_OVERHEAD); if (csize < 0) { BLOSC_TRACE_ERROR("Can not compress the `%s` variable-length metalayer.", name); return csize; } blosc2_free_ctx(cctx); vlmetalayer->content = realloc(content_buf, csize); vlmetalayer->content_len = csize; // Propagate to frames int rc = vlmetalayer_flush(schunk); if (rc < 0) { BLOSC_TRACE_ERROR("Can not propagate de `%s` variable-length metalayer to a frame.", name); return rc; } return nvlmetalayer; } int blosc2_vlmeta_delete(blosc2_schunk *schunk, const char *name) { int nvlmetalayer = blosc2_vlmeta_exists(schunk, name); if (nvlmetalayer < 0) { BLOSC_TRACE_ERROR("User vlmetalayer \"%s\" not found.", name); return nvlmetalayer; } blosc2_metalayer *vlmetalayer = schunk->vlmetalayers[nvlmetalayer]; for (int i = nvlmetalayer; i < (schunk->nvlmetalayers - 1); i++) { schunk->vlmetalayers[i] = schunk->vlmetalayers[i + 1]; } free(vlmetalayer->content); schunk->nvlmetalayers--; // Propagate to frames int rc = vlmetalayer_flush(schunk); if (rc < 0) { BLOSC_TRACE_ERROR("Can not propagate de `%s` variable-length metalayer to a frame.", name); return rc; } return schunk->nvlmetalayers; } int blosc2_vlmeta_get_names(blosc2_schunk *schunk, char **names) { int16_t nvlmetalayers = schunk->nvlmetalayers; for (int i = 0; i < nvlmetalayers; ++i) { names[i] = schunk->vlmetalayers[i]->name; } return nvlmetalayers; } zmat-0.9.9/src/blosc2/blosc/delta.h0000644000175200007730000000137614515254731017265 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef BLOSC_DELTA_H #define BLOSC_DELTA_H #include #include void delta_encoder(const uint8_t* dref, int32_t offset, int32_t nbytes, int32_t typesize, const uint8_t* src, uint8_t* dest); void delta_decoder(const uint8_t* dref, int32_t offset, int32_t nbytes, int32_t typesize, uint8_t* dest); #endif //BLOSC_DELTA_H zmat-0.9.9/src/blosc2/blosc/fastcopy.c0000644000175200007730000004037514515254731020021 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /********************************************************************* The code in this file is heavily based on memcopy.h, from the zlib-ng compression library. See LICENSES/ZLIB.txt for details. See also: https://github.com/Dead2/zlib-ng/blob/develop/zlib.h New implementations by Francesc Alted: * fast_copy() and copy_run() functions * Support for SSE2/AVX2 copy instructions for these routines **********************************************************************/ #include #include "blosc2/blosc2-common.h" /* * Use inlined functions for supported systems. */ #if defined(_MSC_VER) && !defined(__cplusplus) /* Visual Studio */ #define inline __inline /* Visual C is not C99, but supports some kind of inline */ #endif static inline unsigned char *copy_1_bytes(unsigned char *out, const unsigned char *from) { *out++ = *from; return out; } static inline unsigned char *copy_2_bytes(unsigned char *out, const unsigned char *from) { #if defined(BLOSC_STRICT_ALIGN) uint16_t chunk; memcpy(&chunk, from, 2); memcpy(out, &chunk, 2); #else *(uint16_t *) out = *(uint16_t *) from; #endif return out + 2; } static inline unsigned char *copy_3_bytes(unsigned char *out, const unsigned char *from) { out = copy_1_bytes(out, from); return copy_2_bytes(out, from + 1); } static inline unsigned char *copy_4_bytes(unsigned char *out, const unsigned char *from) { #if defined(BLOSC_STRICT_ALIGN) uint32_t chunk; memcpy(&chunk, from, 4); memcpy(out, &chunk, 4); #else *(uint32_t *) out = *(uint32_t *) from; #endif return out + 4; } static inline unsigned char *copy_5_bytes(unsigned char *out, const unsigned char *from) { out = copy_1_bytes(out, from); return copy_4_bytes(out, from + 1); } static inline unsigned char *copy_6_bytes(unsigned char *out, const unsigned char *from) { out = copy_2_bytes(out, from); return copy_4_bytes(out, from + 2); } static inline unsigned char *copy_7_bytes(unsigned char *out, const unsigned char *from) { out = copy_3_bytes(out, from); return copy_4_bytes(out, from + 3); } static inline unsigned char *copy_8_bytes(unsigned char *out, const unsigned char *from) { #if defined(BLOSC_STRICT_ALIGN) uint64_t chunk; memcpy(&chunk, from, 8); memcpy(out, &chunk, 8); #else *(uint64_t *) out = *(uint64_t *) from; #endif return out + 8; } static inline unsigned char *copy_16_bytes(unsigned char *out, const unsigned char *from) { #if defined(__SSE2__) __m128i chunk; chunk = _mm_loadu_si128((__m128i*)from); _mm_storeu_si128((__m128i*)out, chunk); out += 16; #elif !defined(BLOSC_STRICT_ALIGN) *(uint64_t*)out = *(uint64_t*)from; from += 8; out += 8; *(uint64_t*)out = *(uint64_t*)from; from += 8; out += 8; #else int i; for (i = 0; i < 16; i++) { *out++ = *from++; } #endif return out; } static inline unsigned char *copy_32_bytes(unsigned char *out, const unsigned char *from) { #if defined(__AVX2__) __m256i chunk; chunk = _mm256_loadu_si256((__m256i*)from); _mm256_storeu_si256((__m256i*)out, chunk); out += 32; #elif defined(__SSE2__) __m128i chunk; chunk = _mm_loadu_si128((__m128i*)from); _mm_storeu_si128((__m128i*)out, chunk); from += 16; out += 16; chunk = _mm_loadu_si128((__m128i*)from); _mm_storeu_si128((__m128i*)out, chunk); out += 16; #elif !defined(BLOSC_STRICT_ALIGN) *(uint64_t*)out = *(uint64_t*)from; from += 8; out += 8; *(uint64_t*)out = *(uint64_t*)from; from += 8; out += 8; *(uint64_t*)out = *(uint64_t*)from; from += 8; out += 8; *(uint64_t*)out = *(uint64_t*)from; from += 8; out += 8; #else int i; for (i = 0; i < 32; i++) { *out++ = *from++; } #endif return out; } // This is never used, so comment it out //#if defined(__AVX2__) //static inline unsigned char *copy_32_bytes_aligned(unsigned char *out, const unsigned char *from) { // __m256i chunk; // chunk = _mm256_load_si256((__m256i*)from); // _mm256_storeu_si256((__m256i*)out, chunk); // return out + 32; //} //#endif // __AVX2__ /* Copy LEN bytes (7 or fewer) from FROM into OUT. Return OUT + LEN. */ static inline unsigned char *copy_bytes(unsigned char *out, const unsigned char *from, unsigned len) { assert(len < 8); #ifdef BLOSC_STRICT_ALIGN while (len--) { *out++ = *from++; } #else switch (len) { case 7: return copy_7_bytes(out, from); case 6: return copy_6_bytes(out, from); case 5: return copy_5_bytes(out, from); case 4: return copy_4_bytes(out, from); case 3: return copy_3_bytes(out, from); case 2: return copy_2_bytes(out, from); case 1: return copy_1_bytes(out, from); case 0: return out; default: assert(0); } #endif /* BLOSC_STRICT_ALIGN */ return out; } // Define a symbol for avoiding fall-through warnings emitted by gcc >= 7.0 #if ((defined(__GNUC__) && BLOSC_GCC_VERSION >= 700) && !defined(__clang__) && \ !defined(__ICC) && !defined(__ICL)) #define AVOID_FALLTHROUGH_WARNING #endif /* Byte by byte semantics: copy LEN bytes from FROM and write them to OUT. Return OUT + LEN. */ static inline unsigned char *chunk_memcpy(unsigned char *out, const unsigned char *from, unsigned len) { unsigned sz = sizeof(uint64_t); unsigned rem = len % sz; unsigned by8; assert(len >= sz); /* Copy a few bytes to make sure the loop below has a multiple of SZ bytes to be copied. */ copy_8_bytes(out, from); len /= sz; out += rem; from += rem; by8 = len % 8; len -= by8; switch (by8) { case 7: out = copy_8_bytes(out, from); from += sz; #ifdef AVOID_FALLTHROUGH_WARNING __attribute__ ((fallthrough)); // Shut-up -Wimplicit-fallthrough warning in GCC #endif case 6: out = copy_8_bytes(out, from); from += sz; #ifdef AVOID_FALLTHROUGH_WARNING __attribute__ ((fallthrough)); #endif case 5: out = copy_8_bytes(out, from); from += sz; #ifdef AVOID_FALLTHROUGH_WARNING __attribute__ ((fallthrough)); #endif case 4: out = copy_8_bytes(out, from); from += sz; #ifdef AVOID_FALLTHROUGH_WARNING __attribute__ ((fallthrough)); #endif case 3: out = copy_8_bytes(out, from); from += sz; #ifdef AVOID_FALLTHROUGH_WARNING __attribute__ ((fallthrough)); #endif case 2: out = copy_8_bytes(out, from); from += sz; #ifdef AVOID_FALLTHROUGH_WARNING __attribute__ ((fallthrough)); #endif case 1: out = copy_8_bytes(out, from); from += sz; #ifdef AVOID_FALLTHROUGH_WARNING __attribute__ ((fallthrough)); #endif default: break; } while (len) { out = copy_8_bytes(out, from); from += sz; out = copy_8_bytes(out, from); from += sz; out = copy_8_bytes(out, from); from += sz; out = copy_8_bytes(out, from); from += sz; out = copy_8_bytes(out, from); from += sz; out = copy_8_bytes(out, from); from += sz; out = copy_8_bytes(out, from); from += sz; out = copy_8_bytes(out, from); from += sz; len -= 8; } return out; } #if (defined(__SSE2__) && defined(__AVX2__)) /* 16-byte version of chunk_memcpy() */ static inline unsigned char *chunk_memcpy_16(unsigned char *out, const unsigned char *from, unsigned len) { unsigned sz = 16; unsigned rem = len % sz; unsigned ilen; assert(len >= sz); /* Copy a few bytes to make sure the loop below has a multiple of SZ bytes to be copied. */ copy_16_bytes(out, from); len /= sz; out += rem; from += rem; for (ilen = 0; ilen < len; ilen++) { copy_16_bytes(out, from); out += sz; from += sz; } return out; } #endif // NOTE: chunk_memcpy_32() and chunk_memcpy_32_unrolled() are not used, so commenting them ///* 32-byte version of chunk_memcpy() */ //static inline unsigned char *chunk_memcpy_32(unsigned char *out, const unsigned char *from, unsigned len) { // unsigned sz = 32; // unsigned rem = len % sz; // unsigned ilen; // // assert(len >= sz); // // /* Copy a few bytes to make sure the loop below has a multiple of SZ bytes to be copied. */ // copy_32_bytes(out, from); // // len /= sz; // out += rem; // from += rem; // // for (ilen = 0; ilen < len; ilen++) { // copy_32_bytes(out, from); // out += sz; // from += sz; // } // // return out; //} // ///* 32-byte *unrolled* version of chunk_memcpy() */ //static inline unsigned char *chunk_memcpy_32_unrolled(unsigned char *out, const unsigned char *from, unsigned len) { // unsigned sz = 32; // unsigned rem = len % sz; // unsigned by8; // // assert(len >= sz); // // /* Copy a few bytes to make sure the loop below has a multiple of SZ bytes to be copied. */ // copy_32_bytes(out, from); // // len /= sz; // out += rem; // from += rem; // // by8 = len % 8; // len -= by8; // switch (by8) { // case 7: // out = copy_32_bytes(out, from); // from += sz; // case 6: // out = copy_32_bytes(out, from); // from += sz; // case 5: // out = copy_32_bytes(out, from); // from += sz; // case 4: // out = copy_32_bytes(out, from); // from += sz; // case 3: // out = copy_32_bytes(out, from); // from += sz; // case 2: // out = copy_32_bytes(out, from); // from += sz; // case 1: // out = copy_32_bytes(out, from); // from += sz; // default: // break; // } // // while (len) { // out = copy_32_bytes(out, from); // from += sz; // out = copy_32_bytes(out, from); // from += sz; // out = copy_32_bytes(out, from); // from += sz; // out = copy_32_bytes(out, from); // from += sz; // out = copy_32_bytes(out, from); // from += sz; // out = copy_32_bytes(out, from); // from += sz; // out = copy_32_bytes(out, from); // from += sz; // out = copy_32_bytes(out, from); // from += sz; // // len -= 8; // } // // return out; //} /* SSE2/AVX2 *unaligned* version of chunk_memcpy() */ #if defined(__SSE2__) || defined(__AVX2__) static inline unsigned char *chunk_memcpy_unaligned(unsigned char *out, const unsigned char *from, unsigned len) { #if defined(__AVX2__) unsigned sz = sizeof(__m256i); #elif defined(__SSE2__) unsigned sz = sizeof(__m128i); #endif unsigned rem = len % sz; unsigned ilen; assert(len >= sz); /* Copy a few bytes to make sure the loop below has a multiple of SZ bytes to be copied. */ #if defined(__AVX2__) copy_32_bytes(out, from); #elif defined(__SSE2__) copy_16_bytes(out, from); #endif len /= sz; out += rem; from += rem; for (ilen = 0; ilen < len; ilen++) { #if defined(__AVX2__) copy_32_bytes(out, from); #elif defined(__SSE2__) copy_16_bytes(out, from); #endif out += sz; from += sz; } return out; } #endif // __SSE2__ || __AVX2__ // NOTE: chunk_memcpy_aligned() is not used, so commenting it //#if defined(__SSE2__) || defined(__AVX2__) ///* SSE2/AVX2 *aligned* version of chunk_memcpy() */ //static inline unsigned char *chunk_memcpy_aligned(unsigned char *out, const unsigned char *from, unsigned len) { //#if defined(__AVX2__) // unsigned sz = sizeof(__m256i); // __m256i chunk; //#elif defined(__SSE2__) // unsigned sz = sizeof(__m128i); // __m128i chunk; //#endif // unsigned bytes_to_align = sz - (unsigned)(((uintptr_t)(const void *)(from)) % sz); // unsigned corrected_len = len - bytes_to_align; // unsigned rem = corrected_len % sz; // unsigned ilen; // // assert(len >= sz); // // /* Copy a few bytes to make sure the loop below has aligned access. */ //#if defined(__AVX2__) // chunk = _mm256_loadu_si256((__m256i *) from); // _mm256_storeu_si256((__m256i *) out, chunk); //#elif defined(__SSE2__) // chunk = _mm_loadu_si128((__m128i *) from); // _mm_storeu_si128((__m128i *) out, chunk); //#endif // out += bytes_to_align; // from += bytes_to_align; // // len = corrected_len / sz; // for (ilen = 0; ilen < len; ilen++) { //#if defined(__AVX2__) // chunk = _mm256_load_si256((__m256i *) from); /* *aligned* load */ // _mm256_storeu_si256((__m256i *) out, chunk); //#elif defined(__SSE2__) // chunk = _mm_load_si128((__m128i *) from); /* *aligned* load */ // _mm_storeu_si128((__m128i *) out, chunk); //#endif // out += sz; // from += sz; // } // // /* Copy remaining bytes */ // if (rem < 8) { // out = copy_bytes(out, from, rem); // } // else { // out = chunk_memcpy(out, from, rem); // } // // return out; //} //#endif // __AVX2__ || __SSE2__ /* Byte by byte semantics: copy LEN bytes from FROM and write them to OUT. Return OUT + LEN. */ unsigned char *fastcopy(unsigned char *out, const unsigned char *from, unsigned len) { switch (len) { case 32: return copy_32_bytes(out, from); case 16: return copy_16_bytes(out, from); case 8: return copy_8_bytes(out, from); default: { } } if (len < 8) { return copy_bytes(out, from, len); } #if defined(__SSE2__) if (len < 16) { return chunk_memcpy(out, from, len); } #if !defined(__AVX2__) return chunk_memcpy_unaligned(out, from, len); #else if (len < 32) { return chunk_memcpy_16(out, from, len); } return chunk_memcpy_unaligned(out, from, len); #endif // !__AVX2__ #else return chunk_memcpy(out, from, len); #endif // __SSE2__ } /* Copy a run */ unsigned char* copy_match(unsigned char *out, const unsigned char *from, unsigned len) { #if defined(__AVX2__) unsigned sz = sizeof(__m256i); #elif defined(__SSE2__) unsigned sz = sizeof(__m128i); #else unsigned sz = sizeof(uint64_t); #endif #if ((defined(__GNUC__) && BLOSC_GCC_VERSION < 800) && !defined(__clang__) && !defined(__ICC) && !defined(__ICL)) // GCC < 8 in fully optimization mode seems to have problems with the code further below so stop here for (; len > 0; len--) { *out++ = *from++; } return out; #endif // If out and from are away more than the size of the copy, then a fastcopy is safe unsigned overlap_dist = (unsigned) (out - from); if (overlap_dist > sz) { return fastcopy(out, from, len); } // Otherwise we need to be more careful so as not to overwrite destination switch (overlap_dist) { case 32: for (; len >= 32; len -= 32) { out = copy_32_bytes(out, from); } break; case 30: for (; len >= 30; len -= 30) { out = copy_16_bytes(out, from); out = copy_8_bytes(out, from + 16); out = copy_4_bytes(out, from + 24); out = copy_2_bytes(out, from + 28); } break; case 28: for (; len >= 28; len -= 28) { out = copy_16_bytes(out, from); out = copy_8_bytes(out, from + 16); out = copy_4_bytes(out, from + 24); } break; case 26: for (; len >= 26; len -= 26) { out = copy_16_bytes(out, from); out = copy_8_bytes(out, from + 16); out = copy_2_bytes(out, from + 24); } break; case 24: for (; len >= 24; len -= 24) { out = copy_16_bytes(out, from); out = copy_8_bytes(out, from + 16); } break; case 22: for (; len >= 22; len -= 22) { out = copy_16_bytes(out, from); out = copy_4_bytes(out, from + 16); out = copy_2_bytes(out, from + 20); } break; case 20: for (; len >= 20; len -= 20) { out = copy_16_bytes(out, from); out = copy_4_bytes(out, from + 16); } break; case 18: for (; len >= 18; len -= 18) { out = copy_16_bytes(out, from); out = copy_2_bytes(out, from + 16); } break; case 16: for (; len >= 16; len -= 16) { out = copy_16_bytes(out, from); } break; case 8: for (; len >= 8; len -= 8) { out = copy_8_bytes(out, from); } break; case 4: for (; len >= 4; len -= 4) { out = copy_4_bytes(out, from); } break; case 2: for (; len >= 2; len -= 2) { out = copy_2_bytes(out, from); } break; default: for (; len > 0; len--) { *out++ = *from++; } } // Copy the leftovers for (; len > 0; len--) { *out++ = *from++; } return out; } zmat-0.9.9/src/blosc2/blosc/sframe.h0000644000175200007730000000161014515254731017440 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef BLOSC_SFRAME_H #define BLOSC_SFRAME_H void* sframe_open_index(const char* urlpath, const char* mode, const blosc2_io *io); void* sframe_open_chunk(const char* urlpath, int64_t nchunk, const char* mode, const blosc2_io *io); int sframe_delete_chunk(const char* urlpath, int64_t nchunk); void* sframe_create_chunk(blosc2_frame_s* frame, uint8_t* chunk, int64_t nchunk, int64_t cbytes); int32_t sframe_get_chunk(blosc2_frame_s* frame, int64_t nchunk, uint8_t** chunk, bool* needs_free); #endif //BLOSC_SFRAME_H zmat-0.9.9/src/blosc2/blosc/Makefile0000644000175200007730000000600614515254731017456 0ustar rlaboissrlaboiss################################################################# # Makefile for ZMAT # Qianqian Fang # 2019/04/30 ################################################################# BACKEND ?= ROOTDIR ?=.. ZMATDIR ?=$(ROOTDIR) LIBDIR ?=$(ROOTDIR)/lib MKDIR :=mkdir MEX=mex AR=$(CC) ECHO := echo BINARY:=blosc2 OUTPUT_DIR=$(ZMATDIR) DOXY := doxygen DOCDIR := $(ZMATDIR)/doc DOXYCFG=zmat.cfg INCLUDEDIRS=-I../../lz4 -I../include -I../internal-complibs/zstd ifeq ($(HAVE_ZLIB),miniz) HAVE_ZLIB=yes INCLUDEDIRS+=-I../../miniz endif CUOMPLINK= ARCH = $(shell uname -m) PLATFORM = $(shell uname -s) DLLFLAG=-fPIC CPPOPT=-g -Wall -Wextra -O3 -msse2 -DNDEBUG $(DLLFLAG) -std=gnu99 -Wno-unused-variable #-g -Wall -std=c99 # -DUSE_OS_TIMER ifeq ($(HAVE_ZLIB),yes) CPPOPT+=-DHAVE_ZLIB endif ifeq ($(HAVE_ZSTD),yes) CPPOPT+=-DHAVE_ZSTD endif ifeq ($(HAVE_LZ4),yes) CPPOPT+=-DHAVE_LZ4 endif OUTPUTFLAG:=-o OBJSUFFIX=.o EXESUFFIX=.mex* FILES=bitshuffle-sse2 blosc2 blosc2-stdio blosclz delta directories fastcopy \ frame schunk sframe shuffle shuffle-generic bitshuffle-generic \ shuffle-sse2 stune timestamp trunc-prec ifeq ($(findstring CYGWIN,$(PLATFORM)), CYGWIN) CPPOPT =-c -DWIN32 OBJSUFFIX=.obj EXESUFFIX= DLLFLAG= MEX=cmd /c mex else ifeq ($(findstring Darwin,$(PLATFORM)), Darwin) else CPPOPT+= ifeq ($(findstring x86_64,$(ARCH)), x86_64) CPPOPT += endif endif ifeq ($(MAKECMDGOALS),lib) AR :=ar ARFLAGS :=cr BINARY :=libblosc2.a AROUTPUT := LINKOPT := OUTPUT_DIR :=$(LIBDIR) ifeq ($(findstring Darwin,$(PLATFORM)), Darwin) OUTPUTFLAG := endif endif ifeq ($(MAKECMDGOALS),dll) OUTPUTFLAG :=-o BINARY :=libblosc2.so OUTPUT_DIR :=$(LIBDIR) ifeq ($(findstring Darwin,$(PLATFORM)), Darwin) ARFLAGS :=-shared -Wl,-install_name,$(BINARY).1 -lz else ARFLAGS :=-shared -Wl,-soname,$(BINARY).1 -lz endif endif ifeq ($(MAKECMDGOALS),dll) BINARY :=libblosc2.so endif dll: CPPOPT +=$(DLLFLAG) dll: AR :=$(CC) dll: ARFLAGS ?=-shared -Wl,-soname,$(BINARY).1 dll: LINKOPT :=$(LDFLAGS) dll: AROUTPUT :=-o all: lib TARGETSUFFIX:=$(suffix $(BINARY)) doc: makedocdir $(DOXY) $(DOXYCFG) OBJS := $(addsuffix $(OBJSUFFIX), $(FILES)) all dll lib mex oct: $(OUTPUT_DIR)/$(BINARY) makedirs: @if test ! -d $(OUTPUT_DIR); then $(MKDIR) $(OUTPUT_DIR); fi makedocdir: @if test ! -d $(DOCDIR); then $(MKDIR) $(DOCDIR); fi $(OUTPUT_DIR)/$(BINARY): makedirs $(OBJS) $(OUTPUT_DIR)/$(BINARY): $(OBJS) @$(ECHO) Building $@ $(AR) $(ARFLAGS) $(OUTPUTFLAG) $@ $(OBJS) $(LINKOPT) %$(OBJSUFFIX): %.cpp $(CXX) $(INCLUDEDIRS) $(CPPOPT) -c -o $@ $< %$(OBJSUFFIX): %.c @$(ECHO) Building $@ $(CC) $(CPPFLAGS) $(CFLAGS) $(INCLUDEDIRS) $(CPPOPT) -c -o $@ $< %$(OBJSUFFIX): %.cu @$(ECHO) Building $@ $(CUDACC) -c $(CUCCOPT) -o $@ $< clean: -rm -f $(OBJS) $(OUTPUT_DIR)/$(BINARY)$(EXESUFFIX) zmat$(OBJSUFFIX) $(LIBDIR)/* .PHONY: all lib dll .DEFAULT_GOAL := all zmat-0.9.9/src/blosc2/blosc/shuffle-avx2.c0000644000175200007730000007556214515254731020511 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include "shuffle-generic.h" #include "shuffle-avx2.h" /* Make sure AVX2 is available for the compilation target and compiler. */ #if defined(__AVX2__) #include /* The next is useful for debugging purposes */ #if 0 #include #include static void printymm(__m256i ymm0) { uint8_t buf[32]; ((__m256i *)buf)[0] = ymm0; printf("%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x\n", buf[0], buf[1], buf[2], buf[3], buf[4], buf[5], buf[6], buf[7], buf[8], buf[9], buf[10], buf[11], buf[12], buf[13], buf[14], buf[15], buf[16], buf[17], buf[18], buf[19], buf[20], buf[21], buf[22], buf[23], buf[24], buf[25], buf[26], buf[27], buf[28], buf[29], buf[30], buf[31]); } #endif /* GCC doesn't include the split load/store intrinsics needed for the tiled shuffle, so define them here. */ #if defined(__GNUC__) && !defined(__clang__) && !defined(__ICC) static inline __m256i __attribute__((__always_inline__)) _mm256_loadu2_m128i(const __m128i* const hiaddr, const __m128i* const loaddr) { return _mm256_inserti128_si256( _mm256_castsi128_si256(_mm_loadu_si128(loaddr)), _mm_loadu_si128(hiaddr), 1); } static inline void __attribute__((__always_inline__)) _mm256_storeu2_m128i(__m128i* const hiaddr, __m128i* const loaddr, const __m256i a) { _mm_storeu_si128(loaddr, _mm256_castsi256_si128(a)); _mm_storeu_si128(hiaddr, _mm256_extracti128_si256(a, 1)); } #endif /* defined(__GNUC__) */ /* Routine optimized for shuffling a buffer for a type size of 2 bytes. */ static void shuffle2_avx2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 2; int32_t j; int k; __m256i ymm0[2], ymm1[2]; /* Create the shuffle mask. NOTE: The XMM/YMM 'set' intrinsics require the arguments to be ordered from most to least significant (i.e., their order is reversed when compared to loading the mask from an array). */ const __m256i shmask = _mm256_set_epi8( 0x0f, 0x0d, 0x0b, 0x09, 0x07, 0x05, 0x03, 0x01, 0x0e, 0x0c, 0x0a, 0x08, 0x06, 0x04, 0x02, 0x00, 0x0f, 0x0d, 0x0b, 0x09, 0x07, 0x05, 0x03, 0x01, 0x0e, 0x0c, 0x0a, 0x08, 0x06, 0x04, 0x02, 0x00); for (j = 0; j < vectorizable_elements; j += sizeof(__m256i)) { /* Fetch 32 elements (64 bytes) then transpose bytes, words and double words. */ for (k = 0; k < 2; k++) { ymm0[k] = _mm256_loadu_si256((__m256i*)(src + (j * bytesoftype) + (k * sizeof(__m256i)))); ymm1[k] = _mm256_shuffle_epi8(ymm0[k], shmask); } ymm0[0] = _mm256_permute4x64_epi64(ymm1[0], 0xd8); ymm0[1] = _mm256_permute4x64_epi64(ymm1[1], 0x8d); ymm1[0] = _mm256_blend_epi32(ymm0[0], ymm0[1], 0xf0); ymm0[1] = _mm256_blend_epi32(ymm0[0], ymm0[1], 0x0f); ymm1[1] = _mm256_permute4x64_epi64(ymm0[1], 0x4e); /* Store the result vectors */ uint8_t* const dest_for_jth_element = dest + j; for (k = 0; k < 2; k++) { _mm256_storeu_si256((__m256i*)(dest_for_jth_element + (k * total_elements)), ymm1[k]); } } } /* Routine optimized for shuffling a buffer for a type size of 4 bytes. */ static void shuffle4_avx2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 4; int32_t i; int j; __m256i ymm0[4], ymm1[4]; /* Create the shuffle mask. NOTE: The XMM/YMM 'set' intrinsics require the arguments to be ordered from most to least significant (i.e., their order is reversed when compared to loading the mask from an array). */ const __m256i mask = _mm256_set_epi32( 0x07, 0x03, 0x06, 0x02, 0x05, 0x01, 0x04, 0x00); for (i = 0; i < vectorizable_elements; i += sizeof(__m256i)) { /* Fetch 32 elements (128 bytes) then transpose bytes and words. */ for (j = 0; j < 4; j++) { ymm0[j] = _mm256_loadu_si256((__m256i*)(src + (i * bytesoftype) + (j * sizeof(__m256i)))); ymm1[j] = _mm256_shuffle_epi32(ymm0[j], 0xd8); ymm0[j] = _mm256_shuffle_epi32(ymm0[j], 0x8d); ymm0[j] = _mm256_unpacklo_epi8(ymm1[j], ymm0[j]); ymm1[j] = _mm256_shuffle_epi32(ymm0[j], 0x04e); ymm0[j] = _mm256_unpacklo_epi16(ymm0[j], ymm1[j]); } /* Transpose double words */ for (j = 0; j < 2; j++) { ymm1[j * 2] = _mm256_unpacklo_epi32(ymm0[j * 2], ymm0[j * 2 + 1]); ymm1[j * 2 + 1] = _mm256_unpackhi_epi32(ymm0[j * 2], ymm0[j * 2 + 1]); } /* Transpose quad words */ for (j = 0; j < 2; j++) { ymm0[j * 2] = _mm256_unpacklo_epi64(ymm1[j], ymm1[j + 2]); ymm0[j * 2 + 1] = _mm256_unpackhi_epi64(ymm1[j], ymm1[j + 2]); } for (j = 0; j < 4; j++) { ymm0[j] = _mm256_permutevar8x32_epi32(ymm0[j], mask); } /* Store the result vectors */ uint8_t* const dest_for_ith_element = dest + i; for (j = 0; j < 4; j++) { _mm256_storeu_si256((__m256i*)(dest_for_ith_element + (j * total_elements)), ymm0[j]); } } } /* Routine optimized for shuffling a buffer for a type size of 8 bytes. */ static void shuffle8_avx2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 8; int32_t j; int k, l; __m256i ymm0[8], ymm1[8]; for (j = 0; j < vectorizable_elements; j += sizeof(__m256i)) { /* Fetch 32 elements (256 bytes) then transpose bytes. */ for (k = 0; k < 8; k++) { ymm0[k] = _mm256_loadu_si256((__m256i*)(src + (j * bytesoftype) + (k * sizeof(__m256i)))); ymm1[k] = _mm256_shuffle_epi32(ymm0[k], 0x4e); ymm1[k] = _mm256_unpacklo_epi8(ymm0[k], ymm1[k]); } /* Transpose words */ for (k = 0, l = 0; k < 4; k++, l += 2) { ymm0[k * 2] = _mm256_unpacklo_epi16(ymm1[l], ymm1[l + 1]); ymm0[k * 2 + 1] = _mm256_unpackhi_epi16(ymm1[l], ymm1[l + 1]); } /* Transpose double words */ for (k = 0, l = 0; k < 4; k++, l++) { if (k == 2) l += 2; ymm1[k * 2] = _mm256_unpacklo_epi32(ymm0[l], ymm0[l + 2]); ymm1[k * 2 + 1] = _mm256_unpackhi_epi32(ymm0[l], ymm0[l + 2]); } /* Transpose quad words */ for (k = 0; k < 4; k++) { ymm0[k * 2] = _mm256_unpacklo_epi64(ymm1[k], ymm1[k + 4]); ymm0[k * 2 + 1] = _mm256_unpackhi_epi64(ymm1[k], ymm1[k + 4]); } for (k = 0; k < 8; k++) { ymm1[k] = _mm256_permute4x64_epi64(ymm0[k], 0x72); ymm0[k] = _mm256_permute4x64_epi64(ymm0[k], 0xD8); ymm0[k] = _mm256_unpacklo_epi16(ymm0[k], ymm1[k]); } /* Store the result vectors */ uint8_t* const dest_for_jth_element = dest + j; for (k = 0; k < 8; k++) { _mm256_storeu_si256((__m256i*)(dest_for_jth_element + (k * total_elements)), ymm0[k]); } } } /* Routine optimized for shuffling a buffer for a type size of 16 bytes. */ static void shuffle16_avx2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 16; int32_t j; int k, l; __m256i ymm0[16], ymm1[16]; /* Create the shuffle mask. NOTE: The XMM/YMM 'set' intrinsics require the arguments to be ordered from most to least significant (i.e., their order is reversed when compared to loading the mask from an array). */ const __m256i shmask = _mm256_set_epi8( 0x0f, 0x07, 0x0e, 0x06, 0x0d, 0x05, 0x0c, 0x04, 0x0b, 0x03, 0x0a, 0x02, 0x09, 0x01, 0x08, 0x00, 0x0f, 0x07, 0x0e, 0x06, 0x0d, 0x05, 0x0c, 0x04, 0x0b, 0x03, 0x0a, 0x02, 0x09, 0x01, 0x08, 0x00); for (j = 0; j < vectorizable_elements; j += sizeof(__m256i)) { /* Fetch 32 elements (512 bytes) into 16 YMM registers. */ for (k = 0; k < 16; k++) { ymm0[k] = _mm256_loadu_si256((__m256i*)(src + (j * bytesoftype) + (k * sizeof(__m256i)))); } /* Transpose bytes */ for (k = 0, l = 0; k < 8; k++, l += 2) { ymm1[k * 2] = _mm256_unpacklo_epi8(ymm0[l], ymm0[l + 1]); ymm1[k * 2 + 1] = _mm256_unpackhi_epi8(ymm0[l], ymm0[l + 1]); } /* Transpose words */ for (k = 0, l = -2; k < 8; k++, l++) { if ((k % 2) == 0) l += 2; ymm0[k * 2] = _mm256_unpacklo_epi16(ymm1[l], ymm1[l + 2]); ymm0[k * 2 + 1] = _mm256_unpackhi_epi16(ymm1[l], ymm1[l + 2]); } /* Transpose double words */ for (k = 0, l = -4; k < 8; k++, l++) { if ((k % 4) == 0) l += 4; ymm1[k * 2] = _mm256_unpacklo_epi32(ymm0[l], ymm0[l + 4]); ymm1[k * 2 + 1] = _mm256_unpackhi_epi32(ymm0[l], ymm0[l + 4]); } /* Transpose quad words */ for (k = 0; k < 8; k++) { ymm0[k * 2] = _mm256_unpacklo_epi64(ymm1[k], ymm1[k + 8]); ymm0[k * 2 + 1] = _mm256_unpackhi_epi64(ymm1[k], ymm1[k + 8]); } for (k = 0; k < 16; k++) { ymm0[k] = _mm256_permute4x64_epi64(ymm0[k], 0xd8); ymm0[k] = _mm256_shuffle_epi8(ymm0[k], shmask); } /* Store the result vectors */ uint8_t* const dest_for_jth_element = dest + j; for (k = 0; k < 16; k++) { _mm256_storeu_si256((__m256i*)(dest_for_jth_element + (k * total_elements)), ymm0[k]); } } } /* Routine optimized for shuffling a buffer for a type size larger than 16 bytes. */ static void shuffle16_tiled_avx2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements, const int32_t bytesoftype) { int32_t j; int k, l; __m256i ymm0[16], ymm1[16]; const lldiv_t vecs_per_el = lldiv(bytesoftype, sizeof(__m128i)); const int32_t vecs_rem = (int32_t)vecs_per_el.rem; /* Create the shuffle mask. NOTE: The XMM/YMM 'set' intrinsics require the arguments to be ordered from most to least significant (i.e., their order is reversed when compared to loading the mask from an array). */ const __m256i shmask = _mm256_set_epi8( 0x0f, 0x07, 0x0e, 0x06, 0x0d, 0x05, 0x0c, 0x04, 0x0b, 0x03, 0x0a, 0x02, 0x09, 0x01, 0x08, 0x00, 0x0f, 0x07, 0x0e, 0x06, 0x0d, 0x05, 0x0c, 0x04, 0x0b, 0x03, 0x0a, 0x02, 0x09, 0x01, 0x08, 0x00); for (j = 0; j < vectorizable_elements; j += sizeof(__m256i)) { /* Advance the offset into the type by the vector size (in bytes), unless this is the initial iteration and the type size is not a multiple of the vector size. In that case, only advance by the number of bytes necessary so that the number of remaining bytes in the type will be a multiple of the vector size. */ int32_t offset_into_type; for (offset_into_type = 0; offset_into_type < bytesoftype; offset_into_type += (offset_into_type == 0 && vecs_rem > 0 ? vecs_rem : (int32_t)sizeof(__m128i))) { /* Fetch elements in groups of 512 bytes */ const uint8_t* const src_with_offset = src + offset_into_type; for (k = 0; k < 16; k++) { ymm0[k] = _mm256_loadu2_m128i( (__m128i*)(src_with_offset + (j + (2 * k) + 1) * bytesoftype), (__m128i*)(src_with_offset + (j + (2 * k)) * bytesoftype)); } /* Transpose bytes */ for (k = 0, l = 0; k < 8; k++, l += 2) { ymm1[k * 2] = _mm256_unpacklo_epi8(ymm0[l], ymm0[l + 1]); ymm1[k * 2 + 1] = _mm256_unpackhi_epi8(ymm0[l], ymm0[l + 1]); } /* Transpose words */ for (k = 0, l = -2; k < 8; k++, l++) { if ((k % 2) == 0) l += 2; ymm0[k * 2] = _mm256_unpacklo_epi16(ymm1[l], ymm1[l + 2]); ymm0[k * 2 + 1] = _mm256_unpackhi_epi16(ymm1[l], ymm1[l + 2]); } /* Transpose double words */ for (k = 0, l = -4; k < 8; k++, l++) { if ((k % 4) == 0) l += 4; ymm1[k * 2] = _mm256_unpacklo_epi32(ymm0[l], ymm0[l + 4]); ymm1[k * 2 + 1] = _mm256_unpackhi_epi32(ymm0[l], ymm0[l + 4]); } /* Transpose quad words */ for (k = 0; k < 8; k++) { ymm0[k * 2] = _mm256_unpacklo_epi64(ymm1[k], ymm1[k + 8]); ymm0[k * 2 + 1] = _mm256_unpackhi_epi64(ymm1[k], ymm1[k + 8]); } for (k = 0; k < 16; k++) { ymm0[k] = _mm256_permute4x64_epi64(ymm0[k], 0xd8); ymm0[k] = _mm256_shuffle_epi8(ymm0[k], shmask); } /* Store the result vectors */ uint8_t* const dest_for_jth_element = dest + j; for (k = 0; k < 16; k++) { _mm256_storeu_si256((__m256i*)(dest_for_jth_element + (total_elements * (offset_into_type + k))), ymm0[k]); } } } } /* Routine optimized for unshuffling a buffer for a type size of 2 bytes. */ static void unshuffle2_avx2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 2; int32_t i; int j; __m256i ymm0[2], ymm1[2]; for (i = 0; i < vectorizable_elements; i += sizeof(__m256i)) { /* Load 32 elements (64 bytes) into 2 YMM registers. */ const uint8_t* const src_for_ith_element = src + i; for (j = 0; j < 2; j++) { ymm0[j] = _mm256_loadu_si256((__m256i*)(src_for_ith_element + (j * total_elements))); } /* Shuffle bytes */ for (j = 0; j < 2; j++) { ymm0[j] = _mm256_permute4x64_epi64(ymm0[j], 0xd8); } /* Compute the low 64 bytes */ ymm1[0] = _mm256_unpacklo_epi8(ymm0[0], ymm0[1]); /* Compute the hi 64 bytes */ ymm1[1] = _mm256_unpackhi_epi8(ymm0[0], ymm0[1]); /* Store the result vectors in proper order */ _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (0 * sizeof(__m256i))), ymm1[0]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (1 * sizeof(__m256i))), ymm1[1]); } } /* Routine optimized for unshuffling a buffer for a type size of 4 bytes. */ static void unshuffle4_avx2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 4; int32_t i; int j; __m256i ymm0[4], ymm1[4]; for (i = 0; i < vectorizable_elements; i += sizeof(__m256i)) { /* Load 32 elements (128 bytes) into 4 YMM registers. */ const uint8_t* const src_for_ith_element = src + i; for (j = 0; j < 4; j++) { ymm0[j] = _mm256_loadu_si256((__m256i*)(src_for_ith_element + (j * total_elements))); } /* Shuffle bytes */ for (j = 0; j < 2; j++) { /* Compute the low 64 bytes */ ymm1[j] = _mm256_unpacklo_epi8(ymm0[j * 2], ymm0[j * 2 + 1]); /* Compute the hi 64 bytes */ ymm1[2 + j] = _mm256_unpackhi_epi8(ymm0[j * 2], ymm0[j * 2 + 1]); } /* Shuffle 2-byte words */ for (j = 0; j < 2; j++) { /* Compute the low 64 bytes */ ymm0[j] = _mm256_unpacklo_epi16(ymm1[j * 2], ymm1[j * 2 + 1]); /* Compute the hi 64 bytes */ ymm0[2 + j] = _mm256_unpackhi_epi16(ymm1[j * 2], ymm1[j * 2 + 1]); } ymm1[0] = _mm256_permute2x128_si256(ymm0[0], ymm0[2], 0x20); ymm1[1] = _mm256_permute2x128_si256(ymm0[1], ymm0[3], 0x20); ymm1[2] = _mm256_permute2x128_si256(ymm0[0], ymm0[2], 0x31); ymm1[3] = _mm256_permute2x128_si256(ymm0[1], ymm0[3], 0x31); /* Store the result vectors in proper order */ for (j = 0; j < 4; j++) { _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (j * sizeof(__m256i))), ymm1[j]); } } } /* Routine optimized for unshuffling a buffer for a type size of 8 bytes. */ static void unshuffle8_avx2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 8; int32_t i; int j; __m256i ymm0[8], ymm1[8]; for (i = 0; i < vectorizable_elements; i += sizeof(__m256i)) { /* Fetch 32 elements (256 bytes) into 8 YMM registers. */ const uint8_t* const src_for_ith_element = src + i; for (j = 0; j < 8; j++) { ymm0[j] = _mm256_loadu_si256((__m256i*)(src_for_ith_element + (j * total_elements))); } /* Shuffle bytes */ for (j = 0; j < 4; j++) { /* Compute the low 32 bytes */ ymm1[j] = _mm256_unpacklo_epi8(ymm0[j * 2], ymm0[j * 2 + 1]); /* Compute the hi 32 bytes */ ymm1[4 + j] = _mm256_unpackhi_epi8(ymm0[j * 2], ymm0[j * 2 + 1]); } /* Shuffle words */ for (j = 0; j < 4; j++) { /* Compute the low 32 bytes */ ymm0[j] = _mm256_unpacklo_epi16(ymm1[j * 2], ymm1[j * 2 + 1]); /* Compute the hi 32 bytes */ ymm0[4 + j] = _mm256_unpackhi_epi16(ymm1[j * 2], ymm1[j * 2 + 1]); } for (j = 0; j < 8; j++) { ymm0[j] = _mm256_permute4x64_epi64(ymm0[j], 0xd8); } /* Shuffle 4-byte dwords */ for (j = 0; j < 4; j++) { /* Compute the low 32 bytes */ ymm1[j] = _mm256_unpacklo_epi32(ymm0[j * 2], ymm0[j * 2 + 1]); /* Compute the hi 32 bytes */ ymm1[4 + j] = _mm256_unpackhi_epi32(ymm0[j * 2], ymm0[j * 2 + 1]); } /* Store the result vectors in proper order */ _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (0 * sizeof(__m256i))), ymm1[0]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (1 * sizeof(__m256i))), ymm1[2]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (2 * sizeof(__m256i))), ymm1[1]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (3 * sizeof(__m256i))), ymm1[3]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (4 * sizeof(__m256i))), ymm1[4]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (5 * sizeof(__m256i))), ymm1[6]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (6 * sizeof(__m256i))), ymm1[5]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (7 * sizeof(__m256i))), ymm1[7]); } } /* Routine optimized for unshuffling a buffer for a type size of 16 bytes. */ static void unshuffle16_avx2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 16; int32_t i; int j; __m256i ymm0[16], ymm1[16]; for (i = 0; i < vectorizable_elements; i += sizeof(__m256i)) { /* Fetch 32 elements (512 bytes) into 16 YMM registers. */ const uint8_t* const src_for_ith_element = src + i; for (j = 0; j < 16; j++) { ymm0[j] = _mm256_loadu_si256((__m256i*)(src_for_ith_element + (j * total_elements))); } /* Shuffle bytes */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ ymm1[j] = _mm256_unpacklo_epi8(ymm0[j * 2], ymm0[j * 2 + 1]); /* Compute the hi 32 bytes */ ymm1[8 + j] = _mm256_unpackhi_epi8(ymm0[j * 2], ymm0[j * 2 + 1]); } /* Shuffle 2-byte words */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ ymm0[j] = _mm256_unpacklo_epi16(ymm1[j * 2], ymm1[j * 2 + 1]); /* Compute the hi 32 bytes */ ymm0[8 + j] = _mm256_unpackhi_epi16(ymm1[j * 2], ymm1[j * 2 + 1]); } /* Shuffle 4-byte dwords */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ ymm1[j] = _mm256_unpacklo_epi32(ymm0[j * 2], ymm0[j * 2 + 1]); /* Compute the hi 32 bytes */ ymm1[8 + j] = _mm256_unpackhi_epi32(ymm0[j * 2], ymm0[j * 2 + 1]); } /* Shuffle 8-byte qwords */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ ymm0[j] = _mm256_unpacklo_epi64(ymm1[j * 2], ymm1[j * 2 + 1]); /* Compute the hi 32 bytes */ ymm0[8 + j] = _mm256_unpackhi_epi64(ymm1[j * 2], ymm1[j * 2 + 1]); } for (j = 0; j < 8; j++) { ymm1[j] = _mm256_permute2x128_si256(ymm0[j], ymm0[j + 8], 0x20); ymm1[j + 8] = _mm256_permute2x128_si256(ymm0[j], ymm0[j + 8], 0x31); } /* Store the result vectors in proper order */ _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (0 * sizeof(__m256i))), ymm1[0]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (1 * sizeof(__m256i))), ymm1[4]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (2 * sizeof(__m256i))), ymm1[2]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (3 * sizeof(__m256i))), ymm1[6]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (4 * sizeof(__m256i))), ymm1[1]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (5 * sizeof(__m256i))), ymm1[5]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (6 * sizeof(__m256i))), ymm1[3]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (7 * sizeof(__m256i))), ymm1[7]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (8 * sizeof(__m256i))), ymm1[8]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (9 * sizeof(__m256i))), ymm1[12]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (10 * sizeof(__m256i))), ymm1[10]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (11 * sizeof(__m256i))), ymm1[14]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (12 * sizeof(__m256i))), ymm1[9]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (13 * sizeof(__m256i))), ymm1[13]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (14 * sizeof(__m256i))), ymm1[11]); _mm256_storeu_si256((__m256i*)(dest + (i * bytesoftype) + (15 * sizeof(__m256i))), ymm1[15]); } } /* Routine optimized for unshuffling a buffer for a type size larger than 16 bytes. */ static void unshuffle16_tiled_avx2(const uint8_t *dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements, const int32_t bytesoftype) { int32_t i; int j; __m256i ymm0[16], ymm1[16]; const lldiv_t vecs_per_el = lldiv(bytesoftype, sizeof(__m128i)); const int32_t vecs_rem = (int32_t)vecs_per_el.rem; /* The unshuffle loops are inverted (compared to shuffle_tiled16_avx2) to optimize cache utilization. */ int32_t offset_into_type; for (offset_into_type = 0; offset_into_type < bytesoftype; offset_into_type += (offset_into_type == 0 && vecs_rem > 0 ? vecs_rem : (int32_t)sizeof(__m128i))) { for (i = 0; i < vectorizable_elements; i += sizeof(__m256i)) { /* Load the first 16 bytes of 32 adjacent elements (512 bytes) into 16 YMM registers */ const uint8_t* const src_for_ith_element = src + i; for (j = 0; j < 16; j++) { ymm0[j] = _mm256_loadu_si256((__m256i*)(src_for_ith_element + (total_elements * (offset_into_type + j)))); } /* Shuffle bytes */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ ymm1[j] = _mm256_unpacklo_epi8(ymm0[j * 2], ymm0[j * 2 + 1]); /* Compute the hi 32 bytes */ ymm1[8 + j] = _mm256_unpackhi_epi8(ymm0[j * 2], ymm0[j * 2 + 1]); } /* Shuffle 2-byte words */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ ymm0[j] = _mm256_unpacklo_epi16(ymm1[j * 2], ymm1[j * 2 + 1]); /* Compute the hi 32 bytes */ ymm0[8 + j] = _mm256_unpackhi_epi16(ymm1[j * 2], ymm1[j * 2 + 1]); } /* Shuffle 4-byte dwords */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ ymm1[j] = _mm256_unpacklo_epi32(ymm0[j * 2], ymm0[j * 2 + 1]); /* Compute the hi 32 bytes */ ymm1[8 + j] = _mm256_unpackhi_epi32(ymm0[j * 2], ymm0[j * 2 + 1]); } /* Shuffle 8-byte qwords */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ ymm0[j] = _mm256_unpacklo_epi64(ymm1[j * 2], ymm1[j * 2 + 1]); /* Compute the hi 32 bytes */ ymm0[8 + j] = _mm256_unpackhi_epi64(ymm1[j * 2], ymm1[j * 2 + 1]); } for (j = 0; j < 8; j++) { ymm1[j] = _mm256_permute2x128_si256(ymm0[j], ymm0[j + 8], 0x20); ymm1[j + 8] = _mm256_permute2x128_si256(ymm0[j], ymm0[j + 8], 0x31); } /* Store the result vectors in proper order */ const uint8_t* const dest_with_offset = dest + offset_into_type; _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x01) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x00) * bytesoftype), ymm1[0]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x03) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x02) * bytesoftype), ymm1[4]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x05) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x04) * bytesoftype), ymm1[2]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x07) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x06) * bytesoftype), ymm1[6]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x09) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x08) * bytesoftype), ymm1[1]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x0b) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x0a) * bytesoftype), ymm1[5]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x0d) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x0c) * bytesoftype), ymm1[3]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x0f) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x0e) * bytesoftype), ymm1[7]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x11) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x10) * bytesoftype), ymm1[8]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x13) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x12) * bytesoftype), ymm1[12]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x15) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x14) * bytesoftype), ymm1[10]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x17) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x16) * bytesoftype), ymm1[14]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x19) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x18) * bytesoftype), ymm1[9]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x1b) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x1a) * bytesoftype), ymm1[13]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x1d) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x1c) * bytesoftype), ymm1[11]); _mm256_storeu2_m128i( (__m128i*)(dest_with_offset + (i + 0x1f) * bytesoftype), (__m128i*)(dest_with_offset + (i + 0x1e) * bytesoftype), ymm1[15]); } } } /* Shuffle a block. This can never fail. */ void shuffle_avx2(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest) { const int32_t vectorized_chunk_size = bytesoftype * (int32_t)sizeof(__m256i); /* If the block size is too small to be vectorized, use the generic implementation. */ if (blocksize < vectorized_chunk_size) { shuffle_generic(bytesoftype, blocksize, _src, _dest); return; } /* If the blocksize is not a multiple of both the typesize and the vector size, round the blocksize down to the next value which is a multiple of both. The vectorized shuffle can be used for that portion of the data, and the naive implementation can be used for the remaining portion. */ const int32_t vectorizable_bytes = blocksize - (blocksize % vectorized_chunk_size); const int32_t vectorizable_elements = vectorizable_bytes / bytesoftype; const int32_t total_elements = blocksize / bytesoftype; /* Optimized shuffle implementations */ switch (bytesoftype) { case 2: shuffle2_avx2(_dest, _src, vectorizable_elements, total_elements); break; case 4: shuffle4_avx2(_dest, _src, vectorizable_elements, total_elements); break; case 8: shuffle8_avx2(_dest, _src, vectorizable_elements, total_elements); break; case 16: shuffle16_avx2(_dest, _src, vectorizable_elements, total_elements); break; default: /* For types larger than 16 bytes, use the AVX2 tiled shuffle. */ if (bytesoftype > (int32_t)sizeof(__m128i)) { shuffle16_tiled_avx2(_dest, _src, vectorizable_elements, total_elements, bytesoftype); } else { /* Non-optimized shuffle */ shuffle_generic(bytesoftype, blocksize, _src, _dest); /* The non-optimized function covers the whole buffer, so we're done processing here. */ return; } } /* If the buffer had any bytes at the end which couldn't be handled by the vectorized implementations, use the non-optimized version to finish them up. */ if (vectorizable_bytes < blocksize) { shuffle_generic_inline(bytesoftype, vectorizable_bytes, blocksize, _src, _dest); } } /* Unshuffle a block. This can never fail. */ void unshuffle_avx2(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest) { const int32_t vectorized_chunk_size = bytesoftype * (int32_t)sizeof(__m256i); /* If the block size is too small to be vectorized, use the generic implementation. */ if (blocksize < vectorized_chunk_size) { unshuffle_generic(bytesoftype, blocksize, _src, _dest); return; } /* If the blocksize is not a multiple of both the typesize and the vector size, round the blocksize down to the next value which is a multiple of both. The vectorized unshuffle can be used for that portion of the data, and the naive implementation can be used for the remaining portion. */ const int32_t vectorizable_bytes = blocksize - (blocksize % vectorized_chunk_size); const int32_t vectorizable_elements = vectorizable_bytes / bytesoftype; const int32_t total_elements = blocksize / bytesoftype; /* Optimized unshuffle implementations */ switch (bytesoftype) { case 2: unshuffle2_avx2(_dest, _src, vectorizable_elements, total_elements); break; case 4: unshuffle4_avx2(_dest, _src, vectorizable_elements, total_elements); break; case 8: unshuffle8_avx2(_dest, _src, vectorizable_elements, total_elements); break; case 16: unshuffle16_avx2(_dest, _src, vectorizable_elements, total_elements); break; default: /* For types larger than 16 bytes, use the AVX2 tiled unshuffle. */ if (bytesoftype > (int32_t)sizeof(__m128i)) { unshuffle16_tiled_avx2(_dest, _src, vectorizable_elements, total_elements, bytesoftype); } else { /* Non-optimized unshuffle */ unshuffle_generic(bytesoftype, blocksize, _src, _dest); /* The non-optimized function covers the whole buffer, so we're done processing here. */ return; } } /* If the buffer had any bytes at the end which couldn't be handled by the vectorized implementations, use the non-optimized version to finish them up. */ if (vectorizable_bytes < blocksize) { unshuffle_generic_inline(bytesoftype, vectorizable_bytes, blocksize, _src, _dest); } } #endif /* defined(__AVX2__) */ zmat-0.9.9/src/blosc2/blosc/shuffle-generic.c0000644000175200007730000000172314515254731021231 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include "shuffle-generic.h" /* Shuffle a block. This can never fail. */ void shuffle_generic(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest) { /* Non-optimized shuffle */ shuffle_generic_inline(bytesoftype, 0, blocksize, _src, _dest); } /* Unshuffle a block. This can never fail. */ void unshuffle_generic(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest) { /* Non-optimized unshuffle */ unshuffle_generic_inline(bytesoftype, 0, blocksize, _src, _dest); } zmat-0.9.9/src/blosc2/blosc/b2nd.c0000644000175200007730000017401314515254731017013 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include #include "context.h" #include "b2nd_utils.h" #include "blosc2.h" #include "blosc2/blosc2-common.h" #include int b2nd_serialize_meta(int8_t ndim, const int64_t *shape, const int32_t *chunkshape, const int32_t *blockshape, const char *dtype, int8_t dtype_format, uint8_t **smeta) { if (dtype == NULL) { dtype = B2ND_DEFAULT_DTYPE; } // dtype checks if (dtype_format < 0) { BLOSC_TRACE_ERROR("dtype_format cannot be negative"); BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } size_t dtype_len0 = strlen(dtype); if (dtype_len0 > INT32_MAX) { BLOSC_TRACE_ERROR("dtype is too large (len > %d)", INT32_MAX); BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } const int32_t dtype_len = (int32_t) dtype_len0; // Allocate space for b2nd metalayer int32_t max_smeta_len = (int32_t) (1 + 1 + 1 + (1 + ndim * (1 + sizeof(int64_t))) + (1 + ndim * (1 + sizeof(int32_t))) + (1 + ndim * (1 + sizeof(int32_t))) + 1 + 1 + sizeof(int32_t) + dtype_len); *smeta = malloc((size_t) max_smeta_len); BLOSC_ERROR_NULL(*smeta, BLOSC2_ERROR_MEMORY_ALLOC); uint8_t *pmeta = *smeta; // Build an array with 7 entries (version, ndim, shape, chunkshape, blockshape, dtype_format, dtype) *pmeta++ = 0x90 + 7; // version entry *pmeta++ = B2ND_METALAYER_VERSION; // positive fixnum (7-bit positive integer) // ndim entry *pmeta++ = (uint8_t) ndim; // positive fixnum (7-bit positive integer) // shape entry *pmeta++ = (uint8_t) (0x90) + ndim; // fix array with ndim elements for (uint8_t i = 0; i < ndim; i++) { *pmeta++ = 0xd3; // int64 swap_store(pmeta, shape + i, sizeof(int64_t)); pmeta += sizeof(int64_t); } // chunkshape entry *pmeta++ = (uint8_t) (0x90) + ndim; // fix array with ndim elements for (uint8_t i = 0; i < ndim; i++) { *pmeta++ = 0xd2; // int32 swap_store(pmeta, chunkshape + i, sizeof(int32_t)); pmeta += sizeof(int32_t); } // blockshape entry *pmeta++ = (uint8_t) (0x90) + ndim; // fix array with ndim elements for (uint8_t i = 0; i < ndim; i++) { *pmeta++ = 0xd2; // int32 swap_store(pmeta, blockshape + i, sizeof(int32_t)); pmeta += sizeof(int32_t); } // dtype entry *pmeta++ = dtype_format; // positive fixint (7-bit positive integer) *pmeta++ = (uint8_t) (0xdb); // str with up to 2^31 elements swap_store(pmeta, &dtype_len, sizeof(int32_t)); pmeta += sizeof(int32_t); memcpy(pmeta, dtype, dtype_len); pmeta += dtype_len; int32_t slen = (int32_t) (pmeta - *smeta); if (max_smeta_len != slen) { BLOSC_TRACE_ERROR("meta length is inconsistent!"); return BLOSC2_ERROR_FAILURE; } return (int)slen; } int update_shape(b2nd_array_t *array, int8_t ndim, const int64_t *shape, const int32_t *chunkshape, const int32_t *blockshape) { array->ndim = ndim; array->nitems = 1; array->extnitems = 1; array->extchunknitems = 1; array->chunknitems = 1; array->blocknitems = 1; for (int i = 0; i < B2ND_MAX_DIM; ++i) { if (i < ndim) { array->shape[i] = shape[i]; array->chunkshape[i] = chunkshape[i]; array->blockshape[i] = blockshape[i]; if (shape[i] != 0) { if (shape[i] % array->chunkshape[i] == 0) { array->extshape[i] = shape[i]; } else { array->extshape[i] = shape[i] + chunkshape[i] - shape[i] % chunkshape[i]; } if (chunkshape[i] % blockshape[i] == 0) { array->extchunkshape[i] = chunkshape[i]; } else { array->extchunkshape[i] = chunkshape[i] + blockshape[i] - chunkshape[i] % blockshape[i]; } } else { array->extchunkshape[i] = 0; array->extshape[i] = 0; } } else { array->blockshape[i] = 1; array->chunkshape[i] = 1; array->extshape[i] = 1; array->extchunkshape[i] = 1; array->shape[i] = 1; } array->nitems *= array->shape[i]; array->extnitems *= array->extshape[i]; array->extchunknitems *= array->extchunkshape[i]; array->chunknitems *= array->chunkshape[i]; array->blocknitems *= array->blockshape[i]; } // Compute strides array->item_array_strides[ndim - 1] = 1; array->item_extchunk_strides[ndim - 1] = 1; array->item_chunk_strides[ndim - 1] = 1; array->item_block_strides[ndim - 1] = 1; array->block_chunk_strides[ndim - 1] = 1; array->chunk_array_strides[ndim - 1] = 1; for (int i = ndim - 2; i >= 0; --i) { if (shape[i + 1] != 0) { array->item_array_strides[i] = array->item_array_strides[i + 1] * array->shape[i + 1]; array->item_extchunk_strides[i] = array->item_extchunk_strides[i + 1] * array->extchunkshape[i + 1]; array->item_chunk_strides[i] = array->item_chunk_strides[i + 1] * array->chunkshape[i + 1]; array->item_block_strides[i] = array->item_block_strides[i + 1] * array->blockshape[i + 1]; array->block_chunk_strides[i] = array->block_chunk_strides[i + 1] * (array->extchunkshape[i + 1] / array->blockshape[i + 1]); array->chunk_array_strides[i] = array->chunk_array_strides[i + 1] * (array->extshape[i + 1] * array->chunkshape[i + 1]); } else { array->item_array_strides[i] = 0; array->item_extchunk_strides[i] = 0; array->item_chunk_strides[i] = 0; array->item_block_strides[i] = 0; array->block_chunk_strides[i] = 0; array->chunk_array_strides[i] = 0; } } if (array->sc) { uint8_t *smeta = NULL; // Serialize the dimension info ... int32_t smeta_len = b2nd_serialize_meta(array->ndim, array->shape, array->chunkshape, array->blockshape, array->dtype, array->dtype_format, &smeta); if (smeta_len < 0) { fprintf(stderr, "error during serializing dims info for Blosc2 NDim"); return -1; } // ... and update it in its metalayer if (blosc2_meta_exists(array->sc, "b2nd") < 0) { if (blosc2_meta_add(array->sc, "b2nd", smeta, smeta_len) < 0) { BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } } else { if (blosc2_meta_update(array->sc, "b2nd", smeta, smeta_len) < 0) { BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } } free(smeta); } return BLOSC2_ERROR_SUCCESS; } int array_without_schunk(b2nd_context_t *ctx, b2nd_array_t **array) { /* Create a b2nd_array_t buffer */ (*array) = (b2nd_array_t *) malloc(sizeof(b2nd_array_t)); BLOSC_ERROR_NULL(*array, BLOSC2_ERROR_MEMORY_ALLOC); (*array)->sc = NULL; (*array)->ndim = ctx->ndim; int64_t *shape = ctx->shape; int32_t *chunkshape = ctx->chunkshape; int32_t *blockshape = ctx->blockshape; BLOSC_ERROR(update_shape(*array, ctx->ndim, shape, chunkshape, blockshape)); if (ctx->dtype != NULL) { (*array)->dtype = malloc(strlen(ctx->dtype) + 1); strcpy((*array)->dtype, ctx->dtype); } else { (*array)->dtype = NULL; } (*array)->dtype_format = ctx->dtype_format; // The partition cache (empty initially) (*array)->chunk_cache.data = NULL; (*array)->chunk_cache.nchunk = -1; // means no valid cache yet return BLOSC2_ERROR_SUCCESS; } int array_new(b2nd_context_t *ctx, int special_value, b2nd_array_t **array) { BLOSC_ERROR(array_without_schunk(ctx, array)); blosc2_schunk *sc = blosc2_schunk_new(ctx->b2_storage); if (sc == NULL) { BLOSC_TRACE_ERROR("Pointer is NULL"); return BLOSC2_ERROR_FAILURE; } // Serialize the dimension info if (sc->nmetalayers >= BLOSC2_MAX_METALAYERS) { BLOSC_TRACE_ERROR("the number of metalayers for this schunk has been exceeded"); return BLOSC2_ERROR_FAILURE; } uint8_t *smeta = NULL; int32_t smeta_len = b2nd_serialize_meta(ctx->ndim, (*array)->shape, (*array)->chunkshape, (*array)->blockshape, (*array)->dtype, (*array)->dtype_format, &smeta); if (smeta_len < 0) { BLOSC_TRACE_ERROR("error during serializing dims info for Blosc2 NDim"); return BLOSC2_ERROR_FAILURE; } // And store it in b2nd metalayer if (blosc2_meta_add(sc, "b2nd", smeta, smeta_len) < 0) { return BLOSC2_ERROR_FAILURE; } free(smeta); for (int i = 0; i < ctx->nmetalayers; ++i) { char *name = ctx->metalayers[i].name; uint8_t *data = ctx->metalayers[i].content; int32_t size = ctx->metalayers[i].content_len; if (blosc2_meta_add(sc, name, data, size) < 0) { BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } } // Fill schunk with uninit values if ((*array)->nitems != 0) { int32_t chunksize = (int32_t) (*array)->extchunknitems * sc->typesize; int64_t nchunks = (*array)->extnitems / (*array)->chunknitems; int64_t nitems = nchunks * (*array)->extchunknitems; // blosc2_schunk_fill_special(sc, nitems, BLOSC2_SPECIAL_ZERO, chunksize); BLOSC_ERROR(blosc2_schunk_fill_special(sc, nitems, special_value, chunksize)); } (*array)->sc = sc; return BLOSC2_ERROR_SUCCESS; } int b2nd_uninit(b2nd_context_t *ctx, b2nd_array_t **array) { BLOSC_ERROR_NULL(ctx, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR(array_new(ctx, BLOSC2_SPECIAL_UNINIT, array)); return BLOSC2_ERROR_SUCCESS; } int b2nd_empty(b2nd_context_t *ctx, b2nd_array_t **array) { BLOSC_ERROR_NULL(ctx, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); // BLOSC_ERROR(array_new(ctx, BLOSC2_SPECIAL_UNINIT, array)); // Avoid variable cratios BLOSC_ERROR(array_new(ctx, BLOSC2_SPECIAL_ZERO, array)); return BLOSC2_ERROR_SUCCESS; } int b2nd_zeros(b2nd_context_t *ctx, b2nd_array_t **array) { BLOSC_ERROR_NULL(ctx, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR(array_new(ctx, BLOSC2_SPECIAL_ZERO, array)); return BLOSC2_ERROR_SUCCESS; } int b2nd_full(b2nd_context_t *ctx, b2nd_array_t **array, const void *fill_value) { BLOSC_ERROR_NULL(ctx, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR(b2nd_empty(ctx, array)); int32_t chunkbytes = (int32_t) (*array)->extchunknitems * (*array)->sc->typesize; blosc2_cparams *cparams; if (blosc2_schunk_get_cparams((*array)->sc, &cparams) != 0) { BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } int32_t chunksize = BLOSC_EXTENDED_HEADER_LENGTH + (*array)->sc->typesize; uint8_t *chunk = malloc(chunksize); BLOSC_ERROR_NULL(chunk, BLOSC2_ERROR_MEMORY_ALLOC); if (blosc2_chunk_repeatval(*cparams, chunkbytes, chunk, chunksize, fill_value) < 0) { BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } free(cparams); for (int i = 0; i < (*array)->sc->nchunks; ++i) { if (blosc2_schunk_update_chunk((*array)->sc, i, chunk, true) < 0) { BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } } free(chunk); return BLOSC2_ERROR_SUCCESS; } int b2nd_from_schunk(blosc2_schunk *schunk, b2nd_array_t **array) { BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); if (schunk == NULL) { BLOSC_TRACE_ERROR("Schunk is null"); return BLOSC2_ERROR_NULL_POINTER; } blosc2_cparams *cparams; if (blosc2_schunk_get_cparams(schunk, &cparams) < 0) { BLOSC_TRACE_ERROR("Blosc error"); return BLOSC2_ERROR_NULL_POINTER; } free(cparams); b2nd_context_t params = {0}; params.b2_storage = schunk->storage; // Deserialize the b2nd metalayer uint8_t *smeta; int32_t smeta_len; if (blosc2_meta_get(schunk, "b2nd", &smeta, &smeta_len) < 0) { // Try with a caterva metalayer; we are meant to be backward compatible with it if (blosc2_meta_get(schunk, "caterva", &smeta, &smeta_len) < 0) { BLOSC_ERROR(BLOSC2_ERROR_METALAYER_NOT_FOUND); } } BLOSC_ERROR(b2nd_deserialize_meta(smeta, smeta_len, ¶ms.ndim, params.shape, params.chunkshape, params.blockshape, ¶ms.dtype, ¶ms.dtype_format)); free(smeta); BLOSC_ERROR(array_without_schunk(¶ms, array)); free(params.dtype); (*array)->sc = schunk; if ((*array) == NULL) { BLOSC_TRACE_ERROR("Error creating a b2nd container from a frame"); return BLOSC2_ERROR_NULL_POINTER; } return BLOSC2_ERROR_SUCCESS; } int b2nd_to_cframe(const b2nd_array_t *array, uint8_t **cframe, int64_t *cframe_len, bool *needs_free) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(cframe, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(cframe_len, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(needs_free, BLOSC2_ERROR_NULL_POINTER); *cframe_len = blosc2_schunk_to_buffer(array->sc, cframe, needs_free); if (*cframe_len <= 0) { BLOSC_TRACE_ERROR("Error serializing the b2nd array"); return BLOSC2_ERROR_FAILURE; } return BLOSC2_ERROR_SUCCESS; } int b2nd_from_cframe(uint8_t *cframe, int64_t cframe_len, bool copy, b2nd_array_t **array) { BLOSC_ERROR_NULL(cframe, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); blosc2_schunk *sc = blosc2_schunk_from_buffer(cframe, cframe_len, copy); if (sc == NULL) { BLOSC_TRACE_ERROR("Blosc error"); return BLOSC2_ERROR_FAILURE; } // ...and create a b2nd array out of it BLOSC_ERROR(b2nd_from_schunk(sc, array)); return BLOSC2_ERROR_SUCCESS; } int b2nd_open(const char *urlpath, b2nd_array_t **array) { BLOSC_ERROR_NULL(urlpath, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); blosc2_schunk *sc = blosc2_schunk_open(urlpath); // ...and create a b2nd array out of it BLOSC_ERROR(b2nd_from_schunk(sc, array)); return BLOSC2_ERROR_SUCCESS; } int b2nd_open_offset(const char *urlpath, b2nd_array_t **array, int64_t offset) { BLOSC_ERROR_NULL(urlpath, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); blosc2_schunk *sc = blosc2_schunk_open_offset(urlpath, offset); // ...and create a b2nd array out of it BLOSC_ERROR(b2nd_from_schunk(sc, array)); return BLOSC2_ERROR_SUCCESS; } int b2nd_free(b2nd_array_t *array) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); if (array) { if (array->sc != NULL) { blosc2_schunk_free(array->sc); } free(array->dtype); free(array); } return BLOSC2_ERROR_SUCCESS; } int b2nd_from_cbuffer(b2nd_context_t *ctx, b2nd_array_t **array, const void *buffer, int64_t buffersize) { BLOSC_ERROR_NULL(ctx, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(buffer, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR(b2nd_empty(ctx, array)); if (buffersize < (int64_t) (*array)->nitems * (*array)->sc->typesize) { BLOSC_TRACE_ERROR("The buffersize (%lld) is smaller than the array size (%lld)", (long long) buffersize, (long long) (*array)->nitems * (*array)->sc->typesize); BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } if ((*array)->nitems == 0) { return BLOSC2_ERROR_SUCCESS; } int64_t start[B2ND_MAX_DIM] = {0}; int64_t *stop = (*array)->shape; int64_t *shape = (*array)->shape; BLOSC_ERROR(b2nd_set_slice_cbuffer(buffer, shape, buffersize, start, stop, *array)); return BLOSC2_ERROR_SUCCESS; } int b2nd_to_cbuffer(const b2nd_array_t *array, void *buffer, int64_t buffersize) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(buffer, BLOSC2_ERROR_NULL_POINTER); if (buffersize < (int64_t) array->nitems * array->sc->typesize) { BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } if (array->nitems == 0) { return BLOSC2_ERROR_SUCCESS; } int64_t start[B2ND_MAX_DIM] = {0}; const int64_t *stop = array->shape; BLOSC_ERROR(b2nd_get_slice_cbuffer(array, start, stop, buffer, array->shape, buffersize)); return BLOSC2_ERROR_SUCCESS; } // Setting and getting slices int get_set_slice(void *buffer, int64_t buffersize, const int64_t *start, const int64_t *stop, const int64_t *shape, b2nd_array_t *array, bool set_slice) { BLOSC_ERROR_NULL(buffer, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(start, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(stop, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); if (buffersize < 0) { BLOSC_TRACE_ERROR("buffersize is < 0"); BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } uint8_t *buffer_b = (uint8_t *) buffer; const int64_t *buffer_start = start; const int64_t *buffer_stop = stop; const int64_t *buffer_shape = shape; int8_t ndim = array->ndim; // 0-dim case if (ndim == 0) { if (set_slice) { int32_t chunk_size = array->sc->typesize + BLOSC2_MAX_OVERHEAD; uint8_t *chunk = malloc(chunk_size); BLOSC_ERROR_NULL(chunk, BLOSC2_ERROR_MEMORY_ALLOC); if (blosc2_compress_ctx(array->sc->cctx, buffer_b, array->sc->typesize, chunk, chunk_size) < 0) { BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } if (blosc2_schunk_update_chunk(array->sc, 0, chunk, false) < 0) { BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } } else { if (blosc2_schunk_decompress_chunk(array->sc, 0, buffer_b, array->sc->typesize) < 0) { BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } } return BLOSC2_ERROR_SUCCESS; } int32_t data_nbytes = (int32_t) array->extchunknitems * array->sc->typesize; uint8_t *data = malloc(data_nbytes); BLOSC_ERROR_NULL(data, BLOSC2_ERROR_MEMORY_ALLOC); int64_t chunks_in_array[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { chunks_in_array[i] = array->extshape[i] / array->chunkshape[i]; } int64_t chunks_in_array_strides[B2ND_MAX_DIM]; chunks_in_array_strides[ndim - 1] = 1; for (int i = ndim - 2; i >= 0; --i) { chunks_in_array_strides[i] = chunks_in_array_strides[i + 1] * chunks_in_array[i + 1]; } int64_t blocks_in_chunk[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { blocks_in_chunk[i] = array->extchunkshape[i] / array->blockshape[i]; } // Compute the number of chunks to update int64_t update_start[B2ND_MAX_DIM]; int64_t update_shape[B2ND_MAX_DIM]; int64_t update_nchunks = 1; for (int i = 0; i < ndim; ++i) { int64_t pos = 0; while (pos <= buffer_start[i]) { pos += array->chunkshape[i]; } update_start[i] = pos / array->chunkshape[i] - 1; while (pos < buffer_stop[i]) { pos += array->chunkshape[i]; } update_shape[i] = pos / array->chunkshape[i] - update_start[i]; update_nchunks *= update_shape[i]; } for (int update_nchunk = 0; update_nchunk < update_nchunks; ++update_nchunk) { int64_t nchunk_ndim[B2ND_MAX_DIM] = {0}; blosc2_unidim_to_multidim(ndim, update_shape, update_nchunk, nchunk_ndim); for (int i = 0; i < ndim; ++i) { nchunk_ndim[i] += update_start[i]; } int64_t nchunk; blosc2_multidim_to_unidim(nchunk_ndim, ndim, chunks_in_array_strides, &nchunk); // check if the chunk needs to be updated int64_t chunk_start[B2ND_MAX_DIM] = {0}; int64_t chunk_stop[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { chunk_start[i] = nchunk_ndim[i] * array->chunkshape[i]; chunk_stop[i] = chunk_start[i] + array->chunkshape[i]; if (chunk_stop[i] > array->shape[i]) { chunk_stop[i] = array->shape[i]; } } bool chunk_empty = false; for (int i = 0; i < ndim; ++i) { chunk_empty |= (chunk_stop[i] <= buffer_start[i] || chunk_start[i] >= buffer_stop[i]); } if (chunk_empty) { continue; } int32_t nblocks = (int32_t) array->extchunknitems / array->blocknitems; if (set_slice) { // Check if all the chunk is going to be updated and avoid the decompression bool decompress_chunk = false; for (int i = 0; i < ndim; ++i) { decompress_chunk |= (chunk_start[i] < buffer_start[i] || chunk_stop[i] > buffer_stop[i]); } if (decompress_chunk) { int err = blosc2_schunk_decompress_chunk(array->sc, nchunk, data, data_nbytes); if (err < 0) { BLOSC_TRACE_ERROR("Error decompressing chunk"); BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } } else { // Avoid writing non zero padding from previous chunk memset(data, 0, data_nbytes); } } else { bool *block_maskout = malloc(nblocks); BLOSC_ERROR_NULL(block_maskout, BLOSC2_ERROR_MEMORY_ALLOC); for (int nblock = 0; nblock < nblocks; ++nblock) { int64_t nblock_ndim[B2ND_MAX_DIM] = {0}; blosc2_unidim_to_multidim(ndim, blocks_in_chunk, nblock, nblock_ndim); // check if the block needs to be updated int64_t block_start[B2ND_MAX_DIM] = {0}; int64_t block_stop[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { block_start[i] = nblock_ndim[i] * array->blockshape[i]; block_stop[i] = block_start[i] + array->blockshape[i]; block_start[i] += chunk_start[i]; block_stop[i] += chunk_start[i]; if (block_start[i] > chunk_stop[i]) { block_start[i] = chunk_stop[i]; } if (block_stop[i] > chunk_stop[i]) { block_stop[i] = chunk_stop[i]; } } bool block_empty = false; for (int i = 0; i < ndim; ++i) { block_empty |= (block_stop[i] <= start[i] || block_start[i] >= stop[i]); } block_maskout[nblock] = block_empty ? true : false; } if (blosc2_set_maskout(array->sc->dctx, block_maskout, nblocks) != BLOSC2_ERROR_SUCCESS) { BLOSC_TRACE_ERROR("Error setting the maskout"); BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } int err = blosc2_schunk_decompress_chunk(array->sc, nchunk, data, data_nbytes); if (err < 0) { BLOSC_TRACE_ERROR("Error decompressing chunk"); BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } free(block_maskout); } // Iterate over blocks for (int nblock = 0; nblock < nblocks; ++nblock) { int64_t nblock_ndim[B2ND_MAX_DIM] = {0}; blosc2_unidim_to_multidim(ndim, blocks_in_chunk, nblock, nblock_ndim); // check if the block needs to be updated int64_t block_start[B2ND_MAX_DIM] = {0}; int64_t block_stop[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { block_start[i] = nblock_ndim[i] * array->blockshape[i]; block_stop[i] = block_start[i] + array->blockshape[i]; block_start[i] += chunk_start[i]; block_stop[i] += chunk_start[i]; if (block_start[i] > chunk_stop[i]) { block_start[i] = chunk_stop[i]; } if (block_stop[i] > chunk_stop[i]) { block_stop[i] = chunk_stop[i]; } } int64_t block_shape[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { block_shape[i] = block_stop[i] - block_start[i]; } bool block_empty = false; for (int i = 0; i < ndim; ++i) { block_empty |= (block_stop[i] <= start[i] || block_start[i] >= stop[i]); } if (block_empty) { continue; } // compute the start of the slice inside the block int64_t slice_start[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { if (block_start[i] < buffer_start[i]) { slice_start[i] = buffer_start[i] - block_start[i]; } else { slice_start[i] = 0; } slice_start[i] += block_start[i]; } int64_t slice_stop[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { if (block_stop[i] > buffer_stop[i]) { slice_stop[i] = block_shape[i] - (block_stop[i] - buffer_stop[i]); } else { slice_stop[i] = block_stop[i] - block_start[i]; } slice_stop[i] += block_start[i]; } int64_t slice_shape[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { slice_shape[i] = slice_stop[i] - slice_start[i]; } uint8_t *src = &buffer_b[0]; const int64_t *src_pad_shape = buffer_shape; int64_t src_start[B2ND_MAX_DIM] = {0}; int64_t src_stop[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { src_start[i] = slice_start[i] - buffer_start[i]; src_stop[i] = slice_stop[i] - buffer_start[i]; } uint8_t *dst = &data[nblock * array->blocknitems * array->sc->typesize]; int64_t dst_pad_shape[B2ND_MAX_DIM]; for (int i = 0; i < ndim; ++i) { dst_pad_shape[i] = array->blockshape[i]; } int64_t dst_start[B2ND_MAX_DIM] = {0}; int64_t dst_stop[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { dst_start[i] = slice_start[i] - block_start[i]; dst_stop[i] = dst_start[i] + slice_shape[i]; } if (set_slice) { b2nd_copy_buffer(ndim, array->sc->typesize, src, src_pad_shape, src_start, src_stop, dst, dst_pad_shape, dst_start); } else { b2nd_copy_buffer(ndim, array->sc->typesize, dst, dst_pad_shape, dst_start, dst_stop, src, src_pad_shape, src_start); } } if (set_slice) { // Recompress the data int32_t chunk_nbytes = data_nbytes + BLOSC2_MAX_OVERHEAD; uint8_t *chunk = malloc(chunk_nbytes); BLOSC_ERROR_NULL(chunk, BLOSC2_ERROR_MEMORY_ALLOC); int brc; brc = blosc2_compress_ctx(array->sc->cctx, data, data_nbytes, chunk, chunk_nbytes); if (brc < 0) { BLOSC_TRACE_ERROR("Blosc can not compress the data"); BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } int64_t brc_ = blosc2_schunk_update_chunk(array->sc, nchunk, chunk, false); if (brc_ < 0) { BLOSC_TRACE_ERROR("Blosc can not update the chunk"); BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } } } free(data); return BLOSC2_ERROR_SUCCESS; } int b2nd_get_slice_cbuffer(const b2nd_array_t *array, const int64_t *start, const int64_t *stop, void *buffer, const int64_t *buffershape, int64_t buffersize) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(start, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(stop, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(buffershape, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(buffer, BLOSC2_ERROR_NULL_POINTER); int64_t size = array->sc->typesize; for (int i = 0; i < array->ndim; ++i) { if (stop[i] - start[i] > buffershape[i]) { BLOSC_TRACE_ERROR("The buffer shape can not be smaller than the slice shape"); return BLOSC2_ERROR_INVALID_PARAM; } size *= buffershape[i]; } if (array->nitems == 0) { return BLOSC2_ERROR_SUCCESS; } if (buffersize < size) { BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } BLOSC_ERROR(get_set_slice(buffer, buffersize, start, stop, buffershape, (b2nd_array_t *)array, false)); return BLOSC2_ERROR_SUCCESS; } int b2nd_set_slice_cbuffer(const void *buffer, const int64_t *buffershape, int64_t buffersize, const int64_t *start, const int64_t *stop, b2nd_array_t *array) { BLOSC_ERROR_NULL(buffer, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(start, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(stop, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); int64_t size = array->sc->typesize; for (int i = 0; i < array->ndim; ++i) { size *= stop[i] - start[i]; } if (buffersize < size) { BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } if (array->nitems == 0) { return BLOSC2_ERROR_SUCCESS; } BLOSC_ERROR(get_set_slice((void*)buffer, buffersize, start, stop, (int64_t *)buffershape, array, true)); return BLOSC2_ERROR_SUCCESS; } int b2nd_get_slice(b2nd_context_t *ctx, b2nd_array_t **array, const b2nd_array_t *src, const int64_t *start, const int64_t *stop) { BLOSC_ERROR_NULL(src, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(start, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(stop, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); ctx->ndim = src->ndim; for (int i = 0; i < src->ndim; ++i) { ctx->shape[i] = stop[i] - start[i]; } // Add data BLOSC_ERROR(b2nd_empty(ctx, array)); if ((*array)->nitems == 0) { return BLOSC2_ERROR_SUCCESS; } int8_t ndim = (*array)->ndim; int64_t chunks_in_array[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { chunks_in_array[i] = (*array)->extshape[i] / (*array)->chunkshape[i]; } int64_t nchunks = (*array)->sc->nchunks; for (int nchunk = 0; nchunk < nchunks; ++nchunk) { int64_t nchunk_ndim[B2ND_MAX_DIM] = {0}; blosc2_unidim_to_multidim(ndim, chunks_in_array, nchunk, nchunk_ndim); // check if the chunk needs to be updated int64_t chunk_start[B2ND_MAX_DIM] = {0}; int64_t chunk_stop[B2ND_MAX_DIM] = {0}; int64_t chunk_shape[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { chunk_start[i] = nchunk_ndim[i] * (*array)->chunkshape[i]; chunk_stop[i] = chunk_start[i] + (*array)->chunkshape[i]; if (chunk_stop[i] > (*array)->shape[i]) { chunk_stop[i] = (*array)->shape[i]; } chunk_shape[i] = chunk_stop[i] - chunk_start[i]; } int64_t src_start[B2ND_MAX_DIM] = {0}; int64_t src_stop[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { src_start[i] = chunk_start[i] + start[i]; src_stop[i] = chunk_stop[i] + start[i]; } int64_t buffersize = ctx->b2_storage->cparams->typesize; for (int i = 0; i < ndim; ++i) { buffersize *= chunk_shape[i]; } uint8_t *buffer = malloc(buffersize); BLOSC_ERROR_NULL(buffer, BLOSC2_ERROR_MEMORY_ALLOC); BLOSC_ERROR(b2nd_get_slice_cbuffer(src, src_start, src_stop, buffer, chunk_shape, buffersize)); BLOSC_ERROR(b2nd_set_slice_cbuffer(buffer, chunk_shape, buffersize, chunk_start, chunk_stop, *array)); free(buffer); } return BLOSC2_ERROR_SUCCESS; } int b2nd_squeeze(b2nd_array_t *array) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); bool index[B2ND_MAX_DIM]; for (int i = 0; i < array->ndim; ++i) { if (array->shape[i] != 1) { index[i] = false; } else { index[i] = true; } } BLOSC_ERROR(b2nd_squeeze_index(array, index)); return BLOSC2_ERROR_SUCCESS; } int b2nd_squeeze_index(b2nd_array_t *array, const bool *index) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); uint8_t nones = 0; int64_t newshape[B2ND_MAX_DIM]; int32_t newchunkshape[B2ND_MAX_DIM]; int32_t newblockshape[B2ND_MAX_DIM]; for (int i = 0; i < array->ndim; ++i) { if (index[i] == true) { if (array->shape[i] != 1) { BLOSC_ERROR(BLOSC2_ERROR_INVALID_INDEX); } } else { newshape[nones] = array->shape[i]; newchunkshape[nones] = array->chunkshape[i]; newblockshape[nones] = array->blockshape[i]; nones += 1; } } for (int i = 0; i < B2ND_MAX_DIM; ++i) { if (i < nones) { array->chunkshape[i] = newchunkshape[i]; array->blockshape[i] = newblockshape[i]; } else { array->chunkshape[i] = 1; array->blockshape[i] = 1; } } BLOSC_ERROR(update_shape(array, nones, newshape, newchunkshape, newblockshape)); return BLOSC2_ERROR_SUCCESS; } int b2nd_copy(b2nd_context_t *ctx, const b2nd_array_t *src, b2nd_array_t **array) { BLOSC_ERROR_NULL(src, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); ctx->ndim = src->ndim; for (int i = 0; i < src->ndim; ++i) { ctx->shape[i] = src->shape[i]; } bool equals = true; for (int i = 0; i < src->ndim; ++i) { if (src->chunkshape[i] != ctx->chunkshape[i]) { equals = false; break; } if (src->blockshape[i] != ctx->blockshape[i]) { equals = false; break; } } if (equals) { BLOSC_ERROR(array_without_schunk(ctx, array)); blosc2_schunk *new_sc = blosc2_schunk_copy(src->sc, ctx->b2_storage); if (new_sc == NULL) { return BLOSC2_ERROR_FAILURE; } (*array)->sc = new_sc; } else { int64_t start[B2ND_MAX_DIM] = {0}; int64_t stop[B2ND_MAX_DIM]; for (int i = 0; i < src->ndim; ++i) { stop[i] = src->shape[i]; } // Copy metalayers b2nd_context_t params_meta; memcpy(¶ms_meta, ctx, sizeof(params_meta)); int j = 0; for (int i = 0; i < src->sc->nmetalayers; ++i) { if (strcmp(src->sc->metalayers[i]->name, "b2nd") == 0) { continue; } blosc2_metalayer *meta = ¶ms_meta.metalayers[j]; meta->name = src->sc->metalayers[i]->name; meta->content = src->sc->metalayers[i]->content; meta->content_len = src->sc->metalayers[i]->content_len; j++; } params_meta.nmetalayers = j; // Copy data BLOSC_ERROR(b2nd_get_slice(¶ms_meta, array, src, start, stop)); // Copy vlmetayers for (int i = 0; i < src->sc->nvlmetalayers; ++i) { uint8_t *content; int32_t content_len; if (blosc2_vlmeta_get(src->sc, src->sc->vlmetalayers[i]->name, &content, &content_len) < 0) { BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } BLOSC_ERROR(blosc2_vlmeta_add((*array)->sc, src->sc->vlmetalayers[i]->name, content, content_len, (*array)->sc->storage->cparams)); free(content); } } return BLOSC2_ERROR_SUCCESS; } int b2nd_save(const b2nd_array_t *array, char *urlpath) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(urlpath, BLOSC2_ERROR_NULL_POINTER); b2nd_array_t *tmp; blosc2_storage b2_storage = BLOSC2_STORAGE_DEFAULTS; b2nd_context_t params = {.b2_storage=&b2_storage}; b2_storage.urlpath = urlpath; b2_storage.contiguous = array->sc->storage->contiguous; for (int i = 0; i < array->ndim; ++i) { params.chunkshape[i] = array->chunkshape[i]; params.blockshape[i] = array->blockshape[i]; } BLOSC_ERROR(b2nd_copy(¶ms, array, &tmp)); BLOSC_ERROR(b2nd_free(tmp)); return BLOSC2_ERROR_SUCCESS; } int b2nd_print_meta(const b2nd_array_t *array) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); int8_t ndim; int64_t shape[B2ND_MAX_DIM]; int32_t chunkshape[B2ND_MAX_DIM]; int32_t blockshape[B2ND_MAX_DIM]; char *dtype; int8_t dtype_format; uint8_t *smeta; int32_t smeta_len; if (blosc2_meta_get(array->sc, "b2nd", &smeta, &smeta_len) < 0) { // Try with a caterva metalayer; we are meant to be backward compatible with it if (blosc2_meta_get(array->sc, "caterva", &smeta, &smeta_len) < 0) { BLOSC_ERROR(BLOSC2_ERROR_METALAYER_NOT_FOUND); } } BLOSC_ERROR(b2nd_deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape, &dtype, &dtype_format)); free(smeta); printf("b2nd metalayer parameters: \n Ndim: %d", ndim); printf("\n shape: %" PRId64 "", shape[0]); for (int i = 1; i < ndim; ++i) { printf(", %" PRId64 "", shape[i]); } printf("\n chunkshape: %d", chunkshape[0]); for (int i = 1; i < ndim; ++i) { printf(", %d", chunkshape[i]); } if (dtype != NULL) { printf("\n dtype: %s", dtype); free(dtype); } printf("\n blockshape: %d", blockshape[0]); for (int i = 1; i < ndim; ++i) { printf(", %d", blockshape[i]); } printf("\n"); return BLOSC2_ERROR_SUCCESS; } int extend_shape(b2nd_array_t *array, const int64_t *new_shape, const int64_t *start) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(new_shape, BLOSC2_ERROR_NULL_POINTER); int8_t ndim = array->ndim; int64_t diffs_shape[B2ND_MAX_DIM]; int64_t diffs_sum = 0; for (int i = 0; i < ndim; i++) { diffs_shape[i] = new_shape[i] - array->shape[i]; diffs_sum += diffs_shape[i]; if (diffs_shape[i] < 0) { BLOSC_TRACE_ERROR("The new shape must be greater than the old one"); BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } if (array->shape[i] == 0) { BLOSC_TRACE_ERROR("Cannot extend array with shape[%d] = 0", i); BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } } if (diffs_sum == 0) { // Shapes are equal. Do nothing. return BLOSC2_ERROR_SUCCESS; } int64_t old_nchunks = array->sc->nchunks; // aux array to keep old shapes b2nd_array_t *aux = malloc(sizeof(b2nd_array_t)); BLOSC_ERROR_NULL(aux, BLOSC2_ERROR_MEMORY_ALLOC); aux->sc = NULL; BLOSC_ERROR(update_shape(aux, ndim, array->shape, array->chunkshape, array->blockshape)); BLOSC_ERROR(update_shape(array, ndim, new_shape, array->chunkshape, array->blockshape)); int64_t nchunks = array->extnitems / array->chunknitems; int64_t nchunks_; int64_t nchunk_ndim[B2ND_MAX_DIM]; blosc2_cparams *cparams; BLOSC_ERROR(blosc2_schunk_get_cparams(array->sc, &cparams)); void *chunk; int64_t csize; if (nchunks != old_nchunks) { if (start == NULL) { start = aux->shape; } int64_t chunks_in_array[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { chunks_in_array[i] = array->extshape[i] / array->chunkshape[i]; } for (int i = 0; i < nchunks; ++i) { blosc2_unidim_to_multidim(ndim, chunks_in_array, i, nchunk_ndim); for (int j = 0; j < ndim; ++j) { if (start[j] <= (array->chunkshape[j] * nchunk_ndim[j]) && (array->chunkshape[j] * nchunk_ndim[j]) < (start[j] + new_shape[j] - aux->shape[j])) { chunk = malloc(BLOSC_EXTENDED_HEADER_LENGTH); BLOSC_ERROR_NULL(chunk, BLOSC2_ERROR_MEMORY_ALLOC); csize = blosc2_chunk_zeros(*cparams, array->sc->chunksize, chunk, BLOSC_EXTENDED_HEADER_LENGTH); if (csize < 0) { free(aux); free(cparams); BLOSC_TRACE_ERROR("Blosc error when creating a chunk"); return BLOSC2_ERROR_FAILURE; } nchunks_ = blosc2_schunk_insert_chunk(array->sc, i, chunk, false); if (nchunks_ < 0) { free(aux); free(cparams); BLOSC_TRACE_ERROR("Blosc error when inserting a chunk"); return BLOSC2_ERROR_FAILURE; } break; } } } } free(aux); free(cparams); return BLOSC2_ERROR_SUCCESS; } int shrink_shape(b2nd_array_t *array, const int64_t *new_shape, const int64_t *start) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(new_shape, BLOSC2_ERROR_NULL_POINTER); int8_t ndim = array->ndim; int64_t diffs_shape[B2ND_MAX_DIM]; int64_t diffs_sum = 0; for (int i = 0; i < ndim; i++) { diffs_shape[i] = new_shape[i] - array->shape[i]; diffs_sum += diffs_shape[i]; if (diffs_shape[i] > 0) { BLOSC_TRACE_ERROR("The new shape must be smaller than the old one"); BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } if (array->shape[i] == 0) { BLOSC_TRACE_ERROR("Cannot shrink array with shape[%d] = 0", i); BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } } if (diffs_sum == 0) { // Shapes are equal. Do nothing. return BLOSC2_ERROR_SUCCESS; } int64_t old_nchunks = array->sc->nchunks; // aux array to keep old shapes b2nd_array_t *aux = malloc(sizeof(b2nd_array_t)); BLOSC_ERROR_NULL(aux, BLOSC2_ERROR_MEMORY_ALLOC); aux->sc = NULL; BLOSC_ERROR(update_shape(aux, ndim, array->shape, array->chunkshape, array->blockshape)); BLOSC_ERROR(update_shape(array, ndim, new_shape, array->chunkshape, array->blockshape)); // Delete chunks if needed int64_t chunks_in_array_old[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { chunks_in_array_old[i] = aux->extshape[i] / aux->chunkshape[i]; } if (start == NULL) { start = new_shape; } int64_t nchunk_ndim[B2ND_MAX_DIM] = {0}; int64_t nchunks_; for (int i = (int) old_nchunks - 1; i >= 0; --i) { blosc2_unidim_to_multidim(ndim, chunks_in_array_old, i, nchunk_ndim); for (int j = 0; j < ndim; ++j) { if (start[j] <= (array->chunkshape[j] * nchunk_ndim[j]) && (array->chunkshape[j] * nchunk_ndim[j]) < (start[j] + aux->shape[j] - new_shape[j])) { nchunks_ = blosc2_schunk_delete_chunk(array->sc, i); if (nchunks_ < 0) { free(aux); BLOSC_TRACE_ERROR("Blosc error when deleting a chunk"); return BLOSC2_ERROR_FAILURE; } break; } } } free(aux); return BLOSC2_ERROR_SUCCESS; } int b2nd_resize(b2nd_array_t *array, const int64_t *new_shape, const int64_t *start) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(new_shape, BLOSC2_ERROR_NULL_POINTER); if (start != NULL) { for (int i = 0; i < array->ndim; ++i) { if (start[i] > array->shape[i]) { BLOSC_TRACE_ERROR("`start` must be lower or equal than old array shape in all dims"); BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } if ((new_shape[i] > array->shape[i] && start[i] != array->shape[i]) || (new_shape[i] < array->shape[i] && (start[i] + array->shape[i] - new_shape[i]) != array->shape[i])) { // Chunks cannot be cut unless they are in the last position if (start[i] % array->chunkshape[i] != 0) { BLOSC_TRACE_ERROR("If array end is not being modified " "`start` must be a multiple of chunkshape in all dims"); BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } if ((new_shape[i] - array->shape[i]) % array->chunkshape[i] != 0) { BLOSC_TRACE_ERROR("If array end is not being modified " "`(new_shape - shape)` must be multiple of chunkshape in all dims"); BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } } } } // Get shrunk shape int64_t shrunk_shape[B2ND_MAX_DIM] = {0}; for (int i = 0; i < array->ndim; ++i) { if (new_shape[i] <= array->shape[i]) { shrunk_shape[i] = new_shape[i]; } else { shrunk_shape[i] = array->shape[i]; } } BLOSC_ERROR(shrink_shape(array, shrunk_shape, start)); BLOSC_ERROR(extend_shape(array, new_shape, start)); return BLOSC2_ERROR_SUCCESS; } int b2nd_insert(b2nd_array_t *array, const void *buffer, int64_t buffersize, int8_t axis, int64_t insert_start) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(buffer, BLOSC2_ERROR_NULL_POINTER); if (axis >= array->ndim) { BLOSC_TRACE_ERROR("`axis` cannot be greater than the number of dimensions"); BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } int64_t axis_size = array->sc->typesize; int64_t buffershape[B2ND_MAX_DIM]; for (int i = 0; i < array->ndim; ++i) { if (i != axis) { axis_size *= array->shape[i]; buffershape[i] = array->shape[i]; } } if (buffersize % axis_size != 0) { BLOSC_TRACE_ERROR("`buffersize` must be multiple of the array"); BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } int64_t newshape[B2ND_MAX_DIM]; memcpy(newshape, array->shape, array->ndim * sizeof(int64_t)); newshape[axis] += buffersize / axis_size; buffershape[axis] = newshape[axis] - array->shape[axis]; int64_t start[B2ND_MAX_DIM] = {0}; start[axis] = insert_start; if (insert_start == array->shape[axis]) { BLOSC_ERROR(b2nd_resize(array, newshape, NULL)); } else { BLOSC_ERROR(b2nd_resize(array, newshape, start)); } int64_t stop[B2ND_MAX_DIM]; memcpy(stop, array->shape, sizeof(int64_t) * array->ndim); stop[axis] = start[axis] + buffershape[axis]; BLOSC_ERROR(b2nd_set_slice_cbuffer(buffer, buffershape, buffersize, start, stop, array)); return BLOSC2_ERROR_SUCCESS; } int b2nd_append(b2nd_array_t *array, const void *buffer, int64_t buffersize, int8_t axis) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(buffer, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR(b2nd_insert(array, buffer, buffersize, axis, array->shape[axis])); return BLOSC2_ERROR_SUCCESS; } int b2nd_delete(b2nd_array_t *array, const int8_t axis, int64_t delete_start, int64_t delete_len) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); if (axis >= array->ndim) { BLOSC_TRACE_ERROR("axis cannot be greater than the number of dimensions"); BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } int64_t newshape[B2ND_MAX_DIM]; memcpy(newshape, array->shape, array->ndim * sizeof(int64_t)); newshape[axis] -= delete_len; int64_t start[B2ND_MAX_DIM] = {0}; start[axis] = delete_start; if (delete_start == (array->shape[axis] - delete_len)) { BLOSC_ERROR(b2nd_resize(array, newshape, NULL)); } else { BLOSC_ERROR(b2nd_resize(array, newshape, start)); } return BLOSC2_ERROR_SUCCESS; } // Indexing typedef struct { int64_t value; int64_t index; } b2nd_selection_t; int compare_selection(const void *a, const void *b) { int res = (int) (((b2nd_selection_t *) a)->value - ((b2nd_selection_t *) b)->value); // In case values are equal, sort by index if (res == 0) { res = (int) (((b2nd_selection_t *) a)->index - ((b2nd_selection_t *) b)->index); } return res; } int copy_block_buffer_data(b2nd_array_t *array, int8_t ndim, int64_t *block_selection_size, b2nd_selection_t **chunk_selection, b2nd_selection_t **p_block_selection_0, b2nd_selection_t **p_block_selection_1, uint8_t *block, uint8_t *buffer, int64_t *buffershape, int64_t *bufferstrides, bool get) { p_block_selection_0[ndim] = chunk_selection[ndim]; p_block_selection_1[ndim] = chunk_selection[ndim]; while (p_block_selection_1[ndim] - p_block_selection_0[ndim] < block_selection_size[ndim]) { if (ndim == array->ndim - 1) { int64_t index_in_block_n[B2ND_MAX_DIM]; for (int i = 0; i < array->ndim; ++i) { index_in_block_n[i] = p_block_selection_1[i]->value % array->chunkshape[i] % array->blockshape[i]; } int64_t index_in_block = 0; for (int i = 0; i < array->ndim; ++i) { index_in_block += index_in_block_n[i] * array->item_block_strides[i]; } int64_t index_in_buffer_n[B2ND_MAX_DIM]; for (int i = 0; i < array->ndim; ++i) { index_in_buffer_n[i] = p_block_selection_1[i]->index; } int64_t index_in_buffer = 0; for (int i = 0; i < array->ndim; ++i) { index_in_buffer += index_in_buffer_n[i] * bufferstrides[i]; } if (get) { memcpy(&buffer[index_in_buffer * array->sc->typesize], &block[index_in_block * array->sc->typesize], array->sc->typesize); } else { memcpy(&block[index_in_block * array->sc->typesize], &buffer[index_in_buffer * array->sc->typesize], array->sc->typesize); } } else { BLOSC_ERROR(copy_block_buffer_data(array, (int8_t) (ndim + 1), block_selection_size, chunk_selection, p_block_selection_0, p_block_selection_1, block, buffer, buffershape, bufferstrides, get) ); } p_block_selection_1[ndim]++; } return BLOSC2_ERROR_SUCCESS; } int iter_block_copy(b2nd_array_t *array, int8_t ndim, int64_t *chunk_selection_size, b2nd_selection_t **ordered_selection, b2nd_selection_t **chunk_selection_0, b2nd_selection_t **chunk_selection_1, uint8_t *data, uint8_t *buffer, int64_t *buffershape, int64_t *bufferstrides, bool get) { chunk_selection_0[ndim] = ordered_selection[ndim]; chunk_selection_1[ndim] = ordered_selection[ndim]; while (chunk_selection_1[ndim] - ordered_selection[ndim] < chunk_selection_size[ndim]) { int64_t block_index_ndim = ((*chunk_selection_1[ndim]).value % array->chunkshape[ndim]) / array->blockshape[ndim]; while (chunk_selection_1[ndim] - ordered_selection[ndim] < chunk_selection_size[ndim] && block_index_ndim == ((*chunk_selection_1[ndim]).value % array->chunkshape[ndim]) / array->blockshape[ndim]) { chunk_selection_1[ndim]++; } if (ndim == array->ndim - 1) { int64_t block_chunk_strides[B2ND_MAX_DIM]; block_chunk_strides[array->ndim - 1] = 1; for (int i = array->ndim - 2; i >= 0; --i) { block_chunk_strides[i] = block_chunk_strides[i + 1] * (array->extchunkshape[i + 1] / array->blockshape[i + 1]); } int64_t block_index[B2ND_MAX_DIM]; for (int i = 0; i < array->ndim; ++i) { block_index[i] = ((*chunk_selection_0[i]).value % array->chunkshape[i]) / array->blockshape[i]; } int64_t nblock = 0; for (int i = 0; i < array->ndim; ++i) { nblock += block_index[i] * block_chunk_strides[i]; } b2nd_selection_t **p_block_selection_0 = malloc(array->ndim * sizeof(b2nd_selection_t *)); BLOSC_ERROR_NULL(p_block_selection_0, BLOSC2_ERROR_MEMORY_ALLOC); b2nd_selection_t **p_block_selection_1 = malloc(array->ndim * sizeof(b2nd_selection_t *)); BLOSC_ERROR_NULL(p_block_selection_1, BLOSC2_ERROR_MEMORY_ALLOC); int64_t *block_selection_size = malloc(array->ndim * sizeof(int64_t)); BLOSC_ERROR_NULL(block_selection_size, BLOSC2_ERROR_MEMORY_ALLOC); for (int i = 0; i < array->ndim; ++i) { block_selection_size[i] = chunk_selection_1[i] - chunk_selection_0[i]; } BLOSC_ERROR(copy_block_buffer_data(array, (int8_t) 0, block_selection_size, chunk_selection_0, p_block_selection_0, p_block_selection_1, &data[nblock * array->blocknitems * array->sc->typesize], buffer, buffershape, bufferstrides, get) ); free(p_block_selection_0); free(p_block_selection_1); free(block_selection_size); } else { BLOSC_ERROR(iter_block_copy(array, (int8_t) (ndim + 1), chunk_selection_size, ordered_selection, chunk_selection_0, chunk_selection_1, data, buffer, buffershape, bufferstrides, get) ); } chunk_selection_0[ndim] = chunk_selection_1[ndim]; } return BLOSC2_ERROR_SUCCESS; } int iter_block_maskout(b2nd_array_t *array, int8_t ndim, int64_t *sel_block_size, b2nd_selection_t **o_selection, b2nd_selection_t **p_o_sel_block_0, b2nd_selection_t **p_o_sel_block_1, bool *maskout) { p_o_sel_block_0[ndim] = o_selection[ndim]; p_o_sel_block_1[ndim] = o_selection[ndim]; while (p_o_sel_block_1[ndim] - o_selection[ndim] < sel_block_size[ndim]) { int64_t block_index_ndim = ((*p_o_sel_block_1[ndim]).value % array->chunkshape[ndim]) / array->blockshape[ndim]; while (p_o_sel_block_1[ndim] - o_selection[ndim] < sel_block_size[ndim] && block_index_ndim == ((*p_o_sel_block_1[ndim]).value % array->chunkshape[ndim]) / array->blockshape[ndim]) { p_o_sel_block_1[ndim]++; } if (ndim == array->ndim - 1) { int64_t block_chunk_strides[B2ND_MAX_DIM]; block_chunk_strides[array->ndim - 1] = 1; for (int i = array->ndim - 2; i >= 0; --i) { block_chunk_strides[i] = block_chunk_strides[i + 1] * (array->extchunkshape[i + 1] / array->blockshape[i + 1]); } int64_t block_index[B2ND_MAX_DIM]; for (int i = 0; i < array->ndim; ++i) { block_index[i] = ((*p_o_sel_block_0[i]).value % array->chunkshape[i]) / array->blockshape[i]; } int64_t nblock = 0; for (int i = 0; i < array->ndim; ++i) { nblock += block_index[i] * block_chunk_strides[i]; } maskout[nblock] = false; } else { BLOSC_ERROR(iter_block_maskout(array, (int8_t) (ndim + 1), sel_block_size, o_selection, p_o_sel_block_0, p_o_sel_block_1, maskout) ); } p_o_sel_block_0[ndim] = p_o_sel_block_1[ndim]; } return BLOSC2_ERROR_SUCCESS; } int iter_chunk(b2nd_array_t *array, int8_t ndim, int64_t *selection_size, b2nd_selection_t **ordered_selection, b2nd_selection_t **p_ordered_selection_0, b2nd_selection_t **p_ordered_selection_1, uint8_t *buffer, int64_t *buffershape, int64_t *bufferstrides, bool get) { p_ordered_selection_0[ndim] = ordered_selection[ndim]; p_ordered_selection_1[ndim] = ordered_selection[ndim]; while (p_ordered_selection_1[ndim] - ordered_selection[ndim] < selection_size[ndim]) { int64_t chunk_index_ndim = (*p_ordered_selection_1[ndim]).value / array->chunkshape[ndim]; while (p_ordered_selection_1[ndim] - ordered_selection[ndim] < selection_size[ndim] && chunk_index_ndim == (*p_ordered_selection_1[ndim]).value / array->chunkshape[ndim]) { p_ordered_selection_1[ndim]++; } if (ndim == array->ndim - 1) { int64_t chunk_array_strides[B2ND_MAX_DIM]; chunk_array_strides[array->ndim - 1] = 1; for (int i = array->ndim - 2; i >= 0; --i) { chunk_array_strides[i] = chunk_array_strides[i + 1] * (array->extshape[i + 1] / array->chunkshape[i + 1]); } int64_t chunk_index[B2ND_MAX_DIM]; for (int i = 0; i < array->ndim; ++i) { chunk_index[i] = (*p_ordered_selection_0[i]).value / array->chunkshape[i]; } int64_t nchunk = 0; for (int i = 0; i < array->ndim; ++i) { nchunk += chunk_index[i] * chunk_array_strides[i]; } int64_t nblocks = array->extchunknitems / array->blocknitems; b2nd_selection_t **p_chunk_selection_0 = malloc(array->ndim * sizeof(b2nd_selection_t *)); BLOSC_ERROR_NULL(p_chunk_selection_0, BLOSC2_ERROR_MEMORY_ALLOC); b2nd_selection_t **p_chunk_selection_1 = malloc(array->ndim * sizeof(b2nd_selection_t *)); BLOSC_ERROR_NULL(p_chunk_selection_1, BLOSC2_ERROR_MEMORY_ALLOC); int64_t *chunk_selection_size = malloc(array->ndim * sizeof(int64_t)); BLOSC_ERROR_NULL(chunk_selection_size, BLOSC2_ERROR_MEMORY_ALLOC); for (int i = 0; i < array->ndim; ++i) { chunk_selection_size[i] = p_ordered_selection_1[i] - p_ordered_selection_0[i]; } if (get) { bool *maskout = calloc(nblocks, sizeof(bool)); for (int i = 0; i < nblocks; ++i) { maskout[i] = true; } BLOSC_ERROR(iter_block_maskout(array, (int8_t) 0, chunk_selection_size, p_ordered_selection_0, p_chunk_selection_0, p_chunk_selection_1, maskout)); if (blosc2_set_maskout(array->sc->dctx, maskout, (int) nblocks) != BLOSC2_ERROR_SUCCESS) { BLOSC_TRACE_ERROR("Error setting the maskout"); BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } free(maskout); } int data_nitems = (int) array->extchunknitems; int data_nbytes = data_nitems * array->sc->typesize; uint8_t *data = malloc(data_nitems * array->sc->typesize); BLOSC_ERROR_NULL(data, BLOSC2_ERROR_MEMORY_ALLOC); int err = blosc2_schunk_decompress_chunk(array->sc, nchunk, data, data_nbytes); if (err < 0) { BLOSC_TRACE_ERROR("Error decompressing chunk"); BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } BLOSC_ERROR(iter_block_copy(array, 0, chunk_selection_size, p_ordered_selection_0, p_chunk_selection_0, p_chunk_selection_1, data, buffer, buffershape, bufferstrides, get)); if (!get) { int32_t chunk_size = data_nbytes + BLOSC_EXTENDED_HEADER_LENGTH; uint8_t *chunk = malloc(chunk_size); BLOSC_ERROR_NULL(chunk, BLOSC2_ERROR_MEMORY_ALLOC); err = blosc2_compress_ctx(array->sc->cctx, data, data_nbytes, chunk, chunk_size); if (err < 0) { BLOSC_TRACE_ERROR("Error compressing data"); BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } err = (int) blosc2_schunk_update_chunk(array->sc, nchunk, chunk, false); if (err < 0) { BLOSC_TRACE_ERROR("Error updating chunk"); BLOSC_ERROR(BLOSC2_ERROR_FAILURE); } } free(data); free(chunk_selection_size); free(p_chunk_selection_0); free(p_chunk_selection_1); } else { BLOSC_ERROR(iter_chunk(array, (int8_t) (ndim + 1), selection_size, ordered_selection, p_ordered_selection_0, p_ordered_selection_1, buffer, buffershape, bufferstrides, get)); } p_ordered_selection_0[ndim] = p_ordered_selection_1[ndim]; } return BLOSC2_ERROR_SUCCESS; } int orthogonal_selection(b2nd_array_t *array, int64_t **selection, int64_t *selection_size, void *buffer, int64_t *buffershape, int64_t buffersize, bool get) { BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(selection, BLOSC2_ERROR_NULL_POINTER); BLOSC_ERROR_NULL(selection_size, BLOSC2_ERROR_NULL_POINTER); int8_t ndim = array->ndim; for (int i = 0; i < ndim; ++i) { BLOSC_ERROR_NULL(selection[i], BLOSC2_ERROR_NULL_POINTER); // Check that indexes are not larger than array shape for (int j = 0; j < selection_size[i]; ++j) { if (selection[i][j] > array->shape[i]) { BLOSC_ERROR(BLOSC2_ERROR_INVALID_INDEX); } } } // Check buffer size int64_t sel_size = array->sc->typesize; for (int i = 0; i < ndim; ++i) { sel_size *= selection_size[i]; } if (sel_size < buffersize) { BLOSC_ERROR(BLOSC2_ERROR_INVALID_PARAM); } // Sort selections b2nd_selection_t **ordered_selection = malloc(ndim * sizeof(b2nd_selection_t *)); BLOSC_ERROR_NULL(ordered_selection, BLOSC2_ERROR_MEMORY_ALLOC); for (int i = 0; i < ndim; ++i) { ordered_selection[i] = malloc(selection_size[i] * sizeof(b2nd_selection_t)); for (int j = 0; j < selection_size[i]; ++j) { ordered_selection[i][j].index = j; ordered_selection[i][j].value = selection[i][j]; } qsort(ordered_selection[i], selection_size[i], sizeof(b2nd_selection_t), compare_selection); } // Define pointers to iterate over ordered_selection data b2nd_selection_t **p_ordered_selection_0 = malloc(ndim * sizeof(b2nd_selection_t *)); BLOSC_ERROR_NULL(p_ordered_selection_0, BLOSC2_ERROR_MEMORY_ALLOC); b2nd_selection_t **p_ordered_selection_1 = malloc(ndim * sizeof(b2nd_selection_t *)); BLOSC_ERROR_NULL(p_ordered_selection_1, BLOSC2_ERROR_MEMORY_ALLOC); int64_t bufferstrides[B2ND_MAX_DIM]; bufferstrides[array->ndim - 1] = 1; for (int i = array->ndim - 2; i >= 0; --i) { bufferstrides[i] = bufferstrides[i + 1] * buffershape[i + 1]; } BLOSC_ERROR(iter_chunk(array, 0, selection_size, ordered_selection, p_ordered_selection_0, p_ordered_selection_1, buffer, buffershape, bufferstrides, get)); // Free allocated memory free(p_ordered_selection_0); free(p_ordered_selection_1); for (int i = 0; i < ndim; ++i) { free(ordered_selection[i]); } free(ordered_selection); return BLOSC2_ERROR_SUCCESS; } int b2nd_get_orthogonal_selection(const b2nd_array_t *array, int64_t **selection, int64_t *selection_size, void *buffer, int64_t *buffershape, int64_t buffersize) { return orthogonal_selection((b2nd_array_t *)array, selection, selection_size, buffer, buffershape, buffersize, true); } int b2nd_set_orthogonal_selection(b2nd_array_t *array, int64_t **selection, int64_t *selection_size, const void *buffer, int64_t *buffershape, int64_t buffersize) { return orthogonal_selection(array, selection, selection_size, (void*)buffer, buffershape, buffersize, false); } b2nd_context_t * b2nd_create_ctx(const blosc2_storage *b2_storage, int8_t ndim, const int64_t *shape, const int32_t *chunkshape, const int32_t *blockshape, const char *dtype, int8_t dtype_format, const blosc2_metalayer *metalayers, int32_t nmetalayers) { b2nd_context_t *ctx = malloc(sizeof(b2nd_context_t)); BLOSC_ERROR_NULL(ctx, NULL); blosc2_storage *params_b2_storage = malloc(sizeof(blosc2_storage)); BLOSC_ERROR_NULL(params_b2_storage, NULL); if (b2_storage == NULL) { memcpy(params_b2_storage, &BLOSC2_STORAGE_DEFAULTS, sizeof(blosc2_storage)); } else { memcpy(params_b2_storage, b2_storage, sizeof(blosc2_storage)); } blosc2_cparams *cparams = malloc(sizeof(blosc2_cparams)); BLOSC_ERROR_NULL(cparams, NULL); // We need a copy of cparams mainly to be able to modify blocksize if (b2_storage->cparams == NULL) { memcpy(cparams, &BLOSC2_CPARAMS_DEFAULTS, sizeof(blosc2_cparams)); } else { memcpy(cparams, b2_storage->cparams, sizeof(blosc2_cparams)); } if (dtype == NULL) { ctx->dtype = strdup(B2ND_DEFAULT_DTYPE); ctx->dtype_format = 0; // The default is NumPy format } else { ctx->dtype = strdup(dtype); ctx->dtype_format = dtype_format; } params_b2_storage->cparams = cparams; ctx->b2_storage = params_b2_storage; ctx->ndim = ndim; int32_t blocknitems = 1; for (int i = 0; i < ndim; i++) { ctx->shape[i] = shape[i]; ctx->chunkshape[i] = chunkshape[i]; ctx->blockshape[i] = blockshape[i]; blocknitems *= ctx->blockshape[i]; } cparams->blocksize = blocknitems * cparams->typesize; ctx->nmetalayers = nmetalayers; for (int i = 0; i < nmetalayers; ++i) { ctx->metalayers[i] = metalayers[i]; } return ctx; } int b2nd_free_ctx(b2nd_context_t *ctx) { ctx->b2_storage->cparams->schunk = NULL; free(ctx->b2_storage->cparams); free(ctx->b2_storage); free(ctx->dtype); free(ctx); return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/blosc/fastcopy.h0000644000175200007730000000137314515254731020021 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef BLOSC_FASTCOPY_H #define BLOSC_FASTCOPY_H /* Same semantics than memcpy() */ unsigned char *fastcopy(unsigned char *out, const unsigned char *from, unsigned len); /* Same as fastcopy() but without overwriting origin or destination when they overlap */ unsigned char* copy_match(unsigned char *out, const unsigned char *from, unsigned len); #endif //BLOSC_FASTCOPY_H zmat-0.9.9/src/blosc2/blosc/bitshuffle-altivec.c0000644000175200007730000005207514515254731021751 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /********************************************************************* Bitshuffle - Filter for improving compression of typed binary data. Author: Kiyoshi Masui Website: https://github.com/kiyo-masui/bitshuffle Note: Adapted for c-blosc by Francesc Alted Altivec/VSX version by Jerome Kieffer. See LICENSES/BITSHUFFLE.txt file for details about copyright and rights to use. **********************************************************************/ #include "bitshuffle-generic.h" #include "bitshuffle-altivec.h" /* Make sure ALTIVEC is available for the compilation target and compiler. */ #if defined(__ALTIVEC__) #include #include "transpose-altivec.h" /* The next is useful for debugging purposes */ #if 0 #include #include static void helper_print(__vector uint8_t v, char* txt){ printf("%s %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x\n",txt, v[0], v[1], v[2], v[3], v[4], v[5], v[6], v[7], v[8], v[9], v[10], v[11], v[12], v[13], v[14], v[15]); } #endif static inline __vector uint8_t gen_save_mask(size_t offset){ __vector uint8_t mask; size_t k; for (k = 0; k < 16; k++) mask[k] = (k> 3]; *oui16 = tmp[4]; } } count = bshuf_trans_bit_byte_remainder(in, out, size, elem_size, nbyte - nbyte % 16); return count; } /* Transpose bits within elements. */ int64_t bshuf_trans_bit_elem_altivec(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf) { int64_t count; CHECK_MULT_EIGHT(size); count = bshuf_trans_byte_elem_altivec(in, out, size, elem_size, tmp_buf); CHECK_ERR(count); // bshuf_trans_bit_byte_altivec / bitshuffle1_altivec count = bshuf_trans_bit_byte_altivec(out, tmp_buf, size, elem_size); CHECK_ERR(count); count = bshuf_trans_bitrow_eight(tmp_buf, out, size, elem_size); return count; } /* For data organized into a row for each bit (8 * elem_size rows), transpose * the bytes. */ int64_t bshuf_trans_byte_bitrow_altivec(void* in, void* out, const size_t size, const size_t elem_size) { static const __vector uint8_t epi8_low = (const __vector uint8_t) { 0x00, 0x10, 0x01, 0x11, 0x02, 0x12, 0x03, 0x13, 0x04, 0x14, 0x05, 0x15, 0x06, 0x16, 0x07, 0x17}; static const __vector uint8_t epi8_hi = (const __vector uint8_t) { 0x08, 0x18, 0x09, 0x19, 0x0a, 0x1a, 0x0b, 0x1b, 0x0c, 0x1c, 0x0d, 0x1d, 0x0e, 0x1e, 0x0f, 0x1f}; static const __vector uint8_t epi16_low = (const __vector uint8_t) { 0x00, 0x01, 0x10, 0x11, 0x02, 0x03, 0x12, 0x13, 0x04, 0x05, 0x14, 0x15, 0x06, 0x07, 0x16, 0x17}; static const __vector uint8_t epi16_hi = (const __vector uint8_t) { 0x08, 0x09, 0x18, 0x19, 0x0a, 0x0b, 0x1a, 0x1b, 0x0c, 0x0d, 0x1c, 0x1d, 0x0e, 0x0f, 0x1e, 0x1f}; static const __vector uint8_t epi32_low = (const __vector uint8_t) { 0x00, 0x01, 0x02, 0x03, 0x10, 0x11, 0x12, 0x13, 0x04, 0x05, 0x06, 0x07, 0x14, 0x15, 0x16, 0x17}; static const __vector uint8_t epi32_hi = (const __vector uint8_t) { 0x08, 0x09, 0x0a, 0x0b, 0x18, 0x19, 0x1a, 0x1b, 0x0c, 0x0d, 0x0e, 0x0f, 0x1c, 0x1d, 0x1e, 0x1f}; static const __vector uint8_t epi64_low = (const __vector uint8_t) { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17}; static const __vector uint8_t epi64_hi = (const __vector uint8_t) { 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f}; const uint8_t* in_b = (const uint8_t*)in; uint8_t* out_b = (uint8_t*)out; size_t nrows = 8 * elem_size; size_t nbyte_row = size / 8; __vector uint8_t xmm0[16], xmm1[16]; CHECK_MULT_EIGHT(size); // The optimized algorithms can only deal with even values or 1 for elem_size if ((elem_size > 1) && (elem_size % 2)) { return bshuf_trans_byte_bitrow_scal(in, out, size, elem_size); } int nvectors = (elem_size == 1) ? 8 : 16; for (size_t ii = 0; ii + (nvectors - 1) < nrows; ii += nvectors) { for (size_t jj = 0; jj + 15 < nbyte_row; jj += 16) { // vectors of 16 elements if (elem_size == 1) { for (int k = 0; k < 8; k++) { xmm0[k] = vec_xl((ii + k) * nbyte_row + jj, in_b); } xmm1[0] = vec_perm(xmm0[0], xmm0[1], epi8_low); xmm1[1] = vec_perm(xmm0[2], xmm0[3], epi8_low); xmm1[2] = vec_perm(xmm0[4], xmm0[5], epi8_low); xmm1[3] = vec_perm(xmm0[6], xmm0[7], epi8_low); xmm1[4] = vec_perm(xmm0[0], xmm0[1], epi8_hi); xmm1[5] = vec_perm(xmm0[2], xmm0[3], epi8_hi); xmm1[6] = vec_perm(xmm0[4], xmm0[5], epi8_hi); xmm1[7] = vec_perm(xmm0[6], xmm0[7], epi8_hi); xmm0[0] = vec_perm(xmm1[0], xmm1[1], epi16_low); xmm0[1] = vec_perm(xmm1[2], xmm1[3], epi16_low); xmm0[2] = vec_perm(xmm1[0], xmm1[1], epi16_hi); xmm0[3] = vec_perm(xmm1[2], xmm1[3], epi16_hi); xmm0[4] = vec_perm(xmm1[4], xmm1[5], epi16_low); xmm0[5] = vec_perm(xmm1[6], xmm1[7], epi16_low); xmm0[6] = vec_perm(xmm1[4], xmm1[5], epi16_hi); xmm0[7] = vec_perm(xmm1[6], xmm1[7], epi16_hi); xmm1[0] = vec_perm(xmm0[0], xmm0[1], epi32_low); xmm1[1] = vec_perm(xmm0[0], xmm0[1], epi32_hi); xmm1[2] = vec_perm(xmm0[2], xmm0[3], epi32_low); xmm1[3] = vec_perm(xmm0[2], xmm0[3], epi32_hi); xmm1[4] = vec_perm(xmm0[4], xmm0[5], epi32_low); xmm1[5] = vec_perm(xmm0[4], xmm0[5], epi32_hi); xmm1[6] = vec_perm(xmm0[6], xmm0[7], epi32_low); xmm1[7] = vec_perm(xmm0[6], xmm0[7], epi32_hi); for (int k = 0; k < 8; k++) { vec_xst(xmm1[k], (jj + k * 2) * nrows + ii, out_b); } continue; } for (int k = 0; k < 16; k++) { xmm0[k] = vec_xl((ii + k) * nbyte_row + jj, in_b); } for (int k = 0; k < 16; k += 8) { xmm1[k + 0] = vec_perm(xmm0[k + 0], xmm0[k + 1], epi8_low); xmm1[k + 1] = vec_perm(xmm0[k + 2], xmm0[k + 3], epi8_low); xmm1[k + 2] = vec_perm(xmm0[k + 4], xmm0[k + 5], epi8_low); xmm1[k + 3] = vec_perm(xmm0[k + 6], xmm0[k + 7], epi8_low); xmm1[k + 4] = vec_perm(xmm0[k + 0], xmm0[k + 1], epi8_hi); xmm1[k + 5] = vec_perm(xmm0[k + 2], xmm0[k + 3], epi8_hi); xmm1[k + 6] = vec_perm(xmm0[k + 4], xmm0[k + 5], epi8_hi); xmm1[k + 7] = vec_perm(xmm0[k + 6], xmm0[k + 7], epi8_hi); } for (int k = 0; k < 16; k += 8) { xmm0[k + 0] = vec_perm(xmm1[k + 0], xmm1[k + 1], epi16_low); xmm0[k + 1] = vec_perm(xmm1[k + 2], xmm1[k + 3], epi16_low); xmm0[k + 2] = vec_perm(xmm1[k + 0], xmm1[k + 1], epi16_hi); xmm0[k + 3] = vec_perm(xmm1[k + 2], xmm1[k + 3], epi16_hi); xmm0[k + 4] = vec_perm(xmm1[k + 4], xmm1[k + 5], epi16_low); xmm0[k + 5] = vec_perm(xmm1[k + 6], xmm1[k + 7], epi16_low); xmm0[k + 6] = vec_perm(xmm1[k + 4], xmm1[k + 5], epi16_hi); xmm0[k + 7] = vec_perm(xmm1[k + 6], xmm1[k + 7], epi16_hi); } for (int k = 0; k < 16; k += 8) { xmm1[k + 0] = vec_perm(xmm0[k + 0], xmm0[k + 1], epi32_low); xmm1[k + 1] = vec_perm(xmm0[k + 0], xmm0[k + 1], epi32_hi); xmm1[k + 2] = vec_perm(xmm0[k + 2], xmm0[k + 3], epi32_low); xmm1[k + 3] = vec_perm(xmm0[k + 2], xmm0[k + 3], epi32_hi); xmm1[k + 4] = vec_perm(xmm0[k + 4], xmm0[k + 5], epi32_low); xmm1[k + 5] = vec_perm(xmm0[k + 4], xmm0[k + 5], epi32_hi); xmm1[k + 6] = vec_perm(xmm0[k + 6], xmm0[k + 7], epi32_low); xmm1[k + 7] = vec_perm(xmm0[k + 6], xmm0[k + 7], epi32_hi); } for (int k = 0; k < 8; k += 4) { xmm0[k * 2 + 0] = vec_perm(xmm1[k + 0], xmm1[k + 8], epi64_low); xmm0[k * 2 + 1] = vec_perm(xmm1[k + 0], xmm1[k + 8], epi64_hi); xmm0[k * 2 + 2] = vec_perm(xmm1[k + 1], xmm1[k + 9], epi64_low); xmm0[k * 2 + 3] = vec_perm(xmm1[k + 1], xmm1[k + 9], epi64_hi); xmm0[k * 2 + 4] = vec_perm(xmm1[k + 2], xmm1[k + 10], epi64_low); xmm0[k * 2 + 5] = vec_perm(xmm1[k + 2], xmm1[k + 10], epi64_hi); xmm0[k * 2 + 6] = vec_perm(xmm1[k + 3], xmm1[k + 11], epi64_low); xmm0[k * 2 + 7] = vec_perm(xmm1[k + 3], xmm1[k + 11], epi64_hi); } for (int k = 0; k < 16; k++) { vec_xst(xmm0[k], (jj + k) * nrows + ii, out_b); } } // Copy the remainder for (size_t jj = nbyte_row - nbyte_row % 16; jj < nbyte_row; jj++) { for (int k = 0; k < nvectors; k++) { out_b[jj * nrows + ii + k] = in_b[(ii + k) * nbyte_row + jj]; } } } return size * elem_size; } /* Shuffle bits within the bytes of eight element blocks. */ int64_t bshuf_shuffle_bit_eightelem_altivec(void* in, void* out, const size_t size, const size_t elem_size) { /* With a bit of care, this could be written such that such that it is */ /* in_buf = out_buf safe. */ const uint8_t* in_b = (const uint8_t*)in; uint8_t* out_b = (uint8_t*)out; size_t nbyte = elem_size * size; __vector uint8_t masks[8], data; CHECK_MULT_EIGHT(size); // Generate all 8 needed masks for (int kk = 0; kk < 8; kk++){ masks[kk] = make_bitperm_mask(1, kk); } if (elem_size % 2) { bshuf_shuffle_bit_eightelem_scal(in, out, size, elem_size); } else { for (size_t ii = 0; ii + 8 * elem_size - 1 < nbyte; ii += 8 * elem_size) { for (size_t jj = 0; jj + 15 < 8 * elem_size; jj += 16) { data = vec_xl(ii + jj, in_b); for (size_t kk = 0; kk < 8; kk++) { __vector uint16_t tmp; uint16_t* oui16; tmp = (__vector uint16_t) vec_bperm(data, masks[kk]); oui16 = (uint16_t*)&out_b[ii + (jj>>3) + kk * elem_size]; *oui16 = tmp[4]; } } } } return size * elem_size; } /* Untranspose bits within elements. */ int64_t bshuf_untrans_bit_elem_altivec(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf) { int64_t count; CHECK_MULT_EIGHT(size); count = bshuf_trans_byte_bitrow_altivec(in, tmp_buf, size, elem_size); CHECK_ERR(count); count = bshuf_shuffle_bit_eightelem_altivec(tmp_buf, out, size, elem_size); return count; } #endif /* defined(__ALTIVEC__) */ zmat-0.9.9/src/blosc2/blosc/shuffle-avx2.h0000644000175200007730000000200314515254731020472 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /* AVX2-accelerated shuffle/unshuffle routines. */ #ifndef SHUFFLE_AVX2_H #define SHUFFLE_AVX2_H #include "blosc2/blosc2-common.h" #ifdef __cplusplus extern "C" { #endif /** AVX2-accelerated shuffle routine. */ BLOSC_NO_EXPORT void shuffle_avx2(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest); /** AVX2-accelerated unshuffle routine. */ BLOSC_NO_EXPORT void unshuffle_avx2(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest); #ifdef __cplusplus } #endif #endif /* SHUFFLE_AVX2_H */ zmat-0.9.9/src/blosc2/blosc/CTestTestfile.cmake0000644000175200007730000000044714515254731021545 0ustar rlaboissrlaboiss# CMake generated Testfile for # Source directory: /home/fangq/space/git/Temp/c-blosc2/blosc # Build directory: /home/fangq/space/git/Temp/c-blosc2/blosc # # This file includes the relevant testing commands required for # testing this directory and lists subdirectories to be tested as well. zmat-0.9.9/src/blosc2/blosc/config.h.in0000644000175200007730000000047314515254731020043 0ustar rlaboissrlaboiss#ifndef _CONFIGURATION_HEADER_GUARD_H_ #define _CONFIGURATION_HEADER_GUARD_H_ #cmakedefine HAVE_ZLIB @HAVE_ZLIB@ #cmakedefine HAVE_ZLIB_NG @HAVE_ZLIB_NG@ #cmakedefine HAVE_ZSTD @HAVE_ZSTD@ #cmakedefine HAVE_IPP @HAVE_IPP@ #cmakedefine BLOSC_DLL_EXPORT @DLL_EXPORT@ #cmakedefine HAVE_PLUGINS @HAVE_PLUGINS@ #endif zmat-0.9.9/src/blosc2/blosc/bitshuffle-sse2.c0000644000175200007730000003704214515254731021173 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /********************************************************************* Bitshuffle - Filter for improving compression of typed binary data. Author: Kiyoshi Masui Website: https://github.com/kiyo-masui/bitshuffle Note: Adapted for c-blosc by Francesc Alted. See LICENSES/BITSHUFFLE.txt file for details about copyright and rights to use. **********************************************************************/ #include "bitshuffle-generic.h" #include "bitshuffle-sse2.h" /* Make sure SSE2 is available for the compilation target and compiler. */ #if defined(__SSE2__) #include /* The next is useful for debugging purposes */ #if 0 #include #include static void printxmm(__m128i xmm0) { uint8_t buf[32]; ((__m128i *)buf)[0] = xmm0; printf("%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x\n", buf[0], buf[1], buf[2], buf[3], buf[4], buf[5], buf[6], buf[7], buf[8], buf[9], buf[10], buf[11], buf[12], buf[13], buf[14], buf[15]); } #endif /* ---- Worker code that requires SSE2. Intel Petium 4 (2000) and later. ---- */ /* Transpose bytes within elements for 16 bit elements. */ int64_t bshuf_trans_byte_elem_SSE_16(void* in, void* out, const size_t size) { char* in_b = (char*)in; char* out_b = (char*)out; __m128i a0, b0, a1, b1; size_t ii; for (ii = 0; ii + 15 < size; ii += 16) { a0 = _mm_loadu_si128((__m128i*)&in_b[2 * ii + 0 * 16]); b0 = _mm_loadu_si128((__m128i*)&in_b[2 * ii + 1 * 16]); a1 = _mm_unpacklo_epi8(a0, b0); b1 = _mm_unpackhi_epi8(a0, b0); a0 = _mm_unpacklo_epi8(a1, b1); b0 = _mm_unpackhi_epi8(a1, b1); a1 = _mm_unpacklo_epi8(a0, b0); b1 = _mm_unpackhi_epi8(a0, b0); a0 = _mm_unpacklo_epi8(a1, b1); b0 = _mm_unpackhi_epi8(a1, b1); _mm_storeu_si128((__m128i*)&out_b[0 * size + ii], a0); _mm_storeu_si128((__m128i*)&out_b[1 * size + ii], b0); } return bshuf_trans_byte_elem_remainder(in, out, size, 2, size - size % 16); } /* Transpose bytes within elements for 32 bit elements. */ int64_t bshuf_trans_byte_elem_SSE_32(void* in, void* out, const size_t size) { char* in_b = (char*)in; char* out_b = (char*)out; __m128i a0, b0, c0, d0, a1, b1, c1, d1; size_t ii; for (ii = 0; ii + 15 < size; ii += 16) { a0 = _mm_loadu_si128((__m128i*)&in_b[4 * ii + 0 * 16]); b0 = _mm_loadu_si128((__m128i*)&in_b[4 * ii + 1 * 16]); c0 = _mm_loadu_si128((__m128i*)&in_b[4 * ii + 2 * 16]); d0 = _mm_loadu_si128((__m128i*)&in_b[4 * ii + 3 * 16]); a1 = _mm_unpacklo_epi8(a0, b0); b1 = _mm_unpackhi_epi8(a0, b0); c1 = _mm_unpacklo_epi8(c0, d0); d1 = _mm_unpackhi_epi8(c0, d0); a0 = _mm_unpacklo_epi8(a1, b1); b0 = _mm_unpackhi_epi8(a1, b1); c0 = _mm_unpacklo_epi8(c1, d1); d0 = _mm_unpackhi_epi8(c1, d1); a1 = _mm_unpacklo_epi8(a0, b0); b1 = _mm_unpackhi_epi8(a0, b0); c1 = _mm_unpacklo_epi8(c0, d0); d1 = _mm_unpackhi_epi8(c0, d0); a0 = _mm_unpacklo_epi64(a1, c1); b0 = _mm_unpackhi_epi64(a1, c1); c0 = _mm_unpacklo_epi64(b1, d1); d0 = _mm_unpackhi_epi64(b1, d1); _mm_storeu_si128((__m128i*)&out_b[0 * size + ii], a0); _mm_storeu_si128((__m128i*)&out_b[1 * size + ii], b0); _mm_storeu_si128((__m128i*)&out_b[2 * size + ii], c0); _mm_storeu_si128((__m128i*)&out_b[3 * size + ii], d0); } return bshuf_trans_byte_elem_remainder(in, out, size, 4, size - size % 16); } /* Transpose bytes within elements for 64 bit elements. */ int64_t bshuf_trans_byte_elem_SSE_64(void* in, void* out, const size_t size) { char* in_b = (char*)in; char* out_b = (char*)out; __m128i a0, b0, c0, d0, e0, f0, g0, h0; __m128i a1, b1, c1, d1, e1, f1, g1, h1; size_t ii; for (ii = 0; ii + 15 < size; ii += 16) { a0 = _mm_loadu_si128((__m128i*)&in_b[8 * ii + 0 * 16]); b0 = _mm_loadu_si128((__m128i*)&in_b[8 * ii + 1 * 16]); c0 = _mm_loadu_si128((__m128i*)&in_b[8 * ii + 2 * 16]); d0 = _mm_loadu_si128((__m128i*)&in_b[8 * ii + 3 * 16]); e0 = _mm_loadu_si128((__m128i*)&in_b[8 * ii + 4 * 16]); f0 = _mm_loadu_si128((__m128i*)&in_b[8 * ii + 5 * 16]); g0 = _mm_loadu_si128((__m128i*)&in_b[8 * ii + 6 * 16]); h0 = _mm_loadu_si128((__m128i*)&in_b[8 * ii + 7 * 16]); a1 = _mm_unpacklo_epi8(a0, b0); b1 = _mm_unpackhi_epi8(a0, b0); c1 = _mm_unpacklo_epi8(c0, d0); d1 = _mm_unpackhi_epi8(c0, d0); e1 = _mm_unpacklo_epi8(e0, f0); f1 = _mm_unpackhi_epi8(e0, f0); g1 = _mm_unpacklo_epi8(g0, h0); h1 = _mm_unpackhi_epi8(g0, h0); a0 = _mm_unpacklo_epi8(a1, b1); b0 = _mm_unpackhi_epi8(a1, b1); c0 = _mm_unpacklo_epi8(c1, d1); d0 = _mm_unpackhi_epi8(c1, d1); e0 = _mm_unpacklo_epi8(e1, f1); f0 = _mm_unpackhi_epi8(e1, f1); g0 = _mm_unpacklo_epi8(g1, h1); h0 = _mm_unpackhi_epi8(g1, h1); a1 = _mm_unpacklo_epi32(a0, c0); b1 = _mm_unpackhi_epi32(a0, c0); c1 = _mm_unpacklo_epi32(b0, d0); d1 = _mm_unpackhi_epi32(b0, d0); e1 = _mm_unpacklo_epi32(e0, g0); f1 = _mm_unpackhi_epi32(e0, g0); g1 = _mm_unpacklo_epi32(f0, h0); h1 = _mm_unpackhi_epi32(f0, h0); a0 = _mm_unpacklo_epi64(a1, e1); b0 = _mm_unpackhi_epi64(a1, e1); c0 = _mm_unpacklo_epi64(b1, f1); d0 = _mm_unpackhi_epi64(b1, f1); e0 = _mm_unpacklo_epi64(c1, g1); f0 = _mm_unpackhi_epi64(c1, g1); g0 = _mm_unpacklo_epi64(d1, h1); h0 = _mm_unpackhi_epi64(d1, h1); _mm_storeu_si128((__m128i*)&out_b[0 * size + ii], a0); _mm_storeu_si128((__m128i*)&out_b[1 * size + ii], b0); _mm_storeu_si128((__m128i*)&out_b[2 * size + ii], c0); _mm_storeu_si128((__m128i*)&out_b[3 * size + ii], d0); _mm_storeu_si128((__m128i*)&out_b[4 * size + ii], e0); _mm_storeu_si128((__m128i*)&out_b[5 * size + ii], f0); _mm_storeu_si128((__m128i*)&out_b[6 * size + ii], g0); _mm_storeu_si128((__m128i*)&out_b[7 * size + ii], h0); } return bshuf_trans_byte_elem_remainder(in, out, size, 8, size - size % 16); } /* Memory copy with bshuf call signature. */ int64_t bshuf_copy(void* in, void* out, const size_t size, const size_t elem_size) { char* in_b = (char*)in; char* out_b = (char*)out; memcpy(out_b, in_b, size * elem_size); return (int64_t)size * (int64_t)elem_size; } /* Transpose bytes within elements using best SSE algorithm available. */ int64_t bshuf_trans_byte_elem_sse2(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf) { int64_t count; /* Trivial cases: power of 2 bytes. */ switch (elem_size) { case 1: count = bshuf_copy(in, out, size, elem_size); return count; case 2: count = bshuf_trans_byte_elem_SSE_16(in, out, size); return count; case 4: count = bshuf_trans_byte_elem_SSE_32(in, out, size); return count; case 8: count = bshuf_trans_byte_elem_SSE_64(in, out, size); return count; } /* Worst case: odd number of bytes. Turns out that this is faster for */ /* (odd * 2) byte elements as well (hence % 4). */ if (elem_size % 4) { count = bshuf_trans_byte_elem_scal(in, out, size, elem_size); return count; } /* Multiple of power of 2: transpose hierarchically. */ { size_t nchunk_elem; if ((elem_size % 8) == 0) { nchunk_elem = elem_size / 8; TRANS_ELEM_TYPE(in, out, size, nchunk_elem, int64_t); count = bshuf_trans_byte_elem_SSE_64(out, tmp_buf, size * nchunk_elem); bshuf_trans_elem(tmp_buf, out, 8, nchunk_elem, size); } else if ((elem_size % 4) == 0) { nchunk_elem = elem_size / 4; TRANS_ELEM_TYPE(in, out, size, nchunk_elem, int32_t); count = bshuf_trans_byte_elem_SSE_32(out, tmp_buf, size * nchunk_elem); bshuf_trans_elem(tmp_buf, out, 4, nchunk_elem, size); } else { /* Not used since scalar algorithm is faster. */ nchunk_elem = elem_size / 2; TRANS_ELEM_TYPE(in, out, size, nchunk_elem, int16_t); count = bshuf_trans_byte_elem_SSE_16(out, tmp_buf, size * nchunk_elem); bshuf_trans_elem(tmp_buf, out, 2, nchunk_elem, size); } return count; } } /* Transpose bits within bytes. */ int64_t bshuf_trans_bit_byte_sse2(void* in, void* out, const size_t size, const size_t elem_size) { char* in_b = (char*)in; char* out_b = (char*)out; uint16_t* out_ui16; int64_t count; size_t nbyte = elem_size * size; __m128i xmm; int32_t bt; size_t ii, kk; CHECK_MULT_EIGHT(nbyte); for (ii = 0; ii + 15 < nbyte; ii += 16) { xmm = _mm_loadu_si128((__m128i*)&in_b[ii]); for (kk = 0; kk < 8; kk++) { bt = _mm_movemask_epi8(xmm); xmm = _mm_slli_epi16(xmm, 1); out_ui16 = (uint16_t*)&out_b[((7 - kk) * nbyte + ii) / 8]; *out_ui16 = (uint16_t)bt; } } count = bshuf_trans_bit_byte_remainder(in, out, size, elem_size, nbyte - nbyte % 16); return count; } /* Transpose bits within elements. */ int64_t bshuf_trans_bit_elem_sse2(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf) { int64_t count; CHECK_MULT_EIGHT(size); count = bshuf_trans_byte_elem_sse2(in, out, size, elem_size, tmp_buf); CHECK_ERR(count); count = bshuf_trans_bit_byte_sse2(out, tmp_buf, size, elem_size); CHECK_ERR(count); count = bshuf_trans_bitrow_eight(tmp_buf, out, size, elem_size); return count; } /* For data organized into a row for each bit (8 * elem_size rows), transpose * the bytes. */ int64_t bshuf_trans_byte_bitrow_sse2(void* in, void* out, const size_t size, const size_t elem_size) { char* in_b = (char*)in; char* out_b = (char*)out; size_t nrows = 8 * elem_size; size_t nbyte_row = size / 8; size_t ii, jj; __m128i a0, b0, c0, d0, e0, f0, g0, h0; __m128i a1, b1, c1, d1, e1, f1, g1, h1; __m128* as, * bs, * cs, * ds, * es, * fs, * gs, * hs; CHECK_MULT_EIGHT(size); for (ii = 0; ii + 7 < nrows; ii += 8) { for (jj = 0; jj + 15 < nbyte_row; jj += 16) { a0 = _mm_loadu_si128((__m128i*)&in_b[(ii + 0) * nbyte_row + jj]); b0 = _mm_loadu_si128((__m128i*)&in_b[(ii + 1) * nbyte_row + jj]); c0 = _mm_loadu_si128((__m128i*)&in_b[(ii + 2) * nbyte_row + jj]); d0 = _mm_loadu_si128((__m128i*)&in_b[(ii + 3) * nbyte_row + jj]); e0 = _mm_loadu_si128((__m128i*)&in_b[(ii + 4) * nbyte_row + jj]); f0 = _mm_loadu_si128((__m128i*)&in_b[(ii + 5) * nbyte_row + jj]); g0 = _mm_loadu_si128((__m128i*)&in_b[(ii + 6) * nbyte_row + jj]); h0 = _mm_loadu_si128((__m128i*)&in_b[(ii + 7) * nbyte_row + jj]); a1 = _mm_unpacklo_epi8(a0, b0); b1 = _mm_unpacklo_epi8(c0, d0); c1 = _mm_unpacklo_epi8(e0, f0); d1 = _mm_unpacklo_epi8(g0, h0); e1 = _mm_unpackhi_epi8(a0, b0); f1 = _mm_unpackhi_epi8(c0, d0); g1 = _mm_unpackhi_epi8(e0, f0); h1 = _mm_unpackhi_epi8(g0, h0); a0 = _mm_unpacklo_epi16(a1, b1); b0 = _mm_unpacklo_epi16(c1, d1); c0 = _mm_unpackhi_epi16(a1, b1); d0 = _mm_unpackhi_epi16(c1, d1); e0 = _mm_unpacklo_epi16(e1, f1); f0 = _mm_unpacklo_epi16(g1, h1); g0 = _mm_unpackhi_epi16(e1, f1); h0 = _mm_unpackhi_epi16(g1, h1); a1 = _mm_unpacklo_epi32(a0, b0); b1 = _mm_unpackhi_epi32(a0, b0); c1 = _mm_unpacklo_epi32(c0, d0); d1 = _mm_unpackhi_epi32(c0, d0); e1 = _mm_unpacklo_epi32(e0, f0); f1 = _mm_unpackhi_epi32(e0, f0); g1 = _mm_unpacklo_epi32(g0, h0); h1 = _mm_unpackhi_epi32(g0, h0); /* We don't have a storeh instruction for integers, so interpret */ /* as a float. Have a storel (_mm_storel_epi64). */ as = (__m128*)&a1; bs = (__m128*)&b1; cs = (__m128*)&c1; ds = (__m128*)&d1; es = (__m128*)&e1; fs = (__m128*)&f1; gs = (__m128*)&g1; hs = (__m128*)&h1; _mm_storel_pi((__m64*)&out_b[(jj + 0) * nrows + ii], *as); _mm_storel_pi((__m64*)&out_b[(jj + 2) * nrows + ii], *bs); _mm_storel_pi((__m64*)&out_b[(jj + 4) * nrows + ii], *cs); _mm_storel_pi((__m64*)&out_b[(jj + 6) * nrows + ii], *ds); _mm_storel_pi((__m64*)&out_b[(jj + 8) * nrows + ii], *es); _mm_storel_pi((__m64*)&out_b[(jj + 10) * nrows + ii], *fs); _mm_storel_pi((__m64*)&out_b[(jj + 12) * nrows + ii], *gs); _mm_storel_pi((__m64*)&out_b[(jj + 14) * nrows + ii], *hs); _mm_storeh_pi((__m64*)&out_b[(jj + 1) * nrows + ii], *as); _mm_storeh_pi((__m64*)&out_b[(jj + 3) * nrows + ii], *bs); _mm_storeh_pi((__m64*)&out_b[(jj + 5) * nrows + ii], *cs); _mm_storeh_pi((__m64*)&out_b[(jj + 7) * nrows + ii], *ds); _mm_storeh_pi((__m64*)&out_b[(jj + 9) * nrows + ii], *es); _mm_storeh_pi((__m64*)&out_b[(jj + 11) * nrows + ii], *fs); _mm_storeh_pi((__m64*)&out_b[(jj + 13) * nrows + ii], *gs); _mm_storeh_pi((__m64*)&out_b[(jj + 15) * nrows + ii], *hs); } for (jj = nbyte_row - nbyte_row % 16; jj < nbyte_row; jj++) { out_b[jj * nrows + ii + 0] = in_b[(ii + 0) * nbyte_row + jj]; out_b[jj * nrows + ii + 1] = in_b[(ii + 1) * nbyte_row + jj]; out_b[jj * nrows + ii + 2] = in_b[(ii + 2) * nbyte_row + jj]; out_b[jj * nrows + ii + 3] = in_b[(ii + 3) * nbyte_row + jj]; out_b[jj * nrows + ii + 4] = in_b[(ii + 4) * nbyte_row + jj]; out_b[jj * nrows + ii + 5] = in_b[(ii + 5) * nbyte_row + jj]; out_b[jj * nrows + ii + 6] = in_b[(ii + 6) * nbyte_row + jj]; out_b[jj * nrows + ii + 7] = in_b[(ii + 7) * nbyte_row + jj]; } } return (int64_t)size * (int64_t)elem_size; } /* Shuffle bits within the bytes of eight element blocks. */ int64_t bshuf_shuffle_bit_eightelem_sse2(void* in, void* out, const size_t size, const size_t elem_size) { /* With a bit of care, this could be written such that such that it is */ /* in_buf = out_buf safe. */ char* in_b = (char*)in; uint16_t* out_ui16 = (uint16_t*)out; size_t nbyte = elem_size * size; __m128i xmm; int32_t bt; size_t ii, jj, kk; size_t ind; CHECK_MULT_EIGHT(size); if (elem_size % 2) { bshuf_shuffle_bit_eightelem_scal(in, out, size, elem_size); } else { for (ii = 0; ii + 8 * elem_size - 1 < nbyte; ii += 8 * elem_size) { for (jj = 0; jj + 15 < 8 * elem_size; jj += 16) { xmm = _mm_loadu_si128((__m128i*)&in_b[ii + jj]); for (kk = 0; kk < 8; kk++) { bt = _mm_movemask_epi8(xmm); xmm = _mm_slli_epi16(xmm, 1); ind = (ii + jj / 8 + (7 - kk) * elem_size); out_ui16[ind / 2] = (uint16_t)bt; } } } } return (int64_t)size * (int64_t)elem_size; } /* Untranspose bits within elements. */ int64_t bshuf_untrans_bit_elem_sse2(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf) { int64_t count; CHECK_MULT_EIGHT(size); count = bshuf_trans_byte_bitrow_sse2(in, tmp_buf, size, elem_size); CHECK_ERR(count); count = bshuf_shuffle_bit_eightelem_sse2(tmp_buf, out, size, elem_size); return count; } #endif /* defined(__SSE2__) */ zmat-0.9.9/src/blosc2/blosc/CMakeLists.txt0000644000175200007730000002524014515254731020557 0ustar rlaboissrlaboiss# a simple way to detect that we are using CMAKE add_definitions(-DUSING_CMAKE) set(INTERNAL_LIBS ${PROJECT_SOURCE_DIR}/internal-complibs) # Hide symbols by default unless they're specifically exported. # This makes it easier to keep the set of exported symbols the # same across all compilers/platforms. set(CMAKE_C_VISIBILITY_PRESET hidden) # includes set(BLOSC_INCLUDE_DIRS ${BLOSC_INCLUDE_DIRS} ${CMAKE_CURRENT_SOURCE_DIR}) if(LZ4_FOUND) set(BLOSC_INCLUDE_DIRS ${BLOSC_INCLUDE_DIRS} ${LZ4_INCLUDE_DIR}) else() set(LZ4_LOCAL_DIR ${INTERNAL_LIBS}/lz4-1.9.4) set(BLOSC_INCLUDE_DIRS ${BLOSC_INCLUDE_DIRS} ${LZ4_LOCAL_DIR}) endif() if(NOT DEACTIVATE_ZLIB) if(ZLIB_NG_FOUND) set(BLOSC_INCLUDE_DIRS ${BLOSC_INCLUDE_DIRS} ${ZLIB_NG_INCLUDE_DIR}) elseif(ZLIB_FOUND) set(BLOSC_INCLUDE_DIRS ${BLOSC_INCLUDE_DIRS} ${ZLIB_INCLUDE_DIR}) else() set(ZLIB_LOCAL_DIR ${INTERNAL_LIBS}/${ZLIB_NG_DIR}) set(BLOSC_INCLUDE_DIRS ${BLOSC_INCLUDE_DIRS} ${ZLIB_LOCAL_DIR}) endif() endif() if(NOT DEACTIVATE_ZSTD) if(ZSTD_FOUND) set(BLOSC_INCLUDE_DIRS ${BLOSC_INCLUDE_DIRS} ${ZSTD_INCLUDE_DIR}) else() set(ZSTD_LOCAL_DIR ${INTERNAL_LIBS}/zstd) set(BLOSC_INCLUDE_DIRS ${BLOSC_INCLUDE_DIRS} ${ZSTD_LOCAL_DIR} ${ZSTD_LOCAL_DIR}/common) endif() endif() include_directories(${BLOSC_INCLUDE_DIRS}) # library sources set(SOURCES ${SOURCES} blosc2.c blosclz.c fastcopy.c fastcopy.h schunk.c frame.c stune.c stune.h context.h delta.c delta.h shuffle-generic.c bitshuffle-generic.c trunc-prec.c trunc-prec.h timestamp.c sframe.c directories.c blosc2-stdio.c) if(NOT CMAKE_SYSTEM_PROCESSOR STREQUAL arm64) if(COMPILER_SUPPORT_SSE2) message(STATUS "Adding run-time support for SSE2") set(SOURCES ${SOURCES} shuffle-sse2.c bitshuffle-sse2.c) endif() if(COMPILER_SUPPORT_AVX2) message(STATUS "Adding run-time support for AVX2") set(SOURCES ${SOURCES} shuffle-avx2.c bitshuffle-avx2.c) endif() endif() if(COMPILER_SUPPORT_NEON) message(STATUS "Adding run-time support for NEON") set(SOURCES ${SOURCES} shuffle-neon.c bitshuffle-neon.c) endif() if(COMPILER_SUPPORT_ALTIVEC) message(STATUS "Adding run-time support for ALTIVEC") set(SOURCES ${SOURCES} shuffle-altivec.c bitshuffle-altivec.c) endif() set(SOURCES ${SOURCES} shuffle.c) set(version_string ${BLOSC2_VERSION_MAJOR}.${BLOSC2_VERSION_MINOR}.${BLOSC2_VERSION_PATCH}) set(CMAKE_THREAD_PREFER_PTHREAD TRUE) # pre 3.1 set(THREADS_PREFER_PTHREAD_FLAG TRUE) # CMake 3.1+ if(WIN32) # try to use the system library find_package(Threads) if(NOT Threads_FOUND) message(STATUS "using the internal pthread library for win32 systems.") set(SOURCES ${SOURCES} win32/pthread.c) else() if(CMAKE_VERSION VERSION_LESS 3.1) set(LIBS ${LIBS} ${CMAKE_THREAD_LIBS_INIT}) else() set(LIBS ${LIBS} Threads::Threads) endif() endif() else() find_package(Threads REQUIRED) if(CMAKE_VERSION VERSION_LESS 3.1) set(LIBS ${LIBS} ${CMAKE_THREAD_LIBS_INIT}) else() set(LIBS ${LIBS} Threads::Threads) endif() endif() if(LZ4_FOUND) set(LIBS ${LIBS} ${LZ4_LIBRARY}) else() file(GLOB LZ4_FILES ${LZ4_LOCAL_DIR}/*.c) set(SOURCES ${SOURCES} ${LZ4_FILES}) source_group("LZ4" FILES ${LZ4_FILES}) endif() if(NOT DEACTIVATE_ZLIB) if(ZLIB_NG_FOUND) set(LIBS ${LIBS} ${ZLIB_NG_LIBRARY}) elseif(ZLIB_FOUND) set(LIBS ${LIBS} ${ZLIB_LIBRARY}) else() set(ZLIB_LOCAL_DIR ${INTERNAL_LIBS}/${ZLIB_NG_DIR}) file(GLOB ZLIB_FILES ${ZLIB_LOCAL_DIR}/*.c) set(SOURCES ${SOURCES} ${ZLIB_FILES}) source_group("Zlib" FILES ${ZLIB_FILES}) endif() endif() if(NOT DEACTIVATE_ZSTD) if(ZSTD_FOUND) set(LIBS ${LIBS} ${ZSTD_LIBRARY}) else() # Enable assembly code only when not using MSVC *and* x86 is there if((NOT MSVC) AND COMPILER_SUPPORT_SSE2) # if SSE2 is here, this is an x86 platform message(STATUS "Adding support for assembly sources in ZSTD") file(GLOB ZSTD_DECOMPRESS_FILES ${ZSTD_LOCAL_DIR}/decompress/*.S) else() message(STATUS "Disabling support for assembly sources in ZSTD") add_compile_definitions("ZSTD_DISABLE_ASM") endif() file(GLOB ZSTD_DECOMPRESS_FILES ${ZSTD_DECOMPRESS_FILES} ${ZSTD_LOCAL_DIR}/decompress/*.c) file(GLOB ZSTD_COMMON_FILES ${ZSTD_LOCAL_DIR}/common/*.c) file(GLOB ZSTD_COMPRESS_FILES ${ZSTD_LOCAL_DIR}/compress/*.c) file(GLOB ZSTD_DICT_FILES ${ZSTD_LOCAL_DIR}/dictBuilder/*.c) set(ZSTD_FILES ${ZSTD_COMMON_FILES} ${ZSTD_COMPRESS_FILES} ${ZSTD_DECOMPRESS_FILES} ${ZSTD_DICT_FILES}) set(SOURCES ${SOURCES} ${ZSTD_FILES}) source_group("Zstd" FILES ${ZSTD_FILES}) endif() endif() if(HAVE_IPP) set(LIBS ${LIBS} "${IPP_LIBRARIES}") endif() if(UNIX AND NOT APPLE) set(LIBS ${LIBS} "rt") set(LIBS ${LIBS} "m") # set(LIBS ${LIBS} "profiler") endif() # targets if(BUILD_SHARED) add_library(blosc2_shared SHARED ${SOURCES}) set_target_properties(blosc2_shared PROPERTIES OUTPUT_NAME blosc2) if(MSVC OR MINGW) set_target_properties(blosc2_shared PROPERTIES PREFIX lib) endif() set_target_properties(blosc2_shared PROPERTIES VERSION ${version_string} SOVERSION 2 # Change this when an ABI change happens ) set_property( TARGET blosc2_shared APPEND PROPERTY COMPILE_DEFINITIONS BLOSC_SHARED_LIBRARY) endif() # Based on the target architecture and hardware features supported # by the C compiler, set hardware architecture optimization flags # for specific shuffle implementations. if(COMPILER_SUPPORT_SSE2) if(MSVC) # MSVC targets SSE2 by default on 64-bit configurations, but not 32-bit configurations. if(${CMAKE_SIZEOF_VOID_P} EQUAL 4) set_source_files_properties( shuffle-sse2.c bitshuffle-sse2.c blosclz.c fastcopy.c PROPERTIES COMPILE_FLAGS "/arch:SSE2") endif() else() set_source_files_properties( shuffle-sse2.c bitshuffle-sse2.c blosclz.c fastcopy.c PROPERTIES COMPILE_FLAGS -msse2) endif() # Define a symbol for the shuffle-dispatch implementation # so it knows SSE2 is supported even though that file is # compiled without SSE2 support (for portability). set_property( SOURCE shuffle.c APPEND PROPERTY COMPILE_DEFINITIONS SHUFFLE_SSE2_ENABLED) endif() if(COMPILER_SUPPORT_AVX2) if(MSVC) set_source_files_properties( shuffle-avx2.c bitshuffle-avx2.c PROPERTIES COMPILE_FLAGS "/arch:AVX2") else() set_source_files_properties( shuffle-avx2.c bitshuffle-avx2.c PROPERTIES COMPILE_FLAGS -mavx2) endif() # Define a symbol for the shuffle-dispatch implementation # so it knows AVX2 is supported even though that file is # compiled without AVX2 support (for portability). set_property( SOURCE shuffle.c APPEND PROPERTY COMPILE_DEFINITIONS SHUFFLE_AVX2_ENABLED) endif() if(COMPILER_SUPPORT_NEON) set_source_files_properties( shuffle-neon.c bitshuffle-neon.c PROPERTIES COMPILE_FLAGS "-flax-vector-conversions") if(CMAKE_SYSTEM_PROCESSOR STREQUAL armv7l) # Only armv7l needs special -mfpu=neon flag; aarch64 doesn't. set_source_files_properties( shuffle-neon.c bitshuffle-neon.c PROPERTIES COMPILE_FLAGS "-mfpu=neon -flax-vector-conversions") endif() # Define a symbol for the shuffle-dispatch implementation # so it knows NEON is supported even though that file is # compiled without NEON support (for portability). set_property( SOURCE shuffle.c APPEND PROPERTY COMPILE_DEFINITIONS SHUFFLE_NEON_ENABLED) endif() if(COMPILER_SUPPORT_ALTIVEC) set_source_files_properties(shuffle-altivec.c bitshuffle-altivec.c PROPERTIES COMPILE_FLAGS -DNO_WARN_X86_INTRINSICS) # Define a symbol for the shuffle-dispatch implementation # so it knows ALTIVEC is supported even though that file is # compiled without ALTIVEC support (for portability). set_property( SOURCE shuffle.c APPEND PROPERTY COMPILE_DEFINITIONS SHUFFLE_ALTIVEC_ENABLED) endif() # When the option has been selected to compile the test suite, # compile an additional version of blosc2_static which exports # some normally-hidden symbols (to facilitate unit testing). if(BUILD_TESTS) add_library(blosc_testing STATIC ${SOURCES}) set_target_properties(blosc_testing PROPERTIES OUTPUT_NAME blosc_testing) if(MSVC OR MINGW) set_target_properties(blosc_testing PROPERTIES PREFIX lib) endif() set_property( TARGET blosc_testing APPEND PROPERTY COMPILE_DEFINITIONS BLOSC_SHARED_LIBRARY) set_property( TARGET blosc_testing APPEND PROPERTY COMPILE_DEFINITIONS BLOSC_TESTING) endif() if(BUILD_SHARED) target_link_libraries(blosc2_shared ${LIBS}) target_include_directories(blosc2_shared PUBLIC ${BLOSC_INCLUDE_DIRS}) endif() if(BUILD_TESTS) target_link_libraries(blosc_testing ${LIBS}) target_include_directories(blosc_testing PUBLIC ${BLOSC_INCLUDE_DIRS}) endif() if(BUILD_STATIC) add_library(blosc2_static STATIC ${SOURCES}) set_target_properties(blosc2_static PROPERTIES OUTPUT_NAME blosc2) if(MSVC OR MINGW) set_target_properties(blosc2_static PROPERTIES PREFIX lib) endif() target_link_libraries(blosc2_static ${LIBS}) target_include_directories(blosc2_static PUBLIC ${BLOSC_INCLUDE_DIRS}) endif() # install if(BLOSC_INSTALL) install(FILES ${PROJECT_SOURCE_DIR}/include/blosc2.h DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} COMPONENT DEV) install(FILES ${PROJECT_SOURCE_DIR}/include/blosc2/blosc2-export.h ${PROJECT_SOURCE_DIR}/include/blosc2/blosc2-common.h ${PROJECT_SOURCE_DIR}/include/blosc2/blosc2-stdio.h DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/blosc2 COMPONENT DEV) if(BUILD_PLUGINS) install(FILES ${PROJECT_SOURCE_DIR}/include/blosc2/filters-registry.h ${PROJECT_SOURCE_DIR}/include/blosc2/codecs-registry.h DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/blosc2 COMPONENT DEV) endif() if(BUILD_SHARED) install(TARGETS blosc2_shared LIBRARY COMPONENT LIB ARCHIVE COMPONENT DEV RUNTIME COMPONENT LIB) endif() if(BUILD_STATIC) install(TARGETS blosc2_static COMPONENT DEV) endif() endif() zmat-0.9.9/src/blosc2/blosc/timestamp.c0000644000175200007730000000405514515254731020167 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include "blosc2.h" /* System-specific high-precision timing functions. */ #if defined(_WIN32) /* Set a timestamp value to the current time. */ void blosc_set_timestamp(blosc_timestamp_t* timestamp) { /* Ignore the return value, assume the call always succeeds. */ QueryPerformanceCounter(timestamp); } /* Given two timestamp values, return the difference in nanoseconds. */ double blosc_elapsed_nsecs(blosc_timestamp_t start_time, blosc_timestamp_t end_time) { LARGE_INTEGER CounterFreq; QueryPerformanceFrequency(&CounterFreq); return (double)(end_time.QuadPart - start_time.QuadPart) / ((double)CounterFreq.QuadPart / 1e9); } #else /* Set a timestamp value to the current time. */ void blosc_set_timestamp(blosc_timestamp_t* timestamp) { #ifdef __MACH__ // OS X does not have clock_gettime, use clock_get_time clock_serv_t cclock; mach_timespec_t mts; host_get_clock_service(mach_host_self(), CALENDAR_CLOCK, &cclock); clock_get_time(cclock, &mts); mach_port_deallocate(mach_task_self(), cclock); timestamp->tv_sec = mts.tv_sec; timestamp->tv_nsec = mts.tv_nsec; #else clock_gettime(CLOCK_MONOTONIC, timestamp); #endif } /* Given two timestamp values, return the difference in nanoseconds. */ double blosc_elapsed_nsecs(blosc_timestamp_t start_time, blosc_timestamp_t end_time) { return (1e9 * (double)(end_time.tv_sec - start_time.tv_sec)) + (double)(end_time.tv_nsec - start_time.tv_nsec); } #endif /* Given two timeval stamps, return the difference in seconds */ double blosc_elapsed_secs(blosc_timestamp_t last, blosc_timestamp_t current) { return 1e-9 * blosc_elapsed_nsecs(last, current); } zmat-0.9.9/src/blosc2/blosc/blosc2.c0000644000175200007730000042307014515254731017352 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include #include #include #include #include #include #include #include "blosc2.h" #include "blosc-private.h" #include "../plugins/codecs/zfp/blosc2-zfp.h" #include "frame.h" #if defined(USING_CMAKE) #include "config.h" #endif /* USING_CMAKE */ #include "context.h" #include "shuffle.h" #include "delta.h" #include "trunc-prec.h" #include "blosclz.h" #include "stune.h" #include "blosc2/codecs-registry.h" #include "blosc2/filters-registry.h" #include "lz4.h" #include "lz4hc.h" #ifdef HAVE_IPP #include #include #endif #if defined(HAVE_ZLIB_NG) #ifdef ZLIB_COMPAT #include "zlib.h" #else #include "zlib-ng.h" #endif #elif defined(HAVE_ZLIB) #include "zlib.h" #endif /* HAVE_MINIZ */ #if defined(HAVE_ZSTD) #include "zstd.h" #include "zstd_errors.h" // #include "cover.h" // for experimenting with fast cover training for building dicts #include "zdict.h" #endif /* HAVE_ZSTD */ #if defined(_WIN32) && !defined(__MINGW32__) #include #include #include #define getpid _getpid #endif /* _WIN32 */ #if defined(_WIN32) && !defined(__GNUC__) #include "win32/pthread.c" #endif /* Synchronization variables */ /* Global context for non-contextual API */ static blosc2_context* g_global_context; static pthread_mutex_t global_comp_mutex; static int g_compressor = BLOSC_BLOSCLZ; static int g_delta = 0; /* The default splitmode */ static int32_t g_splitmode = BLOSC_FORWARD_COMPAT_SPLIT; /* the compressor to use by default */ static int16_t g_nthreads = 1; static int32_t g_force_blocksize = 0; static int g_initlib = 0; static blosc2_schunk* g_schunk = NULL; /* the pointer to super-chunk */ blosc2_codec g_codecs[256] = {0}; uint8_t g_ncodecs = 0; static blosc2_filter g_filters[256] = {0}; static uint64_t g_nfilters = 0; static blosc2_io_cb g_io[256] = {0}; static uint64_t g_nio = 0; // Forward declarations int init_threadpool(blosc2_context *context); int release_threadpool(blosc2_context *context); /* Macros for synchronization */ /* Wait until all threads are initialized */ #ifdef BLOSC_POSIX_BARRIERS #define WAIT_INIT(RET_VAL, CONTEXT_PTR) \ rc = pthread_barrier_wait(&(CONTEXT_PTR)->barr_init); \ if (rc != 0 && rc != PTHREAD_BARRIER_SERIAL_THREAD) { \ BLOSC_TRACE_ERROR("Could not wait on barrier (init): %d", rc); \ return((RET_VAL)); \ } #else #define WAIT_INIT(RET_VAL, CONTEXT_PTR) \ pthread_mutex_lock(&(CONTEXT_PTR)->count_threads_mutex); \ if ((CONTEXT_PTR)->count_threads < (CONTEXT_PTR)->nthreads) { \ (CONTEXT_PTR)->count_threads++; \ pthread_cond_wait(&(CONTEXT_PTR)->count_threads_cv, \ &(CONTEXT_PTR)->count_threads_mutex); \ } \ else { \ pthread_cond_broadcast(&(CONTEXT_PTR)->count_threads_cv); \ } \ pthread_mutex_unlock(&(CONTEXT_PTR)->count_threads_mutex); #endif /* Wait for all threads to finish */ #ifdef BLOSC_POSIX_BARRIERS #define WAIT_FINISH(RET_VAL, CONTEXT_PTR) \ rc = pthread_barrier_wait(&(CONTEXT_PTR)->barr_finish); \ if (rc != 0 && rc != PTHREAD_BARRIER_SERIAL_THREAD) { \ BLOSC_TRACE_ERROR("Could not wait on barrier (finish)"); \ return((RET_VAL)); \ } #else #define WAIT_FINISH(RET_VAL, CONTEXT_PTR) \ pthread_mutex_lock(&(CONTEXT_PTR)->count_threads_mutex); \ if ((CONTEXT_PTR)->count_threads > 0) { \ (CONTEXT_PTR)->count_threads--; \ pthread_cond_wait(&(CONTEXT_PTR)->count_threads_cv, \ &(CONTEXT_PTR)->count_threads_mutex); \ } \ else { \ pthread_cond_broadcast(&(CONTEXT_PTR)->count_threads_cv); \ } \ pthread_mutex_unlock(&(CONTEXT_PTR)->count_threads_mutex); #endif /* global variable to change threading backend from Blosc-managed to caller-managed */ static blosc_threads_callback threads_callback = 0; static void *threads_callback_data = 0; /* non-threadsafe function should be called before any other Blosc function in order to change how threads are managed */ void blosc2_set_threads_callback(blosc_threads_callback callback, void *callback_data) { threads_callback = callback; threads_callback_data = callback_data; } /* A function for aligned malloc that is portable */ static uint8_t* my_malloc(size_t size) { void* block = NULL; int res = 0; /* Do an alignment to 32 bytes because AVX2 is supported */ #if defined(_WIN32) /* A (void *) cast needed for avoiding a warning with MINGW :-/ */ block = (void *)_aligned_malloc(size, 32); #elif _POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600 /* Platform does have an implementation of posix_memalign */ res = posix_memalign(&block, 32, size); #else block = malloc(size); #endif /* _WIN32 */ if (block == NULL || res != 0) { BLOSC_TRACE_ERROR("Error allocating memory!"); return NULL; } return (uint8_t*)block; } /* Release memory booked by my_malloc */ static void my_free(void* block) { #if defined(_WIN32) _aligned_free(block); #else free(block); #endif /* _WIN32 */ } /* * Conversion routines between compressor and compression libraries */ /* Return the library code associated with the compressor name */ static int compname_to_clibcode(const char* compname) { if (strcmp(compname, BLOSC_BLOSCLZ_COMPNAME) == 0) return BLOSC_BLOSCLZ_LIB; if (strcmp(compname, BLOSC_LZ4_COMPNAME) == 0) return BLOSC_LZ4_LIB; if (strcmp(compname, BLOSC_LZ4HC_COMPNAME) == 0) return BLOSC_LZ4_LIB; if (strcmp(compname, BLOSC_ZLIB_COMPNAME) == 0) return BLOSC_ZLIB_LIB; if (strcmp(compname, BLOSC_ZSTD_COMPNAME) == 0) return BLOSC_ZSTD_LIB; for (int i = 0; i < g_ncodecs; ++i) { if (strcmp(compname, g_codecs[i].compname) == 0) return g_codecs[i].complib; } return BLOSC2_ERROR_NOT_FOUND; } /* Return the library name associated with the compressor code */ static const char* clibcode_to_clibname(int clibcode) { if (clibcode == BLOSC_BLOSCLZ_LIB) return BLOSC_BLOSCLZ_LIBNAME; if (clibcode == BLOSC_LZ4_LIB) return BLOSC_LZ4_LIBNAME; if (clibcode == BLOSC_ZLIB_LIB) return BLOSC_ZLIB_LIBNAME; if (clibcode == BLOSC_ZSTD_LIB) return BLOSC_ZSTD_LIBNAME; for (int i = 0; i < g_ncodecs; ++i) { if (clibcode == g_codecs[i].complib) return g_codecs[i].compname; } return NULL; /* should never happen */ } /* * Conversion routines between compressor names and compressor codes */ /* Get the compressor name associated with the compressor code */ int blosc2_compcode_to_compname(int compcode, const char** compname) { int code = -1; /* -1 means non-existent compressor code */ const char* name = NULL; /* Map the compressor code */ if (compcode == BLOSC_BLOSCLZ) name = BLOSC_BLOSCLZ_COMPNAME; else if (compcode == BLOSC_LZ4) name = BLOSC_LZ4_COMPNAME; else if (compcode == BLOSC_LZ4HC) name = BLOSC_LZ4HC_COMPNAME; else if (compcode == BLOSC_ZLIB) name = BLOSC_ZLIB_COMPNAME; else if (compcode == BLOSC_ZSTD) name = BLOSC_ZSTD_COMPNAME; else { for (int i = 0; i < g_ncodecs; ++i) { if (compcode == g_codecs[i].compcode) { name = g_codecs[i].compname; break; } } } *compname = name; /* Guess if there is support for this code */ if (compcode == BLOSC_BLOSCLZ) code = BLOSC_BLOSCLZ; else if (compcode == BLOSC_LZ4) code = BLOSC_LZ4; else if (compcode == BLOSC_LZ4HC) code = BLOSC_LZ4HC; #if defined(HAVE_ZLIB) else if (compcode == BLOSC_ZLIB) code = BLOSC_ZLIB; #endif /* HAVE_ZLIB */ #if defined(HAVE_ZSTD) else if (compcode == BLOSC_ZSTD) code = BLOSC_ZSTD; #endif /* HAVE_ZSTD */ else if (compcode >= BLOSC_LAST_CODEC) code = compcode; return code; } /* Get the compressor code for the compressor name. -1 if it is not available */ int blosc2_compname_to_compcode(const char* compname) { int code = -1; /* -1 means non-existent compressor code */ if (strcmp(compname, BLOSC_BLOSCLZ_COMPNAME) == 0) { code = BLOSC_BLOSCLZ; } else if (strcmp(compname, BLOSC_LZ4_COMPNAME) == 0) { code = BLOSC_LZ4; } else if (strcmp(compname, BLOSC_LZ4HC_COMPNAME) == 0) { code = BLOSC_LZ4HC; } #if defined(HAVE_ZLIB) else if (strcmp(compname, BLOSC_ZLIB_COMPNAME) == 0) { code = BLOSC_ZLIB; } #endif /* HAVE_ZLIB */ #if defined(HAVE_ZSTD) else if (strcmp(compname, BLOSC_ZSTD_COMPNAME) == 0) { code = BLOSC_ZSTD; } #endif /* HAVE_ZSTD */ else{ for (int i = 0; i < g_ncodecs; ++i) { if (strcmp(compname, g_codecs[i].compname) == 0) { code = g_codecs[i].compcode; break; } } } return code; } /* Convert compressor code to blosc compressor format code */ static int compcode_to_compformat(int compcode) { switch (compcode) { case BLOSC_BLOSCLZ: return BLOSC_BLOSCLZ_FORMAT; case BLOSC_LZ4: return BLOSC_LZ4_FORMAT; case BLOSC_LZ4HC: return BLOSC_LZ4HC_FORMAT; #if defined(HAVE_ZLIB) case BLOSC_ZLIB: return BLOSC_ZLIB_FORMAT; #endif /* HAVE_ZLIB */ #if defined(HAVE_ZSTD) case BLOSC_ZSTD: return BLOSC_ZSTD_FORMAT; break; #endif /* HAVE_ZSTD */ default: return BLOSC_UDCODEC_FORMAT; } return -1; } /* Convert compressor code to blosc compressor format version */ static int compcode_to_compversion(int compcode) { /* Write compressor format */ switch (compcode) { case BLOSC_BLOSCLZ: return BLOSC_BLOSCLZ_VERSION_FORMAT; case BLOSC_LZ4: return BLOSC_LZ4_VERSION_FORMAT; case BLOSC_LZ4HC: return BLOSC_LZ4HC_VERSION_FORMAT; #if defined(HAVE_ZLIB) case BLOSC_ZLIB: return BLOSC_ZLIB_VERSION_FORMAT; break; #endif /* HAVE_ZLIB */ #if defined(HAVE_ZSTD) case BLOSC_ZSTD: return BLOSC_ZSTD_VERSION_FORMAT; break; #endif /* HAVE_ZSTD */ default: for (int i = 0; i < g_ncodecs; ++i) { if (compcode == g_codecs[i].compcode) { return g_codecs[i].compver; } } } return -1; } static int lz4_wrap_compress(const char* input, size_t input_length, char* output, size_t maxout, int accel, void* hash_table) { BLOSC_UNUSED_PARAM(accel); int cbytes; #ifdef HAVE_IPP if (hash_table == NULL) { return BLOSC2_ERROR_INVALID_PARAM; // the hash table should always be initialized } int outlen = (int)maxout; int inlen = (int)input_length; // I have not found any function that uses `accel` like in `LZ4_compress_fast`, but // the IPP LZ4Safe call does a pretty good job on compressing well, so let's use it IppStatus status = ippsEncodeLZ4Safe_8u((const Ipp8u*)input, &inlen, (Ipp8u*)output, &outlen, (Ipp8u*)hash_table); if (status == ippStsDstSizeLessExpected) { return 0; // we cannot compress in required outlen } else if (status != ippStsNoErr) { return BLOSC2_ERROR_FAILURE; // an unexpected error happened } cbytes = outlen; #else BLOSC_UNUSED_PARAM(hash_table); accel = 1; // deactivate acceleration to match IPP behaviour cbytes = LZ4_compress_fast(input, output, (int)input_length, (int)maxout, accel); #endif return cbytes; } static int lz4hc_wrap_compress(const char* input, size_t input_length, char* output, size_t maxout, int clevel) { int cbytes; if (input_length > (size_t)(UINT32_C(2) << 30)) return BLOSC2_ERROR_2GB_LIMIT; /* clevel for lz4hc goes up to 12, at least in LZ4 1.7.5 * but levels larger than 9 do not buy much compression. */ cbytes = LZ4_compress_HC(input, output, (int)input_length, (int)maxout, clevel); return cbytes; } static int lz4_wrap_decompress(const char* input, size_t compressed_length, char* output, size_t maxout) { int nbytes; #ifdef HAVE_IPP int outlen = (int)maxout; int inlen = (int)compressed_length; IppStatus status; status = ippsDecodeLZ4_8u((const Ipp8u*)input, inlen, (Ipp8u*)output, &outlen); //status = ippsDecodeLZ4Dict_8u((const Ipp8u*)input, &inlen, (Ipp8u*)output, 0, &outlen, NULL, 1 << 16); nbytes = (status == ippStsNoErr) ? outlen : -outlen; #else nbytes = LZ4_decompress_safe(input, output, (int)compressed_length, (int)maxout); #endif if (nbytes != (int)maxout) { return 0; } return (int)maxout; } #if defined(HAVE_ZLIB) /* zlib is not very respectful with sharing name space with others. Fortunately, its names do not collide with those already in blosc. */ static int zlib_wrap_compress(const char* input, size_t input_length, char* output, size_t maxout, int clevel) { int status; #if defined(HAVE_ZLIB_NG) && ! defined(ZLIB_COMPAT) size_t cl = maxout; status = zng_compress2( (uint8_t*)output, &cl, (uint8_t*)input, input_length, clevel); #else uLongf cl = (uLongf)maxout; status = compress2( (Bytef*)output, &cl, (Bytef*)input, (uLong)input_length, clevel); #endif if (status != Z_OK) { return 0; } return (int)cl; } static int zlib_wrap_decompress(const char* input, size_t compressed_length, char* output, size_t maxout) { int status; #if defined(HAVE_ZLIB_NG) && ! defined(ZLIB_COMPAT) size_t ul = maxout; status = zng_uncompress( (uint8_t*)output, &ul, (uint8_t*)input, compressed_length); #else uLongf ul = (uLongf)maxout; status = uncompress( (Bytef*)output, &ul, (Bytef*)input, (uLong)compressed_length); #endif if (status != Z_OK) { return 0; } return (int)ul; } #endif /* HAVE_ZLIB */ #if defined(HAVE_ZSTD) static int zstd_wrap_compress(struct thread_context* thread_context, const char* input, size_t input_length, char* output, size_t maxout, int clevel) { size_t code; blosc2_context* context = thread_context->parent_context; clevel = (clevel < 9) ? clevel * 2 - 1 : ZSTD_maxCLevel(); /* Make the level 8 close enough to maxCLevel */ if (clevel == 8) clevel = ZSTD_maxCLevel() - 2; if (thread_context->zstd_cctx == NULL) { thread_context->zstd_cctx = ZSTD_createCCtx(); } if (context->use_dict) { assert(context->dict_cdict != NULL); code = ZSTD_compress_usingCDict( thread_context->zstd_cctx, (void*)output, maxout, (void*)input, input_length, context->dict_cdict); } else { code = ZSTD_compressCCtx(thread_context->zstd_cctx, (void*)output, maxout, (void*)input, input_length, clevel); } if (ZSTD_isError(code) != ZSTD_error_no_error) { // Do not print anything because blosc will just memcpy this buffer // fprintf(stderr, "Error in ZSTD compression: '%s'. Giving up.\n", // ZDICT_getErrorName(code)); return 0; } return (int)code; } static int zstd_wrap_decompress(struct thread_context* thread_context, const char* input, size_t compressed_length, char* output, size_t maxout) { size_t code; blosc2_context* context = thread_context->parent_context; if (thread_context->zstd_dctx == NULL) { thread_context->zstd_dctx = ZSTD_createDCtx(); } if (context->use_dict) { assert(context->dict_ddict != NULL); code = ZSTD_decompress_usingDDict( thread_context->zstd_dctx, (void*)output, maxout, (void*)input, compressed_length, context->dict_ddict); } else { code = ZSTD_decompressDCtx(thread_context->zstd_dctx, (void*)output, maxout, (void*)input, compressed_length); } if (ZSTD_isError(code) != ZSTD_error_no_error) { BLOSC_TRACE_ERROR("Error in ZSTD decompression: '%s'. Giving up.", ZDICT_getErrorName(code)); return 0; } return (int)code; } #endif /* HAVE_ZSTD */ /* Compute acceleration for blosclz */ static int get_accel(const blosc2_context* context) { int clevel = context->clevel; if (context->compcode == BLOSC_LZ4) { /* This acceleration setting based on discussions held in: * https://groups.google.com/forum/#!topic/lz4c/zosy90P8MQw */ return (10 - clevel); } return 1; } int do_nothing(uint8_t filter, char cmode) { if (cmode == 'c') { return (filter == BLOSC_NOFILTER); } else { // TRUNC_PREC do not have to be applied during decompression return ((filter == BLOSC_NOFILTER) || (filter == BLOSC_TRUNC_PREC)); } } int next_filter(const uint8_t* filters, int current_filter, char cmode) { for (int i = current_filter - 1; i >= 0; i--) { if (!do_nothing(filters[i], cmode)) { return filters[i]; } } return BLOSC_NOFILTER; } int last_filter(const uint8_t* filters, char cmode) { int last_index = -1; for (int i = BLOSC2_MAX_FILTERS - 1; i >= 0; i--) { if (!do_nothing(filters[i], cmode)) { last_index = i; } } return last_index; } /* Convert filter pipeline to filter flags */ static uint8_t filters_to_flags(const uint8_t* filters) { uint8_t flags = 0; for (int i = 0; i < BLOSC2_MAX_FILTERS; i++) { switch (filters[i]) { case BLOSC_SHUFFLE: flags |= BLOSC_DOSHUFFLE; break; case BLOSC_BITSHUFFLE: flags |= BLOSC_DOBITSHUFFLE; break; case BLOSC_DELTA: flags |= BLOSC_DODELTA; break; default : break; } } return flags; } /* Convert filter flags to filter pipeline */ static void flags_to_filters(const uint8_t flags, uint8_t* filters) { /* Initialize the filter pipeline */ memset(filters, 0, BLOSC2_MAX_FILTERS); /* Fill the filter pipeline */ if (flags & BLOSC_DOSHUFFLE) filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_SHUFFLE; if (flags & BLOSC_DOBITSHUFFLE) filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_BITSHUFFLE; if (flags & BLOSC_DODELTA) filters[BLOSC2_MAX_FILTERS - 2] = BLOSC_DELTA; } /* Get filter flags from header flags */ static uint8_t get_filter_flags(const uint8_t header_flags, const int32_t typesize) { uint8_t flags = 0; if ((header_flags & BLOSC_DOSHUFFLE) && (typesize > 1)) { flags |= BLOSC_DOSHUFFLE; } if (header_flags & BLOSC_DOBITSHUFFLE) { flags |= BLOSC_DOBITSHUFFLE; } if (header_flags & BLOSC_DODELTA) { flags |= BLOSC_DODELTA; } if (header_flags & BLOSC_MEMCPYED) { flags |= BLOSC_MEMCPYED; } return flags; } typedef struct blosc_header_s { uint8_t version; uint8_t versionlz; uint8_t flags; uint8_t typesize; int32_t nbytes; int32_t blocksize; int32_t cbytes; // Extended Blosc2 header uint8_t filter_codes[BLOSC2_MAX_FILTERS]; uint8_t udcompcode; uint8_t compcode_meta; uint8_t filter_meta[BLOSC2_MAX_FILTERS]; uint8_t reserved2; uint8_t blosc2_flags; } blosc_header; int read_chunk_header(const uint8_t* src, int32_t srcsize, bool extended_header, blosc_header* header) { memset(header, 0, sizeof(blosc_header)); if (srcsize < BLOSC_MIN_HEADER_LENGTH) { BLOSC_TRACE_ERROR("Not enough space to read Blosc header."); return BLOSC2_ERROR_READ_BUFFER; } memcpy(header, src, BLOSC_MIN_HEADER_LENGTH); bool little_endian = is_little_endian(); if (!little_endian) { header->nbytes = bswap32_(header->nbytes); header->blocksize = bswap32_(header->blocksize); header->cbytes = bswap32_(header->cbytes); } if (header->version > BLOSC2_VERSION_FORMAT) { /* Version from future */ return BLOSC2_ERROR_VERSION_SUPPORT; } if (header->cbytes < BLOSC_MIN_HEADER_LENGTH) { BLOSC_TRACE_ERROR("`cbytes` is too small to read min header."); return BLOSC2_ERROR_INVALID_HEADER; } if (header->blocksize <= 0 || (header->nbytes > 0 && (header->blocksize > header->nbytes))) { BLOSC_TRACE_ERROR("`blocksize` is zero or greater than uncompressed size"); return BLOSC2_ERROR_INVALID_HEADER; } if (header->blocksize > BLOSC2_MAXBLOCKSIZE) { BLOSC_TRACE_ERROR("`blocksize` greater than maximum allowed"); return BLOSC2_ERROR_INVALID_HEADER; } if (header->typesize == 0) { BLOSC_TRACE_ERROR("`typesize` is zero."); return BLOSC2_ERROR_INVALID_HEADER; } /* Read extended header if it is wanted */ if ((extended_header) && (header->flags & BLOSC_DOSHUFFLE) && (header->flags & BLOSC_DOBITSHUFFLE)) { if (header->cbytes < BLOSC_EXTENDED_HEADER_LENGTH) { BLOSC_TRACE_ERROR("`cbytes` is too small to read extended header."); return BLOSC2_ERROR_INVALID_HEADER; } if (srcsize < BLOSC_EXTENDED_HEADER_LENGTH) { BLOSC_TRACE_ERROR("Not enough space to read Blosc extended header."); return BLOSC2_ERROR_READ_BUFFER; } memcpy((uint8_t *)header + BLOSC_MIN_HEADER_LENGTH, src + BLOSC_MIN_HEADER_LENGTH, BLOSC_EXTENDED_HEADER_LENGTH - BLOSC_MIN_HEADER_LENGTH); int32_t special_type = (header->blosc2_flags >> 4) & BLOSC2_SPECIAL_MASK; if (special_type != 0) { if (header->nbytes % header->typesize != 0) { BLOSC_TRACE_ERROR("`nbytes` is not a multiple of typesize"); return BLOSC2_ERROR_INVALID_HEADER; } if (special_type == BLOSC2_SPECIAL_VALUE) { if (header->cbytes < BLOSC_EXTENDED_HEADER_LENGTH + header->typesize) { BLOSC_TRACE_ERROR("`cbytes` is too small for run length encoding"); return BLOSC2_ERROR_READ_BUFFER; } } } // The number of filters depends on the version of the header. Blosc2 alpha series // did not initialize filters to zero beyond the max supported. if (header->version == BLOSC2_VERSION_FORMAT_ALPHA) { header->filter_codes[5] = 0; header->filter_meta[5] = 0; } } else { flags_to_filters(header->flags, header->filter_codes); } return 0; } static inline void blosc2_calculate_blocks(blosc2_context* context) { /* Compute number of blocks in buffer */ context->nblocks = context->sourcesize / context->blocksize; context->leftover = context->sourcesize % context->blocksize; context->nblocks = (context->leftover > 0) ? (context->nblocks + 1) : context->nblocks; } static int blosc2_initialize_context_from_header(blosc2_context* context, blosc_header* header) { context->header_flags = header->flags; context->typesize = header->typesize; context->sourcesize = header->nbytes; context->blocksize = header->blocksize; context->blosc2_flags = header->blosc2_flags; context->compcode = header->flags >> 5; if (context->compcode == BLOSC_UDCODEC_FORMAT) { context->compcode = header->udcompcode; } blosc2_calculate_blocks(context); bool is_lazy = false; if ((context->header_flags & BLOSC_DOSHUFFLE) && (context->header_flags & BLOSC_DOBITSHUFFLE)) { /* Extended header */ context->header_overhead = BLOSC_EXTENDED_HEADER_LENGTH; memcpy(context->filters, header->filter_codes, BLOSC2_MAX_FILTERS); memcpy(context->filters_meta, header->filter_meta, BLOSC2_MAX_FILTERS); context->compcode_meta = header->compcode_meta; context->filter_flags = filters_to_flags(header->filter_codes); context->special_type = (header->blosc2_flags >> 4) & BLOSC2_SPECIAL_MASK; is_lazy = (context->blosc2_flags & 0x08u); } else { context->header_overhead = BLOSC_MIN_HEADER_LENGTH; context->filter_flags = get_filter_flags(context->header_flags, context->typesize); flags_to_filters(context->header_flags, context->filters); } // Some checks for malformed headers if (!is_lazy && header->cbytes > context->srcsize) { return BLOSC2_ERROR_INVALID_HEADER; } return 0; } static int blosc2_intialize_header_from_context(blosc2_context* context, blosc_header* header, bool extended_header) { memset(header, 0, sizeof(blosc_header)); header->version = BLOSC2_VERSION_FORMAT; header->versionlz = compcode_to_compversion(context->compcode); header->flags = context->header_flags; header->typesize = (uint8_t)context->typesize; header->nbytes = (int32_t)context->sourcesize; header->blocksize = (int32_t)context->blocksize; int little_endian = is_little_endian(); if (!little_endian) { header->nbytes = bswap32_(header->nbytes); header->blocksize = bswap32_(header->blocksize); // cbytes written after compression } if (extended_header) { /* Store filter pipeline info at the end of the header */ for (int i = 0; i < BLOSC2_MAX_FILTERS; i++) { header->filter_codes[i] = context->filters[i]; header->filter_meta[i] = context->filters_meta[i]; } header->udcompcode = context->compcode; header->compcode_meta = context->compcode_meta; if (!little_endian) { header->blosc2_flags |= BLOSC2_BIGENDIAN; } if (context->use_dict) { header->blosc2_flags |= BLOSC2_USEDICT; } if (context->blosc2_flags & BLOSC2_INSTR_CODEC) { header->blosc2_flags |= BLOSC2_INSTR_CODEC; } } return 0; } uint8_t* pipeline_forward(struct thread_context* thread_context, const int32_t bsize, const uint8_t* src, const int32_t offset, uint8_t* dest, uint8_t* tmp, uint8_t* tmp2) { blosc2_context* context = thread_context->parent_context; uint8_t* _src = (uint8_t*)src + offset; uint8_t* _tmp = tmp; uint8_t* _dest = dest; int32_t typesize = context->typesize; uint8_t* filters = context->filters; uint8_t* filters_meta = context->filters_meta; bool memcpyed = context->header_flags & (uint8_t)BLOSC_MEMCPYED; /* Prefilter function */ if (context->prefilter != NULL) { /* Set unwritten values to zero */ memset(_dest, 0, bsize); // Create new prefilter parameters for this block (must be private for each thread) blosc2_prefilter_params preparams; memcpy(&preparams, context->preparams, sizeof(preparams)); preparams.input = _src; preparams.output = _dest; preparams.output_size = bsize; preparams.output_typesize = typesize; preparams.output_offset = offset; preparams.nblock = offset / context->blocksize; preparams.nchunk = context->schunk != NULL ? context->schunk->current_nchunk : -1; preparams.tid = thread_context->tid; preparams.ttmp = thread_context->tmp; preparams.ttmp_nbytes = thread_context->tmp_nbytes; preparams.ctx = context; if (context->prefilter(&preparams) != 0) { BLOSC_TRACE_ERROR("Execution of prefilter function failed"); return NULL; } if (memcpyed) { // No more filters are required return _dest; } // Cycle buffers _src = _dest; _dest = _tmp; _tmp = _src; } /* Process the filter pipeline */ for (int i = 0; i < BLOSC2_MAX_FILTERS; i++) { int rc = BLOSC2_ERROR_SUCCESS; if (filters[i] <= BLOSC2_DEFINED_FILTERS_STOP) { switch (filters[i]) { case BLOSC_SHUFFLE: for (int j = 0; j <= filters_meta[i]; j++) { shuffle(typesize, bsize, _src, _dest); // Cycle filters when required if (j < filters_meta[i]) { _src = _dest; _dest = _tmp; _tmp = _src; } } break; case BLOSC_BITSHUFFLE: if (bitshuffle(typesize, bsize, _src, _dest, tmp2) < 0) { return NULL; } break; case BLOSC_DELTA: delta_encoder(src, offset, bsize, typesize, _src, _dest); break; case BLOSC_TRUNC_PREC: if (truncate_precision(filters_meta[i], typesize, bsize, _src, _dest) < 0) { return NULL; } break; default: if (filters[i] != BLOSC_NOFILTER) { BLOSC_TRACE_ERROR("Filter %d not handled during compression\n", filters[i]); return NULL; } } } else { // Look for the filters_meta in user filters and run it for (uint64_t j = 0; j < g_nfilters; ++j) { if (g_filters[j].id == filters[i]) { if (g_filters[j].forward != NULL) { blosc2_cparams cparams; blosc2_ctx_get_cparams(context, &cparams); rc = g_filters[j].forward(_src, _dest, bsize, filters_meta[i], &cparams, g_filters[j].id); } else { BLOSC_TRACE_ERROR("Forward function is NULL"); return NULL; } if (rc != BLOSC2_ERROR_SUCCESS) { BLOSC_TRACE_ERROR("User-defined filter %d failed during compression\n", filters[i]); return NULL; } goto urfiltersuccess; } } BLOSC_TRACE_ERROR("User-defined filter %d not found during compression\n", filters[i]); return NULL; urfiltersuccess:; } // Cycle buffers when required if (filters[i] != BLOSC_NOFILTER) { _src = _dest; _dest = _tmp; _tmp = _src; } } return _src; } // Optimized version for detecting runs. It compares 8 bytes values wherever possible. static bool get_run(const uint8_t* ip, const uint8_t* ip_bound) { uint8_t x = *ip; int64_t value, value2; /* Broadcast the value for every byte in a 64-bit register */ memset(&value, x, 8); while (ip < (ip_bound - 8)) { #if defined(BLOSC_STRICT_ALIGN) memcpy(&value2, ip, 8); #else value2 = *(int64_t*)ip; #endif if (value != value2) { // Values differ. We don't have a run. return false; } else { ip += 8; } } /* Look into the remainder */ while ((ip < ip_bound) && (*ip == x)) ip++; return ip == ip_bound ? true : false; } /* Shuffle & compress a single block */ static int blosc_c(struct thread_context* thread_context, int32_t bsize, int32_t leftoverblock, int32_t ntbytes, int32_t destsize, const uint8_t* src, const int32_t offset, uint8_t* dest, uint8_t* tmp, uint8_t* tmp2) { blosc2_context* context = thread_context->parent_context; int dont_split = (context->header_flags & 0x10) >> 4; int dict_training = context->use_dict && context->dict_cdict == NULL; int32_t j, neblock, nstreams; int32_t cbytes; /* number of compressed bytes in split */ int32_t ctbytes = 0; /* number of compressed bytes in block */ int32_t maxout; int32_t typesize = context->typesize; const char* compname; int accel; const uint8_t* _src; uint8_t *_tmp = tmp, *_tmp2 = tmp2; uint8_t *_tmp3 = thread_context->tmp4; int last_filter_index = last_filter(context->filters, 'c'); bool memcpyed = context->header_flags & (uint8_t)BLOSC_MEMCPYED; bool instr_codec = context->blosc2_flags & BLOSC2_INSTR_CODEC; blosc_timestamp_t last, current; float filter_time = 0.f; if (instr_codec) { blosc_set_timestamp(&last); } // See whether we have a run here if (last_filter_index >= 0 || context->prefilter != NULL) { /* Apply the filter pipeline just for the prefilter */ if (memcpyed && context->prefilter != NULL) { // We only need the prefilter output _src = pipeline_forward(thread_context, bsize, src, offset, dest, _tmp2, _tmp3); if (_src == NULL) { return BLOSC2_ERROR_FILTER_PIPELINE; } return bsize; } /* Apply regular filter pipeline */ _src = pipeline_forward(thread_context, bsize, src, offset, _tmp, _tmp2, _tmp3); if (_src == NULL) { return BLOSC2_ERROR_FILTER_PIPELINE; } } else { _src = src + offset; } if (instr_codec) { blosc_set_timestamp(¤t); filter_time = (float) blosc_elapsed_secs(last, current); last = current; } assert(context->clevel > 0); /* Calculate acceleration for different compressors */ accel = get_accel(context); /* The number of compressed data streams for this block */ if (!dont_split && !leftoverblock && !dict_training) { nstreams = (int32_t)typesize; } else { nstreams = 1; } neblock = bsize / nstreams; for (j = 0; j < nstreams; j++) { if (instr_codec) { blosc_set_timestamp(&last); } if (!dict_training) { dest += sizeof(int32_t); ntbytes += sizeof(int32_t); ctbytes += sizeof(int32_t); const uint8_t *ip = (uint8_t *) _src + j * neblock; const uint8_t *ipbound = (uint8_t *) _src + (j + 1) * neblock; if (context->header_overhead == BLOSC_EXTENDED_HEADER_LENGTH && get_run(ip, ipbound)) { // A run int32_t value = _src[j * neblock]; if (ntbytes > destsize) { return 0; /* Non-compressible data */ } if (instr_codec) { blosc_set_timestamp(¤t); int32_t instr_size = sizeof(blosc2_instr); ntbytes += instr_size; ctbytes += instr_size; if (ntbytes > destsize) { return 0; /* Non-compressible data */ } _sw32(dest - 4, instr_size); blosc2_instr *desti = (blosc2_instr *)dest; memset(desti, 0, sizeof(blosc2_instr)); // Special values have an overhead of about 1 int32 int32_t ssize = value == 0 ? sizeof(int32_t) : sizeof(int32_t) + 1; desti->cratio = (float) neblock / (float) ssize; float ctime = (float) blosc_elapsed_secs(last, current); desti->cspeed = (float) neblock / ctime; desti->filter_speed = (float) neblock / filter_time; desti->flags[0] = 1; // mark a runlen dest += instr_size; continue; } // Encode the repeated byte in the first (LSB) byte of the length of the split. _sw32(dest - 4, -value); // write the value in two's complement if (value > 0) { // Mark encoding as a run-length (== 0 is always a 0's run) ntbytes += 1; ctbytes += 1; if (ntbytes > destsize) { return 0; /* Non-compressible data */ } // Set MSB bit (sign) to 1 (not really necessary here, but for demonstration purposes) // dest[-1] |= 0x80; dest[0] = 0x1; // set run-length bit (0) in token dest += 1; } continue; } } maxout = neblock; if (ntbytes + maxout > destsize) { /* avoid buffer * overrun */ maxout = destsize - ntbytes; if (maxout <= 0) { return 0; /* non-compressible block */ } } if (dict_training) { // We are in the build dict state, so don't compress // TODO: copy only a percentage for sampling memcpy(dest, _src + j * neblock, (unsigned int)neblock); cbytes = (int32_t)neblock; } else if (context->compcode == BLOSC_BLOSCLZ) { cbytes = blosclz_compress(context->clevel, _src + j * neblock, (int)neblock, dest, maxout, context); } else if (context->compcode == BLOSC_LZ4) { void *hash_table = NULL; #ifdef HAVE_IPP hash_table = (void*)thread_context->lz4_hash_table; #endif cbytes = lz4_wrap_compress((char*)_src + j * neblock, (size_t)neblock, (char*)dest, (size_t)maxout, accel, hash_table); } else if (context->compcode == BLOSC_LZ4HC) { cbytes = lz4hc_wrap_compress((char*)_src + j * neblock, (size_t)neblock, (char*)dest, (size_t)maxout, context->clevel); } #if defined(HAVE_ZLIB) else if (context->compcode == BLOSC_ZLIB) { cbytes = zlib_wrap_compress((char*)_src + j * neblock, (size_t)neblock, (char*)dest, (size_t)maxout, context->clevel); } #endif /* HAVE_ZLIB */ #if defined(HAVE_ZSTD) else if (context->compcode == BLOSC_ZSTD) { cbytes = zstd_wrap_compress(thread_context, (char*)_src + j * neblock, (size_t)neblock, (char*)dest, (size_t)maxout, context->clevel); } #endif /* HAVE_ZSTD */ else if (context->compcode > BLOSC2_DEFINED_CODECS_STOP) { for (int i = 0; i < g_ncodecs; ++i) { if (g_codecs[i].compcode == context->compcode) { blosc2_cparams cparams; blosc2_ctx_get_cparams(context, &cparams); cbytes = g_codecs[i].encoder(_src + j * neblock, neblock, dest, maxout, context->compcode_meta, &cparams, context->src); goto urcodecsuccess; } } BLOSC_TRACE_ERROR("User-defined compressor codec %d not found during compression", context->compcode); return BLOSC2_ERROR_CODEC_SUPPORT; urcodecsuccess: ; } else { blosc2_compcode_to_compname(context->compcode, &compname); BLOSC_TRACE_ERROR("Blosc has not been compiled with '%s' compression support." "Please use one having it.", compname); return BLOSC2_ERROR_CODEC_SUPPORT; } if (cbytes > maxout) { /* Buffer overrun caused by compression (should never happen) */ return BLOSC2_ERROR_WRITE_BUFFER; } if (cbytes < 0) { /* cbytes should never be negative */ return BLOSC2_ERROR_DATA; } if (cbytes == 0) { // When cbytes is 0, the compressor has not been able to compress anything cbytes = neblock; } if (instr_codec) { blosc_set_timestamp(¤t); int32_t instr_size = sizeof(blosc2_instr); ntbytes += instr_size; ctbytes += instr_size; if (ntbytes > destsize) { return 0; /* Non-compressible data */ } _sw32(dest - 4, instr_size); float ctime = (float)blosc_elapsed_secs(last, current); blosc2_instr *desti = (blosc2_instr *)dest; memset(desti, 0, sizeof(blosc2_instr)); // cratio is computed having into account 1 additional int (csize) desti->cratio = (float)neblock / (float)(cbytes + sizeof(int32_t)); desti->cspeed = (float)neblock / ctime; desti->filter_speed = (float) neblock / filter_time; dest += instr_size; continue; } if (!dict_training) { if (cbytes == neblock) { /* The compressor has been unable to compress data at all. */ /* Before doing the copy, check that we are not running into a buffer overflow. */ if ((ntbytes + neblock) > destsize) { return 0; /* Non-compressible data */ } memcpy(dest, _src + j * neblock, (unsigned int)neblock); cbytes = neblock; } _sw32(dest - 4, cbytes); } dest += cbytes; ntbytes += cbytes; ctbytes += cbytes; } /* Closes j < nstreams */ //printf("c%d", ctbytes); return ctbytes; } /* Process the filter pipeline (decompression mode) */ int pipeline_backward(struct thread_context* thread_context, const int32_t bsize, uint8_t* dest, const int32_t offset, uint8_t* src, uint8_t* tmp, uint8_t* tmp2, int last_filter_index, int32_t nblock) { blosc2_context* context = thread_context->parent_context; int32_t typesize = context->typesize; uint8_t* filters = context->filters; uint8_t* filters_meta = context->filters_meta; uint8_t* _src = src; uint8_t* _dest = tmp; uint8_t* _tmp = tmp2; int errcode = 0; for (int i = BLOSC2_MAX_FILTERS - 1; i >= 0; i--) { // Delta filter requires the whole chunk ready int last_copy_filter = (last_filter_index == i) || (next_filter(filters, i, 'd') == BLOSC_DELTA); if (last_copy_filter && context->postfilter == NULL) { _dest = dest + offset; } int rc = BLOSC2_ERROR_SUCCESS; if (filters[i] <= BLOSC2_DEFINED_FILTERS_STOP) { switch (filters[i]) { case BLOSC_SHUFFLE: for (int j = 0; j <= filters_meta[i]; j++) { unshuffle(typesize, bsize, _src, _dest); // Cycle filters when required if (j < filters_meta[i]) { _src = _dest; _dest = _tmp; _tmp = _src; } // Check whether we have to copy the intermediate _dest buffer to final destination if (last_copy_filter && (filters_meta[i] % 2) == 1 && j == filters_meta[i]) { memcpy(dest + offset, _dest, (unsigned int) bsize); } } break; case BLOSC_BITSHUFFLE: if (bitunshuffle(typesize, bsize, _src, _dest, _tmp, context->src[BLOSC2_CHUNK_VERSION]) < 0) { return BLOSC2_ERROR_FILTER_PIPELINE; } break; case BLOSC_DELTA: if (context->nthreads == 1) { /* Serial mode */ delta_decoder(dest, offset, bsize, typesize, _dest); } else { /* Force the thread in charge of the block 0 to go first */ pthread_mutex_lock(&context->delta_mutex); if (context->dref_not_init) { if (offset != 0) { pthread_cond_wait(&context->delta_cv, &context->delta_mutex); } else { delta_decoder(dest, offset, bsize, typesize, _dest); context->dref_not_init = 0; pthread_cond_broadcast(&context->delta_cv); } } pthread_mutex_unlock(&context->delta_mutex); if (offset != 0) { delta_decoder(dest, offset, bsize, typesize, _dest); } } break; case BLOSC_TRUNC_PREC: // TRUNC_PREC filter does not need to be undone break; default: if (filters[i] != BLOSC_NOFILTER) { BLOSC_TRACE_ERROR("Filter %d not handled during decompression.", filters[i]); errcode = -1; } } } else { // Look for the filters_meta in user filters and run it for (uint64_t j = 0; j < g_nfilters; ++j) { if (g_filters[j].id == filters[i]) { if (g_filters[j].backward != NULL) { blosc2_dparams dparams; blosc2_ctx_get_dparams(context, &dparams); rc = g_filters[j].backward(_src, _dest, bsize, filters_meta[i], &dparams, g_filters[j].id); } else { BLOSC_TRACE_ERROR("Backward function is NULL"); return BLOSC2_ERROR_FILTER_PIPELINE; } if (rc != BLOSC2_ERROR_SUCCESS) { BLOSC_TRACE_ERROR("User-defined filter %d failed during decompression.", filters[i]); return rc; } goto urfiltersuccess; } } BLOSC_TRACE_ERROR("User-defined filter %d not found during decompression.", filters[i]); return BLOSC2_ERROR_FILTER_PIPELINE; urfiltersuccess:; } // Cycle buffers when required if ((filters[i] != BLOSC_NOFILTER) && (filters[i] != BLOSC_TRUNC_PREC)) { _src = _dest; _dest = _tmp; _tmp = _src; } if (last_filter_index == i) { break; } } /* Postfilter function */ if (context->postfilter != NULL) { // Create new postfilter parameters for this block (must be private for each thread) blosc2_postfilter_params postparams; memcpy(&postparams, context->postparams, sizeof(postparams)); postparams.input = _src; postparams.output = dest + offset; postparams.size = bsize; postparams.typesize = typesize; postparams.offset = nblock * context->blocksize; postparams.nchunk = context->schunk != NULL ? context->schunk->current_nchunk : -1; postparams.nblock = nblock; postparams.tid = thread_context->tid; postparams.ttmp = thread_context->tmp; postparams.ttmp_nbytes = thread_context->tmp_nbytes; postparams.ctx = context; if (context->postfilter(&postparams) != 0) { BLOSC_TRACE_ERROR("Execution of postfilter function failed"); return BLOSC2_ERROR_POSTFILTER; } } return errcode; } static int32_t set_nans(int32_t typesize, uint8_t* dest, int32_t destsize) { // destsize can only be a multiple of typesize if (destsize % typesize != 0) { return -1; } int32_t nitems = destsize / typesize; if (nitems == 0) { return 0; } if (typesize == 4) { float* dest_ = (float*)dest; float val = nanf(""); for (int i = 0; i < nitems; i++) { dest_[i] = val; } return nitems; } else if (typesize == 8) { double* dest_ = (double*)dest; double val = nan(""); for (int i = 0; i < nitems; i++) { dest_[i] = val; } return nitems; } BLOSC_TRACE_ERROR("Unsupported typesize for NaN"); return BLOSC2_ERROR_DATA; } static int32_t set_values(int32_t typesize, const uint8_t* src, uint8_t* dest, int32_t destsize) { // destsize can only be a multiple of typesize int64_t val8; int64_t* dest8; int32_t val4; int32_t* dest4; int16_t val2; int16_t* dest2; int8_t val1; int8_t* dest1; if (destsize % typesize != 0) { return -1; } int32_t nitems = destsize / typesize; if (nitems == 0) { return 0; } switch (typesize) { case 8: val8 = ((int64_t*)(src + BLOSC_EXTENDED_HEADER_LENGTH))[0]; dest8 = (int64_t*)dest; for (int i = 0; i < nitems; i++) { dest8[i] = val8; } break; case 4: val4 = ((int32_t*)(src + BLOSC_EXTENDED_HEADER_LENGTH))[0]; dest4 = (int32_t*)dest; for (int i = 0; i < nitems; i++) { dest4[i] = val4; } break; case 2: val2 = ((int16_t*)(src + BLOSC_EXTENDED_HEADER_LENGTH))[0]; dest2 = (int16_t*)dest; for (int i = 0; i < nitems; i++) { dest2[i] = val2; } break; case 1: val1 = ((int8_t*)(src + BLOSC_EXTENDED_HEADER_LENGTH))[0]; dest1 = (int8_t*)dest; for (int i = 0; i < nitems; i++) { dest1[i] = val1; } break; default: for (int i = 0; i < nitems; i++) { memcpy(dest + i * typesize, src + BLOSC_EXTENDED_HEADER_LENGTH, typesize); } } return nitems; } /* Decompress & unshuffle a single block */ static int blosc_d( struct thread_context* thread_context, int32_t bsize, int32_t leftoverblock, bool memcpyed, const uint8_t* src, int32_t srcsize, int32_t src_offset, int32_t nblock, uint8_t* dest, int32_t dest_offset, uint8_t* tmp, uint8_t* tmp2) { blosc2_context* context = thread_context->parent_context; uint8_t* filters = context->filters; uint8_t *tmp3 = thread_context->tmp4; int32_t compformat = (context->header_flags & (uint8_t)0xe0) >> 5u; int dont_split = (context->header_flags & 0x10) >> 4; int32_t chunk_nbytes; int32_t chunk_cbytes; int nstreams; int32_t neblock; int32_t nbytes; /* number of decompressed bytes in split */ int32_t cbytes; /* number of compressed bytes in split */ // int32_t ctbytes = 0; /* number of compressed bytes in block */ int32_t ntbytes = 0; /* number of uncompressed bytes in block */ uint8_t* _dest; int32_t typesize = context->typesize; bool instr_codec = context->blosc2_flags & BLOSC2_INSTR_CODEC; const char* compname; int rc; if (context->block_maskout != NULL && context->block_maskout[nblock]) { // Do not decompress, but act as if we successfully decompressed everything return bsize; } rc = blosc2_cbuffer_sizes(src, &chunk_nbytes, &chunk_cbytes, NULL); if (rc < 0) { return rc; } // In some situations (lazychunks) the context can arrive uninitialized // (but BITSHUFFLE needs it for accessing the format of the chunk) if (context->src == NULL) { context->src = src; } // Chunks with special values cannot be lazy bool is_lazy = ((context->header_overhead == BLOSC_EXTENDED_HEADER_LENGTH) && (context->blosc2_flags & 0x08u) && !context->special_type); if (is_lazy) { // The chunk is on disk, so just lazily load the block if (context->schunk == NULL) { BLOSC_TRACE_ERROR("Lazy chunk needs an associated super-chunk."); return BLOSC2_ERROR_INVALID_PARAM; } if (context->schunk->frame == NULL) { BLOSC_TRACE_ERROR("Lazy chunk needs an associated frame."); return BLOSC2_ERROR_INVALID_PARAM; } blosc2_frame_s* frame = (blosc2_frame_s*)context->schunk->frame; char* urlpath = frame->urlpath; size_t trailer_offset = BLOSC_EXTENDED_HEADER_LENGTH + context->nblocks * sizeof(int32_t); int32_t nchunk; int64_t chunk_offset; // The nchunk and the offset of the current chunk are in the trailer nchunk = *(int32_t*)(src + trailer_offset); chunk_offset = *(int64_t*)(src + trailer_offset + sizeof(int32_t)); // Get the csize of the nblock int32_t *block_csizes = (int32_t *)(src + trailer_offset + sizeof(int32_t) + sizeof(int64_t)); int32_t block_csize = block_csizes[nblock]; // Read the lazy block on disk void* fp = NULL; blosc2_io_cb *io_cb = blosc2_get_io_cb(context->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } if (frame->sframe) { // The chunk is not in the frame char* chunkpath = malloc(strlen(frame->urlpath) + 1 + 8 + strlen(".chunk") + 1); BLOSC_ERROR_NULL(chunkpath, BLOSC2_ERROR_MEMORY_ALLOC); sprintf(chunkpath, "%s/%08X.chunk", frame->urlpath, nchunk); fp = io_cb->open(chunkpath, "rb", context->schunk->storage->io->params); BLOSC_ERROR_NULL(fp, BLOSC2_ERROR_FILE_OPEN); free(chunkpath); // The offset of the block is src_offset io_cb->seek(fp, src_offset, SEEK_SET); } else { fp = io_cb->open(urlpath, "rb", context->schunk->storage->io->params); BLOSC_ERROR_NULL(fp, BLOSC2_ERROR_FILE_OPEN); // The offset of the block is src_offset io_cb->seek(fp, frame->file_offset + chunk_offset + src_offset, SEEK_SET); } // We can make use of tmp3 because it will be used after src is not needed anymore int64_t rbytes = io_cb->read(tmp3, 1, block_csize, fp); io_cb->close(fp); if ((int32_t)rbytes != block_csize) { BLOSC_TRACE_ERROR("Cannot read the (lazy) block out of the fileframe."); return BLOSC2_ERROR_READ_BUFFER; } src = tmp3; src_offset = 0; srcsize = block_csize; } // If the chunk is memcpyed, we just have to copy the block to dest and return if (memcpyed) { int bsize_ = leftoverblock ? chunk_nbytes % context->blocksize : bsize; if (!context->special_type) { if (chunk_nbytes + context->header_overhead != chunk_cbytes) { return BLOSC2_ERROR_WRITE_BUFFER; } if (chunk_cbytes < context->header_overhead + (nblock * context->blocksize) + bsize_) { /* Not enough input to copy block */ return BLOSC2_ERROR_READ_BUFFER; } } if (!is_lazy) { src += context->header_overhead + nblock * context->blocksize; } _dest = dest + dest_offset; if (context->postfilter != NULL) { // We are making use of a postfilter, so use a temp for destination _dest = tmp; } rc = 0; switch (context->special_type) { case BLOSC2_SPECIAL_VALUE: // All repeated values rc = set_values(context->typesize, context->src, _dest, bsize_); if (rc < 0) { BLOSC_TRACE_ERROR("set_values failed"); return BLOSC2_ERROR_DATA; } break; case BLOSC2_SPECIAL_NAN: rc = set_nans(context->typesize, _dest, bsize_); if (rc < 0) { BLOSC_TRACE_ERROR("set_nans failed"); return BLOSC2_ERROR_DATA; } break; case BLOSC2_SPECIAL_ZERO: memset(_dest, 0, bsize_); break; case BLOSC2_SPECIAL_UNINIT: // We do nothing here break; default: memcpy(_dest, src, bsize_); } if (context->postfilter != NULL) { // Create new postfilter parameters for this block (must be private for each thread) blosc2_postfilter_params postparams; memcpy(&postparams, context->postparams, sizeof(postparams)); postparams.input = tmp; postparams.output = dest + dest_offset; postparams.size = bsize; postparams.typesize = typesize; postparams.offset = nblock * context->blocksize; postparams.nchunk = context->schunk != NULL ? context->schunk->current_nchunk : -1; postparams.nblock = nblock; postparams.tid = thread_context->tid; postparams.ttmp = thread_context->tmp; postparams.ttmp_nbytes = thread_context->tmp_nbytes; postparams.ctx = context; // Execute the postfilter (the processed block will be copied to dest) if (context->postfilter(&postparams) != 0) { BLOSC_TRACE_ERROR("Execution of postfilter function failed"); return BLOSC2_ERROR_POSTFILTER; } } thread_context->zfp_cell_nitems = 0; return bsize_; } if (!is_lazy && (src_offset <= 0 || src_offset >= srcsize)) { /* Invalid block src offset encountered */ return BLOSC2_ERROR_DATA; } src += src_offset; srcsize -= src_offset; int last_filter_index = last_filter(filters, 'd'); if (instr_codec) { // If instrumented, we don't want to run the filters _dest = dest + dest_offset; } else if (((last_filter_index >= 0) && (next_filter(filters, BLOSC2_MAX_FILTERS, 'd') != BLOSC_DELTA)) || context->postfilter != NULL) { // We are making use of some filter, so use a temp for destination _dest = tmp; } else { // If no filters, or only DELTA in pipeline _dest = dest + dest_offset; } /* The number of compressed data streams for this block */ if (!dont_split && !leftoverblock && !context->use_dict) { // We don't want to split when in a training dict state nstreams = (int32_t)typesize; } else { nstreams = 1; } neblock = bsize / nstreams; if (neblock == 0) { /* Not enough space to output bytes */ return -1; } for (int j = 0; j < nstreams; j++) { if (srcsize < (signed)sizeof(int32_t)) { /* Not enough input to read compressed size */ return BLOSC2_ERROR_READ_BUFFER; } srcsize -= sizeof(int32_t); cbytes = sw32_(src); /* amount of compressed bytes */ if (cbytes > 0) { if (srcsize < cbytes) { /* Not enough input to read compressed bytes */ return BLOSC2_ERROR_READ_BUFFER; } srcsize -= cbytes; } src += sizeof(int32_t); // ctbytes += (signed)sizeof(int32_t); /* Uncompress */ if (cbytes == 0) { // A run of 0's memset(_dest, 0, (unsigned int)neblock); nbytes = neblock; } else if (cbytes < 0) { // A negative number means some encoding depending on the token that comes next uint8_t token; if (srcsize < (signed)sizeof(uint8_t)) { // Not enough input to read token */ return BLOSC2_ERROR_READ_BUFFER; } srcsize -= sizeof(uint8_t); token = src[0]; src += 1; // ctbytes += 1; if (token & 0x1) { // A run of bytes that are different than 0 if (cbytes < -255) { // Runs can only encode a byte return BLOSC2_ERROR_RUN_LENGTH; } uint8_t value = -cbytes; memset(_dest, value, (unsigned int)neblock); } else { BLOSC_TRACE_ERROR("Invalid or unsupported compressed stream token value - %d", token); return BLOSC2_ERROR_RUN_LENGTH; } nbytes = neblock; cbytes = 0; // everything is encoded in the cbytes token } else if (cbytes == neblock) { memcpy(_dest, src, (unsigned int)neblock); nbytes = (int32_t)neblock; } else { if (compformat == BLOSC_BLOSCLZ_FORMAT) { nbytes = blosclz_decompress(src, cbytes, _dest, (int)neblock); } else if (compformat == BLOSC_LZ4_FORMAT) { nbytes = lz4_wrap_decompress((char*)src, (size_t)cbytes, (char*)_dest, (size_t)neblock); } #if defined(HAVE_ZLIB) else if (compformat == BLOSC_ZLIB_FORMAT) { nbytes = zlib_wrap_decompress((char*)src, (size_t)cbytes, (char*)_dest, (size_t)neblock); } #endif /* HAVE_ZLIB */ #if defined(HAVE_ZSTD) else if (compformat == BLOSC_ZSTD_FORMAT) { nbytes = zstd_wrap_decompress(thread_context, (char*)src, (size_t)cbytes, (char*)_dest, (size_t)neblock); } #endif /* HAVE_ZSTD */ else if (compformat == BLOSC_UDCODEC_FORMAT) { bool getcell = false; #if defined(HAVE_PLUGINS) if ((context->compcode == BLOSC_CODEC_ZFP_FIXED_RATE) && (thread_context->zfp_cell_nitems > 0)) { nbytes = zfp_getcell(thread_context, src, cbytes, _dest, neblock); if (nbytes < 0) { return BLOSC2_ERROR_DATA; } if (nbytes == thread_context->zfp_cell_nitems * typesize) { getcell = true; } } #endif /* HAVE_PLUGINS */ if (!getcell) { thread_context->zfp_cell_nitems = 0; for (int i = 0; i < g_ncodecs; ++i) { if (g_codecs[i].compcode == context->compcode) { blosc2_dparams dparams; blosc2_ctx_get_dparams(context, &dparams); nbytes = g_codecs[i].decoder(src, cbytes, _dest, neblock, context->compcode_meta, &dparams, context->src); goto urcodecsuccess; } } BLOSC_TRACE_ERROR("User-defined compressor codec %d not found during decompression", context->compcode); return BLOSC2_ERROR_CODEC_SUPPORT; } urcodecsuccess: ; } else { compname = clibcode_to_clibname(compformat); BLOSC_TRACE_ERROR( "Blosc has not been compiled with decompression " "support for '%s' format. " "Please recompile for adding this support.", compname); return BLOSC2_ERROR_CODEC_SUPPORT; } /* Check that decompressed bytes number is correct */ if ((nbytes != neblock) && (thread_context->zfp_cell_nitems == 0)) { return BLOSC2_ERROR_DATA; } } src += cbytes; // ctbytes += cbytes; _dest += nbytes; ntbytes += nbytes; } /* Closes j < nstreams */ if (!instr_codec) { if (last_filter_index >= 0 || context->postfilter != NULL) { /* Apply regular filter pipeline */ int errcode = pipeline_backward(thread_context, bsize, dest, dest_offset, tmp, tmp2, tmp3, last_filter_index, nblock); if (errcode < 0) return errcode; } } /* Return the number of uncompressed bytes */ return (int)ntbytes; } /* Serial version for compression/decompression */ static int serial_blosc(struct thread_context* thread_context) { blosc2_context* context = thread_context->parent_context; int32_t j, bsize, leftoverblock; int32_t cbytes; int32_t ntbytes = (int32_t)context->output_bytes; int32_t* bstarts = context->bstarts; uint8_t* tmp = thread_context->tmp; uint8_t* tmp2 = thread_context->tmp2; int dict_training = context->use_dict && (context->dict_cdict == NULL); bool memcpyed = context->header_flags & (uint8_t)BLOSC_MEMCPYED; if (!context->do_compress && context->special_type) { // Fake a runlen as if it was a memcpyed chunk memcpyed = true; } for (j = 0; j < context->nblocks; j++) { if (context->do_compress && !memcpyed && !dict_training) { _sw32(bstarts + j, ntbytes); } bsize = context->blocksize; leftoverblock = 0; if ((j == context->nblocks - 1) && (context->leftover > 0)) { bsize = context->leftover; leftoverblock = 1; } if (context->do_compress) { if (memcpyed && !context->prefilter) { /* We want to memcpy only */ memcpy(context->dest + context->header_overhead + j * context->blocksize, context->src + j * context->blocksize, (unsigned int)bsize); cbytes = (int32_t)bsize; } else { /* Regular compression */ cbytes = blosc_c(thread_context, bsize, leftoverblock, ntbytes, context->destsize, context->src, j * context->blocksize, context->dest + ntbytes, tmp, tmp2); if (cbytes == 0) { ntbytes = 0; /* incompressible data */ break; } } } else { /* Regular decompression */ // If memcpyed we don't have a bstarts section (because it is not needed) int32_t src_offset = memcpyed ? context->header_overhead + j * context->blocksize : sw32_(bstarts + j); cbytes = blosc_d(thread_context, bsize, leftoverblock, memcpyed, context->src, context->srcsize, src_offset, j, context->dest, j * context->blocksize, tmp, tmp2); } if (cbytes < 0) { ntbytes = cbytes; /* error in blosc_c or blosc_d */ break; } ntbytes += cbytes; } return ntbytes; } static void t_blosc_do_job(void *ctxt); /* Threaded version for compression/decompression */ static int parallel_blosc(blosc2_context* context) { #ifdef BLOSC_POSIX_BARRIERS int rc; #endif /* Set sentinels */ context->thread_giveup_code = 1; context->thread_nblock = -1; if (threads_callback) { threads_callback(threads_callback_data, t_blosc_do_job, context->nthreads, sizeof(struct thread_context), (void*) context->thread_contexts); } else { /* Synchronization point for all threads (wait for initialization) */ WAIT_INIT(-1, context); /* Synchronization point for all threads (wait for finalization) */ WAIT_FINISH(-1, context); } if (context->thread_giveup_code <= 0) { /* Compression/decompression gave up. Return error code. */ return context->thread_giveup_code; } /* Return the total bytes (de-)compressed in threads */ return (int)context->output_bytes; } /* initialize a thread_context that has already been allocated */ static int init_thread_context(struct thread_context* thread_context, blosc2_context* context, int32_t tid) { int32_t ebsize; thread_context->parent_context = context; thread_context->tid = tid; ebsize = context->blocksize + context->typesize * (signed)sizeof(int32_t); thread_context->tmp_nbytes = (size_t)4 * ebsize; thread_context->tmp = my_malloc(thread_context->tmp_nbytes); BLOSC_ERROR_NULL(thread_context->tmp, BLOSC2_ERROR_MEMORY_ALLOC); thread_context->tmp2 = thread_context->tmp + ebsize; thread_context->tmp3 = thread_context->tmp2 + ebsize; thread_context->tmp4 = thread_context->tmp3 + ebsize; thread_context->tmp_blocksize = context->blocksize; thread_context->zfp_cell_nitems = 0; thread_context->zfp_cell_start = 0; #if defined(HAVE_ZSTD) thread_context->zstd_cctx = NULL; thread_context->zstd_dctx = NULL; #endif /* Create the hash table for LZ4 in case we are using IPP */ #ifdef HAVE_IPP IppStatus status; int inlen = thread_context->tmp_blocksize > 0 ? thread_context->tmp_blocksize : 1 << 16; int hash_size = 0; status = ippsEncodeLZ4HashTableGetSize_8u(&hash_size); if (status != ippStsNoErr) { BLOSC_TRACE_ERROR("Error in ippsEncodeLZ4HashTableGetSize_8u."); } Ipp8u *hash_table = ippsMalloc_8u(hash_size); status = ippsEncodeLZ4HashTableInit_8u(hash_table, inlen); if (status != ippStsNoErr) { BLOSC_TRACE_ERROR("Error in ippsEncodeLZ4HashTableInit_8u."); } thread_context->lz4_hash_table = hash_table; #endif return 0; } static struct thread_context* create_thread_context(blosc2_context* context, int32_t tid) { struct thread_context* thread_context; thread_context = (struct thread_context*)my_malloc(sizeof(struct thread_context)); BLOSC_ERROR_NULL(thread_context, NULL); int rc = init_thread_context(thread_context, context, tid); if (rc < 0) { return NULL; } return thread_context; } /* free members of thread_context, but not thread_context itself */ static void destroy_thread_context(struct thread_context* thread_context) { my_free(thread_context->tmp); #if defined(HAVE_ZSTD) if (thread_context->zstd_cctx != NULL) { ZSTD_freeCCtx(thread_context->zstd_cctx); } if (thread_context->zstd_dctx != NULL) { ZSTD_freeDCtx(thread_context->zstd_dctx); } #endif #ifdef HAVE_IPP if (thread_context->lz4_hash_table != NULL) { ippsFree(thread_context->lz4_hash_table); } #endif } void free_thread_context(struct thread_context* thread_context) { destroy_thread_context(thread_context); my_free(thread_context); } int check_nthreads(blosc2_context* context) { if (context->nthreads <= 0) { BLOSC_TRACE_ERROR("nthreads must be a positive integer."); return BLOSC2_ERROR_INVALID_PARAM; } if (context->new_nthreads != context->nthreads) { if (context->nthreads > 1) { release_threadpool(context); } context->nthreads = context->new_nthreads; } if (context->new_nthreads > 1 && context->threads_started == 0) { init_threadpool(context); } return context->nthreads; } /* Do the compression or decompression of the buffer depending on the global params. */ static int do_job(blosc2_context* context) { int32_t ntbytes; /* Set sentinels */ context->dref_not_init = 1; /* Check whether we need to restart threads */ check_nthreads(context); /* Run the serial version when nthreads is 1 or when the buffers are not larger than blocksize */ if (context->nthreads == 1 || (context->sourcesize / context->blocksize) <= 1) { /* The context for this 'thread' has no been initialized yet */ if (context->serial_context == NULL) { context->serial_context = create_thread_context(context, 0); } else if (context->blocksize != context->serial_context->tmp_blocksize) { free_thread_context(context->serial_context); context->serial_context = create_thread_context(context, 0); } BLOSC_ERROR_NULL(context->serial_context, BLOSC2_ERROR_THREAD_CREATE); ntbytes = serial_blosc(context->serial_context); } else { ntbytes = parallel_blosc(context); } return ntbytes; } static int initialize_context_compression( blosc2_context* context, const void* src, int32_t srcsize, void* dest, int32_t destsize, int clevel, uint8_t const *filters, uint8_t const *filters_meta, int32_t typesize, int compressor, int32_t blocksize, int16_t new_nthreads, int16_t nthreads, int32_t splitmode, blosc2_btune *udbtune, void *btune_config, blosc2_schunk* schunk) { /* Set parameters */ context->do_compress = 1; context->src = (const uint8_t*)src; context->srcsize = srcsize; context->dest = (uint8_t*)dest; context->output_bytes = 0; context->destsize = destsize; context->sourcesize = srcsize; context->typesize = (int32_t)typesize; context->filter_flags = filters_to_flags(filters); for (int i = 0; i < BLOSC2_MAX_FILTERS; i++) { context->filters[i] = filters[i]; context->filters_meta[i] = filters_meta[i]; } context->compcode = compressor; context->nthreads = nthreads; context->new_nthreads = new_nthreads; context->end_threads = 0; context->clevel = clevel; context->schunk = schunk; context->btune = btune_config; context->udbtune = udbtune; context->splitmode = splitmode; /* Tune some compression parameters */ context->blocksize = (int32_t)blocksize; if (context->btune != NULL) { context->udbtune->btune_next_cparams(context); } else { context->udbtune->btune_next_blocksize(context); } char* envvar = getenv("BLOSC_WARN"); int64_t warnlvl = 0; if (envvar != NULL) { warnlvl = strtol(envvar, NULL, 10); } /* Check buffer size limits */ if (srcsize > BLOSC2_MAX_BUFFERSIZE) { if (warnlvl > 0) { BLOSC_TRACE_ERROR("Input buffer size cannot exceed %d bytes.", BLOSC2_MAX_BUFFERSIZE); } return 0; } if (destsize < BLOSC2_MAX_OVERHEAD) { if (warnlvl > 0) { BLOSC_TRACE_ERROR("Output buffer size should be larger than %d bytes.", BLOSC2_MAX_OVERHEAD); } return 0; } /* Compression level */ if (clevel < 0 || clevel > 9) { /* If clevel not in 0..9, print an error */ BLOSC_TRACE_ERROR("`clevel` parameter must be between 0 and 9!."); return BLOSC2_ERROR_CODEC_PARAM; } /* Check typesize limits */ if (context->typesize > BLOSC_MAX_TYPESIZE) { /* If typesize is too large, treat buffer as an 1-byte stream. */ context->typesize = 1; } blosc2_calculate_blocks(context); return 1; } static int initialize_context_decompression(blosc2_context* context, blosc_header* header, const void* src, int32_t srcsize, void* dest, int32_t destsize) { int32_t bstarts_end; context->do_compress = 0; context->src = (const uint8_t*)src; context->srcsize = srcsize; context->dest = (uint8_t*)dest; context->destsize = destsize; context->output_bytes = 0; context->end_threads = 0; int rc = blosc2_initialize_context_from_header(context, header); if (rc < 0) { return rc; } /* Check that we have enough space to decompress */ if (context->sourcesize > (int32_t)context->destsize) { return BLOSC2_ERROR_WRITE_BUFFER; } if (context->block_maskout != NULL && context->block_maskout_nitems != context->nblocks) { BLOSC_TRACE_ERROR("The number of items in block_maskout (%d) must match the number" " of blocks in chunk (%d).", context->block_maskout_nitems, context->nblocks); return BLOSC2_ERROR_DATA; } context->special_type = (header->blosc2_flags >> 4) & BLOSC2_SPECIAL_MASK; if (context->special_type > BLOSC2_SPECIAL_LASTID) { BLOSC_TRACE_ERROR("Unknown special values ID (%d) ", context->special_type); return BLOSC2_ERROR_DATA; } int memcpyed = (context->header_flags & (uint8_t) BLOSC_MEMCPYED); if (memcpyed && (header->cbytes != header->nbytes + context->header_overhead)) { BLOSC_TRACE_ERROR("Wrong header info for this memcpyed chunk"); return BLOSC2_ERROR_DATA; } if ((header->nbytes == 0) && (header->cbytes == context->header_overhead) && !context->special_type) { // A compressed buffer with only a header can only contain a zero-length buffer return 0; } context->bstarts = (int32_t *) (context->src + context->header_overhead); bstarts_end = context->header_overhead; if (!context->special_type && !memcpyed) { /* If chunk is not special or a memcpyed, we do have a bstarts section */ bstarts_end = (int32_t)(context->header_overhead + (context->nblocks * sizeof(int32_t))); } if (srcsize < bstarts_end) { BLOSC_TRACE_ERROR("`bstarts` exceeds length of source buffer."); return BLOSC2_ERROR_READ_BUFFER; } srcsize -= bstarts_end; /* Read optional dictionary if flag set */ if (context->blosc2_flags & BLOSC2_USEDICT) { #if defined(HAVE_ZSTD) context->use_dict = 1; if (context->dict_ddict != NULL) { // Free the existing dictionary (probably from another chunk) ZSTD_freeDDict(context->dict_ddict); } // The trained dictionary is after the bstarts block if (srcsize < (signed)sizeof(int32_t)) { BLOSC_TRACE_ERROR("Not enough space to read size of dictionary."); return BLOSC2_ERROR_READ_BUFFER; } srcsize -= sizeof(int32_t); // Read dictionary size context->dict_size = sw32_(context->src + bstarts_end); if (context->dict_size <= 0 || context->dict_size > BLOSC2_MAXDICTSIZE) { BLOSC_TRACE_ERROR("Dictionary size is smaller than minimum or larger than maximum allowed."); return BLOSC2_ERROR_CODEC_DICT; } if (srcsize < (int32_t)context->dict_size) { BLOSC_TRACE_ERROR("Not enough space to read entire dictionary."); return BLOSC2_ERROR_READ_BUFFER; } srcsize -= context->dict_size; // Read dictionary context->dict_buffer = (void*)(context->src + bstarts_end + sizeof(int32_t)); context->dict_ddict = ZSTD_createDDict(context->dict_buffer, context->dict_size); #endif // HAVE_ZSTD } return 0; } static int write_compression_header(blosc2_context* context, bool extended_header) { blosc_header header; int dont_split; int dict_training = context->use_dict && (context->dict_cdict == NULL); context->header_flags = 0; if (context->clevel == 0) { /* Compression level 0 means buffer to be memcpy'ed */ context->header_flags |= (uint8_t)BLOSC_MEMCPYED; } if (context->sourcesize < BLOSC_MIN_BUFFERSIZE) { /* Buffer is too small. Try memcpy'ing. */ context->header_flags |= (uint8_t)BLOSC_MEMCPYED; } bool memcpyed = context->header_flags & (uint8_t)BLOSC_MEMCPYED; if (extended_header) { /* Indicate that we are building an extended header */ context->header_overhead = BLOSC_EXTENDED_HEADER_LENGTH; context->header_flags |= (BLOSC_DOSHUFFLE | BLOSC_DOBITSHUFFLE); /* Store filter pipeline info at the end of the header */ if (dict_training || memcpyed) { context->bstarts = NULL; context->output_bytes = context->header_overhead; } else { context->bstarts = (int32_t*)(context->dest + context->header_overhead); context->output_bytes = context->header_overhead + (int32_t)sizeof(int32_t) * context->nblocks; } } else { // Regular header context->header_overhead = BLOSC_MIN_HEADER_LENGTH; if (memcpyed) { context->bstarts = NULL; context->output_bytes = context->header_overhead; } else { context->bstarts = (int32_t *) (context->dest + context->header_overhead); context->output_bytes = context->header_overhead + (int32_t)sizeof(int32_t) * context->nblocks; } } // when memcpyed bit is set, there is no point in dealing with others if (!memcpyed) { if (context->filter_flags & BLOSC_DOSHUFFLE) { /* Byte-shuffle is active */ context->header_flags |= BLOSC_DOSHUFFLE; } if (context->filter_flags & BLOSC_DOBITSHUFFLE) { /* Bit-shuffle is active */ context->header_flags |= BLOSC_DOBITSHUFFLE; } if (context->filter_flags & BLOSC_DODELTA) { /* Delta is active */ context->header_flags |= BLOSC_DODELTA; } dont_split = !split_block(context, context->typesize, context->blocksize); /* dont_split is in bit 4 */ context->header_flags |= dont_split << 4; /* codec starts at bit 5 */ uint8_t compformat = compcode_to_compformat(context->compcode); context->header_flags |= compformat << 5; } // Create blosc header and store to dest blosc2_intialize_header_from_context(context, &header, extended_header); memcpy(context->dest, &header, (extended_header) ? BLOSC_EXTENDED_HEADER_LENGTH : BLOSC_MIN_HEADER_LENGTH); return 1; } static int blosc_compress_context(blosc2_context* context) { int ntbytes = 0; blosc_timestamp_t last, current; bool memcpyed = context->header_flags & (uint8_t)BLOSC_MEMCPYED; blosc_set_timestamp(&last); if (!memcpyed) { /* Do the actual compression */ ntbytes = do_job(context); if (ntbytes < 0) { return ntbytes; } if (ntbytes == 0) { // Try out with a memcpy later on (last chance for fitting src buffer in dest). context->header_flags |= (uint8_t)BLOSC_MEMCPYED; memcpyed = true; } } int dont_split = (context->header_flags & 0x10) >> 4; int nstreams = context->nblocks; if (!dont_split) { // When splitting, the number of streams is computed differently if (context->leftover) { nstreams = (context->nblocks - 1) * context->typesize + 1; } else { nstreams *= context->typesize; } } if (memcpyed) { if (context->sourcesize + context->header_overhead > context->destsize) { /* We are exceeding maximum output size */ ntbytes = 0; } else { context->output_bytes = context->header_overhead; ntbytes = do_job(context); if (ntbytes < 0) { return ntbytes; } // Success! update the memcpy bit in header context->dest[BLOSC2_CHUNK_FLAGS] = context->header_flags; // and clear the memcpy bit in context (for next reuse) context->header_flags &= ~(uint8_t)BLOSC_MEMCPYED; } } else { // Check whether we have a run for the whole chunk int start_csizes = context->header_overhead + 4 * context->nblocks; if (ntbytes == (int)(start_csizes + nstreams * sizeof(int32_t))) { // The streams are all zero runs (by construction). Encode it... context->dest[BLOSC2_CHUNK_BLOSC2_FLAGS] |= BLOSC2_SPECIAL_ZERO << 4; // ...and assign the new chunk length ntbytes = context->header_overhead; } } /* Set the number of compressed bytes in header */ _sw32(context->dest + BLOSC2_CHUNK_CBYTES, ntbytes); if (context->blosc2_flags & BLOSC2_INSTR_CODEC) { dont_split = (context->header_flags & 0x10) >> 4; int32_t blocksize = dont_split ? (int32_t)sizeof(blosc2_instr) : (int32_t)sizeof(blosc2_instr) * context->typesize; _sw32(context->dest + BLOSC2_CHUNK_NBYTES, nstreams * (int32_t)sizeof(blosc2_instr)); _sw32(context->dest + BLOSC2_CHUNK_BLOCKSIZE, blocksize); } /* Set the number of bytes in dest buffer (might be useful for btune) */ context->destsize = ntbytes; if (context->btune != NULL) { blosc_set_timestamp(¤t); double ctime = blosc_elapsed_secs(last, current); context->udbtune->btune_update(context, ctime); } return ntbytes; } /* The public secure routine for compression with context. */ int blosc2_compress_ctx(blosc2_context* context, const void* src, int32_t srcsize, void* dest, int32_t destsize) { int error, cbytes; if (context->do_compress != 1) { BLOSC_TRACE_ERROR("Context is not meant for compression. Giving up."); return BLOSC2_ERROR_INVALID_PARAM; } error = initialize_context_compression( context, src, srcsize, dest, destsize, context->clevel, context->filters, context->filters_meta, context->typesize, context->compcode, context->blocksize, context->new_nthreads, context->nthreads, context->splitmode, context->udbtune, context->btune, context->schunk); if (error <= 0) { return error; } /* Write the extended header */ error = write_compression_header(context, true); if (error < 0) { return error; } cbytes = blosc_compress_context(context); if (cbytes < 0) { return cbytes; } if (context->use_dict && context->dict_cdict == NULL) { if (context->compcode != BLOSC_ZSTD) { const char* compname; compname = clibcode_to_clibname(context->compcode); BLOSC_TRACE_ERROR("Codec %s does not support dicts. Giving up.", compname); return BLOSC2_ERROR_CODEC_DICT; } #ifdef HAVE_ZSTD // Build the dictionary out of the filters outcome and compress with it int32_t dict_maxsize = BLOSC2_MAXDICTSIZE; // Do not make the dict more than 5% larger than uncompressed buffer if (dict_maxsize > srcsize / 20) { dict_maxsize = srcsize / 20; } void* samples_buffer = context->dest + context->header_overhead; unsigned nblocks = 8; // the minimum that accepts zstd as of 1.4.0 unsigned sample_fraction = 1; // 1 allows to use most of the chunk for training size_t sample_size = context->sourcesize / nblocks / sample_fraction; // Populate the samples sizes for training the dictionary size_t* samples_sizes = malloc(nblocks * sizeof(void*)); BLOSC_ERROR_NULL(samples_sizes, BLOSC2_ERROR_MEMORY_ALLOC); for (size_t i = 0; i < nblocks; i++) { samples_sizes[i] = sample_size; } // Train from samples void* dict_buffer = malloc(dict_maxsize); BLOSC_ERROR_NULL(dict_buffer, BLOSC2_ERROR_MEMORY_ALLOC); int32_t dict_actual_size = (int32_t)ZDICT_trainFromBuffer(dict_buffer, dict_maxsize, samples_buffer, samples_sizes, nblocks); // TODO: experiment with parameters of low-level fast cover algorithm // Note that this API is still unstable. See: https://github.com/facebook/zstd/issues/1599 // ZDICT_fastCover_params_t fast_cover_params; // memset(&fast_cover_params, 0, sizeof(fast_cover_params)); // fast_cover_params.d = nblocks; // fast_cover_params.steps = 4; // fast_cover_params.zParams.compressionLevel = context->clevel; //size_t dict_actual_size = ZDICT_optimizeTrainFromBuffer_fastCover(dict_buffer, dict_maxsize, samples_buffer, samples_sizes, nblocks, &fast_cover_params); if (ZDICT_isError(dict_actual_size) != ZSTD_error_no_error) { BLOSC_TRACE_ERROR("Error in ZDICT_trainFromBuffer(): '%s'." " Giving up.", ZDICT_getErrorName(dict_actual_size)); return BLOSC2_ERROR_CODEC_DICT; } assert(dict_actual_size > 0); free(samples_sizes); // Update bytes counter and pointers to bstarts for the new compressed buffer context->bstarts = (int32_t*)(context->dest + context->header_overhead); context->output_bytes = context->header_overhead + (int32_t)sizeof(int32_t) * context->nblocks; /* Write the size of trained dict at the end of bstarts */ _sw32(context->dest + context->output_bytes, (int32_t)dict_actual_size); context->output_bytes += sizeof(int32_t); /* Write the trained dict afterwards */ context->dict_buffer = context->dest + context->output_bytes; memcpy(context->dict_buffer, dict_buffer, (unsigned int)dict_actual_size); context->dict_cdict = ZSTD_createCDict(dict_buffer, dict_actual_size, 1); // TODO: use get_accel() free(dict_buffer); // the dictionary is copied in the header now context->output_bytes += (int32_t)dict_actual_size; context->dict_size = dict_actual_size; /* Compress with dict */ cbytes = blosc_compress_context(context); // Invalidate the dictionary for compressing other chunks using the same context context->dict_buffer = NULL; ZSTD_freeCDict(context->dict_cdict); context->dict_cdict = NULL; #endif // HAVE_ZSTD } return cbytes; } void build_filters(const int doshuffle, const int delta, const int32_t typesize, uint8_t* filters) { /* Fill the end part of the filter pipeline */ if ((doshuffle == BLOSC_SHUFFLE) && (typesize > 1)) filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_SHUFFLE; if (doshuffle == BLOSC_BITSHUFFLE) filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_BITSHUFFLE; if (doshuffle == BLOSC_NOSHUFFLE) filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_NOSHUFFLE; if (delta) filters[BLOSC2_MAX_FILTERS - 2] = BLOSC_DELTA; } /* The public secure routine for compression. */ int blosc2_compress(int clevel, int doshuffle, int32_t typesize, const void* src, int32_t srcsize, void* dest, int32_t destsize) { int error; int result; char* envvar; /* Check whether the library should be initialized */ if (!g_initlib) blosc2_init(); /* Check for a BLOSC_CLEVEL environment variable */ envvar = getenv("BLOSC_CLEVEL"); if (envvar != NULL) { long value; value = strtol(envvar, NULL, 10); if ((value != EINVAL) && (value >= 0)) { clevel = (int)value; } else { BLOSC_TRACE_WARNING("BLOSC_CLEVEL environment variable '%s' not recognized\n", envvar); } } /* Check for a BLOSC_SHUFFLE environment variable */ envvar = getenv("BLOSC_SHUFFLE"); if (envvar != NULL) { if (strcmp(envvar, "NOSHUFFLE") == 0) { doshuffle = BLOSC_NOSHUFFLE; } else if (strcmp(envvar, "SHUFFLE") == 0) { doshuffle = BLOSC_SHUFFLE; } else if (strcmp(envvar, "BITSHUFFLE") == 0) { doshuffle = BLOSC_BITSHUFFLE; } else { BLOSC_TRACE_WARNING("BLOSC_SHUFFLE environment variable '%s' not recognized\n", envvar); } } /* Check for a BLOSC_DELTA environment variable */ envvar = getenv("BLOSC_DELTA"); if (envvar != NULL) { if (strcmp(envvar, "1") == 0) { blosc2_set_delta(1); } else if (strcmp(envvar, "0") == 0) { blosc2_set_delta(0); } else { BLOSC_TRACE_WARNING("BLOSC_DELTA environment variable '%s' not recognized\n", envvar); } } /* Check for a BLOSC_TYPESIZE environment variable */ envvar = getenv("BLOSC_TYPESIZE"); if (envvar != NULL) { long value; value = strtol(envvar, NULL, 10); if ((value != EINVAL) && (value > 0)) { typesize = (int32_t)value; } else { BLOSC_TRACE_WARNING("BLOSC_TYPESIZE environment variable '%s' not recognized\n", envvar); } } /* Check for a BLOSC_COMPRESSOR environment variable */ envvar = getenv("BLOSC_COMPRESSOR"); if (envvar != NULL) { result = blosc1_set_compressor(envvar); if (result < 0) { BLOSC_TRACE_WARNING("BLOSC_COMPRESSOR environment variable '%s' not recognized\n", envvar); } } /* Check for a BLOSC_BLOCKSIZE environment variable */ envvar = getenv("BLOSC_BLOCKSIZE"); if (envvar != NULL) { long blocksize; blocksize = strtol(envvar, NULL, 10); if ((blocksize != EINVAL) && (blocksize > 0)) { blosc1_set_blocksize((size_t) blocksize); } else { BLOSC_TRACE_WARNING("BLOSC_BLOCKSIZE environment variable '%s' not recognized\n", envvar); } } /* Check for a BLOSC_NTHREADS environment variable */ envvar = getenv("BLOSC_NTHREADS"); if (envvar != NULL) { long nthreads; nthreads = strtol(envvar, NULL, 10); if ((nthreads != EINVAL) && (nthreads > 0)) { result = blosc2_set_nthreads((int16_t) nthreads); if (result < 0) { BLOSC_TRACE_WARNING("BLOSC_NTHREADS environment variable '%s' not recognized\n", envvar); } } } /* Check for a BLOSC_SPLITMODE environment variable */ envvar = getenv("BLOSC_SPLITMODE"); if (envvar != NULL) { int32_t splitmode = -1; if (strcmp(envvar, "ALWAYS") == 0) { splitmode = BLOSC_ALWAYS_SPLIT; } else if (strcmp(envvar, "NEVER") == 0) { splitmode = BLOSC_NEVER_SPLIT; } else if (strcmp(envvar, "AUTO") == 0) { splitmode = BLOSC_AUTO_SPLIT; } else if (strcmp(envvar, "FORWARD_COMPAT") == 0) { splitmode = BLOSC_FORWARD_COMPAT_SPLIT; } else { BLOSC_TRACE_WARNING("BLOSC_SPLITMODE environment variable '%s' not recognized\n", envvar); } if (splitmode >= 0) { blosc1_set_splitmode(splitmode); } } /* Check for a BLOSC_NOLOCK environment variable. It is important that this should be the last env var so that it can take the previous ones into account */ envvar = getenv("BLOSC_NOLOCK"); if (envvar != NULL) { // TODO: here is the only place that returns an extended header from // a blosc1_compress() call. This should probably be fixed. const char *compname; blosc2_context *cctx; blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; blosc2_compcode_to_compname(g_compressor, &compname); /* Create a context for compression */ build_filters(doshuffle, g_delta, typesize, cparams.filters); // TODO: cparams can be shared in a multithreaded environment. do a copy! cparams.typesize = (uint8_t)typesize; cparams.compcode = (uint8_t)g_compressor; cparams.clevel = (uint8_t)clevel; cparams.nthreads = g_nthreads; cparams.splitmode = g_splitmode; cctx = blosc2_create_cctx(cparams); /* Do the actual compression */ result = blosc2_compress_ctx(cctx, src, srcsize, dest, destsize); /* Release context resources */ blosc2_free_ctx(cctx); return result; } pthread_mutex_lock(&global_comp_mutex); /* Initialize a context compression */ uint8_t* filters = calloc(1, BLOSC2_MAX_FILTERS); BLOSC_ERROR_NULL(filters, BLOSC2_ERROR_MEMORY_ALLOC); uint8_t* filters_meta = calloc(1, BLOSC2_MAX_FILTERS); BLOSC_ERROR_NULL(filters_meta, BLOSC2_ERROR_MEMORY_ALLOC); build_filters(doshuffle, g_delta, typesize, filters); error = initialize_context_compression( g_global_context, src, srcsize, dest, destsize, clevel, filters, filters_meta, (int32_t)typesize, g_compressor, g_force_blocksize, g_nthreads, g_nthreads, g_splitmode, &BTUNE_DEFAULTS, NULL, g_schunk); free(filters); free(filters_meta); if (error <= 0) { pthread_mutex_unlock(&global_comp_mutex); return error; } envvar = getenv("BLOSC_BLOSC1_COMPAT"); if (envvar != NULL) { /* Write chunk header without extended header (Blosc1 compatibility mode) */ error = write_compression_header(g_global_context, false); } else { error = write_compression_header(g_global_context, true); } if (error < 0) { pthread_mutex_unlock(&global_comp_mutex); return error; } result = blosc_compress_context(g_global_context); pthread_mutex_unlock(&global_comp_mutex); return result; } /* The public routine for compression. */ int blosc1_compress(int clevel, int doshuffle, size_t typesize, size_t nbytes, const void* src, void* dest, size_t destsize) { return blosc2_compress(clevel, doshuffle, (int32_t)typesize, src, (int32_t)nbytes, dest, (int32_t)destsize); } static int blosc_run_decompression_with_context(blosc2_context* context, const void* src, int32_t srcsize, void* dest, int32_t destsize) { blosc_header header; int32_t ntbytes; int rc; rc = read_chunk_header(src, srcsize, true, &header); if (rc < 0) { return rc; } if (header.nbytes > destsize) { // Not enough space for writing into the destination return BLOSC2_ERROR_WRITE_BUFFER; } rc = initialize_context_decompression(context, &header, src, srcsize, dest, destsize); if (rc < 0) { return rc; } /* Do the actual decompression */ ntbytes = do_job(context); if (ntbytes < 0) { return ntbytes; } assert(ntbytes <= (int32_t)destsize); return ntbytes; } /* The public secure routine for decompression with context. */ int blosc2_decompress_ctx(blosc2_context* context, const void* src, int32_t srcsize, void* dest, int32_t destsize) { int result; if (context->do_compress != 0) { BLOSC_TRACE_ERROR("Context is not meant for decompression. Giving up."); return BLOSC2_ERROR_INVALID_PARAM; } result = blosc_run_decompression_with_context(context, src, srcsize, dest, destsize); // Reset a possible block_maskout if (context->block_maskout != NULL) { free(context->block_maskout); context->block_maskout = NULL; } context->block_maskout_nitems = 0; return result; } /* The public secure routine for decompression. */ int blosc2_decompress(const void* src, int32_t srcsize, void* dest, int32_t destsize) { int result; char* envvar; long nthreads; blosc2_context *dctx; blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; /* Check whether the library should be initialized */ if (!g_initlib) blosc2_init(); /* Check for a BLOSC_NTHREADS environment variable */ envvar = getenv("BLOSC_NTHREADS"); if (envvar != NULL) { nthreads = strtol(envvar, NULL, 10); if ((nthreads != EINVAL) && (nthreads > 0)) { result = blosc2_set_nthreads((int16_t) nthreads); if (result < 0) { return result; } } } /* Check for a BLOSC_NOLOCK environment variable. It is important that this should be the last env var so that it can take the previous ones into account */ envvar = getenv("BLOSC_NOLOCK"); if (envvar != NULL) { dparams.nthreads = g_nthreads; dctx = blosc2_create_dctx(dparams); result = blosc2_decompress_ctx(dctx, src, srcsize, dest, destsize); blosc2_free_ctx(dctx); return result; } pthread_mutex_lock(&global_comp_mutex); result = blosc_run_decompression_with_context( g_global_context, src, srcsize, dest, destsize); pthread_mutex_unlock(&global_comp_mutex); return result; } /* The public routine for decompression. */ int blosc1_decompress(const void* src, void* dest, size_t destsize) { return blosc2_decompress(src, INT32_MAX, dest, (int32_t)destsize); } /* Specific routine optimized for decompression a small number of items out of a compressed chunk. This does not use threads because it would affect negatively to performance. */ int _blosc_getitem(blosc2_context* context, blosc_header* header, const void* src, int32_t srcsize, int start, int nitems, void* dest, int32_t destsize) { uint8_t* _src = (uint8_t*)(src); /* current pos for source buffer */ uint8_t* _dest = (uint8_t*)(dest); int32_t ntbytes = 0; /* the number of uncompressed bytes */ int32_t bsize, bsize2, ebsize, leftoverblock; int32_t startb, stopb; int32_t stop = start + nitems; int j, rc; if (nitems == 0) { // We have nothing to do return 0; } if (nitems * header->typesize > destsize) { BLOSC_TRACE_ERROR("`nitems`*`typesize` out of dest bounds."); return BLOSC2_ERROR_WRITE_BUFFER; } context->bstarts = (int32_t*)(_src + context->header_overhead); /* Check region boundaries */ if ((start < 0) || (start * header->typesize > header->nbytes)) { BLOSC_TRACE_ERROR("`start` out of bounds."); return BLOSC2_ERROR_INVALID_PARAM; } if ((stop < 0) || (stop * header->typesize > header->nbytes)) { BLOSC_TRACE_ERROR("`start`+`nitems` out of bounds."); return BLOSC2_ERROR_INVALID_PARAM; } int chunk_memcpy = header->flags & 0x1; if (!context->special_type && !chunk_memcpy && ((uint8_t *)(_src + srcsize) < (uint8_t *)(context->bstarts + context->nblocks))) { BLOSC_TRACE_ERROR("`bstarts` out of bounds."); return BLOSC2_ERROR_READ_BUFFER; } bool memcpyed = header->flags & (uint8_t)BLOSC_MEMCPYED; if (context->special_type) { // Fake a runlen as if its a memcpyed chunk memcpyed = true; } bool is_lazy = ((context->header_overhead == BLOSC_EXTENDED_HEADER_LENGTH) && (context->blosc2_flags & 0x08u) && !context->special_type); if (memcpyed && !is_lazy && !context->postfilter) { // Short-circuit for (non-lazy) memcpyed or special values ntbytes = nitems * header->typesize; switch (context->special_type) { case BLOSC2_SPECIAL_VALUE: // All repeated values rc = set_values(context->typesize, _src, _dest, ntbytes); if (rc < 0) { BLOSC_TRACE_ERROR("set_values failed"); return BLOSC2_ERROR_DATA; } break; case BLOSC2_SPECIAL_NAN: rc = set_nans(context->typesize, _dest, ntbytes); if (rc < 0) { BLOSC_TRACE_ERROR("set_nans failed"); return BLOSC2_ERROR_DATA; } break; case BLOSC2_SPECIAL_ZERO: memset(_dest, 0, ntbytes); break; case BLOSC2_SPECIAL_UNINIT: // We do nothing here break; case BLOSC2_NO_SPECIAL: _src += context->header_overhead + start * context->typesize; memcpy(_dest, _src, ntbytes); break; default: BLOSC_TRACE_ERROR("Unhandled special value case"); return -1; } return ntbytes; } ebsize = header->blocksize + header->typesize * (signed)sizeof(int32_t); struct thread_context* scontext = context->serial_context; /* Resize the temporaries in serial context if needed */ if (header->blocksize > scontext->tmp_blocksize) { my_free(scontext->tmp); scontext->tmp_nbytes = (size_t)4 * ebsize; scontext->tmp = my_malloc(scontext->tmp_nbytes); BLOSC_ERROR_NULL(scontext->tmp, BLOSC2_ERROR_MEMORY_ALLOC); scontext->tmp2 = scontext->tmp + ebsize; scontext->tmp3 = scontext->tmp2 + ebsize; scontext->tmp4 = scontext->tmp3 + ebsize; scontext->tmp_blocksize = (int32_t)header->blocksize; } for (j = 0; j < context->nblocks; j++) { bsize = header->blocksize; leftoverblock = 0; if ((j == context->nblocks - 1) && (context->leftover > 0)) { bsize = context->leftover; leftoverblock = 1; } /* Compute start & stop for each block */ startb = start * header->typesize - j * header->blocksize; stopb = stop * header->typesize - j * header->blocksize; if (stopb <= 0) { // We can exit as soon as this block is beyond stop break; } if (startb >= header->blocksize) { continue; } if (startb < 0) { startb = 0; } if (stopb > header->blocksize) { stopb = header->blocksize; } bsize2 = stopb - startb; #if defined(HAVE_PLUGINS) if (context->compcode == BLOSC_CODEC_ZFP_FIXED_RATE) { scontext->zfp_cell_start = startb / context->typesize; scontext->zfp_cell_nitems = nitems; } #endif /* HAVE_PLUGINS */ /* Do the actual data copy */ // Regular decompression. Put results in tmp2. // If the block is aligned and the worst case fits in destination, let's avoid a copy bool get_single_block = ((startb == 0) && (bsize == nitems * header->typesize)); uint8_t* tmp2 = get_single_block ? dest : scontext->tmp2; // If memcpyed we don't have a bstarts section (because it is not needed) int32_t src_offset = memcpyed ? context->header_overhead + j * bsize : sw32_(context->bstarts + j); int32_t cbytes = blosc_d(context->serial_context, bsize, leftoverblock, memcpyed, src, srcsize, src_offset, j, tmp2, 0, scontext->tmp, scontext->tmp3); if (cbytes < 0) { ntbytes = cbytes; break; } if (scontext->zfp_cell_nitems > 0) { if (cbytes == bsize2) { memcpy((uint8_t *) dest, tmp2, (unsigned int) bsize2); } else if (cbytes == context->blocksize) { memcpy((uint8_t *) dest, tmp2 + scontext->zfp_cell_start * context->typesize, (unsigned int) bsize2); cbytes = bsize2; } } else if (!get_single_block) { /* Copy to destination */ memcpy((uint8_t *) dest + ntbytes, tmp2 + startb, (unsigned int) bsize2); } ntbytes += bsize2; } scontext->zfp_cell_nitems = 0; return ntbytes; } int blosc2_getitem(const void* src, int32_t srcsize, int start, int nitems, void* dest, int32_t destsize) { blosc2_context context; int result; /* Minimally populate the context */ memset(&context, 0, sizeof(blosc2_context)); context.schunk = g_schunk; context.nthreads = 1; // force a serial decompression; fixes #95 /* Call the actual getitem function */ result = blosc2_getitem_ctx(&context, src, srcsize, start, nitems, dest, destsize); /* Release resources */ if (context.serial_context != NULL) { free_thread_context(context.serial_context); } return result; } /* Specific routine optimized for decompression a small number of items out of a compressed chunk. Public non-contextual API. */ int blosc1_getitem(const void* src, int start, int nitems, void* dest) { return blosc2_getitem(src, INT32_MAX, start, nitems, dest, INT32_MAX); } int blosc2_getitem_ctx(blosc2_context* context, const void* src, int32_t srcsize, int start, int nitems, void* dest, int32_t destsize) { blosc_header header; int result; /* Minimally populate the context */ result = read_chunk_header((uint8_t *) src, srcsize, true, &header); if (result < 0) { return result; } context->src = src; context->srcsize = srcsize; context->dest = dest; context->destsize = destsize; result = blosc2_initialize_context_from_header(context, &header); if (result < 0) { return result; } if (context->serial_context == NULL) { context->serial_context = create_thread_context(context, 0); } BLOSC_ERROR_NULL(context->serial_context, BLOSC2_ERROR_THREAD_CREATE); /* Call the actual getitem function */ result = _blosc_getitem(context, &header, src, srcsize, start, nitems, dest, destsize); return result; } /* execute single compression/decompression job for a single thread_context */ static void t_blosc_do_job(void *ctxt) { struct thread_context* thcontext = (struct thread_context*)ctxt; blosc2_context* context = thcontext->parent_context; int32_t cbytes; int32_t ntdest; int32_t tblocks; /* number of blocks per thread */ int32_t tblock; /* limit block on a thread */ int32_t nblock_; /* private copy of nblock */ int32_t bsize; int32_t leftoverblock; /* Parameters for threads */ int32_t blocksize; int32_t ebsize; int32_t srcsize; bool compress = context->do_compress != 0; int32_t maxbytes; int32_t nblocks; int32_t leftover; int32_t leftover2; int32_t* bstarts; const uint8_t* src; uint8_t* dest; uint8_t* tmp; uint8_t* tmp2; uint8_t* tmp3; /* Get parameters for this thread before entering the main loop */ blocksize = context->blocksize; ebsize = blocksize + context->typesize * (int32_t)sizeof(int32_t); maxbytes = context->destsize; nblocks = context->nblocks; leftover = context->leftover; bstarts = context->bstarts; src = context->src; srcsize = context->srcsize; dest = context->dest; /* Resize the temporaries if needed */ if (blocksize > thcontext->tmp_blocksize) { my_free(thcontext->tmp); thcontext->tmp_nbytes = (size_t) 4 * ebsize; thcontext->tmp = my_malloc(thcontext->tmp_nbytes); thcontext->tmp2 = thcontext->tmp + ebsize; thcontext->tmp3 = thcontext->tmp2 + ebsize; thcontext->tmp4 = thcontext->tmp3 + ebsize; thcontext->tmp_blocksize = blocksize; } tmp = thcontext->tmp; tmp2 = thcontext->tmp2; tmp3 = thcontext->tmp3; // Determine whether we can do a static distribution of workload among different threads bool memcpyed = context->header_flags & (uint8_t)BLOSC_MEMCPYED; if (!context->do_compress && context->special_type) { // Fake a runlen as if its a memcpyed chunk memcpyed = true; } bool static_schedule = (!compress || memcpyed) && context->block_maskout == NULL; if (static_schedule) { /* Blocks per thread */ tblocks = nblocks / context->nthreads; leftover2 = nblocks % context->nthreads; tblocks = (leftover2 > 0) ? tblocks + 1 : tblocks; nblock_ = thcontext->tid * tblocks; tblock = nblock_ + tblocks; if (tblock > nblocks) { tblock = nblocks; } } else { // Use dynamic schedule via a queue. Get the next block. pthread_mutex_lock(&context->count_mutex); context->thread_nblock++; nblock_ = context->thread_nblock; pthread_mutex_unlock(&context->count_mutex); tblock = nblocks; } /* Loop over blocks */ leftoverblock = 0; while ((nblock_ < tblock) && (context->thread_giveup_code > 0)) { bsize = blocksize; if (nblock_ == (nblocks - 1) && (leftover > 0)) { bsize = leftover; leftoverblock = 1; } if (compress) { if (memcpyed) { if (!context->prefilter) { /* We want to memcpy only */ memcpy(dest + context->header_overhead + nblock_ * blocksize, src + nblock_ * blocksize, (unsigned int) bsize); cbytes = (int32_t) bsize; } else { /* Only the prefilter has to be executed, and this is done in blosc_c(). * However, no further actions are needed, so we can put the result * directly in dest. */ cbytes = blosc_c(thcontext, bsize, leftoverblock, 0, ebsize, src, nblock_ * blocksize, dest + context->header_overhead + nblock_ * blocksize, tmp, tmp3); } } else { /* Regular compression */ cbytes = blosc_c(thcontext, bsize, leftoverblock, 0, ebsize, src, nblock_ * blocksize, tmp2, tmp, tmp3); } } else { /* Regular decompression */ if (context->special_type == BLOSC2_NO_SPECIAL && !memcpyed && (srcsize < (int32_t)(context->header_overhead + (sizeof(int32_t) * nblocks)))) { /* Not enough input to read all `bstarts` */ cbytes = -1; } else { // If memcpyed we don't have a bstarts section (because it is not needed) int32_t src_offset = memcpyed ? context->header_overhead + nblock_ * blocksize : sw32_(bstarts + nblock_); cbytes = blosc_d(thcontext, bsize, leftoverblock, memcpyed, src, srcsize, src_offset, nblock_, dest, nblock_ * blocksize, tmp, tmp2); } } /* Check whether current thread has to giveup */ if (context->thread_giveup_code <= 0) { break; } /* Check results for the compressed/decompressed block */ if (cbytes < 0) { /* compr/decompr failure */ /* Set giveup_code error */ pthread_mutex_lock(&context->count_mutex); context->thread_giveup_code = cbytes; pthread_mutex_unlock(&context->count_mutex); break; } if (compress && !memcpyed) { /* Start critical section */ pthread_mutex_lock(&context->count_mutex); ntdest = context->output_bytes; // Note: do not use a typical local dict_training variable here // because it is probably cached from previous calls if the number of // threads does not change (the usual thing). if (!(context->use_dict && context->dict_cdict == NULL)) { _sw32(bstarts + nblock_, (int32_t) ntdest); } if ((cbytes == 0) || (ntdest + cbytes > maxbytes)) { context->thread_giveup_code = 0; /* incompressible buf */ pthread_mutex_unlock(&context->count_mutex); break; } context->thread_nblock++; nblock_ = context->thread_nblock; context->output_bytes += cbytes; pthread_mutex_unlock(&context->count_mutex); /* End of critical section */ /* Copy the compressed buffer to destination */ memcpy(dest + ntdest, tmp2, (unsigned int) cbytes); } else if (static_schedule) { nblock_++; } else { pthread_mutex_lock(&context->count_mutex); context->thread_nblock++; nblock_ = context->thread_nblock; context->output_bytes += cbytes; pthread_mutex_unlock(&context->count_mutex); } } /* closes while (nblock_) */ if (static_schedule) { pthread_mutex_lock(&context->count_mutex); context->output_bytes = context->sourcesize; if (compress) { context->output_bytes += context->header_overhead; } pthread_mutex_unlock(&context->count_mutex); } } /* Decompress & unshuffle several blocks in a single thread */ static void* t_blosc(void* ctxt) { struct thread_context* thcontext = (struct thread_context*)ctxt; blosc2_context* context = thcontext->parent_context; #ifdef BLOSC_POSIX_BARRIERS int rc; #endif while (1) { /* Synchronization point for all threads (wait for initialization) */ WAIT_INIT(NULL, context); if (context->end_threads) { break; } t_blosc_do_job(ctxt); /* Meeting point for all threads (wait for finalization) */ WAIT_FINISH(NULL, context); } /* Cleanup our working space and context */ free_thread_context(thcontext); return (NULL); } int init_threadpool(blosc2_context *context) { int32_t tid; int rc2; /* Initialize mutex and condition variable objects */ pthread_mutex_init(&context->count_mutex, NULL); pthread_mutex_init(&context->delta_mutex, NULL); pthread_mutex_init(&context->nchunk_mutex, NULL); pthread_cond_init(&context->delta_cv, NULL); /* Set context thread sentinels */ context->thread_giveup_code = 1; context->thread_nblock = -1; /* Barrier initialization */ #ifdef BLOSC_POSIX_BARRIERS pthread_barrier_init(&context->barr_init, NULL, context->nthreads + 1); pthread_barrier_init(&context->barr_finish, NULL, context->nthreads + 1); #else pthread_mutex_init(&context->count_threads_mutex, NULL); pthread_cond_init(&context->count_threads_cv, NULL); context->count_threads = 0; /* Reset threads counter */ #endif if (threads_callback) { /* Create thread contexts to store data for callback threads */ context->thread_contexts = (struct thread_context *)my_malloc( context->nthreads * sizeof(struct thread_context)); BLOSC_ERROR_NULL(context->thread_contexts, BLOSC2_ERROR_MEMORY_ALLOC); for (tid = 0; tid < context->nthreads; tid++) init_thread_context(context->thread_contexts + tid, context, tid); } else { #if !defined(_WIN32) /* Initialize and set thread detached attribute */ pthread_attr_init(&context->ct_attr); pthread_attr_setdetachstate(&context->ct_attr, PTHREAD_CREATE_JOINABLE); #endif /* Make space for thread handlers */ context->threads = (pthread_t*)my_malloc( context->nthreads * sizeof(pthread_t)); BLOSC_ERROR_NULL(context->threads, BLOSC2_ERROR_MEMORY_ALLOC); /* Finally, create the threads */ for (tid = 0; tid < context->nthreads; tid++) { /* Create a thread context (will destroy when finished) */ struct thread_context *thread_context = create_thread_context(context, tid); BLOSC_ERROR_NULL(thread_context, BLOSC2_ERROR_THREAD_CREATE); #if !defined(_WIN32) rc2 = pthread_create(&context->threads[tid], &context->ct_attr, t_blosc, (void*)thread_context); #else rc2 = pthread_create(&context->threads[tid], NULL, t_blosc, (void *)thread_context); #endif if (rc2) { BLOSC_TRACE_ERROR("Return code from pthread_create() is %d.\n" "\tError detail: %s\n", rc2, strerror(rc2)); return BLOSC2_ERROR_THREAD_CREATE; } } } /* We have now started/initialized the threads */ context->threads_started = context->nthreads; context->new_nthreads = context->nthreads; return 0; } int16_t blosc2_get_nthreads(void) { return g_nthreads; } int16_t blosc2_set_nthreads(int16_t nthreads) { int16_t ret = g_nthreads; /* the previous number of threads */ /* Check whether the library should be initialized */ if (!g_initlib) blosc2_init(); if (nthreads != ret) { g_nthreads = nthreads; g_global_context->new_nthreads = nthreads; check_nthreads(g_global_context); } return ret; } const char* blosc1_get_compressor(void) { const char* compname; blosc2_compcode_to_compname(g_compressor, &compname); return compname; } int blosc1_set_compressor(const char* compname) { int code = blosc2_compname_to_compcode(compname); if (code >= BLOSC_LAST_CODEC) { BLOSC_TRACE_ERROR("User defined codecs cannot be set here. Use Blosc2 mechanism instead."); return -1; } g_compressor = code; /* Check whether the library should be initialized */ if (!g_initlib) blosc2_init(); return code; } void blosc2_set_delta(int dodelta) { g_delta = dodelta; /* Check whether the library should be initialized */ if (!g_initlib) blosc2_init(); } const char* blosc2_list_compressors(void) { static int compressors_list_done = 0; static char ret[256]; if (compressors_list_done) return ret; ret[0] = '\0'; strcat(ret, BLOSC_BLOSCLZ_COMPNAME); strcat(ret, ","); strcat(ret, BLOSC_LZ4_COMPNAME); strcat(ret, ","); strcat(ret, BLOSC_LZ4HC_COMPNAME); #if defined(HAVE_ZLIB) strcat(ret, ","); strcat(ret, BLOSC_ZLIB_COMPNAME); #endif /* HAVE_ZLIB */ #if defined(HAVE_ZSTD) strcat(ret, ","); strcat(ret, BLOSC_ZSTD_COMPNAME); #endif /* HAVE_ZSTD */ compressors_list_done = 1; return ret; } const char* blosc2_get_version_string(void) { return BLOSC2_VERSION_STRING; } int blosc2_get_complib_info(const char* compname, char** complib, char** version) { int clibcode; const char* clibname; const char* clibversion = "unknown"; char sbuffer[256]; clibcode = compname_to_clibcode(compname); clibname = clibcode_to_clibname(clibcode); /* complib version */ if (clibcode == BLOSC_BLOSCLZ_LIB) { clibversion = BLOSCLZ_VERSION_STRING; } else if (clibcode == BLOSC_LZ4_LIB) { sprintf(sbuffer, "%d.%d.%d", LZ4_VERSION_MAJOR, LZ4_VERSION_MINOR, LZ4_VERSION_RELEASE); clibversion = sbuffer; } #if defined(HAVE_ZLIB) else if (clibcode == BLOSC_ZLIB_LIB) { #ifdef ZLIB_COMPAT clibversion = ZLIB_VERSION; #elif defined(HAVE_ZLIB_NG) clibversion = ZLIBNG_VERSION; #else clibversion = ZLIB_VERSION; #endif } #endif /* HAVE_ZLIB */ #if defined(HAVE_ZSTD) else if (clibcode == BLOSC_ZSTD_LIB) { sprintf(sbuffer, "%d.%d.%d", ZSTD_VERSION_MAJOR, ZSTD_VERSION_MINOR, ZSTD_VERSION_RELEASE); clibversion = sbuffer; } #endif /* HAVE_ZSTD */ #ifdef _MSC_VER *complib = _strdup(clibname); *version = _strdup(clibversion); #else *complib = strdup(clibname); *version = strdup(clibversion); #endif return clibcode; } /* Return `nbytes`, `cbytes` and `blocksize` from a compressed buffer. */ void blosc1_cbuffer_sizes(const void* cbuffer, size_t* nbytes, size_t* cbytes, size_t* blocksize) { int32_t nbytes32, cbytes32, blocksize32; blosc2_cbuffer_sizes(cbuffer, &nbytes32, &cbytes32, &blocksize32); *nbytes = nbytes32; *cbytes = cbytes32; *blocksize = blocksize32; } int blosc2_cbuffer_sizes(const void* cbuffer, int32_t* nbytes, int32_t* cbytes, int32_t* blocksize) { blosc_header header; int rc = read_chunk_header((uint8_t *) cbuffer, BLOSC_MIN_HEADER_LENGTH, false, &header); if (rc < 0) { /* Return zeros if error reading header */ memset(&header, 0, sizeof(header)); } /* Read the interesting values */ if (nbytes != NULL) *nbytes = header.nbytes; if (cbytes != NULL) *cbytes = header.cbytes; if (blocksize != NULL) *blocksize = header.blocksize; return rc; } int blosc1_cbuffer_validate(const void* cbuffer, size_t cbytes, size_t* nbytes) { int32_t header_cbytes; int32_t header_nbytes; if (cbytes < BLOSC_MIN_HEADER_LENGTH) { /* Compressed data should contain enough space for header */ *nbytes = 0; return BLOSC2_ERROR_WRITE_BUFFER; } int rc = blosc2_cbuffer_sizes(cbuffer, &header_nbytes, &header_cbytes, NULL); if (rc < 0) { *nbytes = 0; return rc; } *nbytes = header_nbytes; if (header_cbytes != (int32_t)cbytes) { /* Compressed size from header does not match `cbytes` */ *nbytes = 0; return BLOSC2_ERROR_INVALID_HEADER; } if (*nbytes > BLOSC2_MAX_BUFFERSIZE) { /* Uncompressed size is larger than allowed */ *nbytes = 0; return BLOSC2_ERROR_MEMORY_ALLOC; } return 0; } /* Return `typesize` and `flags` from a compressed buffer. */ void blosc1_cbuffer_metainfo(const void* cbuffer, size_t* typesize, int* flags) { blosc_header header; int rc = read_chunk_header((uint8_t *) cbuffer, BLOSC_MIN_HEADER_LENGTH, false, &header); if (rc < 0) { *typesize = *flags = 0; return; } /* Read the interesting values */ *flags = header.flags; *typesize = header.typesize; } /* Return version information from a compressed buffer. */ void blosc2_cbuffer_versions(const void* cbuffer, int* version, int* versionlz) { blosc_header header; int rc = read_chunk_header((uint8_t *) cbuffer, BLOSC_MIN_HEADER_LENGTH, false, &header); if (rc < 0) { *version = *versionlz = 0; return; } /* Read the version info */ *version = header.version; *versionlz = header.versionlz; } /* Return the compressor library/format used in a compressed buffer. */ const char* blosc2_cbuffer_complib(const void* cbuffer) { blosc_header header; int clibcode; const char* complib; int rc = read_chunk_header((uint8_t *) cbuffer, BLOSC_MIN_HEADER_LENGTH, false, &header); if (rc < 0) { return NULL; } /* Read the compressor format/library info */ clibcode = (header.flags & 0xe0) >> 5; complib = clibcode_to_clibname(clibcode); return complib; } /* Get the internal blocksize to be used during compression. 0 means that an automatic blocksize is computed internally. */ int blosc1_get_blocksize(void) { return (int)g_force_blocksize; } /* Force the use of a specific blocksize. If 0, an automatic blocksize will be used (the default). */ void blosc1_set_blocksize(size_t blocksize) { g_force_blocksize = (int32_t)blocksize; } /* Force the use of a specific split mode. */ void blosc1_set_splitmode(int mode) { g_splitmode = mode; } /* Set pointer to super-chunk. If NULL, no super-chunk will be reachable (the default). */ void blosc_set_schunk(blosc2_schunk* schunk) { g_schunk = schunk; g_global_context->schunk = schunk; } blosc2_io *blosc2_io_global = NULL; void blosc2_init(void) { /* Return if Blosc is already initialized */ if (g_initlib) return; BLOSC2_IO_CB_DEFAULTS.id = BLOSC2_IO_FILESYSTEM; BLOSC2_IO_CB_DEFAULTS.open = (blosc2_open_cb) blosc2_stdio_open; BLOSC2_IO_CB_DEFAULTS.close = (blosc2_close_cb) blosc2_stdio_close; BLOSC2_IO_CB_DEFAULTS.tell = (blosc2_tell_cb) blosc2_stdio_tell; BLOSC2_IO_CB_DEFAULTS.seek = (blosc2_seek_cb) blosc2_stdio_seek; BLOSC2_IO_CB_DEFAULTS.write = (blosc2_write_cb) blosc2_stdio_write; BLOSC2_IO_CB_DEFAULTS.read = (blosc2_read_cb) blosc2_stdio_read; BLOSC2_IO_CB_DEFAULTS.truncate = (blosc2_truncate_cb) blosc2_stdio_truncate; g_ncodecs = 0; g_nfilters = 0; #if defined(HAVE_PLUGINS) #include "blosc2/blosc2-common.h" #include "blosc2/blosc2-stdio.h" register_codecs(); register_filters(); #endif pthread_mutex_init(&global_comp_mutex, NULL); /* Create a global context */ g_global_context = (blosc2_context*)my_malloc(sizeof(blosc2_context)); memset(g_global_context, 0, sizeof(blosc2_context)); g_global_context->nthreads = g_nthreads; g_global_context->new_nthreads = g_nthreads; g_initlib = 1; } int blosc2_free_resources(void) { /* Return if Blosc is not initialized */ if (!g_initlib) return BLOSC2_ERROR_FAILURE; return release_threadpool(g_global_context); } void blosc2_destroy(void) { /* Return if Blosc is not initialized */ if (!g_initlib) return; blosc2_free_resources(); g_initlib = 0; blosc2_free_ctx(g_global_context); pthread_mutex_destroy(&global_comp_mutex); } int release_threadpool(blosc2_context *context) { int32_t t; void* status; int rc; if (context->threads_started > 0) { if (threads_callback) { /* free context data for user-managed threads */ for (t=0; tthreads_started; t++) destroy_thread_context(context->thread_contexts + t); my_free(context->thread_contexts); } else { /* Tell all existing threads to finish */ context->end_threads = 1; WAIT_INIT(-1, context); /* Join exiting threads */ for (t = 0; t < context->threads_started; t++) { rc = pthread_join(context->threads[t], &status); if (rc) { BLOSC_TRACE_ERROR("Return code from pthread_join() is %d\n" "\tError detail: %s.", rc, strerror(rc)); } } /* Thread attributes */ #if !defined(_WIN32) pthread_attr_destroy(&context->ct_attr); #endif /* Release thread handlers */ my_free(context->threads); } /* Release mutex and condition variable objects */ pthread_mutex_destroy(&context->count_mutex); pthread_mutex_destroy(&context->delta_mutex); pthread_mutex_destroy(&context->nchunk_mutex); pthread_cond_destroy(&context->delta_cv); /* Barriers */ #ifdef BLOSC_POSIX_BARRIERS pthread_barrier_destroy(&context->barr_init); pthread_barrier_destroy(&context->barr_finish); #else pthread_mutex_destroy(&context->count_threads_mutex); pthread_cond_destroy(&context->count_threads_cv); context->count_threads = 0; /* Reset threads counter */ #endif /* Reset flags and counters */ context->end_threads = 0; context->threads_started = 0; } return 0; } /* Contexts */ /* Create a context for compression */ blosc2_context* blosc2_create_cctx(blosc2_cparams cparams) { blosc2_context* context = (blosc2_context*)my_malloc(sizeof(blosc2_context)); BLOSC_ERROR_NULL(context, NULL); /* Populate the context, using zeros as default values */ memset(context, 0, sizeof(blosc2_context)); context->do_compress = 1; /* meant for compression */ context->use_dict = cparams.use_dict; if (cparams.instr_codec) { context->blosc2_flags = BLOSC2_INSTR_CODEC; } for (int i = 0; i < BLOSC2_MAX_FILTERS; i++) { context->filters[i] = cparams.filters[i]; context->filters_meta[i] = cparams.filters_meta[i]; if (context->filters[i] >= BLOSC_LAST_FILTER && context->filters[i] <= BLOSC2_DEFINED_FILTERS_STOP) { BLOSC_TRACE_ERROR("filter (%d) is not yet defined", context->filters[i]); free(context); return NULL; } if (context->filters[i] > BLOSC_LAST_REGISTERED_FILTER && context->filters[i] <= BLOSC2_GLOBAL_REGISTERED_FILTERS_STOP) { BLOSC_TRACE_ERROR("filter (%d) is not yet defined", context->filters[i]); free(context); return NULL; } } /* Check for a BLOSC_SHUFFLE environment variable */ int doshuffle = -1; char* envvar = getenv("BLOSC_SHUFFLE"); if (envvar != NULL) { if (strcmp(envvar, "NOSHUFFLE") == 0) { doshuffle = BLOSC_NOSHUFFLE; } else if (strcmp(envvar, "SHUFFLE") == 0) { doshuffle = BLOSC_SHUFFLE; } else if (strcmp(envvar, "BITSHUFFLE") == 0) { doshuffle = BLOSC_BITSHUFFLE; } else { BLOSC_TRACE_WARNING("BLOSC_SHUFFLE environment variable '%s' not recognized\n", envvar); } } /* Check for a BLOSC_DELTA environment variable */ int dodelta = BLOSC_NOFILTER; envvar = getenv("BLOSC_DELTA"); if (envvar != NULL) { if (strcmp(envvar, "1") == 0) { dodelta = BLOSC_DELTA; } else if (strcmp(envvar, "0") == 0){ dodelta = BLOSC_NOFILTER; } else { BLOSC_TRACE_WARNING("BLOSC_DELTA environment variable '%s' not recognized\n", envvar); } } /* Check for a BLOSC_TYPESIZE environment variable */ context->typesize = cparams.typesize; envvar = getenv("BLOSC_TYPESIZE"); if (envvar != NULL) { int32_t value; value = (int32_t) strtol(envvar, NULL, 10); if ((value != EINVAL) && (value > 0)) { context->typesize = value; } else { BLOSC_TRACE_WARNING("BLOSC_TYPESIZE environment variable '%s' not recognized\n", envvar); } } build_filters(doshuffle, dodelta, context->typesize, context->filters); context->clevel = cparams.clevel; /* Check for a BLOSC_CLEVEL environment variable */ envvar = getenv("BLOSC_CLEVEL"); if (envvar != NULL) { int value; value = (int)strtol(envvar, NULL, 10); if ((value != EINVAL) && (value >= 0)) { context->clevel = value; } else { BLOSC_TRACE_WARNING("BLOSC_CLEVEL environment variable '%s' not recognized\n", envvar); } } context->compcode = cparams.compcode; /* Check for a BLOSC_COMPRESSOR environment variable */ envvar = getenv("BLOSC_COMPRESSOR"); if (envvar != NULL) { int codec = blosc2_compname_to_compcode(envvar); if (codec >= BLOSC_LAST_CODEC) { BLOSC_TRACE_ERROR("User defined codecs cannot be set here. Use Blosc2 mechanism instead."); return NULL; } context->compcode = codec; } context->compcode_meta = cparams.compcode_meta; context->blocksize = cparams.blocksize; /* Check for a BLOSC_BLOCKSIZE environment variable */ envvar = getenv("BLOSC_BLOCKSIZE"); if (envvar != NULL) { int32_t blocksize; blocksize = (int32_t) strtol(envvar, NULL, 10); if ((blocksize != EINVAL) && (blocksize > 0)) { context->blocksize = blocksize; } else { BLOSC_TRACE_WARNING("BLOSC_BLOCKSIZE environment variable '%s' not recognized\n", envvar); } } context->nthreads = cparams.nthreads; /* Check for a BLOSC_NTHREADS environment variable */ envvar = getenv("BLOSC_NTHREADS"); if (envvar != NULL) { int16_t nthreads = (int16_t) strtol(envvar, NULL, 10); if ((nthreads != EINVAL) && (nthreads > 0)) { context->nthreads = nthreads; } else { BLOSC_TRACE_WARNING("BLOSC_NTHREADS environment variable '%s' not recognized\n", envvar); } } context->new_nthreads = context->nthreads; context->splitmode = cparams.splitmode; /* Check for a BLOSC_SPLITMODE environment variable */ envvar = getenv("BLOSC_SPLITMODE"); if (envvar != NULL) { int32_t splitmode = -1; if (strcmp(envvar, "ALWAYS") == 0) { splitmode = BLOSC_ALWAYS_SPLIT; } else if (strcmp(envvar, "NEVER") == 0) { splitmode = BLOSC_NEVER_SPLIT; } else if (strcmp(envvar, "AUTO") == 0) { splitmode = BLOSC_AUTO_SPLIT; } else if (strcmp(envvar, "FORWARD_COMPAT") == 0) { splitmode = BLOSC_FORWARD_COMPAT_SPLIT; } else { BLOSC_TRACE_WARNING("BLOSC_SPLITMODE environment variable '%s' not recognized\n", envvar); } if (splitmode >= 0) { context->splitmode = splitmode; } } context->threads_started = 0; context->schunk = cparams.schunk; if (cparams.prefilter != NULL) { context->prefilter = cparams.prefilter; context->preparams = (blosc2_prefilter_params*)my_malloc(sizeof(blosc2_prefilter_params)); BLOSC_ERROR_NULL(context->preparams, NULL); memcpy(context->preparams, cparams.preparams, sizeof(blosc2_prefilter_params)); } if (cparams.udbtune == NULL) { context->udbtune = &BTUNE_DEFAULTS; } else { context->udbtune = cparams.udbtune; } context->codec_params = cparams.codec_params; memcpy(context->filter_params, cparams.filter_params, BLOSC2_MAX_FILTERS * sizeof(void*)); return context; } /* Create a context for decompression */ blosc2_context* blosc2_create_dctx(blosc2_dparams dparams) { blosc2_context* context = (blosc2_context*)my_malloc(sizeof(blosc2_context)); BLOSC_ERROR_NULL(context, NULL); /* Populate the context, using zeros as default values */ memset(context, 0, sizeof(blosc2_context)); context->do_compress = 0; /* Meant for decompression */ context->nthreads = dparams.nthreads; char* envvar = getenv("BLOSC_NTHREADS"); if (envvar != NULL) { long nthreads = strtol(envvar, NULL, 10); if ((nthreads != EINVAL) && (nthreads > 0)) { context->nthreads = (int16_t) nthreads; } } context->new_nthreads = context->nthreads; context->threads_started = 0; context->block_maskout = NULL; context->block_maskout_nitems = 0; context->schunk = dparams.schunk; if (dparams.postfilter != NULL) { context->postfilter = dparams.postfilter; context->postparams = (blosc2_postfilter_params*)my_malloc(sizeof(blosc2_postfilter_params)); BLOSC_ERROR_NULL(context->postparams, NULL); memcpy(context->postparams, dparams.postparams, sizeof(blosc2_postfilter_params)); } return context; } void blosc2_free_ctx(blosc2_context* context) { release_threadpool(context); if (context->serial_context != NULL) { free_thread_context(context->serial_context); } if (context->dict_cdict != NULL) { #ifdef HAVE_ZSTD ZSTD_freeCDict(context->dict_cdict); #endif } if (context->dict_ddict != NULL) { #ifdef HAVE_ZSTD ZSTD_freeDDict(context->dict_ddict); #endif } if (context->btune != NULL) { context->udbtune->btune_free(context); } if (context->prefilter != NULL) { my_free(context->preparams); } if (context->postfilter != NULL) { my_free(context->postparams); } if (context->block_maskout != NULL) { free(context->block_maskout); } my_free(context); } int blosc2_ctx_get_cparams(blosc2_context *ctx, blosc2_cparams *cparams) { cparams->compcode = ctx->compcode; cparams->compcode_meta = ctx->compcode_meta; cparams->clevel = ctx->clevel; cparams->use_dict = ctx->use_dict; cparams->instr_codec = ctx->blosc2_flags & BLOSC2_INSTR_CODEC; cparams->typesize = ctx->typesize; cparams->nthreads = ctx->nthreads; cparams->blocksize = ctx->blocksize; cparams->splitmode = ctx->splitmode; cparams->schunk = ctx->schunk; for (int i = 0; i < BLOSC2_MAX_FILTERS; ++i) { cparams->filters[i] = ctx->filters[i]; cparams->filters_meta[i] = ctx->filters_meta[i]; } cparams->prefilter = ctx->prefilter; cparams->preparams = ctx->preparams; cparams->udbtune = ctx->udbtune; cparams->codec_params = ctx->codec_params; return BLOSC2_ERROR_SUCCESS; } int blosc2_ctx_get_dparams(blosc2_context *ctx, blosc2_dparams *dparams) { dparams->nthreads = ctx->nthreads; dparams->schunk = ctx->schunk; dparams->postfilter = ctx->postfilter; dparams->postparams = ctx->postparams; return BLOSC2_ERROR_SUCCESS; } /* Set a maskout in decompression context */ int blosc2_set_maskout(blosc2_context *ctx, bool *maskout, int nblocks) { if (ctx->block_maskout != NULL) { // Get rid of a possible mask here free(ctx->block_maskout); } bool *maskout_ = malloc(nblocks); BLOSC_ERROR_NULL(maskout_, BLOSC2_ERROR_MEMORY_ALLOC); memcpy(maskout_, maskout, nblocks); ctx->block_maskout = maskout_; ctx->block_maskout_nitems = nblocks; return 0; } /* Create a chunk made of zeros */ int blosc2_chunk_zeros(blosc2_cparams cparams, const int32_t nbytes, void* dest, int32_t destsize) { if (destsize < BLOSC_EXTENDED_HEADER_LENGTH) { BLOSC_TRACE_ERROR("dest buffer is not long enough"); return BLOSC2_ERROR_DATA; } if (nbytes % cparams.typesize) { BLOSC_TRACE_ERROR("nbytes must be a multiple of typesize"); return BLOSC2_ERROR_DATA; } blosc_header header; blosc2_context* context = blosc2_create_cctx(cparams); int error = initialize_context_compression( context, NULL, nbytes, dest, destsize, context->clevel, context->filters, context->filters_meta, context->typesize, context->compcode, context->blocksize, context->new_nthreads, context->nthreads, context->splitmode, context->udbtune, context->btune, context->schunk); if (error <= 0) { blosc2_free_ctx(context); return error; } memset(&header, 0, sizeof(header)); header.version = BLOSC2_VERSION_FORMAT; header.versionlz = BLOSC_BLOSCLZ_VERSION_FORMAT; header.flags = BLOSC_DOSHUFFLE | BLOSC_DOBITSHUFFLE; // extended header header.typesize = context->typesize; header.nbytes = (int32_t)nbytes; header.blocksize = context->blocksize; header.cbytes = BLOSC_EXTENDED_HEADER_LENGTH; header.blosc2_flags = BLOSC2_SPECIAL_ZERO << 4; // mark chunk as all zeros memcpy((uint8_t *)dest, &header, sizeof(header)); blosc2_free_ctx(context); return BLOSC_EXTENDED_HEADER_LENGTH; } /* Create a chunk made of uninitialized values */ int blosc2_chunk_uninit(blosc2_cparams cparams, const int32_t nbytes, void* dest, int32_t destsize) { if (destsize < BLOSC_EXTENDED_HEADER_LENGTH) { BLOSC_TRACE_ERROR("dest buffer is not long enough"); return BLOSC2_ERROR_DATA; } if (nbytes % cparams.typesize) { BLOSC_TRACE_ERROR("nbytes must be a multiple of typesize"); return BLOSC2_ERROR_DATA; } blosc_header header; blosc2_context* context = blosc2_create_cctx(cparams); int error = initialize_context_compression( context, NULL, nbytes, dest, destsize, context->clevel, context->filters, context->filters_meta, context->typesize, context->compcode, context->blocksize, context->new_nthreads, context->nthreads, context->splitmode, context->udbtune, context->btune, context->schunk); if (error <= 0) { blosc2_free_ctx(context); return error; } memset(&header, 0, sizeof(header)); header.version = BLOSC2_VERSION_FORMAT; header.versionlz = BLOSC_BLOSCLZ_VERSION_FORMAT; header.flags = BLOSC_DOSHUFFLE | BLOSC_DOBITSHUFFLE; // extended header header.typesize = context->typesize; header.nbytes = (int32_t)nbytes; header.blocksize = context->blocksize; header.cbytes = BLOSC_EXTENDED_HEADER_LENGTH; header.blosc2_flags = BLOSC2_SPECIAL_UNINIT << 4; // mark chunk as uninitialized memcpy((uint8_t *)dest, &header, sizeof(header)); blosc2_free_ctx(context); return BLOSC_EXTENDED_HEADER_LENGTH; } /* Create a chunk made of nans */ int blosc2_chunk_nans(blosc2_cparams cparams, const int32_t nbytes, void* dest, int32_t destsize) { if (destsize < BLOSC_EXTENDED_HEADER_LENGTH) { BLOSC_TRACE_ERROR("dest buffer is not long enough"); return BLOSC2_ERROR_DATA; } if (nbytes % cparams.typesize) { BLOSC_TRACE_ERROR("nbytes must be a multiple of typesize"); return BLOSC2_ERROR_DATA; } blosc_header header; blosc2_context* context = blosc2_create_cctx(cparams); int error = initialize_context_compression( context, NULL, nbytes, dest, destsize, context->clevel, context->filters, context->filters_meta, context->typesize, context->compcode, context->blocksize, context->new_nthreads, context->nthreads, context->splitmode, context->udbtune, context->btune, context->schunk); if (error <= 0) { blosc2_free_ctx(context); return error; } memset(&header, 0, sizeof(header)); header.version = BLOSC2_VERSION_FORMAT; header.versionlz = BLOSC_BLOSCLZ_VERSION_FORMAT; header.flags = BLOSC_DOSHUFFLE | BLOSC_DOBITSHUFFLE; // extended header header.typesize = context->typesize; header.nbytes = (int32_t)nbytes; header.blocksize = context->blocksize; header.cbytes = BLOSC_EXTENDED_HEADER_LENGTH; header.blosc2_flags = BLOSC2_SPECIAL_NAN << 4; // mark chunk as all NaNs memcpy((uint8_t *)dest, &header, sizeof(header)); blosc2_free_ctx(context); return BLOSC_EXTENDED_HEADER_LENGTH; } /* Create a chunk made of repeated values */ int blosc2_chunk_repeatval(blosc2_cparams cparams, const int32_t nbytes, void* dest, int32_t destsize, const void* repeatval) { uint8_t typesize = cparams.typesize; if (destsize < BLOSC_EXTENDED_HEADER_LENGTH + typesize) { BLOSC_TRACE_ERROR("dest buffer is not long enough"); return BLOSC2_ERROR_DATA; } if (nbytes % cparams.typesize) { BLOSC_TRACE_ERROR("nbytes must be a multiple of typesize"); return BLOSC2_ERROR_DATA; } blosc_header header; blosc2_context* context = blosc2_create_cctx(cparams); int error = initialize_context_compression( context, NULL, nbytes, dest, destsize, context->clevel, context->filters, context->filters_meta, context->typesize, context->compcode, context->blocksize, context->new_nthreads, context->nthreads, context->splitmode, context->udbtune, context->btune, context->schunk); if (error <= 0) { blosc2_free_ctx(context); return error; } memset(&header, 0, sizeof(header)); header.version = BLOSC2_VERSION_FORMAT; header.versionlz = BLOSC_BLOSCLZ_VERSION_FORMAT; header.flags = BLOSC_DOSHUFFLE | BLOSC_DOBITSHUFFLE; // extended header header.typesize = (uint8_t)typesize; header.nbytes = (int32_t)nbytes; header.blocksize = context->blocksize; header.cbytes = BLOSC_EXTENDED_HEADER_LENGTH + (int32_t)typesize; header.blosc2_flags = BLOSC2_SPECIAL_VALUE << 4; // mark chunk as all repeated value memcpy((uint8_t *)dest, &header, sizeof(header)); memcpy((uint8_t *)dest + sizeof(header), repeatval, typesize); blosc2_free_ctx(context); return BLOSC_EXTENDED_HEADER_LENGTH + (uint8_t)typesize; } /* Register filters */ int register_filter_private(blosc2_filter *filter) { BLOSC_ERROR_NULL(filter, BLOSC2_ERROR_INVALID_PARAM); if (g_nfilters == UINT8_MAX) { BLOSC_TRACE_ERROR("Can not register more filters"); return BLOSC2_ERROR_CODEC_SUPPORT; } if (filter->id < BLOSC2_GLOBAL_REGISTERED_FILTERS_START) { BLOSC_TRACE_ERROR("The id must be greater or equal than %d", BLOSC2_GLOBAL_REGISTERED_FILTERS_START); return BLOSC2_ERROR_FAILURE; } /* This condition can never be fulfilled if (filter->id > BLOSC2_USER_REGISTERED_FILTERS_STOP) { BLOSC_TRACE_ERROR("The id must be less than or equal to %d", BLOSC2_USER_REGISTERED_FILTERS_STOP); return BLOSC2_ERROR_FAILURE; } */ // Check if the filter is already registered for (uint64_t i = 0; i < g_nfilters; ++i) { if (g_filters[i].id == filter->id) { BLOSC_TRACE_ERROR("The filter is already registered!"); return BLOSC2_ERROR_FAILURE; } } blosc2_filter *filter_new = &g_filters[g_nfilters++]; memcpy(filter_new, filter, sizeof(blosc2_filter)); return BLOSC2_ERROR_SUCCESS; } int blosc2_register_filter(blosc2_filter *filter) { if (filter->id < BLOSC2_USER_REGISTERED_FILTERS_START) { BLOSC_TRACE_ERROR("The id must be greater or equal to %d", BLOSC2_USER_REGISTERED_FILTERS_START); return BLOSC2_ERROR_FAILURE; } return register_filter_private(filter); } /* Register codecs */ int register_codec_private(blosc2_codec *codec) { BLOSC_ERROR_NULL(codec, BLOSC2_ERROR_INVALID_PARAM); if (g_ncodecs == UINT8_MAX) { BLOSC_TRACE_ERROR("Can not register more codecs"); return BLOSC2_ERROR_CODEC_SUPPORT; } if (codec->compcode < BLOSC2_GLOBAL_REGISTERED_CODECS_START) { BLOSC_TRACE_ERROR("The id must be greater or equal than %d", BLOSC2_GLOBAL_REGISTERED_CODECS_START); return BLOSC2_ERROR_FAILURE; } /* This condition can never be fulfilled if (codec->compcode > BLOSC2_USER_REGISTERED_CODECS_STOP) { BLOSC_TRACE_ERROR("The id must be less or equal to %d", BLOSC2_USER_REGISTERED_CODECS_STOP); return BLOSC2_ERROR_FAILURE; } */ // Check if the code is already registered for (int i = 0; i < g_ncodecs; ++i) { if (g_codecs[i].compcode == codec->compcode) { BLOSC_TRACE_ERROR("The codec is already registered!"); return BLOSC2_ERROR_CODEC_PARAM; } } blosc2_codec *codec_new = &g_codecs[g_ncodecs++]; memcpy(codec_new, codec, sizeof(blosc2_codec)); return BLOSC2_ERROR_SUCCESS; } int blosc2_register_codec(blosc2_codec *codec) { if (codec->compcode < BLOSC2_USER_REGISTERED_CODECS_START) { BLOSC_TRACE_ERROR("The compcode must be greater or equal than %d", BLOSC2_USER_REGISTERED_CODECS_START); return BLOSC2_ERROR_CODEC_PARAM; } return register_codec_private(codec); } int _blosc2_register_io_cb(const blosc2_io_cb *io) { // Check if the io is already registered for (uint64_t i = 0; i < g_nio; ++i) { if (g_io[i].id == io->id) { BLOSC_TRACE_ERROR("The codec is already registered!"); return BLOSC2_ERROR_PLUGIN_IO; } } blosc2_io_cb *io_new = &g_io[g_nio++]; memcpy(io_new, io, sizeof(blosc2_io_cb)); return BLOSC2_ERROR_SUCCESS; } int blosc2_register_io_cb(const blosc2_io_cb *io) { BLOSC_ERROR_NULL(io, BLOSC2_ERROR_INVALID_PARAM); if (g_nio == UINT8_MAX) { BLOSC_TRACE_ERROR("Can not register more codecs"); return BLOSC2_ERROR_PLUGIN_IO; } if (io->id < BLOSC2_IO_REGISTERED) { BLOSC_TRACE_ERROR("The compcode must be greater or equal than %d", BLOSC2_IO_REGISTERED); return BLOSC2_ERROR_PLUGIN_IO; } return _blosc2_register_io_cb(io); } blosc2_io_cb *blosc2_get_io_cb(uint8_t id) { for (uint64_t i = 0; i < g_nio; ++i) { if (g_io[i].id == id) { return &g_io[i]; } } if (id == BLOSC2_IO_FILESYSTEM) { if (_blosc2_register_io_cb(&BLOSC2_IO_CB_DEFAULTS) < 0) { BLOSC_TRACE_ERROR("Error registering the default IO API"); return NULL; } return blosc2_get_io_cb(id); } return NULL; } void blosc2_unidim_to_multidim(uint8_t ndim, int64_t *shape, int64_t i, int64_t *index) { int64_t strides[BLOSC2_MAX_DIM]; if (ndim == 0) { return; } strides[ndim - 1] = 1; for (int j = ndim - 2; j >= 0; --j) { strides[j] = shape[j + 1] * strides[j + 1]; } index[0] = i / strides[0]; for (int j = 1; j < ndim; ++j) { index[j] = (i % strides[j - 1]) / strides[j]; } } void blosc2_multidim_to_unidim(const int64_t *index, int8_t ndim, const int64_t *strides, int64_t *i) { *i = 0; for (int j = 0; j < ndim; ++j) { *i += index[j] * strides[j]; } } zmat-0.9.9/src/blosc2/blosc/shuffle-generic.h0000644000175200007730000000747314515254731021246 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /********************************************************************* Generic (non-hardware-accelerated) shuffle/unshuffle routines. These are used when hardware-accelerated functions aren't available for a particular platform; they are also used by the hardware- accelerated functions to handle any remaining elements in a block which isn't a multiple of the hardware's vector size. **********************************************************************/ #ifndef SHUFFLE_GENERIC_H #define SHUFFLE_GENERIC_H #include "blosc2/blosc2-common.h" #include #ifdef __cplusplus extern "C" { #endif /** Generic (non-hardware-accelerated) shuffle routine. This is the pure element-copying nested loop. It is used by the generic shuffle implementation and also by the vectorized shuffle implementations to process any remaining elements in a block which is not a multiple of (type_size * vector_size). */ inline static void shuffle_generic_inline(const int32_t type_size, const int32_t vectorizable_blocksize, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest) { int32_t i, j; /* Calculate the number of elements in the block. */ const int32_t neblock_quot = blocksize / type_size; const int32_t neblock_rem = blocksize % type_size; const int32_t vectorizable_elements = vectorizable_blocksize / type_size; /* Non-optimized shuffle */ for (j = 0; j < type_size; j++) { for (i = vectorizable_elements; i < (int32_t)neblock_quot; i++) { _dest[j * neblock_quot + i] = _src[i * type_size + j]; } } /* Copy any leftover bytes in the block without shuffling them. */ memcpy(_dest + (blocksize - neblock_rem), _src + (blocksize - neblock_rem), neblock_rem); } /** Generic (non-hardware-accelerated) unshuffle routine. This is the pure element-copying nested loop. It is used by the generic unshuffle implementation and also by the vectorized unshuffle implementations to process any remaining elements in a block which is not a multiple of (type_size * vector_size). */ inline static void unshuffle_generic_inline(const int32_t type_size, const int32_t vectorizable_blocksize, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest) { int32_t i, j; /* Calculate the number of elements in the block. */ const int32_t neblock_quot = blocksize / type_size; const int32_t neblock_rem = blocksize % type_size; const int32_t vectorizable_elements = vectorizable_blocksize / type_size; /* Non-optimized unshuffle */ for (i = vectorizable_elements; i < (int32_t)neblock_quot; i++) { for (j = 0; j < type_size; j++) { _dest[i * type_size + j] = _src[j * neblock_quot + i]; } } /* Copy any leftover bytes in the block without unshuffling them. */ memcpy(_dest + (blocksize - neblock_rem), _src + (blocksize - neblock_rem), neblock_rem); } /** Generic (non-hardware-accelerated) shuffle routine. */ BLOSC_NO_EXPORT void shuffle_generic(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest); /** Generic (non-hardware-accelerated) unshuffle routine. */ BLOSC_NO_EXPORT void unshuffle_generic(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest); #ifdef __cplusplus } #endif #endif /* SHUFFLE_GENERIC_H */ zmat-0.9.9/src/blosc2/blosc/shuffle-sse2.c0000644000175200007730000006302314515254731020472 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include "shuffle-generic.h" #include "shuffle-sse2.h" /* Make sure SSE2 is available for the compilation target and compiler. */ #if defined(__SSE2__) #include /* The next is useful for debugging purposes */ #if 0 #include #include static void printxmm(__m128i xmm0) { uint8_t buf[16]; ((__m128i *)buf)[0] = xmm0; printf("%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x\n", buf[0], buf[1], buf[2], buf[3], buf[4], buf[5], buf[6], buf[7], buf[8], buf[9], buf[10], buf[11], buf[12], buf[13], buf[14], buf[15]); } #endif /* Routine optimized for shuffling a buffer for a type size of 2 bytes. */ static void shuffle2_sse2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 2; int32_t j; int k; uint8_t* dest_for_jth_element; __m128i xmm0[2], xmm1[2]; for (j = 0; j < vectorizable_elements; j += sizeof(__m128i)) { /* Fetch 16 elements (32 bytes) then transpose bytes, words and double words. */ for (k = 0; k < 2; k++) { xmm0[k] = _mm_loadu_si128((__m128i*)(src + (j * bytesoftype) + (k * sizeof(__m128i)))); xmm0[k] = _mm_shufflelo_epi16(xmm0[k], 0xd8); xmm0[k] = _mm_shufflehi_epi16(xmm0[k], 0xd8); xmm0[k] = _mm_shuffle_epi32(xmm0[k], 0xd8); xmm1[k] = _mm_shuffle_epi32(xmm0[k], 0x4e); xmm0[k] = _mm_unpacklo_epi8(xmm0[k], xmm1[k]); xmm0[k] = _mm_shuffle_epi32(xmm0[k], 0xd8); xmm1[k] = _mm_shuffle_epi32(xmm0[k], 0x4e); xmm0[k] = _mm_unpacklo_epi16(xmm0[k], xmm1[k]); xmm0[k] = _mm_shuffle_epi32(xmm0[k], 0xd8); } /* Transpose quad words */ for (k = 0; k < 1; k++) { xmm1[k * 2] = _mm_unpacklo_epi64(xmm0[k], xmm0[k + 1]); xmm1[k * 2 + 1] = _mm_unpackhi_epi64(xmm0[k], xmm0[k + 1]); } /* Store the result vectors */ dest_for_jth_element = dest + j; for (k = 0; k < 2; k++) { _mm_storeu_si128((__m128i*)(dest_for_jth_element + (k * total_elements)), xmm1[k]); } } } /* Routine optimized for shuffling a buffer for a type size of 4 bytes. */ static void shuffle4_sse2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 4; int32_t i; int j; uint8_t* dest_for_ith_element; __m128i xmm0[4], xmm1[4]; for (i = 0; i < vectorizable_elements; i += sizeof(__m128i)) { /* Fetch 16 elements (64 bytes) then transpose bytes and words. */ for (j = 0; j < 4; j++) { xmm0[j] = _mm_loadu_si128((__m128i*)(src + (i * bytesoftype) + (j * sizeof(__m128i)))); xmm1[j] = _mm_shuffle_epi32(xmm0[j], 0xd8); xmm0[j] = _mm_shuffle_epi32(xmm0[j], 0x8d); xmm0[j] = _mm_unpacklo_epi8(xmm1[j], xmm0[j]); xmm1[j] = _mm_shuffle_epi32(xmm0[j], 0x04e); xmm0[j] = _mm_unpacklo_epi16(xmm0[j], xmm1[j]); } /* Transpose double words */ for (j = 0; j < 2; j++) { xmm1[j * 2] = _mm_unpacklo_epi32(xmm0[j * 2], xmm0[j * 2 + 1]); xmm1[j * 2 + 1] = _mm_unpackhi_epi32(xmm0[j * 2], xmm0[j * 2 + 1]); } /* Transpose quad words */ for (j = 0; j < 2; j++) { xmm0[j * 2] = _mm_unpacklo_epi64(xmm1[j], xmm1[j + 2]); xmm0[j * 2 + 1] = _mm_unpackhi_epi64(xmm1[j], xmm1[j + 2]); } /* Store the result vectors */ dest_for_ith_element = dest + i; for (j = 0; j < 4; j++) { _mm_storeu_si128((__m128i*)(dest_for_ith_element + (j * total_elements)), xmm0[j]); } } } /* Routine optimized for shuffling a buffer for a type size of 8 bytes. */ static void shuffle8_sse2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 8; int32_t j; int k, l; uint8_t* dest_for_jth_element; __m128i xmm0[8], xmm1[8]; for (j = 0; j < vectorizable_elements; j += sizeof(__m128i)) { /* Fetch 16 elements (128 bytes) then transpose bytes. */ for (k = 0; k < 8; k++) { xmm0[k] = _mm_loadu_si128((__m128i*)(src + (j * bytesoftype) + (k * sizeof(__m128i)))); xmm1[k] = _mm_shuffle_epi32(xmm0[k], 0x4e); xmm1[k] = _mm_unpacklo_epi8(xmm0[k], xmm1[k]); } /* Transpose words */ for (k = 0, l = 0; k < 4; k++, l += 2) { xmm0[k * 2] = _mm_unpacklo_epi16(xmm1[l], xmm1[l + 1]); xmm0[k * 2 + 1] = _mm_unpackhi_epi16(xmm1[l], xmm1[l + 1]); } /* Transpose double words */ for (k = 0, l = 0; k < 4; k++, l++) { if (k == 2) l += 2; xmm1[k * 2] = _mm_unpacklo_epi32(xmm0[l], xmm0[l + 2]); xmm1[k * 2 + 1] = _mm_unpackhi_epi32(xmm0[l], xmm0[l + 2]); } /* Transpose quad words */ for (k = 0; k < 4; k++) { xmm0[k * 2] = _mm_unpacklo_epi64(xmm1[k], xmm1[k + 4]); xmm0[k * 2 + 1] = _mm_unpackhi_epi64(xmm1[k], xmm1[k + 4]); } /* Store the result vectors */ dest_for_jth_element = dest + j; for (k = 0; k < 8; k++) { _mm_storeu_si128((__m128i*)(dest_for_jth_element + (k * total_elements)), xmm0[k]); } } } /* Routine optimized for shuffling a buffer for a type size of 16 bytes. */ static void shuffle16_sse2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 16; int32_t j; int k, l; uint8_t* dest_for_jth_element; __m128i xmm0[16], xmm1[16]; for (j = 0; j < vectorizable_elements; j += sizeof(__m128i)) { /* Fetch 16 elements (256 bytes). */ for (k = 0; k < 16; k++) { xmm0[k] = _mm_loadu_si128((__m128i*)(src + (j * bytesoftype) + (k * sizeof(__m128i)))); } /* Transpose bytes */ for (k = 0, l = 0; k < 8; k++, l += 2) { xmm1[k * 2] = _mm_unpacklo_epi8(xmm0[l], xmm0[l + 1]); xmm1[k * 2 + 1] = _mm_unpackhi_epi8(xmm0[l], xmm0[l + 1]); } /* Transpose words */ for (k = 0, l = -2; k < 8; k++, l++) { if ((k % 2) == 0) l += 2; xmm0[k * 2] = _mm_unpacklo_epi16(xmm1[l], xmm1[l + 2]); xmm0[k * 2 + 1] = _mm_unpackhi_epi16(xmm1[l], xmm1[l + 2]); } /* Transpose double words */ for (k = 0, l = -4; k < 8; k++, l++) { if ((k % 4) == 0) l += 4; xmm1[k * 2] = _mm_unpacklo_epi32(xmm0[l], xmm0[l + 4]); xmm1[k * 2 + 1] = _mm_unpackhi_epi32(xmm0[l], xmm0[l + 4]); } /* Transpose quad words */ for (k = 0; k < 8; k++) { xmm0[k * 2] = _mm_unpacklo_epi64(xmm1[k], xmm1[k + 8]); xmm0[k * 2 + 1] = _mm_unpackhi_epi64(xmm1[k], xmm1[k + 8]); } /* Store the result vectors */ dest_for_jth_element = dest + j; for (k = 0; k < 16; k++) { _mm_storeu_si128((__m128i*)(dest_for_jth_element + (k * total_elements)), xmm0[k]); } } } /* Routine optimized for shuffling a buffer for a type size larger than 16 bytes. */ static void shuffle16_tiled_sse2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements, const int32_t bytesoftype) { int32_t j; const int32_t vecs_per_el_rem = bytesoftype % (int32_t)sizeof(__m128i); int k, l; uint8_t* dest_for_jth_element; __m128i xmm0[16], xmm1[16]; for (j = 0; j < vectorizable_elements; j += sizeof(__m128i)) { /* Advance the offset into the type by the vector size (in bytes), unless this is the initial iteration and the type size is not a multiple of the vector size. In that case, only advance by the number of bytes necessary so that the number of remaining bytes in the type will be a multiple of the vector size. */ int32_t offset_into_type; for (offset_into_type = 0; offset_into_type < bytesoftype; offset_into_type += (offset_into_type == 0 && vecs_per_el_rem > 0 ? vecs_per_el_rem : (int32_t)sizeof(__m128i))) { /* Fetch elements in groups of 256 bytes */ const uint8_t* const src_with_offset = src + offset_into_type; for (k = 0; k < 16; k++) { xmm0[k] = _mm_loadu_si128((__m128i*)(src_with_offset + (j + k) * bytesoftype)); } /* Transpose bytes */ for (k = 0, l = 0; k < 8; k++, l += 2) { xmm1[k * 2] = _mm_unpacklo_epi8(xmm0[l], xmm0[l + 1]); xmm1[k * 2 + 1] = _mm_unpackhi_epi8(xmm0[l], xmm0[l + 1]); } /* Transpose words */ for (k = 0, l = -2; k < 8; k++, l++) { if ((k % 2) == 0) l += 2; xmm0[k * 2] = _mm_unpacklo_epi16(xmm1[l], xmm1[l + 2]); xmm0[k * 2 + 1] = _mm_unpackhi_epi16(xmm1[l], xmm1[l + 2]); } /* Transpose double words */ for (k = 0, l = -4; k < 8; k++, l++) { if ((k % 4) == 0) l += 4; xmm1[k * 2] = _mm_unpacklo_epi32(xmm0[l], xmm0[l + 4]); xmm1[k * 2 + 1] = _mm_unpackhi_epi32(xmm0[l], xmm0[l + 4]); } /* Transpose quad words */ for (k = 0; k < 8; k++) { xmm0[k * 2] = _mm_unpacklo_epi64(xmm1[k], xmm1[k + 8]); xmm0[k * 2 + 1] = _mm_unpackhi_epi64(xmm1[k], xmm1[k + 8]); } /* Store the result vectors */ dest_for_jth_element = dest + j; for (k = 0; k < 16; k++) { _mm_storeu_si128((__m128i*)(dest_for_jth_element + (total_elements * (offset_into_type + k))), xmm0[k]); } } } } /* Routine optimized for unshuffling a buffer for a type size of 2 bytes. */ static void unshuffle2_sse2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 2; int32_t i; int j; __m128i xmm0[2], xmm1[2]; for (i = 0; i < vectorizable_elements; i += sizeof(__m128i)) { /* Load 16 elements (32 bytes) into 2 XMM registers. */ const uint8_t* const src_for_ith_element = src + i; for (j = 0; j < 2; j++) { xmm0[j] = _mm_loadu_si128((__m128i*)(src_for_ith_element + (j * total_elements))); } /* Shuffle bytes */ /* Compute the low 32 bytes */ xmm1[0] = _mm_unpacklo_epi8(xmm0[0], xmm0[1]); /* Compute the hi 32 bytes */ xmm1[1] = _mm_unpackhi_epi8(xmm0[0], xmm0[1]); /* Store the result vectors in proper order */ _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (0 * sizeof(__m128i))), xmm1[0]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (1 * sizeof(__m128i))), xmm1[1]); } } /* Routine optimized for unshuffling a buffer for a type size of 4 bytes. */ static void unshuffle4_sse2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 4; int32_t i; int j; __m128i xmm0[4], xmm1[4]; for (i = 0; i < vectorizable_elements; i += sizeof(__m128i)) { /* Load 16 elements (64 bytes) into 4 XMM registers. */ const uint8_t* const src_for_ith_element = src + i; for (j = 0; j < 4; j++) { xmm0[j] = _mm_loadu_si128((__m128i*)(src_for_ith_element + (j * total_elements))); } /* Shuffle bytes */ for (j = 0; j < 2; j++) { /* Compute the low 32 bytes */ xmm1[j] = _mm_unpacklo_epi8(xmm0[j * 2], xmm0[j * 2 + 1]); /* Compute the hi 32 bytes */ xmm1[2 + j] = _mm_unpackhi_epi8(xmm0[j * 2], xmm0[j * 2 + 1]); } /* Shuffle 2-byte words */ for (j = 0; j < 2; j++) { /* Compute the low 32 bytes */ xmm0[j] = _mm_unpacklo_epi16(xmm1[j * 2], xmm1[j * 2 + 1]); /* Compute the hi 32 bytes */ xmm0[2 + j] = _mm_unpackhi_epi16(xmm1[j * 2], xmm1[j * 2 + 1]); } /* Store the result vectors in proper order */ _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (0 * sizeof(__m128i))), xmm0[0]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (1 * sizeof(__m128i))), xmm0[2]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (2 * sizeof(__m128i))), xmm0[1]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (3 * sizeof(__m128i))), xmm0[3]); } } /* Routine optimized for unshuffling a buffer for a type size of 8 bytes. */ static void unshuffle8_sse2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 8; int32_t i; int j; __m128i xmm0[8], xmm1[8]; for (i = 0; i < vectorizable_elements; i += sizeof(__m128i)) { /* Load 16 elements (128 bytes) into 8 XMM registers. */ const uint8_t* const src_for_ith_element = src + i; for (j = 0; j < 8; j++) { xmm0[j] = _mm_loadu_si128((__m128i*)(src_for_ith_element + (j * total_elements))); } /* Shuffle bytes */ for (j = 0; j < 4; j++) { /* Compute the low 32 bytes */ xmm1[j] = _mm_unpacklo_epi8(xmm0[j * 2], xmm0[j * 2 + 1]); /* Compute the hi 32 bytes */ xmm1[4 + j] = _mm_unpackhi_epi8(xmm0[j * 2], xmm0[j * 2 + 1]); } /* Shuffle 2-byte words */ for (j = 0; j < 4; j++) { /* Compute the low 32 bytes */ xmm0[j] = _mm_unpacklo_epi16(xmm1[j * 2], xmm1[j * 2 + 1]); /* Compute the hi 32 bytes */ xmm0[4 + j] = _mm_unpackhi_epi16(xmm1[j * 2], xmm1[j * 2 + 1]); } /* Shuffle 4-byte dwords */ for (j = 0; j < 4; j++) { /* Compute the low 32 bytes */ xmm1[j] = _mm_unpacklo_epi32(xmm0[j * 2], xmm0[j * 2 + 1]); /* Compute the hi 32 bytes */ xmm1[4 + j] = _mm_unpackhi_epi32(xmm0[j * 2], xmm0[j * 2 + 1]); } /* Store the result vectors in proper order */ _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (0 * sizeof(__m128i))), xmm1[0]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (1 * sizeof(__m128i))), xmm1[4]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (2 * sizeof(__m128i))), xmm1[2]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (3 * sizeof(__m128i))), xmm1[6]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (4 * sizeof(__m128i))), xmm1[1]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (5 * sizeof(__m128i))), xmm1[5]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (6 * sizeof(__m128i))), xmm1[3]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (7 * sizeof(__m128i))), xmm1[7]); } } /* Routine optimized for unshuffling a buffer for a type size of 16 bytes. */ static void unshuffle16_sse2(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 16; int32_t i; int j; __m128i xmm1[16], xmm2[16]; for (i = 0; i < vectorizable_elements; i += sizeof(__m128i)) { /* Load 16 elements (256 bytes) into 16 XMM registers. */ const uint8_t* const src_for_ith_element = src + i; for (j = 0; j < 16; j++) { xmm1[j] = _mm_loadu_si128((__m128i*)(src_for_ith_element + (j * total_elements))); } /* Shuffle bytes */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ xmm2[j] = _mm_unpacklo_epi8(xmm1[j * 2], xmm1[j * 2 + 1]); /* Compute the hi 32 bytes */ xmm2[8 + j] = _mm_unpackhi_epi8(xmm1[j * 2], xmm1[j * 2 + 1]); } /* Shuffle 2-byte words */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ xmm1[j] = _mm_unpacklo_epi16(xmm2[j * 2], xmm2[j * 2 + 1]); /* Compute the hi 32 bytes */ xmm1[8 + j] = _mm_unpackhi_epi16(xmm2[j * 2], xmm2[j * 2 + 1]); } /* Shuffle 4-byte dwords */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ xmm2[j] = _mm_unpacklo_epi32(xmm1[j * 2], xmm1[j * 2 + 1]); /* Compute the hi 32 bytes */ xmm2[8 + j] = _mm_unpackhi_epi32(xmm1[j * 2], xmm1[j * 2 + 1]); } /* Shuffle 8-byte qwords */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ xmm1[j] = _mm_unpacklo_epi64(xmm2[j * 2], xmm2[j * 2 + 1]); /* Compute the hi 32 bytes */ xmm1[8 + j] = _mm_unpackhi_epi64(xmm2[j * 2], xmm2[j * 2 + 1]); } /* Store the result vectors in proper order */ _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (0 * sizeof(__m128i))), xmm1[0]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (1 * sizeof(__m128i))), xmm1[8]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (2 * sizeof(__m128i))), xmm1[4]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (3 * sizeof(__m128i))), xmm1[12]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (4 * sizeof(__m128i))), xmm1[2]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (5 * sizeof(__m128i))), xmm1[10]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (6 * sizeof(__m128i))), xmm1[6]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (7 * sizeof(__m128i))), xmm1[14]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (8 * sizeof(__m128i))), xmm1[1]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (9 * sizeof(__m128i))), xmm1[9]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (10 * sizeof(__m128i))), xmm1[5]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (11 * sizeof(__m128i))), xmm1[13]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (12 * sizeof(__m128i))), xmm1[3]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (13 * sizeof(__m128i))), xmm1[11]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (14 * sizeof(__m128i))), xmm1[7]); _mm_storeu_si128((__m128i*)(dest + (i * bytesoftype) + (15 * sizeof(__m128i))), xmm1[15]); } } /* Routine optimized for unshuffling a buffer for a type size larger than 16 bytes. */ static void unshuffle16_tiled_sse2(uint8_t* const dest, const uint8_t* const orig, const int32_t vectorizable_elements, const int32_t total_elements, const int32_t bytesoftype) { int32_t i; const int32_t vecs_per_el_rem = bytesoftype % (int32_t)sizeof(__m128i); int j; uint8_t* dest_with_offset; __m128i xmm1[16], xmm2[16]; /* The unshuffle loops are inverted (compared to shuffle_tiled16_sse2) to optimize cache utilization. */ int32_t offset_into_type; for (offset_into_type = 0; offset_into_type < bytesoftype; offset_into_type += (offset_into_type == 0 && vecs_per_el_rem > 0 ? vecs_per_el_rem : (int32_t)sizeof(__m128i))) { for (i = 0; i < vectorizable_elements; i += sizeof(__m128i)) { /* Load the first 128 bytes in 16 XMM registers */ const uint8_t* const src_for_ith_element = orig + i; for (j = 0; j < 16; j++) { xmm1[j] = _mm_loadu_si128((__m128i*)(src_for_ith_element + (total_elements * (offset_into_type + j)))); } /* Shuffle bytes */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ xmm2[j] = _mm_unpacklo_epi8(xmm1[j * 2], xmm1[j * 2 + 1]); /* Compute the hi 32 bytes */ xmm2[8 + j] = _mm_unpackhi_epi8(xmm1[j * 2], xmm1[j * 2 + 1]); } /* Shuffle 2-byte words */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ xmm1[j] = _mm_unpacklo_epi16(xmm2[j * 2], xmm2[j * 2 + 1]); /* Compute the hi 32 bytes */ xmm1[8 + j] = _mm_unpackhi_epi16(xmm2[j * 2], xmm2[j * 2 + 1]); } /* Shuffle 4-byte dwords */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ xmm2[j] = _mm_unpacklo_epi32(xmm1[j * 2], xmm1[j * 2 + 1]); /* Compute the hi 32 bytes */ xmm2[8 + j] = _mm_unpackhi_epi32(xmm1[j * 2], xmm1[j * 2 + 1]); } /* Shuffle 8-byte qwords */ for (j = 0; j < 8; j++) { /* Compute the low 32 bytes */ xmm1[j] = _mm_unpacklo_epi64(xmm2[j * 2], xmm2[j * 2 + 1]); /* Compute the hi 32 bytes */ xmm1[8 + j] = _mm_unpackhi_epi64(xmm2[j * 2], xmm2[j * 2 + 1]); } /* Store the result vectors in proper order */ dest_with_offset = dest + offset_into_type; _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 0) * bytesoftype), xmm1[0]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 1) * bytesoftype), xmm1[8]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 2) * bytesoftype), xmm1[4]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 3) * bytesoftype), xmm1[12]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 4) * bytesoftype), xmm1[2]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 5) * bytesoftype), xmm1[10]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 6) * bytesoftype), xmm1[6]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 7) * bytesoftype), xmm1[14]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 8) * bytesoftype), xmm1[1]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 9) * bytesoftype), xmm1[9]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 10) * bytesoftype), xmm1[5]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 11) * bytesoftype), xmm1[13]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 12) * bytesoftype), xmm1[3]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 13) * bytesoftype), xmm1[11]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 14) * bytesoftype), xmm1[7]); _mm_storeu_si128((__m128i*)(dest_with_offset + (i + 15) * bytesoftype), xmm1[15]); } } } /* Shuffle a block. This can never fail. */ void shuffle_sse2(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest) { const int32_t vectorized_chunk_size = bytesoftype * (int32_t)sizeof(__m128i); /* If the blocksize is not a multiple of both the typesize and the vector size, round the blocksize down to the next value which is a multiple of both. The vectorized shuffle can be used for that portion of the data, and the naive implementation can be used for the remaining portion. */ const int32_t vectorizable_bytes = blocksize - (blocksize % vectorized_chunk_size); const int32_t vectorizable_elements = vectorizable_bytes / bytesoftype; const int32_t total_elements = blocksize / bytesoftype; /* If the block size is too small to be vectorized, use the generic implementation. */ if (blocksize < vectorized_chunk_size) { shuffle_generic(bytesoftype, blocksize, _src, _dest); return; } /* Optimized shuffle implementations */ switch (bytesoftype) { case 2: shuffle2_sse2(_dest, _src, vectorizable_elements, total_elements); break; case 4: shuffle4_sse2(_dest, _src, vectorizable_elements, total_elements); break; case 8: shuffle8_sse2(_dest, _src, vectorizable_elements, total_elements); break; case 16: shuffle16_sse2(_dest, _src, vectorizable_elements, total_elements); break; default: if (bytesoftype > (int32_t)sizeof(__m128i)) { shuffle16_tiled_sse2(_dest, _src, vectorizable_elements, total_elements, bytesoftype); } else { /* Non-optimized shuffle */ shuffle_generic(bytesoftype, blocksize, _src, _dest); /* The non-optimized function covers the whole buffer, so we're done processing here. */ return; } } /* If the buffer had any bytes at the end which couldn't be handled by the vectorized implementations, use the non-optimized version to finish them up. */ if (vectorizable_bytes < blocksize) { shuffle_generic_inline(bytesoftype, vectorizable_bytes, blocksize, _src, _dest); } } /* Unshuffle a block. This can never fail. */ void unshuffle_sse2(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest) { const int32_t vectorized_chunk_size = bytesoftype * (int32_t)sizeof(__m128i); /* If the blocksize is not a multiple of both the typesize and the vector size, round the blocksize down to the next value which is a multiple of both. The vectorized unshuffle can be used for that portion of the data, and the naive implementation can be used for the remaining portion. */ const int32_t vectorizable_bytes = blocksize - (blocksize % vectorized_chunk_size); const int32_t vectorizable_elements = vectorizable_bytes / bytesoftype; const int32_t total_elements = blocksize / bytesoftype; /* If the block size is too small to be vectorized, use the generic implementation. */ if (blocksize < vectorized_chunk_size) { unshuffle_generic(bytesoftype, blocksize, _src, _dest); return; } /* Optimized unshuffle implementations */ switch (bytesoftype) { case 2: unshuffle2_sse2(_dest, _src, vectorizable_elements, total_elements); break; case 4: unshuffle4_sse2(_dest, _src, vectorizable_elements, total_elements); break; case 8: unshuffle8_sse2(_dest, _src, vectorizable_elements, total_elements); break; case 16: unshuffle16_sse2(_dest, _src, vectorizable_elements, total_elements); break; default: if (bytesoftype > (int32_t)sizeof(__m128i)) { unshuffle16_tiled_sse2(_dest, _src, vectorizable_elements, total_elements, bytesoftype); } else { /* Non-optimized unshuffle */ unshuffle_generic(bytesoftype, blocksize, _src, _dest); /* The non-optimized function covers the whole buffer, so we're done processing here. */ return; } } /* If the buffer had any bytes at the end which couldn't be handled by the vectorized implementations, use the non-optimized version to finish them up. */ if (vectorizable_bytes < blocksize) { unshuffle_generic_inline(bytesoftype, vectorizable_bytes, blocksize, _src, _dest); } } #endif /* defined(__SSE2__) */ zmat-0.9.9/src/blosc2/blosc/delta.c0000644000175200007730000001101114515254731017243 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include #include "delta.h" /* Apply the delta filters to src. This can never fail. */ void delta_encoder(const uint8_t* dref, int32_t offset, int32_t nbytes, int32_t typesize, const uint8_t* src, uint8_t* dest) { int32_t i; if (offset == 0) { /* This is the reference block, use delta coding in elements */ switch (typesize) { case 1: dest[0] = dref[0]; for (i = 1; i < nbytes; i++) { dest[i] = src[i] ^ dref[i-1]; } break; case 2: ((uint16_t *)dest)[0] = ((uint16_t *)dref)[0]; for (i = 1; i < nbytes / 2; i++) { ((uint16_t *)dest)[i] = ((uint16_t *)src)[i] ^ ((uint16_t *)dref)[i-1]; } break; case 4: ((uint32_t *)dest)[0] = ((uint32_t *)dref)[0]; for (i = 1; i < nbytes / 4; i++) { ((uint32_t *)dest)[i] = ((uint32_t *)src)[i] ^ ((uint32_t *)dref)[i-1]; } break; case 8: ((uint64_t *)dest)[0] = ((uint64_t *)dref)[0]; for (i = 1; i < nbytes / 8; i++) { ((uint64_t *)dest)[i] = ((uint64_t *)src)[i] ^ ((uint64_t *)dref)[i-1]; } break; default: if ((typesize % 8) == 0) { delta_encoder(dref, offset, nbytes, 8, src, dest); } else { delta_encoder(dref, offset, nbytes, 1, src, dest); } } } else { /* Use delta coding wrt reference block */ switch (typesize) { case 1: for (i = 0; i < nbytes; i++) { dest[i] = src[i] ^ dref[i]; } break; case 2: for (i = 0; i < nbytes / 2; i++) { ((uint16_t *) dest)[i] = ((uint16_t *) src)[i] ^ ((uint16_t *) dref)[i]; } break; case 4: for (i = 0; i < nbytes / 4; i++) { ((uint32_t *) dest)[i] = ((uint32_t *) src)[i] ^ ((uint32_t *) dref)[i]; } break; case 8: for (i = 0; i < nbytes / 8; i++) { ((uint64_t *) dest)[i] = ((uint64_t *) src)[i] ^ ((uint64_t *) dref)[i]; } break; default: if ((typesize % 8) == 0) { delta_encoder(dref, offset, nbytes, 8, src, dest); } else { delta_encoder(dref, offset, nbytes, 1, src, dest); } } } } /* Undo the delta filter in dest. This can never fail. */ void delta_decoder(const uint8_t* dref, int32_t offset, int32_t nbytes, int32_t typesize, uint8_t* dest) { int32_t i; if (offset == 0) { /* Decode delta for the reference block */ switch (typesize) { case 1: for (i = 1; i < nbytes; i++) { dest[i] ^= dref[i-1]; } break; case 2: for (i = 1; i < nbytes / 2; i++) { ((uint16_t *)dest)[i] ^= ((uint16_t *)dref)[i-1]; } break; case 4: for (i = 1; i < nbytes / 4; i++) { ((uint32_t *)dest)[i] ^= ((uint32_t *)dref)[i-1]; } break; case 8: for (i = 1; i < nbytes / 8; i++) { ((uint64_t *)dest)[i] ^= ((uint64_t *)dref)[i-1]; } break; default: if ((typesize % 8) == 0) { delta_decoder(dref, offset, nbytes, 8, dest); } else { delta_decoder(dref, offset, nbytes, 1, dest); } } } else { /* Decode delta for the non-reference blocks */ switch (typesize) { case 1: for (i = 0; i < nbytes; i++) { dest[i] ^= dref[i]; } break; case 2: for (i = 0; i < nbytes / 2; i++) { ((uint16_t *)dest)[i] ^= ((uint16_t *)dref)[i]; } break; case 4: for (i = 0; i < nbytes / 4; i++) { ((uint32_t *)dest)[i] ^= ((uint32_t *)dref)[i]; } break; case 8: for (i = 0; i < nbytes / 8; i++) { ((uint64_t *)dest)[i] ^= ((uint64_t *)dref)[i]; } break; default: if ((typesize % 8) == 0) { delta_decoder(dref, offset, nbytes, 8, dest); } else { delta_decoder(dref, offset, nbytes, 1, dest); } } } } zmat-0.9.9/src/blosc2/blosc/bitshuffle-neon.c0000644000175200007730000014067114515254731021261 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include "bitshuffle-generic.h" #include "bitshuffle-neon.h" /* Make sure NEON is available for the compilation target and compiler. */ #if defined(__ARM_NEON) #include /* The next is useful for debugging purposes */ #if 0 #include #include static void printmem(uint8_t* buf) { printf("%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,\n", buf[0], buf[1], buf[2], buf[3], buf[4], buf[5], buf[6], buf[7], buf[8], buf[9], buf[10], buf[11], buf[12], buf[13], buf[14], buf[15]); } #endif /* Routine optimized for bit-shuffling a buffer for a type size of 1 byte. */ static void bitshuffle1_neon(void* src, void* dest, const size_t size, const size_t elem_size) { uint16x8_t x0; size_t i, j, k; uint8x8_t lo_x, hi_x, lo, hi; const int8_t __attribute__ ((aligned (16))) xr[8] = {0, 1, 2, 3, 4, 5, 6, 7}; uint8x8_t mask_and = vdup_n_u8(0x01); int8x8_t mask_shift = vld1_s8(xr); for (i = 0, k = 0; i < size * elem_size; i += 16, k++) { /* Load 16-byte groups */ x0 = vld1q_u8(src + k * 16); /* Split in 8-bytes grops */ lo_x = vget_low_u8(x0); hi_x = vget_high_u8(x0); for (j = 0; j < 8; j++) { /* Create mask from the most significant bit of each 8-bit element */ lo = vand_u8(lo_x, mask_and); lo = vshl_u8(lo, mask_shift); hi = vand_u8(hi_x, mask_and); hi = vshl_u8(hi, mask_shift); lo = vpadd_u8(lo, lo); lo = vpadd_u8(lo, lo); lo = vpadd_u8(lo, lo); hi = vpadd_u8(hi, hi); hi = vpadd_u8(hi, hi); hi = vpadd_u8(hi, hi); /* Shift packed 8-bit */ lo_x = vshr_n_u8(lo_x, 1); hi_x = vshr_n_u8(hi_x, 1); /* Store the created mask to the destination vector */ vst1_lane_u8(dest + 2 * k + j * size * elem_size / (8 * elem_size), lo, 0); vst1_lane_u8(dest + 2 * k + 1 + j * size * elem_size / (8 * elem_size), hi, 0); } } } /* Routine optimized for bit-shuffling a buffer for a type size of 2 bytes. */ static void bitshuffle2_neon(void* src, void* dest, const size_t size, const size_t elem_size) { uint8x16x2_t x0; size_t i, j, k; uint8x8_t lo_x[2], hi_x[2], lo[2], hi[2]; const int8_t __attribute__ ((aligned (16))) xr[8] = {0, 1, 2, 3, 4, 5, 6, 7}; uint8x8_t mask_and = vdup_n_u8(0x01); int8x8_t mask_shift = vld1_s8(xr); for (i = 0, k = 0; i < size * elem_size; i += 32, k++) { /* Load 32-byte groups */ x0 = vld2q_u8(src + i); /* Split in 8-bytes grops */ lo_x[0] = vget_low_u8(x0.val[0]); hi_x[0] = vget_high_u8(x0.val[0]); lo_x[1] = vget_low_u8(x0.val[1]); hi_x[1] = vget_high_u8(x0.val[1]); for (j = 0; j < 8; j++) { /* Create mask from the most significant bit of each 8-bit element */ lo[0] = vand_u8(lo_x[0], mask_and); lo[0] = vshl_u8(lo[0], mask_shift); lo[1] = vand_u8(lo_x[1], mask_and); lo[1] = vshl_u8(lo[1], mask_shift); hi[0] = vand_u8(hi_x[0], mask_and); hi[0] = vshl_u8(hi[0], mask_shift); hi[1] = vand_u8(hi_x[1], mask_and); hi[1] = vshl_u8(hi[1], mask_shift); lo[0] = vpadd_u8(lo[0], lo[0]); lo[0] = vpadd_u8(lo[0], lo[0]); lo[0] = vpadd_u8(lo[0], lo[0]); lo[1] = vpadd_u8(lo[1], lo[1]); lo[1] = vpadd_u8(lo[1], lo[1]); lo[1] = vpadd_u8(lo[1], lo[1]); hi[0] = vpadd_u8(hi[0], hi[0]); hi[0] = vpadd_u8(hi[0], hi[0]); hi[0] = vpadd_u8(hi[0], hi[0]); hi[1] = vpadd_u8(hi[1], hi[1]); hi[1] = vpadd_u8(hi[1], hi[1]); hi[1] = vpadd_u8(hi[1], hi[1]); /* Shift packed 8-bit */ lo_x[0] = vshr_n_u8(lo_x[0], 1); hi_x[0] = vshr_n_u8(hi_x[0], 1); lo_x[1] = vshr_n_u8(lo_x[1], 1); hi_x[1] = vshr_n_u8(hi_x[1], 1); /* Store the created mask to the destination vector */ vst1_lane_u8(dest + 2 * k + j * size * elem_size / (8 * elem_size), lo[0], 0); vst1_lane_u8(dest + 2 * k + j * size * elem_size / (8 * elem_size) + size * elem_size / 2, lo[1], 0); vst1_lane_u8(dest + 2 * k + 1 + j * size * elem_size / (8 * elem_size), hi[0], 0); vst1_lane_u8(dest + 2 * k + 1 + j * size * elem_size / (8 * elem_size) + size * elem_size / 2, hi[1], 0); } } } /* Routine optimized for bit-shuffling a buffer for a type size of 4 bytes. */ static void bitshuffle4_neon(void* src, void* dest, const size_t size, const size_t elem_size) { uint8x16x4_t x0; size_t i, j, k; uint8x8_t lo_x[4], hi_x[4], lo[4], hi[4]; const int8_t __attribute__ ((aligned (16))) xr[8] = {0, 1, 2, 3, 4, 5, 6, 7}; uint8x8_t mask_and = vdup_n_u8(0x01); int8x8_t mask_shift = vld1_s8(xr); for (i = 0, k = 0; i < size * elem_size; i += 64, k++) { /* Load 64-byte groups */ x0 = vld4q_u8(src + i); /* Split in 8-bytes grops */ lo_x[0] = vget_low_u8(x0.val[0]); hi_x[0] = vget_high_u8(x0.val[0]); lo_x[1] = vget_low_u8(x0.val[1]); hi_x[1] = vget_high_u8(x0.val[1]); lo_x[2] = vget_low_u8(x0.val[2]); hi_x[2] = vget_high_u8(x0.val[2]); lo_x[3] = vget_low_u8(x0.val[3]); hi_x[3] = vget_high_u8(x0.val[3]); for (j = 0; j < 8; j++) { /* Create mask from the most significant bit of each 8-bit element */ lo[0] = vand_u8(lo_x[0], mask_and); lo[0] = vshl_u8(lo[0], mask_shift); lo[1] = vand_u8(lo_x[1], mask_and); lo[1] = vshl_u8(lo[1], mask_shift); lo[2] = vand_u8(lo_x[2], mask_and); lo[2] = vshl_u8(lo[2], mask_shift); lo[3] = vand_u8(lo_x[3], mask_and); lo[3] = vshl_u8(lo[3], mask_shift); hi[0] = vand_u8(hi_x[0], mask_and); hi[0] = vshl_u8(hi[0], mask_shift); hi[1] = vand_u8(hi_x[1], mask_and); hi[1] = vshl_u8(hi[1], mask_shift); hi[2] = vand_u8(hi_x[2], mask_and); hi[2] = vshl_u8(hi[2], mask_shift); hi[3] = vand_u8(hi_x[3], mask_and); hi[3] = vshl_u8(hi[3], mask_shift); lo[0] = vpadd_u8(lo[0], lo[0]); lo[0] = vpadd_u8(lo[0], lo[0]); lo[0] = vpadd_u8(lo[0], lo[0]); lo[1] = vpadd_u8(lo[1], lo[1]); lo[1] = vpadd_u8(lo[1], lo[1]); lo[1] = vpadd_u8(lo[1], lo[1]); lo[2] = vpadd_u8(lo[2], lo[2]); lo[2] = vpadd_u8(lo[2], lo[2]); lo[2] = vpadd_u8(lo[2], lo[2]); lo[3] = vpadd_u8(lo[3], lo[3]); lo[3] = vpadd_u8(lo[3], lo[3]); lo[3] = vpadd_u8(lo[3], lo[3]); hi[0] = vpadd_u8(hi[0], hi[0]); hi[0] = vpadd_u8(hi[0], hi[0]); hi[0] = vpadd_u8(hi[0], hi[0]); hi[1] = vpadd_u8(hi[1], hi[1]); hi[1] = vpadd_u8(hi[1], hi[1]); hi[1] = vpadd_u8(hi[1], hi[1]); hi[2] = vpadd_u8(hi[2], hi[2]); hi[2] = vpadd_u8(hi[2], hi[2]); hi[2] = vpadd_u8(hi[2], hi[2]); hi[3] = vpadd_u8(hi[3], hi[3]); hi[3] = vpadd_u8(hi[3], hi[3]); hi[3] = vpadd_u8(hi[3], hi[3]); /* Shift packed 8-bit */ lo_x[0] = vshr_n_u8(lo_x[0], 1); hi_x[0] = vshr_n_u8(hi_x[0], 1); lo_x[1] = vshr_n_u8(lo_x[1], 1); hi_x[1] = vshr_n_u8(hi_x[1], 1); lo_x[2] = vshr_n_u8(lo_x[2], 1); hi_x[2] = vshr_n_u8(hi_x[2], 1); lo_x[3] = vshr_n_u8(lo_x[3], 1); hi_x[3] = vshr_n_u8(hi_x[3], 1); /* Store the created mask to the destination vector */ vst1_lane_u8(dest + 2 * k + j * size * elem_size / (8 * elem_size) + 0 * size * elem_size / 4, lo[0], 0); vst1_lane_u8(dest + 2 * k + j * size * elem_size / (8 * elem_size) + 1 * size * elem_size / 4, lo[1], 0); vst1_lane_u8(dest + 2 * k + j * size * elem_size / (8 * elem_size) + 2 * size * elem_size / 4, lo[2], 0); vst1_lane_u8(dest + 2 * k + j * size * elem_size / (8 * elem_size) + 3 * size * elem_size / 4, lo[3], 0); vst1_lane_u8(dest + 2 * k + 1 + j * size * elem_size / (8 * elem_size) + 0 * size * elem_size / 4, hi[0], 0); vst1_lane_u8(dest + 2 * k + 1 + j * size * elem_size / (8 * elem_size) + 1 * size * elem_size / 4, hi[1], 0); vst1_lane_u8(dest + 2 * k + 1 + j * size * elem_size / (8 * elem_size) + 2 * size * elem_size / 4, hi[2], 0); vst1_lane_u8(dest + 2 * k + 1 + j * size * elem_size / (8 * elem_size) + 3 * size * elem_size / 4, hi[3], 0); } } } /* Routine optimized for bit-shuffling a buffer for a type size of 8 bytes. */ static void bitshuffle8_neon(void* src, void* dest, const size_t size, const size_t elem_size) { size_t i, j, k; uint8x8x2_t r0[4]; uint16x4x2_t r1[4]; uint32x2x2_t r2[4]; const int8_t __attribute__ ((aligned (16))) xr[8] = {0, 1, 2, 3, 4, 5, 6, 7}; uint8x8_t mask_and = vdup_n_u8(0x01); int8x8_t mask_shift = vld1_s8(xr); for (i = 0, k = 0; i < size * elem_size; i += 64, k++) { /* Load and interleave groups of 8 bytes (64 bytes) to the structure r0 */ r0[0] = vzip_u8(vld1_u8(src + i + 0 * 8), vld1_u8(src + i + 1 * 8)); r0[1] = vzip_u8(vld1_u8(src + i + 2 * 8), vld1_u8(src + i + 3 * 8)); r0[2] = vzip_u8(vld1_u8(src + i + 4 * 8), vld1_u8(src + i + 5 * 8)); r0[3] = vzip_u8(vld1_u8(src + i + 6 * 8), vld1_u8(src + i + 7 * 8)); /* Interleave 16 bytes */ r1[0] = vzip_u16(vreinterpret_u16_u8(r0[0].val[0]), vreinterpret_u16_u8(r0[1].val[0])); r1[1] = vzip_u16(vreinterpret_u16_u8(r0[0].val[1]), vreinterpret_u16_u8(r0[1].val[1])); r1[2] = vzip_u16(vreinterpret_u16_u8(r0[2].val[0]), vreinterpret_u16_u8(r0[3].val[0])); r1[3] = vzip_u16(vreinterpret_u16_u8(r0[2].val[1]), vreinterpret_u16_u8(r0[3].val[1])); /* Interleave 32 bytes */ r2[0] = vzip_u32(vreinterpret_u32_u16(r1[0].val[0]), vreinterpret_u32_u16(r1[2].val[0])); r2[1] = vzip_u32(vreinterpret_u32_u16(r1[0].val[1]), vreinterpret_u32_u16(r1[2].val[1])); r2[2] = vzip_u32(vreinterpret_u32_u16(r1[1].val[0]), vreinterpret_u32_u16(r1[3].val[0])); r2[3] = vzip_u32(vreinterpret_u32_u16(r1[1].val[1]), vreinterpret_u32_u16(r1[3].val[1])); for (j = 0; j < 8; j++) { /* Create mask from the most significant bit of each 8-bit element */ r0[0].val[0] = vand_u8(vreinterpret_u8_u32(r2[0].val[0]), mask_and); r0[0].val[0] = vshl_u8(r0[0].val[0], mask_shift); r0[0].val[1] = vand_u8(vreinterpret_u8_u32(r2[0].val[1]), mask_and); r0[0].val[1] = vshl_u8(r0[0].val[1], mask_shift); r0[1].val[0] = vand_u8(vreinterpret_u8_u32(r2[1].val[0]), mask_and); r0[1].val[0] = vshl_u8(r0[1].val[0], mask_shift); r0[1].val[1] = vand_u8(vreinterpret_u8_u32(r2[1].val[1]), mask_and); r0[1].val[1] = vshl_u8(r0[1].val[1], mask_shift); r0[2].val[0] = vand_u8(vreinterpret_u8_u32(r2[2].val[0]), mask_and); r0[2].val[0] = vshl_u8(r0[2].val[0], mask_shift); r0[2].val[1] = vand_u8(vreinterpret_u8_u32(r2[2].val[1]), mask_and); r0[2].val[1] = vshl_u8(r0[2].val[1], mask_shift); r0[3].val[0] = vand_u8(vreinterpret_u8_u32(r2[3].val[0]), mask_and); r0[3].val[0] = vshl_u8(r0[3].val[0], mask_shift); r0[3].val[1] = vand_u8(vreinterpret_u8_u32(r2[3].val[1]), mask_and); r0[3].val[1] = vshl_u8(r0[3].val[1], mask_shift); r0[0].val[0] = vpadd_u8(r0[0].val[0], r0[0].val[0]); r0[0].val[0] = vpadd_u8(r0[0].val[0], r0[0].val[0]); r0[0].val[0] = vpadd_u8(r0[0].val[0], r0[0].val[0]); r0[0].val[1] = vpadd_u8(r0[0].val[1], r0[0].val[1]); r0[0].val[1] = vpadd_u8(r0[0].val[1], r0[0].val[1]); r0[0].val[1] = vpadd_u8(r0[0].val[1], r0[0].val[1]); r0[1].val[0] = vpadd_u8(r0[1].val[0], r0[1].val[0]); r0[1].val[0] = vpadd_u8(r0[1].val[0], r0[1].val[0]); r0[1].val[0] = vpadd_u8(r0[1].val[0], r0[1].val[0]); r0[1].val[1] = vpadd_u8(r0[1].val[1], r0[1].val[1]); r0[1].val[1] = vpadd_u8(r0[1].val[1], r0[1].val[1]); r0[1].val[1] = vpadd_u8(r0[1].val[1], r0[1].val[1]); r0[2].val[0] = vpadd_u8(r0[2].val[0], r0[2].val[0]); r0[2].val[0] = vpadd_u8(r0[2].val[0], r0[2].val[0]); r0[2].val[0] = vpadd_u8(r0[2].val[0], r0[2].val[0]); r0[2].val[1] = vpadd_u8(r0[2].val[1], r0[2].val[1]); r0[2].val[1] = vpadd_u8(r0[2].val[1], r0[2].val[1]); r0[2].val[1] = vpadd_u8(r0[2].val[1], r0[2].val[1]); r0[3].val[0] = vpadd_u8(r0[3].val[0], r0[3].val[0]); r0[3].val[0] = vpadd_u8(r0[3].val[0], r0[3].val[0]); r0[3].val[0] = vpadd_u8(r0[3].val[0], r0[3].val[0]); r0[3].val[1] = vpadd_u8(r0[3].val[1], r0[3].val[1]); r0[3].val[1] = vpadd_u8(r0[3].val[1], r0[3].val[1]); r0[3].val[1] = vpadd_u8(r0[3].val[1], r0[3].val[1]); /* Shift packed 8-bit */ r2[0].val[0] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[0].val[0]), 1)); r2[0].val[1] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[0].val[1]), 1)); r2[1].val[0] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[1].val[0]), 1)); r2[1].val[1] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[1].val[1]), 1)); r2[2].val[0] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[2].val[0]), 1)); r2[2].val[1] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[2].val[1]), 1)); r2[3].val[0] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[3].val[0]), 1)); r2[3].val[1] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[3].val[1]), 1)); /* Store the created mask to the destination vector */ vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 0 * size * elem_size / 8, r0[0].val[0], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 1 * size * elem_size / 8, r0[0].val[1], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 2 * size * elem_size / 8, r0[1].val[0], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 3 * size * elem_size / 8, r0[1].val[1], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 4 * size * elem_size / 8, r0[2].val[0], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 5 * size * elem_size / 8, r0[2].val[1], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 6 * size * elem_size / 8, r0[3].val[0], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 7 * size * elem_size / 8, r0[3].val[1], 0); } } } /* Routine optimized for bit-shuffling a buffer for a type size of 16 bytes. */ static void bitshuffle16_neon(void* src, void* dest, const size_t size, const size_t elem_size) { size_t i, j, k; uint8x8x2_t r0[8]; uint16x4x2_t r1[8]; uint32x2x2_t r2[8]; const int8_t __attribute__ ((aligned (16))) xr[8] = {0, 1, 2, 3, 4, 5, 6, 7}; uint8x8_t mask_and = vdup_n_u8(0x01); int8x8_t mask_shift = vld1_s8(xr); for (i = 0, k = 0; i < size * elem_size; i += 128, k++) { /* Load and interleave groups of 16 bytes (128 bytes) to the structure r0 */ r0[0] = vzip_u8(vld1_u8(src + i + 0 * 8), vld1_u8(src + i + 2 * 8)); r0[1] = vzip_u8(vld1_u8(src + i + 1 * 8), vld1_u8(src + i + 3 * 8)); r0[2] = vzip_u8(vld1_u8(src + i + 4 * 8), vld1_u8(src + i + 6 * 8)); r0[3] = vzip_u8(vld1_u8(src + i + 5 * 8), vld1_u8(src + i + 7 * 8)); r0[4] = vzip_u8(vld1_u8(src + i + 8 * 8), vld1_u8(src + i + 10 * 8)); r0[5] = vzip_u8(vld1_u8(src + i + 9 * 8), vld1_u8(src + i + 11 * 8)); r0[6] = vzip_u8(vld1_u8(src + i + 12 * 8), vld1_u8(src + i + 14 * 8)); r0[7] = vzip_u8(vld1_u8(src + i + 13 * 8), vld1_u8(src + i + 15 * 8)); /* Interleave 16 bytes */ r1[0] = vzip_u16(vreinterpret_u16_u8(r0[0].val[0]), vreinterpret_u16_u8(r0[2].val[0])); r1[1] = vzip_u16(vreinterpret_u16_u8(r0[0].val[1]), vreinterpret_u16_u8(r0[2].val[1])); r1[2] = vzip_u16(vreinterpret_u16_u8(r0[1].val[0]), vreinterpret_u16_u8(r0[3].val[0])); r1[3] = vzip_u16(vreinterpret_u16_u8(r0[1].val[1]), vreinterpret_u16_u8(r0[3].val[1])); r1[4] = vzip_u16(vreinterpret_u16_u8(r0[4].val[0]), vreinterpret_u16_u8(r0[6].val[0])); r1[5] = vzip_u16(vreinterpret_u16_u8(r0[4].val[1]), vreinterpret_u16_u8(r0[6].val[1])); r1[6] = vzip_u16(vreinterpret_u16_u8(r0[5].val[0]), vreinterpret_u16_u8(r0[7].val[0])); r1[7] = vzip_u16(vreinterpret_u16_u8(r0[5].val[1]), vreinterpret_u16_u8(r0[7].val[1])); /* Interleave 32 bytes */ r2[0] = vzip_u32(vreinterpret_u32_u16(r1[0].val[0]), vreinterpret_u32_u16(r1[4].val[0])); r2[1] = vzip_u32(vreinterpret_u32_u16(r1[0].val[1]), vreinterpret_u32_u16(r1[4].val[1])); r2[2] = vzip_u32(vreinterpret_u32_u16(r1[1].val[0]), vreinterpret_u32_u16(r1[5].val[0])); r2[3] = vzip_u32(vreinterpret_u32_u16(r1[1].val[1]), vreinterpret_u32_u16(r1[5].val[1])); r2[4] = vzip_u32(vreinterpret_u32_u16(r1[2].val[0]), vreinterpret_u32_u16(r1[6].val[0])); r2[5] = vzip_u32(vreinterpret_u32_u16(r1[2].val[1]), vreinterpret_u32_u16(r1[6].val[1])); r2[6] = vzip_u32(vreinterpret_u32_u16(r1[3].val[0]), vreinterpret_u32_u16(r1[7].val[0])); r2[7] = vzip_u32(vreinterpret_u32_u16(r1[3].val[1]), vreinterpret_u32_u16(r1[7].val[1])); for (j = 0; j < 8; j++) { /* Create mask from the most significant bit of each 8-bit element */ r0[0].val[0] = vand_u8(vreinterpret_u8_u32(r2[0].val[0]), mask_and); r0[0].val[0] = vshl_u8(r0[0].val[0], mask_shift); r0[0].val[1] = vand_u8(vreinterpret_u8_u32(r2[0].val[1]), mask_and); r0[0].val[1] = vshl_u8(r0[0].val[1], mask_shift); r0[1].val[0] = vand_u8(vreinterpret_u8_u32(r2[1].val[0]), mask_and); r0[1].val[0] = vshl_u8(r0[1].val[0], mask_shift); r0[1].val[1] = vand_u8(vreinterpret_u8_u32(r2[1].val[1]), mask_and); r0[1].val[1] = vshl_u8(r0[1].val[1], mask_shift); r0[2].val[0] = vand_u8(vreinterpret_u8_u32(r2[2].val[0]), mask_and); r0[2].val[0] = vshl_u8(r0[2].val[0], mask_shift); r0[2].val[1] = vand_u8(vreinterpret_u8_u32(r2[2].val[1]), mask_and); r0[2].val[1] = vshl_u8(r0[2].val[1], mask_shift); r0[3].val[0] = vand_u8(vreinterpret_u8_u32(r2[3].val[0]), mask_and); r0[3].val[0] = vshl_u8(r0[3].val[0], mask_shift); r0[3].val[1] = vand_u8(vreinterpret_u8_u32(r2[3].val[1]), mask_and); r0[3].val[1] = vshl_u8(r0[3].val[1], mask_shift); r0[4].val[0] = vand_u8(vreinterpret_u8_u32(r2[4].val[0]), mask_and); r0[4].val[0] = vshl_u8(r0[4].val[0], mask_shift); r0[4].val[1] = vand_u8(vreinterpret_u8_u32(r2[4].val[1]), mask_and); r0[4].val[1] = vshl_u8(r0[4].val[1], mask_shift); r0[5].val[0] = vand_u8(vreinterpret_u8_u32(r2[5].val[0]), mask_and); r0[5].val[0] = vshl_u8(r0[5].val[0], mask_shift); r0[5].val[1] = vand_u8(vreinterpret_u8_u32(r2[5].val[1]), mask_and); r0[5].val[1] = vshl_u8(r0[5].val[1], mask_shift); r0[6].val[0] = vand_u8(vreinterpret_u8_u32(r2[6].val[0]), mask_and); r0[6].val[0] = vshl_u8(r0[6].val[0], mask_shift); r0[6].val[1] = vand_u8(vreinterpret_u8_u32(r2[6].val[1]), mask_and); r0[6].val[1] = vshl_u8(r0[6].val[1], mask_shift); r0[7].val[0] = vand_u8(vreinterpret_u8_u32(r2[7].val[0]), mask_and); r0[7].val[0] = vshl_u8(r0[7].val[0], mask_shift); r0[7].val[1] = vand_u8(vreinterpret_u8_u32(r2[7].val[1]), mask_and); r0[7].val[1] = vshl_u8(r0[7].val[1], mask_shift); r0[0].val[0] = vpadd_u8(r0[0].val[0], r0[0].val[0]); r0[0].val[0] = vpadd_u8(r0[0].val[0], r0[0].val[0]); r0[0].val[0] = vpadd_u8(r0[0].val[0], r0[0].val[0]); r0[0].val[1] = vpadd_u8(r0[0].val[1], r0[0].val[1]); r0[0].val[1] = vpadd_u8(r0[0].val[1], r0[0].val[1]); r0[0].val[1] = vpadd_u8(r0[0].val[1], r0[0].val[1]); r0[1].val[0] = vpadd_u8(r0[1].val[0], r0[1].val[0]); r0[1].val[0] = vpadd_u8(r0[1].val[0], r0[1].val[0]); r0[1].val[0] = vpadd_u8(r0[1].val[0], r0[1].val[0]); r0[1].val[1] = vpadd_u8(r0[1].val[1], r0[1].val[1]); r0[1].val[1] = vpadd_u8(r0[1].val[1], r0[1].val[1]); r0[1].val[1] = vpadd_u8(r0[1].val[1], r0[1].val[1]); r0[2].val[0] = vpadd_u8(r0[2].val[0], r0[2].val[0]); r0[2].val[0] = vpadd_u8(r0[2].val[0], r0[2].val[0]); r0[2].val[0] = vpadd_u8(r0[2].val[0], r0[2].val[0]); r0[2].val[1] = vpadd_u8(r0[2].val[1], r0[2].val[1]); r0[2].val[1] = vpadd_u8(r0[2].val[1], r0[2].val[1]); r0[2].val[1] = vpadd_u8(r0[2].val[1], r0[2].val[1]); r0[3].val[0] = vpadd_u8(r0[3].val[0], r0[3].val[0]); r0[3].val[0] = vpadd_u8(r0[3].val[0], r0[3].val[0]); r0[3].val[0] = vpadd_u8(r0[3].val[0], r0[3].val[0]); r0[3].val[1] = vpadd_u8(r0[3].val[1], r0[3].val[1]); r0[3].val[1] = vpadd_u8(r0[3].val[1], r0[3].val[1]); r0[3].val[1] = vpadd_u8(r0[3].val[1], r0[3].val[1]); r0[4].val[0] = vpadd_u8(r0[4].val[0], r0[4].val[0]); r0[4].val[0] = vpadd_u8(r0[4].val[0], r0[4].val[0]); r0[4].val[0] = vpadd_u8(r0[4].val[0], r0[4].val[0]); r0[4].val[1] = vpadd_u8(r0[4].val[1], r0[4].val[1]); r0[4].val[1] = vpadd_u8(r0[4].val[1], r0[4].val[1]); r0[4].val[1] = vpadd_u8(r0[4].val[1], r0[4].val[1]); r0[5].val[0] = vpadd_u8(r0[5].val[0], r0[5].val[0]); r0[5].val[0] = vpadd_u8(r0[5].val[0], r0[5].val[0]); r0[5].val[0] = vpadd_u8(r0[5].val[0], r0[5].val[0]); r0[5].val[1] = vpadd_u8(r0[5].val[1], r0[5].val[1]); r0[5].val[1] = vpadd_u8(r0[5].val[1], r0[5].val[1]); r0[5].val[1] = vpadd_u8(r0[5].val[1], r0[5].val[1]); r0[6].val[0] = vpadd_u8(r0[6].val[0], r0[6].val[0]); r0[6].val[0] = vpadd_u8(r0[6].val[0], r0[6].val[0]); r0[6].val[0] = vpadd_u8(r0[6].val[0], r0[6].val[0]); r0[6].val[1] = vpadd_u8(r0[6].val[1], r0[6].val[1]); r0[6].val[1] = vpadd_u8(r0[6].val[1], r0[6].val[1]); r0[6].val[1] = vpadd_u8(r0[6].val[1], r0[6].val[1]); r0[7].val[0] = vpadd_u8(r0[7].val[0], r0[7].val[0]); r0[7].val[0] = vpadd_u8(r0[7].val[0], r0[7].val[0]); r0[7].val[0] = vpadd_u8(r0[7].val[0], r0[7].val[0]); r0[7].val[1] = vpadd_u8(r0[7].val[1], r0[7].val[1]); r0[7].val[1] = vpadd_u8(r0[7].val[1], r0[7].val[1]); r0[7].val[1] = vpadd_u8(r0[7].val[1], r0[7].val[1]); /* Shift packed 8-bit */ r2[0].val[0] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[0].val[0]), 1)); r2[0].val[1] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[0].val[1]), 1)); r2[1].val[0] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[1].val[0]), 1)); r2[1].val[1] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[1].val[1]), 1)); r2[2].val[0] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[2].val[0]), 1)); r2[2].val[1] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[2].val[1]), 1)); r2[3].val[0] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[3].val[0]), 1)); r2[3].val[1] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[3].val[1]), 1)); r2[4].val[0] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[4].val[0]), 1)); r2[4].val[1] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[4].val[1]), 1)); r2[5].val[0] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[5].val[0]), 1)); r2[5].val[1] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[5].val[1]), 1)); r2[6].val[0] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[6].val[0]), 1)); r2[6].val[1] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[6].val[1]), 1)); r2[7].val[0] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[7].val[0]), 1)); r2[7].val[1] = vreinterpret_u8_u32(vshr_n_u8(vreinterpret_u8_u32(r2[7].val[1]), 1)); /* Store the created mask to the destination vector */ vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 0 * size * elem_size / 16, r0[0].val[0], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 1 * size * elem_size / 16, r0[0].val[1], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 2 * size * elem_size / 16, r0[1].val[0], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 3 * size * elem_size / 16, r0[1].val[1], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 4 * size * elem_size / 16, r0[2].val[0], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 5 * size * elem_size / 16, r0[2].val[1], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 6 * size * elem_size / 16, r0[3].val[0], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 7 * size * elem_size / 16, r0[3].val[1], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 8 * size * elem_size / 16, r0[4].val[0], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 9 * size * elem_size / 16, r0[4].val[1], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 10 * size * elem_size / 16, r0[5].val[0], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 11 * size * elem_size / 16, r0[5].val[1], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 12 * size * elem_size / 16, r0[6].val[0], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 13 * size * elem_size / 16, r0[6].val[1], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 14 * size * elem_size / 16, r0[7].val[0], 0); vst1_lane_u8(dest + k + j * size * elem_size / (8 * elem_size) + 15 * size * elem_size / 16, r0[7].val[1], 0); } } } /* Routine optimized for bit-unshuffling a buffer for a type size of 1 byte. */ static void bitunshuffle1_neon(void* _src, void* dest, const size_t size, const size_t elem_size) { uint8x8_t lo_x, hi_x, lo, hi; size_t i, j, k; uint8_t* src = _src; const int8_t __attribute__ ((aligned (16))) xr[8] = {0, 1, 2, 3, 4, 5, 6, 7}; uint8x8_t mask_and = vdup_n_u8(0x01); int8x8_t mask_shift = vld1_s8(xr); for (i = 0, k = 0; i < size * elem_size; i += 16, k++) { for (j = 0; j < 8; j++) { /* Load lanes */ lo_x[j] = src[2 * k + 0 + j * size * elem_size / (8 * elem_size)]; hi_x[j] = src[2 * k + 1 + j * size * elem_size / (8 * elem_size)]; } for (j = 0; j < 8; j++) { /* Create mask from the most significant bit of each 8-bit element */ lo = vand_u8(lo_x, mask_and); lo = vshl_u8(lo, mask_shift); hi = vand_u8(hi_x, mask_and); hi = vshl_u8(hi, mask_shift); lo = vpadd_u8(lo, lo); lo = vpadd_u8(lo, lo); lo = vpadd_u8(lo, lo); hi = vpadd_u8(hi, hi); hi = vpadd_u8(hi, hi); hi = vpadd_u8(hi, hi); /* Shift packed 8-bit */ lo_x = vshr_n_u8(lo_x, 1); hi_x = vshr_n_u8(hi_x, 1); /* Store the created mask to the destination vector */ vst1_lane_u8(dest + j + i, lo, 0); vst1_lane_u8(dest + j + i + 8, hi, 0); } } } /* Routine optimized for bit-unshuffling a buffer for a type size of 2 byte. */ static void bitunshuffle2_neon(void* _src, void* dest, const size_t size, const size_t elem_size) { size_t i, j, k; uint8x8_t lo_x[2], hi_x[2], lo[2], hi[2]; uint8_t* src = _src; const int8_t __attribute__ ((aligned (16))) xr[8] = {0, 1, 2, 3, 4, 5, 6, 7}; uint8x8_t mask_and = vdup_n_u8(0x01); int8x8_t mask_shift = vld1_s8(xr); for (i = 0, k = 0; i < size * elem_size; i += 32, k++) { for (j = 0; j < 8; j++) { /* Load lanes */ lo_x[0][j] = src[2 * k + j * size * elem_size / (8 * elem_size)]; lo_x[1][j] = src[2 * k + j * size * elem_size / (8 * elem_size) + size * elem_size / 2]; hi_x[0][j] = src[2 * k + 1 + j * size * elem_size / (8 * elem_size)]; hi_x[1][j] = src[2 * k + 1 + j * size * elem_size / (8 * elem_size) + size * elem_size / 2]; } for (j = 0; j < 8; j++) { /* Create mask from the most significant bit of each 8-bit element */ lo[0] = vand_u8(lo_x[0], mask_and); lo[0] = vshl_u8(lo[0], mask_shift); lo[1] = vand_u8(lo_x[1], mask_and); lo[1] = vshl_u8(lo[1], mask_shift); hi[0] = vand_u8(hi_x[0], mask_and); hi[0] = vshl_u8(hi[0], mask_shift); hi[1] = vand_u8(hi_x[1], mask_and); hi[1] = vshl_u8(hi[1], mask_shift); lo[0] = vpadd_u8(lo[0], lo[0]); lo[0] = vpadd_u8(lo[0], lo[0]); lo[0] = vpadd_u8(lo[0], lo[0]); lo[1] = vpadd_u8(lo[1], lo[1]); lo[1] = vpadd_u8(lo[1], lo[1]); lo[1] = vpadd_u8(lo[1], lo[1]); hi[0] = vpadd_u8(hi[0], hi[0]); hi[0] = vpadd_u8(hi[0], hi[0]); hi[0] = vpadd_u8(hi[0], hi[0]); hi[1] = vpadd_u8(hi[1], hi[1]); hi[1] = vpadd_u8(hi[1], hi[1]); hi[1] = vpadd_u8(hi[1], hi[1]); /* Shift packed 8-bit */ lo_x[0] = vshr_n_u8(lo_x[0], 1); hi_x[0] = vshr_n_u8(hi_x[0], 1); lo_x[1] = vshr_n_u8(lo_x[1], 1); hi_x[1] = vshr_n_u8(hi_x[1], 1); /* Store the created mask to the destination vector */ vst1_lane_u8(dest + 2 * j + i, lo[0], 0); vst1_lane_u8(dest + 2 * j + 1 + i, lo[1], 0); vst1_lane_u8(dest + 2 * j + i + 16, hi[0], 0); vst1_lane_u8(dest + 2 * j + 1 + i + 16, hi[1], 0); } } } /* Routine optimized for bit-unshuffling a buffer for a type size of 4 byte. */ static void bitunshuffle4_neon(void* _src, void* dest, const size_t size, const size_t elem_size) { size_t i, j, k; uint8x8_t lo_x[4], hi_x[4], lo[4], hi[4]; uint8_t* src = _src; const int8_t __attribute__ ((aligned (16))) xr[8] = {0, 1, 2, 3, 4, 5, 6, 7}; uint8x8_t mask_and = vdup_n_u8(0x01); int8x8_t mask_shift = vld1_s8(xr); for (i = 0, k = 0; i < size * elem_size; i += 64, k++) { for (j = 0; j < 8; j++) { /* Load lanes */ lo_x[0][j] = src[2 * k + j * size * elem_size / (8 * elem_size) + 0 * size * elem_size / 4]; hi_x[0][j] = src[2 * k + 1 + j * size * elem_size / (8 * elem_size) + 0 * size * elem_size / 4]; lo_x[1][j] = src[2 * k + j * size * elem_size / (8 * elem_size) + 1 * size * elem_size / 4]; hi_x[1][j] = src[2 * k + 1 + j * size * elem_size / (8 * elem_size) + 1 * size * elem_size / 4]; lo_x[2][j] = src[2 * k + j * size * elem_size / (8 * elem_size) + 2 * size * elem_size / 4]; hi_x[2][j] = src[2 * k + 1 + j * size * elem_size / (8 * elem_size) + 2 * size * elem_size / 4]; lo_x[3][j] = src[2 * k + j * size * elem_size / (8 * elem_size) + 3 * size * elem_size / 4]; hi_x[3][j] = src[2 * k + 1 + j * size * elem_size / (8 * elem_size) + 3 * size * elem_size / 4]; } for (j = 0; j < 8; j++) { /* Create mask from the most significant bit of each 8-bit element */ lo[0] = vand_u8(lo_x[0], mask_and); lo[0] = vshl_u8(lo[0], mask_shift); lo[1] = vand_u8(lo_x[1], mask_and); lo[1] = vshl_u8(lo[1], mask_shift); lo[2] = vand_u8(lo_x[2], mask_and); lo[2] = vshl_u8(lo[2], mask_shift); lo[3] = vand_u8(lo_x[3], mask_and); lo[3] = vshl_u8(lo[3], mask_shift); hi[0] = vand_u8(hi_x[0], mask_and); hi[0] = vshl_u8(hi[0], mask_shift); hi[1] = vand_u8(hi_x[1], mask_and); hi[1] = vshl_u8(hi[1], mask_shift); hi[2] = vand_u8(hi_x[2], mask_and); hi[2] = vshl_u8(hi[2], mask_shift); hi[3] = vand_u8(hi_x[3], mask_and); hi[3] = vshl_u8(hi[3], mask_shift); lo[0] = vpadd_u8(lo[0], lo[0]); lo[0] = vpadd_u8(lo[0], lo[0]); lo[0] = vpadd_u8(lo[0], lo[0]); lo[1] = vpadd_u8(lo[1], lo[1]); lo[1] = vpadd_u8(lo[1], lo[1]); lo[1] = vpadd_u8(lo[1], lo[1]); lo[2] = vpadd_u8(lo[2], lo[2]); lo[2] = vpadd_u8(lo[2], lo[2]); lo[2] = vpadd_u8(lo[2], lo[2]); lo[3] = vpadd_u8(lo[3], lo[3]); lo[3] = vpadd_u8(lo[3], lo[3]); lo[3] = vpadd_u8(lo[3], lo[3]); hi[0] = vpadd_u8(hi[0], hi[0]); hi[0] = vpadd_u8(hi[0], hi[0]); hi[0] = vpadd_u8(hi[0], hi[0]); hi[1] = vpadd_u8(hi[1], hi[1]); hi[1] = vpadd_u8(hi[1], hi[1]); hi[1] = vpadd_u8(hi[1], hi[1]); hi[2] = vpadd_u8(hi[2], hi[2]); hi[2] = vpadd_u8(hi[2], hi[2]); hi[2] = vpadd_u8(hi[2], hi[2]); hi[3] = vpadd_u8(hi[3], hi[3]); hi[3] = vpadd_u8(hi[3], hi[3]); hi[3] = vpadd_u8(hi[3], hi[3]); /* Shift packed 8-bit */ lo_x[0] = vshr_n_u8(lo_x[0], 1); hi_x[0] = vshr_n_u8(hi_x[0], 1); lo_x[1] = vshr_n_u8(lo_x[1], 1); hi_x[1] = vshr_n_u8(hi_x[1], 1); lo_x[2] = vshr_n_u8(lo_x[2], 1); hi_x[2] = vshr_n_u8(hi_x[2], 1); lo_x[3] = vshr_n_u8(lo_x[3], 1); hi_x[3] = vshr_n_u8(hi_x[3], 1); /* Store the created mask to the destination vector */ vst1_lane_u8(dest + 4 * j + i, lo[0], 0); vst1_lane_u8(dest + 4 * j + 1 + i, lo[1], 0); vst1_lane_u8(dest + 4 * j + 2 + i, lo[2], 0); vst1_lane_u8(dest + 4 * j + 3 + i, lo[3], 0); vst1_lane_u8(dest + 4 * j + i + 32, hi[0], 0); vst1_lane_u8(dest + 4 * j + 1 + i + 32, hi[1], 0); vst1_lane_u8(dest + 4 * j + 2 + i + 32, hi[2], 0); vst1_lane_u8(dest + 4 * j + 3 + i + 32, hi[3], 0); } } } /* Routine optimized for bit-unshuffling a buffer for a type size of 8 byte. */ static void bitunshuffle8_neon(void* _src, void* dest, const size_t size, const size_t elem_size) { size_t i, j, k; uint8x8x2_t r0[4], r1[4]; uint8_t* src = _src; const int8_t __attribute__ ((aligned (16))) xr[8] = {0, 1, 2, 3, 4, 5, 6, 7}; uint8x8_t mask_and = vdup_n_u8(0x01); int8x8_t mask_shift = vld1_s8(xr); for (i = 0, k = 0; i < size * elem_size; i += 64, k++) { for (j = 0; j < 8; j++) { /* Load lanes */ r0[0].val[0][j] = src[k + j * size * elem_size / (8 * elem_size) + 0 * size * elem_size / 8]; r0[0].val[1][j] = src[k + j * size * elem_size / (8 * elem_size) + 1 * size * elem_size / 8]; r0[1].val[0][j] = src[k + j * size * elem_size / (8 * elem_size) + 2 * size * elem_size / 8]; r0[1].val[1][j] = src[k + j * size * elem_size / (8 * elem_size) + 3 * size * elem_size / 8]; r0[2].val[0][j] = src[k + j * size * elem_size / (8 * elem_size) + 4 * size * elem_size / 8]; r0[2].val[1][j] = src[k + j * size * elem_size / (8 * elem_size) + 5 * size * elem_size / 8]; r0[3].val[0][j] = src[k + j * size * elem_size / (8 * elem_size) + 6 * size * elem_size / 8]; r0[3].val[1][j] = src[k + j * size * elem_size / (8 * elem_size) + 7 * size * elem_size / 8]; } for (j = 0; j < 8; j++) { /* Create mask from the most significant bit of each 8-bit element */ r1[0].val[0] = vand_u8(r0[0].val[0], mask_and); r1[0].val[0] = vshl_u8(r1[0].val[0], mask_shift); r1[0].val[1] = vand_u8(r0[0].val[1], mask_and); r1[0].val[1] = vshl_u8(r1[0].val[1], mask_shift); r1[1].val[0] = vand_u8(r0[1].val[0], mask_and); r1[1].val[0] = vshl_u8(r1[1].val[0], mask_shift); r1[1].val[1] = vand_u8(r0[1].val[1], mask_and); r1[1].val[1] = vshl_u8(r1[1].val[1], mask_shift); r1[2].val[0] = vand_u8(r0[2].val[0], mask_and); r1[2].val[0] = vshl_u8(r1[2].val[0], mask_shift); r1[2].val[1] = vand_u8(r0[2].val[1], mask_and); r1[2].val[1] = vshl_u8(r1[2].val[1], mask_shift); r1[3].val[0] = vand_u8(r0[3].val[0], mask_and); r1[3].val[0] = vshl_u8(r1[3].val[0], mask_shift); r1[3].val[1] = vand_u8(r0[3].val[1], mask_and); r1[3].val[1] = vshl_u8(r1[3].val[1], mask_shift); r1[0].val[0] = vpadd_u8(r1[0].val[0], r1[0].val[0]); r1[0].val[0] = vpadd_u8(r1[0].val[0], r1[0].val[0]); r1[0].val[0] = vpadd_u8(r1[0].val[0], r1[0].val[0]); r1[0].val[1] = vpadd_u8(r1[0].val[1], r1[0].val[1]); r1[0].val[1] = vpadd_u8(r1[0].val[1], r1[0].val[1]); r1[0].val[1] = vpadd_u8(r1[0].val[1], r1[0].val[1]); r1[1].val[0] = vpadd_u8(r1[1].val[0], r1[1].val[0]); r1[1].val[0] = vpadd_u8(r1[1].val[0], r1[1].val[0]); r1[1].val[0] = vpadd_u8(r1[1].val[0], r1[1].val[0]); r1[1].val[1] = vpadd_u8(r1[1].val[1], r1[1].val[1]); r1[1].val[1] = vpadd_u8(r1[1].val[1], r1[1].val[1]); r1[1].val[1] = vpadd_u8(r1[1].val[1], r1[1].val[1]); r1[2].val[0] = vpadd_u8(r1[2].val[0], r1[2].val[0]); r1[2].val[0] = vpadd_u8(r1[2].val[0], r1[2].val[0]); r1[2].val[0] = vpadd_u8(r1[2].val[0], r1[2].val[0]); r1[2].val[1] = vpadd_u8(r1[2].val[1], r1[2].val[1]); r1[2].val[1] = vpadd_u8(r1[2].val[1], r1[2].val[1]); r1[2].val[1] = vpadd_u8(r1[2].val[1], r1[2].val[1]); r1[3].val[0] = vpadd_u8(r1[3].val[0], r1[3].val[0]); r1[3].val[0] = vpadd_u8(r1[3].val[0], r1[3].val[0]); r1[3].val[0] = vpadd_u8(r1[3].val[0], r1[3].val[0]); r1[3].val[1] = vpadd_u8(r1[3].val[1], r1[3].val[1]); r1[3].val[1] = vpadd_u8(r1[3].val[1], r1[3].val[1]); r1[3].val[1] = vpadd_u8(r1[3].val[1], r1[3].val[1]); /* Shift packed 8-bit */ r0[0].val[0] = vshr_n_u8(r0[0].val[0], 1); r0[0].val[1] = vshr_n_u8(r0[0].val[1], 1); r0[1].val[0] = vshr_n_u8(r0[1].val[0], 1); r0[1].val[1] = vshr_n_u8(r0[1].val[1], 1); r0[2].val[0] = vshr_n_u8(r0[2].val[0], 1); r0[2].val[1] = vshr_n_u8(r0[2].val[1], 1); r0[3].val[0] = vshr_n_u8(r0[3].val[0], 1); r0[3].val[1] = vshr_n_u8(r0[3].val[1], 1); /* Store the created mask to the destination vector */ vst1_lane_u8(dest + 8 * j + 0 + i, r1[0].val[0], 0); vst1_lane_u8(dest + 8 * j + 1 + i, r1[0].val[1], 0); vst1_lane_u8(dest + 8 * j + 2 + i, r1[1].val[0], 0); vst1_lane_u8(dest + 8 * j + 3 + i, r1[1].val[1], 0); vst1_lane_u8(dest + 8 * j + 4 + i, r1[2].val[0], 0); vst1_lane_u8(dest + 8 * j + 5 + i, r1[2].val[1], 0); vst1_lane_u8(dest + 8 * j + 6 + i, r1[3].val[0], 0); vst1_lane_u8(dest + 8 * j + 7 + i, r1[3].val[1], 0); } } } /* Routine optimized for bit-unshuffling a buffer for a type size of 16 byte. */ static void bitunshuffle16_neon(void* _src, void* dest, const size_t size, const size_t elem_size) { size_t i, j, k; uint8x8x2_t r0[8], r1[8]; uint8_t* src = _src; const int8_t __attribute__ ((aligned (16))) xr[8] = {0, 1, 2, 3, 4, 5, 6, 7}; uint8x8_t mask_and = vdup_n_u8(0x01); int8x8_t mask_shift = vld1_s8(xr); for (i = 0, k = 0; i < size * elem_size; i += 128, k++) { for (j = 0; j < 8; j++) { /* Load lanes */ r0[0].val[0][j] = src[k + j * size * elem_size / (8 * elem_size) + 0 * size * elem_size / 16]; r0[0].val[1][j] = src[k + j * size * elem_size / (8 * elem_size) + 1 * size * elem_size / 16]; r0[1].val[0][j] = src[k + j * size * elem_size / (8 * elem_size) + 2 * size * elem_size / 16]; r0[1].val[1][j] = src[k + j * size * elem_size / (8 * elem_size) + 3 * size * elem_size / 16]; r0[2].val[0][j] = src[k + j * size * elem_size / (8 * elem_size) + 4 * size * elem_size / 16]; r0[2].val[1][j] = src[k + j * size * elem_size / (8 * elem_size) + 5 * size * elem_size / 16]; r0[3].val[0][j] = src[k + j * size * elem_size / (8 * elem_size) + 6 * size * elem_size / 16]; r0[3].val[1][j] = src[k + j * size * elem_size / (8 * elem_size) + 7 * size * elem_size / 16]; r0[4].val[0][j] = src[k + j * size * elem_size / (8 * elem_size) + 8 * size * elem_size / 16]; r0[4].val[1][j] = src[k + j * size * elem_size / (8 * elem_size) + 9 * size * elem_size / 16]; r0[5].val[0][j] = src[k + j * size * elem_size / (8 * elem_size) + 10 * size * elem_size / 16]; r0[5].val[1][j] = src[k + j * size * elem_size / (8 * elem_size) + 11 * size * elem_size / 16]; r0[6].val[0][j] = src[k + j * size * elem_size / (8 * elem_size) + 12 * size * elem_size / 16]; r0[6].val[1][j] = src[k + j * size * elem_size / (8 * elem_size) + 13 * size * elem_size / 16]; r0[7].val[0][j] = src[k + j * size * elem_size / (8 * elem_size) + 14 * size * elem_size / 16]; r0[7].val[1][j] = src[k + j * size * elem_size / (8 * elem_size) + 15 * size * elem_size / 16]; } for (j = 0; j < 8; j++) { /* Create mask from the most significant bit of each 8-bit element */ r1[0].val[0] = vand_u8(r0[0].val[0], mask_and); r1[0].val[0] = vshl_u8(r1[0].val[0], mask_shift); r1[0].val[1] = vand_u8(r0[0].val[1], mask_and); r1[0].val[1] = vshl_u8(r1[0].val[1], mask_shift); r1[1].val[0] = vand_u8(r0[1].val[0], mask_and); r1[1].val[0] = vshl_u8(r1[1].val[0], mask_shift); r1[1].val[1] = vand_u8(r0[1].val[1], mask_and); r1[1].val[1] = vshl_u8(r1[1].val[1], mask_shift); r1[2].val[0] = vand_u8(r0[2].val[0], mask_and); r1[2].val[0] = vshl_u8(r1[2].val[0], mask_shift); r1[2].val[1] = vand_u8(r0[2].val[1], mask_and); r1[2].val[1] = vshl_u8(r1[2].val[1], mask_shift); r1[3].val[0] = vand_u8(r0[3].val[0], mask_and); r1[3].val[0] = vshl_u8(r1[3].val[0], mask_shift); r1[3].val[1] = vand_u8(r0[3].val[1], mask_and); r1[3].val[1] = vshl_u8(r1[3].val[1], mask_shift); r1[4].val[0] = vand_u8(r0[4].val[0], mask_and); r1[4].val[0] = vshl_u8(r1[4].val[0], mask_shift); r1[4].val[1] = vand_u8(r0[4].val[1], mask_and); r1[4].val[1] = vshl_u8(r1[4].val[1], mask_shift); r1[5].val[0] = vand_u8(r0[5].val[0], mask_and); r1[5].val[0] = vshl_u8(r1[5].val[0], mask_shift); r1[5].val[1] = vand_u8(r0[5].val[1], mask_and); r1[5].val[1] = vshl_u8(r1[5].val[1], mask_shift); r1[6].val[0] = vand_u8(r0[6].val[0], mask_and); r1[6].val[0] = vshl_u8(r1[6].val[0], mask_shift); r1[6].val[1] = vand_u8(r0[6].val[1], mask_and); r1[6].val[1] = vshl_u8(r1[6].val[1], mask_shift); r1[7].val[0] = vand_u8(r0[7].val[0], mask_and); r1[7].val[0] = vshl_u8(r1[7].val[0], mask_shift); r1[7].val[1] = vand_u8(r0[7].val[1], mask_and); r1[7].val[1] = vshl_u8(r1[7].val[1], mask_shift); r1[0].val[0] = vpadd_u8(r1[0].val[0], r1[0].val[0]); r1[0].val[0] = vpadd_u8(r1[0].val[0], r1[0].val[0]); r1[0].val[0] = vpadd_u8(r1[0].val[0], r1[0].val[0]); r1[0].val[1] = vpadd_u8(r1[0].val[1], r1[0].val[1]); r1[0].val[1] = vpadd_u8(r1[0].val[1], r1[0].val[1]); r1[0].val[1] = vpadd_u8(r1[0].val[1], r1[0].val[1]); r1[1].val[0] = vpadd_u8(r1[1].val[0], r1[1].val[0]); r1[1].val[0] = vpadd_u8(r1[1].val[0], r1[1].val[0]); r1[1].val[0] = vpadd_u8(r1[1].val[0], r1[1].val[0]); r1[1].val[1] = vpadd_u8(r1[1].val[1], r1[1].val[1]); r1[1].val[1] = vpadd_u8(r1[1].val[1], r1[1].val[1]); r1[1].val[1] = vpadd_u8(r1[1].val[1], r1[1].val[1]); r1[2].val[0] = vpadd_u8(r1[2].val[0], r1[2].val[0]); r1[2].val[0] = vpadd_u8(r1[2].val[0], r1[2].val[0]); r1[2].val[0] = vpadd_u8(r1[2].val[0], r1[2].val[0]); r1[2].val[1] = vpadd_u8(r1[2].val[1], r1[2].val[1]); r1[2].val[1] = vpadd_u8(r1[2].val[1], r1[2].val[1]); r1[2].val[1] = vpadd_u8(r1[2].val[1], r1[2].val[1]); r1[3].val[0] = vpadd_u8(r1[3].val[0], r1[3].val[0]); r1[3].val[0] = vpadd_u8(r1[3].val[0], r1[3].val[0]); r1[3].val[0] = vpadd_u8(r1[3].val[0], r1[3].val[0]); r1[3].val[1] = vpadd_u8(r1[3].val[1], r1[3].val[1]); r1[3].val[1] = vpadd_u8(r1[3].val[1], r1[3].val[1]); r1[3].val[1] = vpadd_u8(r1[3].val[1], r1[3].val[1]); r1[4].val[0] = vpadd_u8(r1[4].val[0], r1[4].val[0]); r1[4].val[0] = vpadd_u8(r1[4].val[0], r1[4].val[0]); r1[4].val[0] = vpadd_u8(r1[4].val[0], r1[4].val[0]); r1[4].val[1] = vpadd_u8(r1[4].val[1], r1[4].val[1]); r1[4].val[1] = vpadd_u8(r1[4].val[1], r1[4].val[1]); r1[4].val[1] = vpadd_u8(r1[4].val[1], r1[4].val[1]); r1[5].val[0] = vpadd_u8(r1[5].val[0], r1[5].val[0]); r1[5].val[0] = vpadd_u8(r1[5].val[0], r1[5].val[0]); r1[5].val[0] = vpadd_u8(r1[5].val[0], r1[5].val[0]); r1[5].val[1] = vpadd_u8(r1[5].val[1], r1[5].val[1]); r1[5].val[1] = vpadd_u8(r1[5].val[1], r1[5].val[1]); r1[5].val[1] = vpadd_u8(r1[5].val[1], r1[5].val[1]); r1[6].val[0] = vpadd_u8(r1[6].val[0], r1[6].val[0]); r1[6].val[0] = vpadd_u8(r1[6].val[0], r1[6].val[0]); r1[6].val[0] = vpadd_u8(r1[6].val[0], r1[6].val[0]); r1[6].val[1] = vpadd_u8(r1[6].val[1], r1[6].val[1]); r1[6].val[1] = vpadd_u8(r1[6].val[1], r1[6].val[1]); r1[6].val[1] = vpadd_u8(r1[6].val[1], r1[6].val[1]); r1[7].val[0] = vpadd_u8(r1[7].val[0], r1[7].val[0]); r1[7].val[0] = vpadd_u8(r1[7].val[0], r1[7].val[0]); r1[7].val[0] = vpadd_u8(r1[7].val[0], r1[7].val[0]); r1[7].val[1] = vpadd_u8(r1[7].val[1], r1[7].val[1]); r1[7].val[1] = vpadd_u8(r1[7].val[1], r1[7].val[1]); r1[7].val[1] = vpadd_u8(r1[7].val[1], r1[7].val[1]); /* Shift packed 8-bit */ r0[0].val[0] = vshr_n_u8(r0[0].val[0], 1); r0[0].val[1] = vshr_n_u8(r0[0].val[1], 1); r0[1].val[0] = vshr_n_u8(r0[1].val[0], 1); r0[1].val[1] = vshr_n_u8(r0[1].val[1], 1); r0[2].val[0] = vshr_n_u8(r0[2].val[0], 1); r0[2].val[1] = vshr_n_u8(r0[2].val[1], 1); r0[3].val[0] = vshr_n_u8(r0[3].val[0], 1); r0[3].val[1] = vshr_n_u8(r0[3].val[1], 1); r0[4].val[0] = vshr_n_u8(r0[4].val[0], 1); r0[4].val[1] = vshr_n_u8(r0[4].val[1], 1); r0[5].val[0] = vshr_n_u8(r0[5].val[0], 1); r0[5].val[1] = vshr_n_u8(r0[5].val[1], 1); r0[6].val[0] = vshr_n_u8(r0[6].val[0], 1); r0[6].val[1] = vshr_n_u8(r0[6].val[1], 1); r0[7].val[0] = vshr_n_u8(r0[7].val[0], 1); r0[7].val[1] = vshr_n_u8(r0[7].val[1], 1); /* Store the created mask to the destination vector */ vst1_lane_u8(dest + 16 * j + 0 + i, r1[0].val[0], 0); vst1_lane_u8(dest + 16 * j + 1 + i, r1[0].val[1], 0); vst1_lane_u8(dest + 16 * j + 2 + i, r1[1].val[0], 0); vst1_lane_u8(dest + 16 * j + 3 + i, r1[1].val[1], 0); vst1_lane_u8(dest + 16 * j + 4 + i, r1[2].val[0], 0); vst1_lane_u8(dest + 16 * j + 5 + i, r1[2].val[1], 0); vst1_lane_u8(dest + 16 * j + 6 + i, r1[3].val[0], 0); vst1_lane_u8(dest + 16 * j + 7 + i, r1[3].val[1], 0); vst1_lane_u8(dest + 16 * j + 8 + i, r1[4].val[0], 0); vst1_lane_u8(dest + 16 * j + 9 + i, r1[4].val[1], 0); vst1_lane_u8(dest + 16 * j + 10 + i, r1[5].val[0], 0); vst1_lane_u8(dest + 16 * j + 11 + i, r1[5].val[1], 0); vst1_lane_u8(dest + 16 * j + 12 + i, r1[6].val[0], 0); vst1_lane_u8(dest + 16 * j + 13 + i, r1[6].val[1], 0); vst1_lane_u8(dest + 16 * j + 14 + i, r1[7].val[0], 0); vst1_lane_u8(dest + 16 * j + 15 + i, r1[7].val[1], 0); } } } /* Shuffle a block. This can never fail. */ int64_t bitshuffle_neon(void* _src, void* _dest, const size_t size, const size_t elem_size, void* tmp_buf) { size_t vectorized_chunk_size = 0; int64_t count; if (elem_size == 1 || elem_size == 2 || elem_size == 4) { vectorized_chunk_size = elem_size * 16; } else if (elem_size == 8 || elem_size == 16) { vectorized_chunk_size = elem_size * 8; } /* If the block size is too small to be vectorized, use the generic implementation. */ if (size * elem_size < vectorized_chunk_size) { count = bshuf_trans_bit_elem_scal((void*)_src, (void*)_dest, size, elem_size, tmp_buf); return count; } /* Optimized bitshuffle implementations */ switch (elem_size) { case 1: bitshuffle1_neon(_src, _dest, size, elem_size); break; case 2: bitshuffle2_neon(_src, _dest, size, elem_size); break; case 4: bitshuffle4_neon(_src, _dest, size, elem_size); break; case 8: bitshuffle8_neon(_src, _dest, size, elem_size); break; case 16: bitshuffle16_neon(_src, _dest, size, elem_size); break; default: /* Non-optimized bitshuffle */ count = bshuf_trans_bit_elem_scal((void*)_src, (void*)_dest, size, elem_size, tmp_buf); /* The non-optimized function covers the whole buffer, so we're done processing here. */ return count; } return (int64_t)size * (int64_t)elem_size; } /* Bitunshuffle a block. This can never fail. */ int64_t bitunshuffle_neon(void* _src, void* _dest, const size_t size, const size_t elem_size, void* tmp_buf) { size_t vectorized_chunk_size = 0; int64_t count; if (size * elem_size == 1 || size * elem_size == 2 || size * elem_size == 4) { vectorized_chunk_size = size * elem_size * 16; } else if (size * elem_size == 8 || size * elem_size == 16) { vectorized_chunk_size = size * elem_size * 8; } /* If the block size is too small to be vectorized, use the generic implementation. */ if (size * elem_size < vectorized_chunk_size) { count = bshuf_untrans_bit_elem_scal((void*)_src, (void*)_dest, size, elem_size, tmp_buf); return count; } /* Optimized bitunshuffle implementations */ switch (elem_size) { case 1: bitunshuffle1_neon(_src, _dest, size, elem_size); break; case 2: bitunshuffle2_neon(_src, _dest, size, elem_size); break; case 4: bitunshuffle4_neon(_src, _dest, size, elem_size); break; case 8: bitunshuffle8_neon(_src, _dest, size, elem_size); break; case 16: bitunshuffle16_neon(_src, _dest, size, elem_size); break; default: /* Non-optimized bitunshuffle */ count = bshuf_untrans_bit_elem_scal((void*)_src, (void*)_dest, size, elem_size, tmp_buf); /* The non-optimized function covers the whole buffer, so we're done processing here. */ return count; } return (int64_t)size * (int64_t)elem_size; } #endif /* defined(__ARM_NEON) */ zmat-0.9.9/src/blosc2/blosc/bitshuffle-neon.h0000644000175200007730000000213414515254731021255 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) Note: Adapted for NEON by Lucian Marc. See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /* NEON-accelerated bitshuffle/bitunshuffle routines. */ #ifndef BITSHUFFLE_NEON_H #define BITSHUFFLE_NEON_H #include "blosc2/blosc2-common.h" #ifdef __cplusplus extern "C" { #endif /** NEON-accelerated bitshuffle routine. */ BLOSC_NO_EXPORT int64_t bitshuffle_neon(void* _src, void* _dest, const size_t blocksize, const size_t bytesoftype, void* tmp_buf); /** NEON-accelerated bitunshuffle routine. */ BLOSC_NO_EXPORT int64_t bitunshuffle_neon(void* _src, void* _dest, const size_t blocksize, const size_t bytesoftype, void* tmp_buf); #ifdef __cplusplus } #endif #endif /* BITSHUFFLE_NEON_H */ zmat-0.9.9/src/blosc2/blosc/blosclz.h0000644000175200007730000000436514515254731017645 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /********************************************************************* The code in this file is heavily based on FastLZ, a lightning-fast lossless compression library. See LICENSES/FASTLZ.txt for details about copyright and rights to use. **********************************************************************/ #ifndef BLOSCLZ_H #define BLOSCLZ_H #include "context.h" #if defined (__cplusplus) extern "C" { #endif #define BLOSCLZ_VERSION_STRING "2.5.2" /** Compress a block of data in the input buffer and returns the size of compressed block. The size of input buffer is specified by length. The minimum input buffer size is 16. The output buffer must be at least 5% larger than the input buffer and can not be smaller than 66 bytes. If the input is not compressible, or output does not fit in maxout bytes, the return value will be 0 and you will have to discard the output buffer. The acceleration parameter is related with the frequency for updating the internal hash. An acceleration of 1 means that the internal hash is updated at full rate. A value < 1 is not allowed and will be silently set to 1. The input buffer and the output buffer can not overlap. */ int blosclz_compress(int opt_level, const void* input, int length, void* output, int maxout, blosc2_context* ctx); /** Decompress a block of compressed data and returns the size of the decompressed block. If error occurs, e.g. the compressed data is corrupted or the output buffer is not large enough, then 0 (zero) will be returned instead. The input buffer and the output buffer can not overlap. Decompression is memory safe and guaranteed not to write the output buffer more than what is specified in maxout. */ int blosclz_decompress(const void* input, int length, void* output, int maxout); #if defined (__cplusplus) } #endif #endif /* BLOSCLZ_H */ zmat-0.9.9/src/blosc2/blosc/shuffle.c0000644000175200007730000004601114515254731017616 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include "shuffle.h" #include "blosc2/blosc2-common.h" #include "shuffle-generic.h" #include "bitshuffle-generic.h" #include #include #include #if !defined(__clang__) && defined(__GNUC__) && defined(__GNUC_MINOR__) && \ __GNUC__ >= 5 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 8) #define HAVE_CPU_FEAT_INTRIN #endif /* Include hardware-accelerated shuffle/unshuffle routines based on the target architecture. Note that a target architecture may support more than one type of acceleration!*/ #if defined(SHUFFLE_USE_AVX2) #include "shuffle-avx2.h" #include "bitshuffle-avx2.h" #endif /* defined(SHUFFLE_USE_AVX2) */ #if defined(SHUFFLE_USE_SSE2) #include "shuffle-sse2.h" #include "bitshuffle-sse2.h" #endif /* defined(SHUFFLE_USE_SSE2) */ #if defined(SHUFFLE_USE_NEON) #if defined(__linux__) #include #ifdef ARM_ASM_HWCAP #include #endif #endif #include "shuffle-neon.h" #include "bitshuffle-neon.h" #endif /* defined(SHUFFLE_USE_NEON) */ #if defined(SHUFFLE_USE_ALTIVEC) #include "shuffle-altivec.h" #include "bitshuffle-altivec.h" #endif /* defined(SHUFFLE_USE_ALTIVEC) */ /* Define function pointer types for shuffle/unshuffle routines. */ typedef void(* shuffle_func)(const int32_t, const int32_t, const uint8_t*, const uint8_t*); typedef void(* unshuffle_func)(const int32_t, const int32_t, const uint8_t*, const uint8_t*); // For bitshuffle, everything is done in terms of size_t and int64_t (return value) // and although this is not strictly necessary for Blosc, it does not hurt either typedef int64_t(* bitshuffle_func)(void*, void*, const size_t, const size_t, void*); typedef int64_t(* bitunshuffle_func)(void*, void*, const size_t, const size_t, void*); /* An implementation of shuffle/unshuffle routines. */ typedef struct shuffle_implementation { /* Name of this implementation. */ const char* name; /* Function pointer to the shuffle routine for this implementation. */ shuffle_func shuffle; /* Function pointer to the unshuffle routine for this implementation. */ unshuffle_func unshuffle; /* Function pointer to the bitshuffle routine for this implementation. */ bitshuffle_func bitshuffle; /* Function pointer to the bitunshuffle routine for this implementation. */ bitunshuffle_func bitunshuffle; } shuffle_implementation_t; typedef enum { BLOSC_HAVE_NOTHING = 0, BLOSC_HAVE_SSE2 = 1, BLOSC_HAVE_AVX2 = 2, BLOSC_HAVE_NEON = 4, BLOSC_HAVE_ALTIVEC = 8 } blosc_cpu_features; /* Detect hardware and set function pointers to the best shuffle/unshuffle implementations supported by the host processor. */ #if defined(SHUFFLE_USE_AVX2) || defined(SHUFFLE_USE_SSE2) /* Intel/i686 */ /* Disabled the __builtin_cpu_supports() call, as it has issues with new versions of gcc (like 5.3.1 in forthcoming ubuntu/xenial: "undefined symbol: __cpu_model" For a similar report, see: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/ZM2L65WIZEEQHHLFERZYD5FAG7QY2OGB/ */ #if defined(HAVE_CPU_FEAT_INTRIN) && 0 static blosc_cpu_features blosc_get_cpu_features(void) { blosc_cpu_features cpu_features = BLOSC_HAVE_NOTHING; if (__builtin_cpu_supports("sse2")) { cpu_features |= BLOSC_HAVE_SSE2; } if (__builtin_cpu_supports("avx2")) { cpu_features |= BLOSC_HAVE_AVX2; } return cpu_features; } #else #if defined(_MSC_VER) && !defined(__clang__) #include /* Needed for _xgetbv */ #include /* Needed for __cpuid */ #else /* Implement the __cpuid and __cpuidex intrinsics for GCC, Clang, and others using inline assembly. */ __attribute__((always_inline)) static inline void __cpuidex(int32_t cpuInfo[4], int32_t function_id, int32_t subfunction_id) { __asm__ __volatile__ ( # if defined(__i386__) && defined (__PIC__) /* Can't clobber ebx with PIC running under 32-bit, so it needs to be manually restored. https://software.intel.com/en-us/articles/how-to-detect-new-instruction-support-in-the-4th-generation-intel-core-processor-family */ "movl %%ebx, %%edi\n\t" "cpuid\n\t" "xchgl %%ebx, %%edi": "=D" (cpuInfo[1]), #else "cpuid": "=b" (cpuInfo[1]), #endif /* defined(__i386) && defined(__PIC__) */ "=a" (cpuInfo[0]), "=c" (cpuInfo[2]), "=d" (cpuInfo[3]) : "a" (function_id), "c" (subfunction_id) ); } #define __cpuid(cpuInfo, function_id) __cpuidex(cpuInfo, function_id, 0) #define _XCR_XFEATURE_ENABLED_MASK 0 // GCC folks added _xgetbv in immintrin.h starting in GCC 9 // See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71659 #if !(defined(_IMMINTRIN_H_INCLUDED) && (BLOSC_GCC_VERSION >= 900)) && !defined(__IMMINTRIN_H) /* Reads the content of an extended control register. https://software.intel.com/en-us/articles/how-to-detect-new-instruction-support-in-the-4th-generation-intel-core-processor-family */ static inline uint64_t _xgetbv(uint32_t xcr) { uint32_t eax, edx; __asm__ __volatile__ ( /* "xgetbv" This is specified as raw instruction bytes due to some older compilers having issues with the mnemonic form. */ ".byte 0x0f, 0x01, 0xd0": "=a" (eax), "=d" (edx) : "c" (xcr) ); return ((uint64_t)edx << 32) | eax; } #endif // !(defined(_IMMINTRIN_H_INCLUDED) && (BLOSC_GCC_VERSION >= 900)) && !defined(__IMMINTRIN_H) #endif /* defined(_MSC_VER) */ #ifndef _XCR_XFEATURE_ENABLED_MASK #define _XCR_XFEATURE_ENABLED_MASK 0x0 #endif static blosc_cpu_features blosc_get_cpu_features(void) { blosc_cpu_features result = BLOSC_HAVE_NOTHING; /* Holds the values of eax, ebx, ecx, edx set by the `cpuid` instruction */ int32_t cpu_info[4]; /* Get the number of basic functions available. */ __cpuid(cpu_info, 0); int32_t max_basic_function_id = cpu_info[0]; /* Check for SSE-based features and required OS support */ __cpuid(cpu_info, 1); const bool sse2_available = (cpu_info[3] & (1 << 26)) != 0; const bool sse3_available = (cpu_info[2] & (1 << 0)) != 0; const bool ssse3_available = (cpu_info[2] & (1 << 9)) != 0; const bool sse41_available = (cpu_info[2] & (1 << 19)) != 0; const bool sse42_available = (cpu_info[2] & (1 << 20)) != 0; const bool xsave_available = (cpu_info[2] & (1 << 26)) != 0; const bool xsave_enabled_by_os = (cpu_info[2] & (1 << 27)) != 0; /* Check for AVX-based features, if the processor supports extended features. */ bool avx2_available = false; bool avx512bw_available = false; if (max_basic_function_id >= 7) { __cpuid(cpu_info, 7); avx2_available = (cpu_info[1] & (1 << 5)) != 0; avx512bw_available = (cpu_info[1] & (1 << 30)) != 0; } /* Even if certain features are supported by the CPU, they may not be supported by the OS (in which case using them would crash the process or system). If xsave is available and enabled by the OS, check the contents of the extended control register XCR0 to see if the CPU features are enabled. */ bool xmm_state_enabled = false; bool ymm_state_enabled = false; //bool zmm_state_enabled = false; // commented this out for avoiding an 'unused variable' warning #if defined(_XCR_XFEATURE_ENABLED_MASK) if (xsave_available && xsave_enabled_by_os && ( sse2_available || sse3_available || ssse3_available || sse41_available || sse42_available || avx2_available || avx512bw_available)) { /* Determine which register states can be restored by the OS. */ uint64_t xcr0_contents = _xgetbv(_XCR_XFEATURE_ENABLED_MASK); xmm_state_enabled = (xcr0_contents & (1UL << 1)) != 0; ymm_state_enabled = (xcr0_contents & (1UL << 2)) != 0; /* Require support for both the upper 256-bits of zmm0-zmm15 to be restored as well as all of zmm16-zmm31 and the opmask registers. */ //zmm_state_enabled = (xcr0_contents & 0x70) == 0x70; } #endif /* defined(_XCR_XFEATURE_ENABLED_MASK) */ #if defined(BLOSC_DUMP_CPU_INFO) printf("Shuffle CPU Information:\n"); printf("SSE2 available: %s\n", sse2_available ? "True" : "False"); printf("SSE3 available: %s\n", sse3_available ? "True" : "False"); printf("SSSE3 available: %s\n", ssse3_available ? "True" : "False"); printf("SSE4.1 available: %s\n", sse41_available ? "True" : "False"); printf("SSE4.2 available: %s\n", sse42_available ? "True" : "False"); printf("AVX2 available: %s\n", avx2_available ? "True" : "False"); printf("AVX512BW available: %s\n", avx512bw_available ? "True" : "False"); printf("XSAVE available: %s\n", xsave_available ? "True" : "False"); printf("XSAVE enabled: %s\n", xsave_enabled_by_os ? "True" : "False"); printf("XMM state enabled: %s\n", xmm_state_enabled ? "True" : "False"); printf("YMM state enabled: %s\n", ymm_state_enabled ? "True" : "False"); //printf("ZMM state enabled: %s\n", zmm_state_enabled ? "True" : "False"); #endif /* defined(BLOSC_DUMP_CPU_INFO) */ /* Using the gathered CPU information, determine which implementation to use. */ /* technically could fail on sse2 cpu on os without xmm support, but that * shouldn't exist anymore */ if (sse2_available) { result |= BLOSC_HAVE_SSE2; } if (xmm_state_enabled && ymm_state_enabled && avx2_available) { result |= BLOSC_HAVE_AVX2; } return result; } #endif /* HAVE_CPU_FEAT_INTRIN */ #elif defined(SHUFFLE_USE_NEON) /* ARM-NEON */ static blosc_cpu_features blosc_get_cpu_features(void) { blosc_cpu_features cpu_features = BLOSC_HAVE_NOTHING; #if defined(__aarch64__) /* aarch64 always has NEON */ cpu_features |= BLOSC_HAVE_NEON; #else if (getauxval(AT_HWCAP) & HWCAP_ARM_NEON) { cpu_features |= BLOSC_HAVE_NEON; } #endif return cpu_features; } #elif defined(SHUFFLE_USE_ALTIVEC) /* POWER9-ALTIVEC preliminary test*/ static blosc_cpu_features blosc_get_cpu_features(void) { blosc_cpu_features cpu_features = BLOSC_HAVE_NOTHING; cpu_features |= BLOSC_HAVE_ALTIVEC; return cpu_features; } #else /* No hardware acceleration supported for the target architecture. */ #if defined(_MSC_VER) #pragma message("Hardware-acceleration detection not implemented for the target architecture. Only the generic shuffle/unshuffle routines will be available.") #else #warning Hardware-acceleration detection not implemented for the target architecture. Only the generic shuffle/unshuffle routines will be available. #endif static blosc_cpu_features blosc_get_cpu_features(void) { return BLOSC_HAVE_NOTHING; } #endif /* defined(SHUFFLE_USE_AVX2) || defined(SHUFFLE_USE_SSE2) */ static shuffle_implementation_t get_shuffle_implementation(void) { blosc_cpu_features cpu_features = blosc_get_cpu_features(); #if defined(SHUFFLE_USE_AVX2) if (cpu_features & BLOSC_HAVE_AVX2) { shuffle_implementation_t impl_avx2; impl_avx2.name = "avx2"; impl_avx2.shuffle = (shuffle_func)shuffle_avx2; impl_avx2.unshuffle = (unshuffle_func)unshuffle_avx2; impl_avx2.bitshuffle = (bitshuffle_func)bshuf_trans_bit_elem_avx2; impl_avx2.bitunshuffle = (bitunshuffle_func)bshuf_untrans_bit_elem_avx2; return impl_avx2; } #endif /* defined(SHUFFLE_USE_AVX2) */ #if defined(SHUFFLE_USE_SSE2) if (cpu_features & BLOSC_HAVE_SSE2) { shuffle_implementation_t impl_sse2; impl_sse2.name = "sse2"; impl_sse2.shuffle = (shuffle_func)shuffle_sse2; impl_sse2.unshuffle = (unshuffle_func)unshuffle_sse2; impl_sse2.bitshuffle = (bitshuffle_func)bshuf_trans_bit_elem_sse2; impl_sse2.bitunshuffle = (bitunshuffle_func)bshuf_untrans_bit_elem_sse2; return impl_sse2; } #endif /* defined(SHUFFLE_USE_SSE2) */ #if defined(SHUFFLE_USE_NEON) if (cpu_features & BLOSC_HAVE_NEON) { shuffle_implementation_t impl_neon; impl_neon.name = "neon"; impl_neon.shuffle = (shuffle_func)shuffle_neon; impl_neon.unshuffle = (unshuffle_func)unshuffle_neon; //impl_neon.shuffle = (shuffle_func)shuffle_generic; //impl_neon.unshuffle = (unshuffle_func)unshuffle_generic; //impl_neon.bitshuffle = (bitshuffle_func)bitshuffle_neon; //impl_neon.bitunshuffle = (bitunshuffle_func)bitunshuffle_neon; // The current bitshuffle optimized for NEON is not any faster // (in fact, it is pretty much slower) than the scalar implementation. // Also, bitshuffle_neon (forward direction) is broken for 1, 2 and 4 bytes. // So, let's use the the scalar one, which is pretty fast, at least on a M1 CPU. impl_neon.bitshuffle = (bitshuffle_func)bshuf_trans_bit_elem_scal; impl_neon.bitunshuffle = (bitunshuffle_func)bshuf_untrans_bit_elem_scal; return impl_neon; } #endif /* defined(SHUFFLE_USE_NEON) */ #if defined(SHUFFLE_USE_ALTIVEC) if (cpu_features & BLOSC_HAVE_ALTIVEC) { shuffle_implementation_t impl_altivec; impl_altivec.name = "altivec"; impl_altivec.shuffle = (shuffle_func)shuffle_altivec; impl_altivec.unshuffle = (unshuffle_func)unshuffle_altivec; impl_altivec.bitshuffle = (bitshuffle_func)bshuf_trans_bit_elem_altivec; impl_altivec.bitunshuffle = (bitunshuffle_func)bshuf_untrans_bit_elem_altivec; return impl_altivec; } #endif /* defined(SHUFFLE_USE_ALTIVEC) */ /* Processor doesn't support any of the hardware-accelerated implementations, so use the generic implementation. */ shuffle_implementation_t impl_generic; impl_generic.name = "generic"; impl_generic.shuffle = (shuffle_func)shuffle_generic; impl_generic.unshuffle = (unshuffle_func)unshuffle_generic; impl_generic.bitshuffle = (bitshuffle_func)bshuf_trans_bit_elem_scal; impl_generic.bitunshuffle = (bitunshuffle_func)bshuf_untrans_bit_elem_scal; return impl_generic; } /* Flag indicating whether the implementation has been initialized. Zero means it hasn't been initialized, non-zero means it has. */ static int32_t implementation_initialized; /* The dynamically-chosen shuffle/unshuffle implementation. This is only safe to use once `implementation_initialized` is set. */ static shuffle_implementation_t host_implementation; /* Initialize the shuffle implementation, if necessary. */ #if defined(__GNUC__) || defined(__clang__) __attribute__((always_inline)) #endif static #if defined(_MSC_VER) __forceinline #else inline #endif void init_shuffle_implementation(void) { /* Initialization could (in rare cases) take place concurrently on multiple threads, but it shouldn't matter because the initialization should return the same result on each thread (so the implementation will be the same). Since that's the case we can avoid complicated synchronization here and get a small performance benefit because we don't need to perform a volatile load on the initialization variable each time this function is called. */ #if defined(__GNUC__) || defined(__clang__) if (__builtin_expect(!implementation_initialized, 0)) { #else if (!implementation_initialized) { #endif /* Initialize the implementation. */ host_implementation = get_shuffle_implementation(); /* Set the flag indicating the implementation has been initialized. */ implementation_initialized = 1; } } /* Shuffle a block by dynamically dispatching to the appropriate hardware-accelerated routine at run-time. */ void shuffle(const int32_t bytesoftype, const int32_t blocksize, const uint8_t* _src, const uint8_t* _dest) { /* Initialize the shuffle implementation if necessary. */ init_shuffle_implementation(); /* The implementation is initialized. Dispatch to it's shuffle routine. */ (host_implementation.shuffle)(bytesoftype, blocksize, _src, _dest); } /* Unshuffle a block by dynamically dispatching to the appropriate hardware-accelerated routine at run-time. */ void unshuffle(const int32_t bytesoftype, const int32_t blocksize, const uint8_t* _src, const uint8_t* _dest) { /* Initialize the shuffle implementation if necessary. */ init_shuffle_implementation(); /* The implementation is initialized. Dispatch to it's unshuffle routine. */ (host_implementation.unshuffle)(bytesoftype, blocksize, _src, _dest); } /* Bit-shuffle a block by dynamically dispatching to the appropriate hardware-accelerated routine at run-time. */ int32_t bitshuffle(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, const uint8_t *_dest, const uint8_t *_tmp) { /* Initialize the shuffle implementation if necessary. */ init_shuffle_implementation(); size_t size = blocksize / bytesoftype; /* bitshuffle only supports a number of elements that is a multiple of 8. */ size -= size % 8; int ret = (int) (host_implementation.bitshuffle)((void *) _src, (void *) _dest, size, bytesoftype, (void *) _tmp); if (ret < 0) { // Some error in bitshuffle (should not happen) fprintf(stderr, "the impossible happened: the bitshuffle filter failed!"); return ret; } // Copy the leftovers size_t offset = size * bytesoftype; memcpy((void *) (_dest + offset), (void *) (_src + offset), blocksize - offset); return blocksize; } /* Bit-unshuffle a block by dynamically dispatching to the appropriate hardware-accelerated routine at run-time. */ int32_t bitunshuffle(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, const uint8_t *_dest, const uint8_t *_tmp, const uint8_t format_version) { /* Initialize the shuffle implementation if necessary. */ init_shuffle_implementation(); size_t size = blocksize / bytesoftype; if (format_version == 2) { /* Starting from version 3, bitshuffle() works differently */ if ((size % 8) == 0) { /* The number of elems is a multiple of 8 which is supported by bitshuffle. */ int ret = (int) (host_implementation.bitunshuffle)((void *) _src, (void *) _dest, blocksize / bytesoftype, bytesoftype, (void *) _tmp); if (ret < 0) { // Some error in bitshuffle (should not happen) fprintf(stderr, "the impossible happened: the bitunshuffle filter failed!"); return ret; } /* Copy the leftovers (we do so starting from c-blosc 1.18 on) */ size_t offset = size * bytesoftype; memcpy((void *) (_dest + offset), (void *) (_src + offset), blocksize - offset); } else { memcpy((void *) _dest, (void *) _src, blocksize); } } else { /* bitshuffle only supports a number of bytes that is a multiple of 8. */ size -= size % 8; int ret = (int) (host_implementation.bitunshuffle)((void *) _src, (void *) _dest, size, bytesoftype, (void *) _tmp); if (ret < 0) { fprintf(stderr, "the impossible happened: the bitunshuffle filter failed!"); return ret; } /* Copy the leftovers */ size_t offset = size * bytesoftype; memcpy((void *) (_dest + offset), (void *) (_src + offset), blocksize - offset); } return blocksize; } zmat-0.9.9/src/blosc2/blosc/win32/0000755000175200007730000000000014515254731016756 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/blosc/win32/pthread.h0000644000175200007730000000600514515254731020557 0ustar rlaboissrlaboiss/* * Code for simulating pthreads API on Windows. This is Git-specific, * but it is enough for Numexpr needs too. * * Copyright (C) 2009 Andrzej K. Haczewski * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal * in the Software without restriction, including without limitation the rights * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN * THE SOFTWARE. * * DISCLAIMER: The implementation is Git-specific, it is subset of original * Pthreads API, without lots of other features that Git doesn't use. * Git also makes sure that the passed arguments are valid, so there's * no need for double-checking. */ #ifndef PTHREAD_H #define PTHREAD_H #ifndef WIN32_LEAN_AND_MEAN #define WIN32_LEAN_AND_MEAN #endif #include "windows.h" /* * Defines that adapt Windows API threads to pthreads API */ #define pthread_mutex_t CRITICAL_SECTION #define pthread_mutex_init(a,b) InitializeCriticalSection((a)) #define pthread_mutex_destroy(a) DeleteCriticalSection((a)) #define pthread_mutex_lock EnterCriticalSection #define pthread_mutex_unlock LeaveCriticalSection /* * Implement simple condition variable for Windows threads, based on ACE * implementation. */ typedef struct { LONG waiters; int was_broadcast; CRITICAL_SECTION waiters_lock; HANDLE sema; HANDLE continue_broadcast; } pthread_cond_t; extern int pthread_cond_init(pthread_cond_t *cond, const void *unused); extern int pthread_cond_destroy(pthread_cond_t *cond); extern int pthread_cond_wait(pthread_cond_t *cond, CRITICAL_SECTION *mutex); extern int pthread_cond_signal(pthread_cond_t *cond); extern int pthread_cond_broadcast(pthread_cond_t *cond); /* * Simple thread creation implementation using pthread API */ typedef struct { HANDLE handle; void *(*start_routine)(void*); void *arg; } pthread_t; extern int pthread_create(pthread_t *thread, const void *unused, void *(*start_routine)(void*), void *arg); /* * To avoid the need of copying a struct, we use small macro wrapper to pass * pointer to win32_pthread_join instead. */ #define pthread_join(a, b) win32_pthread_join(&(a), (b)) extern int win32_pthread_join(pthread_t *thread, void **value_ptr); #endif /* PTHREAD_H */ zmat-0.9.9/src/blosc2/blosc/win32/stdint-windows.h0000644000175200007730000001764514515254731022141 0ustar rlaboissrlaboiss// ISO C9x compliant stdint.h for Microsoft Visual Studio // Based on ISO/IEC 9899:TC2 Committee draft (May 6, 2005) WG14/N1124 // // Copyright (c) 2006-2013 Alexander Chemeris // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, // this list of conditions and the following disclaimer. // // 2. Redistributions in binary form must reproduce the above copyright // notice, this list of conditions and the following disclaimer in the // documentation and/or other materials provided with the distribution. // // 3. Neither the name of the product nor the names of its contributors may // be used to endorse or promote products derived from this software // without specific prior written permission. // // THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED // WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF // MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO // EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, // PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; // OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, // WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR // OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF // ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. // /////////////////////////////////////////////////////////////////////////////// #ifndef _MSC_VER // [ #error "Use this header only with Microsoft Visual C++ compilers!" #endif // _MSC_VER ] #ifndef _MSC_STDINT_H_ // [ #define _MSC_STDINT_H_ #if _MSC_VER > 1000 #pragma once #endif #if _MSC_VER >= 1600 // [ #include #else // ] _MSC_VER >= 1600 [ #include // For Visual Studio 6 in C++ mode and for many Visual Studio versions when // compiling for ARM we should wrap include with 'extern "C++" {}' // or compiler give many errors like this: // error C2733: second C linkage of overloaded function 'wmemchr' not allowed #ifdef __cplusplus extern "C" { #endif # include #ifdef __cplusplus } #endif // Define _W64 macros to mark types changing their size, like intptr_t. #ifndef _W64 # if !defined(__midl) && (defined(_X86_) || defined(_M_IX86)) && _MSC_VER >= 1300 # define _W64 __w64 # else # define _W64 # endif #endif // 7.18.1 Integer types // 7.18.1.1 Exact-width integer types // Visual Studio 6 and Embedded Visual C++ 4 doesn't // realize that, e.g. char has the same size as __int8 // so we give up on __intX for them. #if (_MSC_VER < 1300) typedef signed char int8_t; typedef signed short int16_t; typedef signed int int32_t; typedef unsigned char uint8_t; typedef unsigned short uint16_t; typedef unsigned int uint32_t; #else typedef signed __int8 int8_t; typedef signed __int16 int16_t; typedef signed __int32 int32_t; typedef unsigned __int8 uint8_t; typedef unsigned __int16 uint16_t; typedef unsigned __int32 uint32_t; #endif typedef signed __int64 int64_t; typedef unsigned __int64 uint64_t; // 7.18.1.2 Minimum-width integer types typedef int8_t int_least8_t; typedef int16_t int_least16_t; typedef int32_t int_least32_t; typedef int64_t int_least64_t; typedef uint8_t uint_least8_t; typedef uint16_t uint_least16_t; typedef uint32_t uint_least32_t; typedef uint64_t uint_least64_t; // 7.18.1.3 Fastest minimum-width integer types typedef int8_t int_fast8_t; typedef int16_t int_fast16_t; typedef int32_t int_fast32_t; typedef int64_t int_fast64_t; typedef uint8_t uint_fast8_t; typedef uint16_t uint_fast16_t; typedef uint32_t uint_fast32_t; typedef uint64_t uint_fast64_t; // 7.18.1.4 Integer types capable of holding object pointers #ifdef _WIN64 // [ typedef signed __int64 intptr_t; typedef unsigned __int64 uintptr_t; #else // _WIN64 ][ typedef _W64 signed int intptr_t; typedef _W64 unsigned int uintptr_t; #endif // _WIN64 ] // 7.18.1.5 Greatest-width integer types typedef int64_t intmax_t; typedef uint64_t uintmax_t; // 7.18.2 Limits of specified-width integer types #if !defined(__cplusplus) || defined(__STDC_LIMIT_MACROS) // [ See footnote 220 at page 257 and footnote 221 at page 259 // 7.18.2.1 Limits of exact-width integer types #define INT8_MIN ((int8_t)_I8_MIN) #define INT8_MAX _I8_MAX #define INT16_MIN ((int16_t)_I16_MIN) #define INT16_MAX _I16_MAX #define INT32_MIN ((int32_t)_I32_MIN) #define INT32_MAX _I32_MAX #define INT64_MIN ((int64_t)_I64_MIN) #define INT64_MAX _I64_MAX #define UINT8_MAX _UI8_MAX #define UINT16_MAX _UI16_MAX #define UINT32_MAX _UI32_MAX #define UINT64_MAX _UI64_MAX // 7.18.2.2 Limits of minimum-width integer types #define INT_LEAST8_MIN INT8_MIN #define INT_LEAST8_MAX INT8_MAX #define INT_LEAST16_MIN INT16_MIN #define INT_LEAST16_MAX INT16_MAX #define INT_LEAST32_MIN INT32_MIN #define INT_LEAST32_MAX INT32_MAX #define INT_LEAST64_MIN INT64_MIN #define INT_LEAST64_MAX INT64_MAX #define UINT_LEAST8_MAX UINT8_MAX #define UINT_LEAST16_MAX UINT16_MAX #define UINT_LEAST32_MAX UINT32_MAX #define UINT_LEAST64_MAX UINT64_MAX // 7.18.2.3 Limits of fastest minimum-width integer types #define INT_FAST8_MIN INT8_MIN #define INT_FAST8_MAX INT8_MAX #define INT_FAST16_MIN INT16_MIN #define INT_FAST16_MAX INT16_MAX #define INT_FAST32_MIN INT32_MIN #define INT_FAST32_MAX INT32_MAX #define INT_FAST64_MIN INT64_MIN #define INT_FAST64_MAX INT64_MAX #define UINT_FAST8_MAX UINT8_MAX #define UINT_FAST16_MAX UINT16_MAX #define UINT_FAST32_MAX UINT32_MAX #define UINT_FAST64_MAX UINT64_MAX // 7.18.2.4 Limits of integer types capable of holding object pointers #ifdef _WIN64 // [ # define INTPTR_MIN INT64_MIN # define INTPTR_MAX INT64_MAX # define UINTPTR_MAX UINT64_MAX #else // _WIN64 ][ # define INTPTR_MIN INT32_MIN # define INTPTR_MAX INT32_MAX # define UINTPTR_MAX UINT32_MAX #endif // _WIN64 ] // 7.18.2.5 Limits of greatest-width integer types #define INTMAX_MIN INT64_MIN #define INTMAX_MAX INT64_MAX #define UINTMAX_MAX UINT64_MAX // 7.18.3 Limits of other integer types #ifdef _WIN64 // [ # define PTRDIFF_MIN _I64_MIN # define PTRDIFF_MAX _I64_MAX #else // _WIN64 ][ # define PTRDIFF_MIN _I32_MIN # define PTRDIFF_MAX _I32_MAX #endif // _WIN64 ] #define SIG_ATOMIC_MIN INT_MIN #define SIG_ATOMIC_MAX INT_MAX #ifndef SIZE_MAX // [ # ifdef _WIN64 // [ # define SIZE_MAX _UI64_MAX # else // _WIN64 ][ # define SIZE_MAX _UI32_MAX # endif // _WIN64 ] #endif // SIZE_MAX ] // WCHAR_MIN and WCHAR_MAX are also defined in #ifndef WCHAR_MIN // [ # define WCHAR_MIN 0 #endif // WCHAR_MIN ] #ifndef WCHAR_MAX // [ # define WCHAR_MAX _UI16_MAX #endif // WCHAR_MAX ] #define WINT_MIN 0 #define WINT_MAX _UI16_MAX #endif // __STDC_LIMIT_MACROS ] // 7.18.4 Limits of other integer types #if !defined(__cplusplus) || defined(__STDC_CONSTANT_MACROS) // [ See footnote 224 at page 260 // 7.18.4.1 Macros for minimum-width integer constants #define INT8_C(val) val##i8 #define INT16_C(val) val##i16 #define INT32_C(val) val##i32 #define INT64_C(val) val##i64 #define UINT8_C(val) val##ui8 #define UINT16_C(val) val##ui16 #define UINT32_C(val) val##ui32 #define UINT64_C(val) val##ui64 // 7.18.4.2 Macros for greatest-width integer constants // These #ifndef's are needed to prevent collisions with . // Check out Issue 9 for the details. #ifndef INTMAX_C // [ # define INTMAX_C INT64_C #endif // INTMAX_C ] #ifndef UINTMAX_C // [ # define UINTMAX_C UINT64_C #endif // UINTMAX_C ] #endif // __STDC_CONSTANT_MACROS ] #endif // _MSC_VER >= 1600 ] #endif // _MSC_STDINT_H_ ] zmat-0.9.9/src/blosc2/blosc/win32/pthread.c0000644000175200007730000001503114515254731020551 0ustar rlaboissrlaboiss/* * Code for simulating pthreads API on Windows. This is Git-specific, * but it is enough for Numexpr needs too. * * Copyright (C) 2009 Andrzej K. Haczewski * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal * in the Software without restriction, including without limitation the rights * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN * THE SOFTWARE. * * DISCLAIMER: The implementation is Git-specific, it is subset of original * Pthreads API, without lots of other features that Git doesn't use. * Git also makes sure that the passed arguments are valid, so there's * no need for double-checking. */ #ifndef PTHREAD_C #define PTHREAD_C #include "pthread.h" #include "stdio.h" #include "stdlib.h" #include "process.h" #include "errno.h" #include "limits.h" #define PTHREAD_UNUSED_PARAM(x) ((void)(x)) void die(const char *err, ...) { printf("%s", err); exit(-1); } static unsigned __stdcall win32_start_routine(void *arg) { pthread_t *thread = (pthread_t*)arg; thread->arg = thread->start_routine(thread->arg); return 0; } int pthread_create(pthread_t *thread, const void *unused, void *(*start_routine)(void*), void *arg) { PTHREAD_UNUSED_PARAM(unused); thread->arg = arg; thread->start_routine = start_routine; thread->handle = (HANDLE) _beginthreadex(NULL, 0, win32_start_routine, thread, 0, NULL); if (!thread->handle) return errno; else return 0; } int win32_pthread_join(pthread_t *thread, void **value_ptr) { DWORD result = WaitForSingleObject(thread->handle, INFINITE); switch (result) { case WAIT_OBJECT_0: if (value_ptr) *value_ptr = thread->arg; return 0; case WAIT_ABANDONED: return EINVAL; default: return GetLastError(); } } int pthread_cond_init(pthread_cond_t *cond, const void *unused) { PTHREAD_UNUSED_PARAM(unused); cond->waiters = 0; cond->was_broadcast = 0; InitializeCriticalSection(&cond->waiters_lock); cond->sema = CreateSemaphore(NULL, 0, LONG_MAX, NULL); if (!cond->sema) die("CreateSemaphore() failed"); cond->continue_broadcast = CreateEvent(NULL, /* security */ FALSE, /* auto-reset */ FALSE, /* not signaled */ NULL); /* name */ if (!cond->continue_broadcast) die("CreateEvent() failed"); return 0; } int pthread_cond_destroy(pthread_cond_t *cond) { CloseHandle(cond->sema); CloseHandle(cond->continue_broadcast); DeleteCriticalSection(&cond->waiters_lock); return 0; } int pthread_cond_wait(pthread_cond_t *cond, CRITICAL_SECTION *mutex) { int last_waiter; EnterCriticalSection(&cond->waiters_lock); cond->waiters++; LeaveCriticalSection(&cond->waiters_lock); /* * Unlock external mutex and wait for signal. * NOTE: we've held mutex locked long enough to increment * waiters count above, so there's no problem with * leaving mutex unlocked before we wait on semaphore. */ LeaveCriticalSection(mutex); /* let's wait - ignore return value */ WaitForSingleObject(cond->sema, INFINITE); /* * Decrease waiters count. If we are the last waiter, then we must * notify the broadcasting thread that it can continue. * But if we continued due to cond_signal, we do not have to do that * because the signaling thread knows that only one waiter continued. */ EnterCriticalSection(&cond->waiters_lock); cond->waiters--; last_waiter = cond->was_broadcast && cond->waiters == 0; LeaveCriticalSection(&cond->waiters_lock); if (last_waiter) { /* * cond_broadcast was issued while mutex was held. This means * that all other waiters have continued, but are contending * for the mutex at the end of this function because the * broadcasting thread did not leave cond_broadcast, yet. * (This is so that it can be sure that each waiter has * consumed exactly one slice of the semaphore.) * The last waiter must tell the broadcasting thread that it * can go on. */ SetEvent(cond->continue_broadcast); /* * Now we go on to contend with all other waiters for * the mutex. Auf in den Kampf! */ } /* lock external mutex again */ EnterCriticalSection(mutex); return 0; } /* * IMPORTANT: This implementation requires that pthread_cond_signal * is called while the mutex is held that is used in the corresponding * pthread_cond_wait calls! */ int pthread_cond_signal(pthread_cond_t *cond) { int have_waiters; EnterCriticalSection(&cond->waiters_lock); have_waiters = cond->waiters > 0; LeaveCriticalSection(&cond->waiters_lock); /* * Signal only when there are waiters */ if (have_waiters) return ReleaseSemaphore(cond->sema, 1, NULL) ? 0 : GetLastError(); else return 0; } /* * DOUBLY IMPORTANT: This implementation requires that pthread_cond_broadcast * is called while the mutex is held that is used in the corresponding * pthread_cond_wait calls! */ int pthread_cond_broadcast(pthread_cond_t *cond) { EnterCriticalSection(&cond->waiters_lock); if ((cond->was_broadcast = cond->waiters > 0)) { /* wake up all waiters */ ReleaseSemaphore(cond->sema, cond->waiters, NULL); LeaveCriticalSection(&cond->waiters_lock); /* * At this point all waiters continue. Each one takes its * slice of the semaphore. Now it's our turn to wait: Since * the external mutex is held, no thread can leave cond_wait, * yet. For this reason, we can be sure that no thread gets * a chance to eat *more* than one slice. OTOH, it means * that the last waiter must send us a wake-up. */ WaitForSingleObject(cond->continue_broadcast, INFINITE); /* * Since the external mutex is held, no thread can enter * cond_wait, and, hence, it is safe to reset this flag * without cond->waiters_lock held. */ cond->was_broadcast = 0; } else { LeaveCriticalSection(&cond->waiters_lock); } return 0; } #endif /* PTHREAD_C */ zmat-0.9.9/src/blosc2/blosc/trunc-prec.c0000644000175200007730000000644514515254731020253 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include #include #include "assert.h" #include "trunc-prec.h" #include "blosc2.h" #define BITS_MANTISSA_FLOAT 23 #define BITS_MANTISSA_DOUBLE 52 int truncate_precision32(int8_t prec_bits, int32_t nelems, const int32_t* src, int32_t* dest) { // Make sure that we don't remove all the bits in mantissa so that we // don't mess with NaNs or Infinite representation in IEEE 754: // https://en.wikipedia.org/wiki/NaN if ((abs(prec_bits) > BITS_MANTISSA_FLOAT)) { BLOSC_TRACE_ERROR("The precision cannot be larger than %d bits for floats (asking for %d bits)", BITS_MANTISSA_FLOAT, prec_bits); return -1; } int zeroed_bits = (prec_bits >= 0) ? BITS_MANTISSA_FLOAT - prec_bits : -prec_bits; if (zeroed_bits >= BITS_MANTISSA_FLOAT) { BLOSC_TRACE_ERROR("The reduction in precision cannot be larger or equal than %d bits for floats (asking for %d bits)", BITS_MANTISSA_FLOAT, zeroed_bits); return -1; } int32_t mask = ~((1 << zeroed_bits) - 1); for (int i = 0; i < nelems; i++) { dest[i] = src[i] & mask; } return 0; } int truncate_precision64(int8_t prec_bits, int32_t nelems, const int64_t* src, int64_t* dest) { // Make sure that we don't remove all the bits in mantissa so that we // don't mess with NaNs or Infinite representation in IEEE 754: // https://en.wikipedia.org/wiki/NaN if ((abs(prec_bits) > BITS_MANTISSA_DOUBLE)) { BLOSC_TRACE_ERROR("The precision cannot be larger than %d bits for floats (asking for %d bits)", BITS_MANTISSA_DOUBLE, prec_bits); return -1; } int zeroed_bits = (prec_bits >= 0) ? BITS_MANTISSA_DOUBLE - prec_bits : -prec_bits; if (zeroed_bits >= BITS_MANTISSA_DOUBLE) { BLOSC_TRACE_ERROR("The reduction in precision cannot be larger or equal than %d bits for floats (asking for %d bits)", BITS_MANTISSA_DOUBLE, zeroed_bits); return -1; } uint64_t mask = ~((1ULL << zeroed_bits) - 1ULL); for (int i = 0; i < nelems; i++) { dest[i] = (int64_t)(src[i] & mask); } return 0; } /* Apply the truncate precision to src. This can never fail. */ int truncate_precision(int8_t prec_bits, int32_t typesize, int32_t nbytes, const uint8_t* src, uint8_t* dest) { // Positive values of prec_bits will set absolute precision bits, whereas negative // values will reduce the precision bits (similar to Python slicing convention). switch (typesize) { case 4: return truncate_precision32(prec_bits, nbytes / typesize, (int32_t *)src, (int32_t *)dest); case 8: return truncate_precision64(prec_bits, nbytes / typesize, (int64_t *)src, (int64_t *)dest); default: fprintf(stderr, "Error in trunc-prec filter: Precision for typesize %d " "not handled", (int)typesize); return -1; } } zmat-0.9.9/src/blosc2/blosc/b2nd_utils.c0000644000175200007730000002620114515254731020226 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include // copyNdim where N = {2-8} - specializations of copy loops to be used by b2nd_copy_buffer // since we don't have c++ templates, substitute manual specializations for up to known B2ND_MAX_DIM (8) // it's not pretty, but it substantially reduces overhead vs. the generic method void copy8dim(const uint8_t itemsize, const int64_t *copy_shape, const uint8_t *bsrc, const int64_t *src_strides, uint8_t *bdst, const int64_t *dst_strides) { int64_t copy_nbytes = copy_shape[7] * itemsize; int64_t copy_start[7] = {0}; do { do { do { do { do { do { do { int64_t src_copy_start = 0; int64_t dst_copy_start = 0; for (int j = 0; j < 7; ++j) { src_copy_start += copy_start[j] * src_strides[j]; dst_copy_start += copy_start[j] * dst_strides[j]; } memcpy(&bdst[dst_copy_start * itemsize], &bsrc[src_copy_start * itemsize], copy_nbytes); ++copy_start[6]; } while (copy_start[6] < copy_shape[6]); ++copy_start[5]; copy_start[6] = 0; } while (copy_start[5] < copy_shape[5]); ++copy_start[4]; copy_start[5] = 0; } while (copy_start[4] < copy_shape[4]); ++copy_start[3]; copy_start[4] = 0; } while (copy_start[3] < copy_shape[3]); ++copy_start[2]; copy_start[3] = 0; } while (copy_start[2] < copy_shape[2]); ++copy_start[1]; copy_start[2] = 0; } while (copy_start[1] < copy_shape[1]); ++copy_start[0]; copy_start[1] = 0; } while (copy_start[0] < copy_shape[0]); } void copy7dim(const uint8_t itemsize, const int64_t *copy_shape, const uint8_t *bsrc, const int64_t *src_strides, uint8_t *bdst, const int64_t *dst_strides) { int64_t copy_nbytes = copy_shape[6] * itemsize; int64_t copy_start[6] = {0}; do { do { do { do { do { do { int64_t src_copy_start = 0; int64_t dst_copy_start = 0; for (int j = 0; j < 6; ++j) { src_copy_start += copy_start[j] * src_strides[j]; dst_copy_start += copy_start[j] * dst_strides[j]; } memcpy(&bdst[dst_copy_start * itemsize], &bsrc[src_copy_start * itemsize], copy_nbytes); ++copy_start[5]; } while (copy_start[5] < copy_shape[5]); ++copy_start[4]; copy_start[5] = 0; } while (copy_start[4] < copy_shape[4]); ++copy_start[3]; copy_start[4] = 0; } while (copy_start[3] < copy_shape[3]); ++copy_start[2]; copy_start[3] = 0; } while (copy_start[2] < copy_shape[2]); ++copy_start[1]; copy_start[2] = 0; } while (copy_start[1] < copy_shape[1]); ++copy_start[0]; copy_start[1] = 0; } while (copy_start[0] < copy_shape[0]); } void copy6dim(const uint8_t itemsize, const int64_t *copy_shape, const uint8_t *bsrc, const int64_t *src_strides, uint8_t *bdst, const int64_t *dst_strides) { int64_t copy_nbytes = copy_shape[5] * itemsize; int64_t copy_start[5] = {0}; do { do { do { do { do { int64_t src_copy_start = 0; int64_t dst_copy_start = 0; for (int j = 0; j < 5; ++j) { src_copy_start += copy_start[j] * src_strides[j]; dst_copy_start += copy_start[j] * dst_strides[j]; } memcpy(&bdst[dst_copy_start * itemsize], &bsrc[src_copy_start * itemsize], copy_nbytes); ++copy_start[4]; } while (copy_start[4] < copy_shape[4]); ++copy_start[3]; copy_start[4] = 0; } while (copy_start[3] < copy_shape[3]); ++copy_start[2]; copy_start[3] = 0; } while (copy_start[2] < copy_shape[2]); ++copy_start[1]; copy_start[2] = 0; } while (copy_start[1] < copy_shape[1]); ++copy_start[0]; copy_start[1] = 0; } while (copy_start[0] < copy_shape[0]); } void copy5dim(const uint8_t itemsize, const int64_t *copy_shape, const uint8_t *bsrc, const int64_t *src_strides, uint8_t *bdst, const int64_t *dst_strides) { int64_t copy_nbytes = copy_shape[4] * itemsize; int64_t copy_start[4] = {0}; do { do { do { do { int64_t src_copy_start = 0; int64_t dst_copy_start = 0; for (int j = 0; j < 4; ++j) { src_copy_start += copy_start[j] * src_strides[j]; dst_copy_start += copy_start[j] * dst_strides[j]; } memcpy(&bdst[dst_copy_start * itemsize], &bsrc[src_copy_start * itemsize], copy_nbytes); ++copy_start[3]; } while (copy_start[3] < copy_shape[3]); ++copy_start[2]; copy_start[3] = 0; } while (copy_start[2] < copy_shape[2]); ++copy_start[1]; copy_start[2] = 0; } while (copy_start[1] < copy_shape[1]); ++copy_start[0]; copy_start[1] = 0; } while (copy_start[0] < copy_shape[0]); } void copy4dim(const uint8_t itemsize, const int64_t *copy_shape, const uint8_t *bsrc, const int64_t *src_strides, uint8_t *bdst, const int64_t *dst_strides) { int64_t copy_nbytes = copy_shape[3] * itemsize; int64_t copy_start[3] = {0}; do { do { do { int64_t src_copy_start = 0; int64_t dst_copy_start = 0; for (int j = 0; j < 3; ++j) { src_copy_start += copy_start[j] * src_strides[j]; dst_copy_start += copy_start[j] * dst_strides[j]; } memcpy(&bdst[dst_copy_start * itemsize], &bsrc[src_copy_start * itemsize], copy_nbytes); ++copy_start[2]; } while (copy_start[2] < copy_shape[2]); ++copy_start[1]; copy_start[2] = 0; } while (copy_start[1] < copy_shape[1]); ++copy_start[0]; copy_start[1] = 0; } while (copy_start[0] < copy_shape[0]); } void copy3dim(const uint8_t itemsize, const int64_t *copy_shape, const uint8_t *bsrc, const int64_t *src_strides, uint8_t *bdst, const int64_t *dst_strides) { int64_t copy_nbytes = copy_shape[2] * itemsize; int64_t copy_start[2] = {0}; do { do { int64_t src_copy_start = 0; int64_t dst_copy_start = 0; for (int j = 0; j < 2; ++j) { src_copy_start += copy_start[j] * src_strides[j]; dst_copy_start += copy_start[j] * dst_strides[j]; } memcpy(&bdst[dst_copy_start * itemsize], &bsrc[src_copy_start * itemsize], copy_nbytes); ++copy_start[1]; } while (copy_start[1] < copy_shape[1]); ++copy_start[0]; copy_start[1] = 0; } while (copy_start[0] < copy_shape[0]); } void copy2dim(const uint8_t itemsize, const int64_t *copy_shape, const uint8_t *bsrc, const int64_t *src_strides, uint8_t *bdst, const int64_t *dst_strides) { int64_t copy_nbytes = copy_shape[1] * itemsize; int64_t copy_start = 0; do { int64_t src_copy_start = copy_start * src_strides[0]; int64_t dst_copy_start = copy_start * dst_strides[0]; memcpy(&bdst[dst_copy_start * itemsize], &bsrc[src_copy_start * itemsize], copy_nbytes); ++copy_start; } while (copy_start < copy_shape[0]); } void copy_ndim_fallback(const int8_t ndim, const uint8_t itemsize, int64_t *copy_shape, const uint8_t *bsrc, int64_t *src_strides, uint8_t *bdst, int64_t *dst_strides) { int64_t copy_nbytes = copy_shape[ndim - 1] * itemsize; int64_t number_of_copies = 1; for (int i = 0; i < ndim - 1; ++i) { number_of_copies *= copy_shape[i]; } for (int ncopy = 0; ncopy < number_of_copies; ++ncopy) { // Compute the start of the copy int64_t copy_start[B2ND_MAX_DIM] = {0}; blosc2_unidim_to_multidim((int8_t) (ndim - 1), copy_shape, ncopy, copy_start); // Translate this index to the src buffer int64_t src_copy_start; blosc2_multidim_to_unidim(copy_start, (int8_t) (ndim - 1), src_strides, &src_copy_start); // Translate this index to the dst buffer int64_t dst_copy_start; blosc2_multidim_to_unidim(copy_start, (int8_t) (ndim - 1), dst_strides, &dst_copy_start); // Perform the copy memcpy(&bdst[dst_copy_start * itemsize], &bsrc[src_copy_start * itemsize], copy_nbytes); } } int b2nd_copy_buffer(int8_t ndim, uint8_t itemsize, void *src, const int64_t *src_pad_shape, int64_t *src_start, const int64_t *src_stop, void *dst, const int64_t *dst_pad_shape, int64_t *dst_start) { // Compute the shape of the copy int64_t copy_shape[B2ND_MAX_DIM] = {0}; for (int i = 0; i < ndim; ++i) { copy_shape[i] = src_stop[i] - src_start[i]; if (copy_shape[i] == 0) { return BLOSC2_ERROR_SUCCESS; } } // Compute the strides int64_t src_strides[B2ND_MAX_DIM]; src_strides[ndim - 1] = 1; for (int i = ndim - 2; i >= 0; --i) { src_strides[i] = src_strides[i + 1] * src_pad_shape[i + 1]; } int64_t dst_strides[B2ND_MAX_DIM]; dst_strides[ndim - 1] = 1; for (int i = ndim - 2; i >= 0; --i) { dst_strides[i] = dst_strides[i + 1] * dst_pad_shape[i + 1]; } // Align the buffers removing unnecessary data int64_t src_start_n; blosc2_multidim_to_unidim(src_start, ndim, src_strides, &src_start_n); uint8_t *bsrc = (uint8_t *) src; bsrc = &bsrc[src_start_n * itemsize]; int64_t dst_start_n; blosc2_multidim_to_unidim(dst_start, ndim, dst_strides, &dst_start_n); uint8_t *bdst = (uint8_t *) dst; bdst = &bdst[dst_start_n * itemsize]; switch (ndim) { case 1: memcpy(&bdst[0], &bsrc[0], copy_shape[0] * itemsize); break; case 2: copy2dim(itemsize, copy_shape, bsrc, src_strides, bdst, dst_strides); break; case 3: copy3dim(itemsize, copy_shape, bsrc, src_strides, bdst, dst_strides); break; case 4: copy4dim(itemsize, copy_shape, bsrc, src_strides, bdst, dst_strides); break; case 5: copy5dim(itemsize, copy_shape, bsrc, src_strides, bdst, dst_strides); break; case 6: copy6dim(itemsize, copy_shape, bsrc, src_strides, bdst, dst_strides); break; case 7: copy7dim(itemsize, copy_shape, bsrc, src_strides, bdst, dst_strides); break; case 8: copy8dim(itemsize, copy_shape, bsrc, src_strides, bdst, dst_strides); break; default: // guard against potential future increase to B2ND_MAX_DIM copy_ndim_fallback(ndim, itemsize, copy_shape, bsrc, src_strides, bdst, dst_strides); break; } return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/blosc/transpose-altivec.h0000644000175200007730000000703114515254731021631 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc developers and Jerome Kieffer https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef BLOSC_TRANSPOSE_ALTIVEC_H #define BLOSC_TRANSPOSE_ALTIVEC_H #ifdef __cplusplus extern "C" { #endif static const __vector uint8_t even = (const __vector uint8_t) { 0x00, 0x02, 0x04, 0x06, 0x08, 0x0a, 0x0c, 0x0e, 0x10, 0x12, 0x14, 0x16, 0x18, 0x1a, 0x1c, 0x1e}; static const __vector uint8_t odd = (const __vector uint8_t) { 0x01, 0x03, 0x05, 0x07, 0x09, 0x0b, 0x0d, 0x0f, 0x11, 0x13, 0x15, 0x17, 0x19, 0x1b, 0x1d, 0x1f}; /* Transpose inplace 2 vectors of 16 bytes in src into dst. */ static void transpose2x16(__vector uint8_t *xmm0) { __vector uint8_t xmm1[2]; xmm1[0] = vec_perm(xmm0[0], xmm0[1], even); xmm1[1] = vec_perm(xmm0[0], xmm0[1], odd); for (int i = 0; i < 2; i++) { xmm0[i] = xmm1[i]; } } /* Transpose inplace 4 vectors of 16 bytes in src into dst. * Total cost: 8 calls to vec_perm. */ static void transpose4x16(__vector uint8_t *xmm0) { __vector uint8_t xmm1[4]; /* Transpose vectors 0-1*/ xmm1[0] = vec_perm(xmm0[0], xmm0[1], even); xmm1[1] = vec_perm(xmm0[0], xmm0[1], odd); xmm1[2] = vec_perm(xmm0[2], xmm0[3], even); xmm1[3] = vec_perm(xmm0[2], xmm0[3], odd); /* Transpose vectors 0-2*/ xmm0[0] = vec_perm(xmm1[0], xmm1[2], even); xmm0[1] = vec_perm(xmm1[1], xmm1[3], even); xmm0[2] = vec_perm(xmm1[0], xmm1[2], odd); xmm0[3] = vec_perm(xmm1[1], xmm1[3], odd); } /* Transpose inplace 8 vectors of 16 bytes in src into dst. * Total cost: 24 calls to vec_perm. */ static void transpose8x16(__vector uint8_t *xmm0) { __vector uint8_t xmm1[8]; /* Transpose vectors 0-1*/ for (int i = 0; i < 8; i += 2){ xmm1[i] = vec_perm(xmm0[i], xmm0[i+1], even); xmm1[i+1] = vec_perm(xmm0[i], xmm0[i+1], odd); } /* Transpose vectors 0-2*/ for (int i = 0; i < 8; i += 4){ for (int k = 0; k < 2; k++){ xmm0[i+k] = vec_perm(xmm1[i+k], xmm1[i+k+2], even); xmm0[i+k+2] = vec_perm(xmm1[i+k], xmm1[i+k+2], odd); } } /* Transpose vectors 0-4*/ for (int k = 0; k < 4; k++){ xmm1[k] = vec_perm(xmm0[k], xmm0[k+4], even); xmm1[k+4] = vec_perm(xmm0[k], xmm0[k+4], odd); } for (int i = 0; i < 8; i++) { xmm0[i] = xmm1[i]; } } /* Transpose inplace 16 vectors of 16 bytes in src into dst. * Total cost: 64 calls to vec_perm. */ static void transpose16x16(__vector uint8_t * xmm0){ __vector uint8_t xmm1[16]; /* Transpose vectors 0-1*/ for (int i = 0; i < 16; i += 2){ xmm1[i] = vec_perm(xmm0[i], xmm0[i+1], even); xmm1[i+1] = vec_perm(xmm0[i], xmm0[i+1], odd); } /* Transpose vectors 0-2*/ for (int i = 0; i < 16; i += 4){ for (int k = 0; k < 2; k++){ xmm0[i+k] = vec_perm(xmm1[i+k], xmm1[i+k+2], even); xmm0[i+k+2] = vec_perm(xmm1[i+k], xmm1[i+k+2], odd); } } /* Transpose vectors 0-4*/ for (int i = 0; i < 16; i += 8){ for (int k = 0; k < 4; k++){ xmm1[i+k] = vec_perm(xmm0[i+k], xmm0[i+k+4], even); xmm1[i+k+4] = vec_perm(xmm0[i+k], xmm0[i+k+4], odd); } } /* Transpose vectors 0-8*/ for (int k = 0; k < 8; k++){ xmm0[k] = vec_perm(xmm1[k], xmm1[k+8], even); xmm0[k+8] = vec_perm(xmm1[k], xmm1[k+8], odd); } } #ifdef __cplusplus } #endif #endif //BLOSC_TRANSPOSE_ALTIVEC_H zmat-0.9.9/src/blosc2/blosc/blosclz.c0000644000175200007730000005726714515254731017651 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /********************************************************************* The code in this file is heavily based on FastLZ, a lightning-fast lossless compression library. See LICENSES/FASTLZ.txt for details. **********************************************************************/ #include #include #include "blosclz.h" #include "fastcopy.h" #include "blosc2/blosc2-common.h" /* * Give hints to the compiler for branch prediction optimization. * This is not necessary anymore with modern CPUs. */ #if 0 && defined(__GNUC__) && (__GNUC__ > 2) #define BLOSCLZ_LIKELY(c) (__builtin_expect((c), 1)) #define BLOSCLZ_UNLIKELY(c) (__builtin_expect((c), 0)) #else #define BLOSCLZ_LIKELY(c) (c) #define BLOSCLZ_UNLIKELY(c) (c) #endif /* * Use inlined functions for supported systems. */ #if defined(_MSC_VER) && !defined(__cplusplus) /* Visual Studio */ #define inline __inline /* Visual C is not C99, but supports some kind of inline */ #endif #define MAX_COPY 32U #define MAX_DISTANCE 8191 #define MAX_FARDISTANCE (65535 + MAX_DISTANCE - 1) #ifdef BLOSC_STRICT_ALIGN #define BLOSCLZ_READU16(p) ((p)[0] | (p)[1]<<8) #define BLOSCLZ_READU32(p) ((p)[0] | (p)[1]<<8 | (p)[2]<<16 | (p)[3]<<24) #else #define BLOSCLZ_READU16(p) *((const uint16_t*)(p)) #define BLOSCLZ_READU32(p) *((const uint32_t*)(p)) #endif #define HASH_LOG (14U) // This is used in LZ4 and seems to work pretty well here too #define HASH_FUNCTION(v, s, h) { \ (v) = ((s) * 2654435761U) >> (32U - (h)); \ } #if defined(__AVX2__) static uint8_t *get_run_32(uint8_t *ip, const uint8_t *ip_bound, const uint8_t *ref) { uint8_t x = ip[-1]; while (ip < (ip_bound - (sizeof(__m256i)))) { __m256i value, value2, cmp; /* Broadcast the value for every byte in a 256-bit register */ memset(&value, x, sizeof(__m256i)); value2 = _mm256_loadu_si256((__m256i *)ref); cmp = _mm256_cmpeq_epi64(value, value2); if ((unsigned)_mm256_movemask_epi8(cmp) != 0xFFFFFFFF) { /* Return the byte that starts to differ */ while (*ref++ == x) ip++; return ip; } else { ip += sizeof(__m256i); ref += sizeof(__m256i); } } /* Look into the remainder */ while ((ip < ip_bound) && (*ref++ == x)) ip++; return ip; } #endif #if defined(__SSE2__) uint8_t *get_run_16(uint8_t *ip, const uint8_t *ip_bound, const uint8_t *ref) { uint8_t x = ip[-1]; while (ip < (ip_bound - sizeof(__m128i))) { __m128i value, value2, cmp; /* Broadcast the value for every byte in a 128-bit register */ memset(&value, x, sizeof(__m128i)); value2 = _mm_loadu_si128((__m128i *)ref); cmp = _mm_cmpeq_epi32(value, value2); if (_mm_movemask_epi8(cmp) != 0xFFFF) { /* Return the byte that starts to differ */ while (*ref++ == x) ip++; return ip; } else { ip += sizeof(__m128i); ref += sizeof(__m128i); } } /* Look into the remainder */ while ((ip < ip_bound) && (*ref++ == x)) ip++; return ip; } #endif static uint8_t *get_run(uint8_t *ip, const uint8_t *ip_bound, const uint8_t *ref) { uint8_t x = ip[-1]; int64_t value, value2; /* Broadcast the value for every byte in a 64-bit register */ memset(&value, x, 8); /* safe because the outer check against ip limit */ while (ip < (ip_bound - sizeof(int64_t))) { #if defined(BLOSC_STRICT_ALIGN) memcpy(&value2, ref, 8); #else value2 = ((int64_t*)ref)[0]; #endif if (value != value2) { /* Return the byte that starts to differ */ while (*ref++ == x) ip++; return ip; } else { ip += 8; ref += 8; } } /* Look into the remainder */ while ((ip < ip_bound) && (*ref++ == x)) ip++; return ip; } /* Return the byte that starts to differ */ uint8_t *get_match(uint8_t *ip, const uint8_t *ip_bound, const uint8_t *ref) { #if !defined(BLOSC_STRICT_ALIGN) while (ip < (ip_bound - sizeof(int64_t))) { if (*(int64_t*)ref != *(int64_t*)ip) { /* Return the byte that starts to differ */ while (*ref++ == *ip++) {} return ip; } else { ip += sizeof(int64_t); ref += sizeof(int64_t); } } #endif /* Look into the remainder */ while ((ip < ip_bound) && (*ref++ == *ip++)) {} return ip; } #if defined(__SSE2__) static uint8_t *get_match_16(uint8_t *ip, const uint8_t *ip_bound, const uint8_t *ref) { __m128i value, value2, cmp; while (ip < (ip_bound - sizeof(__m128i))) { value = _mm_loadu_si128((__m128i *) ip); value2 = _mm_loadu_si128((__m128i *) ref); cmp = _mm_cmpeq_epi32(value, value2); if (_mm_movemask_epi8(cmp) != 0xFFFF) { /* Return the byte that starts to differ */ while (*ref++ == *ip++) {} return ip; } else { ip += sizeof(__m128i); ref += sizeof(__m128i); } } /* Look into the remainder */ while ((ip < ip_bound) && (*ref++ == *ip++)) {} return ip; } #endif #if defined(__AVX2__) static uint8_t *get_match_32(uint8_t *ip, const uint8_t *ip_bound, const uint8_t *ref) { while (ip < (ip_bound - sizeof(__m256i))) { __m256i value, value2, cmp; value = _mm256_loadu_si256((__m256i *) ip); value2 = _mm256_loadu_si256((__m256i *)ref); cmp = _mm256_cmpeq_epi64(value, value2); if ((unsigned)_mm256_movemask_epi8(cmp) != 0xFFFFFFFF) { /* Return the byte that starts to differ */ while (*ref++ == *ip++) {} return ip; } else { ip += sizeof(__m256i); ref += sizeof(__m256i); } } /* Look into the remainder */ while ((ip < ip_bound) && (*ref++ == *ip++)) {} return ip; } #endif static uint8_t* get_run_or_match(uint8_t* ip, uint8_t* ip_bound, const uint8_t* ref, bool run) { if (BLOSCLZ_UNLIKELY(run)) { #if defined(__AVX2__) // Extensive experiments on AMD Ryzen3 say that regular get_run is faster // ip = get_run_32(ip, ip_bound, ref); ip = get_run(ip, ip_bound, ref); #elif defined(__SSE2__) // Extensive experiments on AMD Ryzen3 say that regular get_run is faster // ip = get_run_16(ip, ip_bound, ref); ip = get_run(ip, ip_bound, ref); #else ip = get_run(ip, ip_bound, ref); #endif } else { #if defined(__AVX2__) // Extensive experiments on AMD Ryzen3 say that regular get_match_16 is faster // ip = get_match_32(ip, ip_bound, ref); ip = get_match_16(ip, ip_bound, ref); #elif defined(__SSE2__) ip = get_match_16(ip, ip_bound, ref); #else ip = get_match(ip, ip_bound, ref); #endif } return ip; } #define LITERAL(ip, op, op_limit, anchor, copy) { \ if (BLOSCLZ_UNLIKELY((op) + 2 > (op_limit))) \ goto out; \ *(op)++ = *(anchor)++; \ (ip) = (anchor); \ (copy)++; \ if (BLOSCLZ_UNLIKELY((copy) == MAX_COPY)) { \ (copy) = 0; \ *(op)++ = MAX_COPY-1; \ } \ } #define LITERAL2(ip, anchor, copy) { \ oc++; (anchor)++; \ (ip) = (anchor); \ (copy)++; \ if (BLOSCLZ_UNLIKELY((copy) == MAX_COPY)) { \ (copy) = 0; \ oc++; \ } \ } #define MATCH_SHORT(op, op_limit, len, distance) { \ if (BLOSCLZ_UNLIKELY((op) + 2 > (op_limit))) \ goto out; \ *(op)++ = (uint8_t)(((len) << 5U) + ((distance) >> 8U));\ *(op)++ = (uint8_t)(((distance) & 255U)); \ } #define MATCH_LONG(op, op_limit, len, distance) { \ if (BLOSCLZ_UNLIKELY((op) + 1 > (op_limit))) \ goto out; \ *(op)++ = (uint8_t)((7U << 5U) + ((distance) >> 8U)); \ for ((len) -= 7; (len) >= 255; (len) -= 255) { \ if (BLOSCLZ_UNLIKELY((op) + 1 > (op_limit))) \ goto out; \ *(op)++ = 255; \ } \ if (BLOSCLZ_UNLIKELY((op) + 2 > (op_limit))) \ goto out; \ *(op)++ = (uint8_t)(len); \ *(op)++ = (uint8_t)(((distance) & 255U)); \ } #define MATCH_SHORT_FAR(op, op_limit, len, distance) { \ if (BLOSCLZ_UNLIKELY((op) + 4 > (op_limit))) \ goto out; \ *(op)++ = (uint8_t)(((len) << 5U) + 31); \ *(op)++ = 255; \ *(op)++ = (uint8_t)((distance) >> 8U); \ *(op)++ = (uint8_t)((distance) & 255U); \ } #define MATCH_LONG_FAR(op, op_limit, len, distance) { \ if (BLOSCLZ_UNLIKELY((op) + 1 > (op_limit))) \ goto out; \ *(op)++ = (7U << 5U) + 31; \ for ((len) -= 7; (len) >= 255; (len) -= 255) { \ if (BLOSCLZ_UNLIKELY((op) + 1 > (op_limit))) \ goto out; \ *(op)++ = 255; \ } \ if (BLOSCLZ_UNLIKELY((op) + 4 > (op_limit))) \ goto out; \ *(op)++ = (uint8_t)(len); \ *(op)++ = 255; \ *(op)++ = (uint8_t)((distance) >> 8U); \ *(op)++ = (uint8_t)((distance) & 255U); \ } // Get a guess for the compressed size of a buffer static double get_cratio(uint8_t* ibase, int maxlen, int minlen, int ipshift, uint32_t htab[], int8_t hashlog) { uint8_t* ip = ibase; int32_t oc = 0; const uint16_t hashlen = (1U << (uint8_t)hashlog); uint32_t hval; uint32_t seq; uint8_t copy; // Make a tradeoff between testing too much and too little uint16_t limit = (maxlen > hashlen) ? hashlen : maxlen; uint8_t* ip_bound = ibase + limit - 1; uint8_t* ip_limit = ibase + limit - 12; // Initialize the hash table to distances of 0 memset(htab, 0, hashlen * sizeof(uint32_t)); /* we start with literal copy */ copy = 4; oc += 5; /* main loop */ while (BLOSCLZ_LIKELY(ip < ip_limit)) { const uint8_t* ref; unsigned distance; uint8_t* anchor = ip; /* comparison starting-point */ /* find potential match */ seq = BLOSCLZ_READU32(ip); HASH_FUNCTION(hval, seq, hashlog) ref = ibase + htab[hval]; /* calculate distance to the match */ distance = (unsigned int)(anchor - ref); /* update hash table */ htab[hval] = (uint32_t) (anchor - ibase); if (distance == 0 || (distance >= MAX_FARDISTANCE)) { LITERAL2(ip, anchor, copy) continue; } /* is this a match? check the first 4 bytes */ if (BLOSCLZ_READU32(ref) == BLOSCLZ_READU32(ip)) { ref += 4; } else { /* no luck, copy as a literal */ LITERAL2(ip, anchor, copy) continue; } /* last matched byte */ ip = anchor + 4; /* distance is biased */ distance--; /* get runs or matches; zero distance means a run */ ip = get_run_or_match(ip, ip_bound, ref, !distance); ip -= ipshift; int len = (int)(ip - anchor); if (len < minlen) { LITERAL2(ip, anchor, copy) continue; } /* if we haven't copied anything, adjust the output counter */ if (!copy) oc--; /* reset literal counter */ copy = 0; /* encode the match */ if (distance < MAX_DISTANCE) { if (len >= 7) { oc += ((len - 7) / 255) + 1; } oc += 2; } else { /* far away, but not yet in the another galaxy... */ if (len >= 7) { oc += ((len - 7) / 255) + 1; } oc += 4; } /* update the hash at match boundary */ seq = BLOSCLZ_READU32(ip); HASH_FUNCTION(hval, seq, hashlog) htab[hval] = (uint32_t)(ip++ - ibase); ip++; /* assuming literal copy */ oc++; } double ic = (double)(ip - ibase); return ic / (double)oc; } int blosclz_compress(const int clevel, const void* input, int length, void* output, int maxout, blosc2_context* ctx) { BLOSC_UNUSED_PARAM(ctx); uint8_t* ibase = (uint8_t*)input; uint32_t htab[1U << (uint8_t)HASH_LOG]; /* When we go back in a match (shift), we obtain quite different compression properties. * It looks like 4 is more useful in combination with bitshuffle and small typesizes * Fallback to 4 because it provides more consistent results for large cratios. * UPDATE: new experiments show that using a value of 3 is a bit better, at least for ERA5. * * In this block we also check cratios for the beginning of the buffers and * eventually discard those that are small (take too long to decompress). * This process is called _entropy probing_. */ unsigned ipshift = 3; // Minimum lengths for encoding (normally it is good to match the shift value) unsigned minlen = 3; uint8_t hashlog_[10] = {0, HASH_LOG - 2, HASH_LOG - 1, HASH_LOG, HASH_LOG, HASH_LOG, HASH_LOG, HASH_LOG, HASH_LOG, HASH_LOG}; uint8_t hashlog = hashlog_[clevel]; // Experiments say that checking 1/4 of the buffer is enough to figure out approx cratio // UPDATE: new experiments with ERA5 datasets (float32) say that checking the whole buffer // is better (specially when combined with bitshuffle). // The loss in speed for checking the whole buffer is pretty negligible too. int maxlen = length; if (clevel < 4) { maxlen /= 4; } else if (clevel < 7) { maxlen /= 2; } // Start probing somewhere inside the buffer int shift = length - maxlen; // Actual entropy probing! double cratio = get_cratio(ibase + shift, maxlen, minlen, ipshift, htab, hashlog); // discard probes with small compression ratios (too expensive) double cratio_[10] = {0, 2, 1.5, 1.2, 1.2, 1.2, 1.2, 1.15, 1.1, 1.0}; if (cratio < cratio_[clevel]) { goto out; } uint8_t* ip = ibase; uint8_t* ip_bound = ibase + length - 1; uint8_t* ip_limit = ibase + length - 12; uint8_t* op = (uint8_t*)output; const uint8_t* op_limit = op + maxout; uint32_t seq; uint8_t copy; uint32_t hval; /* input and output buffer cannot be less than 16 and 66 bytes or we can get into trouble */ if (length < 16 || maxout < 66) { return 0; } // Initialize the hash table memset(htab, 0, (1U << hashlog) * sizeof(uint32_t)); /* we start with literal copy */ copy = 4; *op++ = MAX_COPY - 1; *op++ = *ip++; *op++ = *ip++; *op++ = *ip++; *op++ = *ip++; /* main loop */ while (BLOSCLZ_LIKELY(ip < ip_limit)) { const uint8_t* ref; unsigned distance; uint8_t* anchor = ip; /* comparison starting-point */ /* find potential match */ seq = BLOSCLZ_READU32(ip); HASH_FUNCTION(hval, seq, hashlog) ref = ibase + htab[hval]; /* calculate distance to the match */ distance = (unsigned int)(anchor - ref); /* update hash table */ htab[hval] = (uint32_t) (anchor - ibase); if (distance == 0 || (distance >= MAX_FARDISTANCE)) { LITERAL(ip, op, op_limit, anchor, copy) continue; } /* is this a match? check the first 4 bytes */ if (BLOSCLZ_UNLIKELY(BLOSCLZ_READU32(ref) == BLOSCLZ_READU32(ip))) { ref += 4; } else { /* no luck, copy as a literal */ LITERAL(ip, op, op_limit, anchor, copy) continue; } /* last matched byte */ ip = anchor + 4; /* distance is biased */ distance--; /* get runs or matches; zero distance means a run */ ip = get_run_or_match(ip, ip_bound, ref, !distance); /* length is biased, '1' means a match of 3 bytes */ ip -= ipshift; unsigned len = (int)(ip - anchor); // Encoding short lengths is expensive during decompression if (len < minlen || (len <= 5 && distance >= MAX_DISTANCE)) { LITERAL(ip, op, op_limit, anchor, copy) continue; } /* if we have copied something, adjust the copy count */ if (copy) /* copy is biased, '0' means 1 byte copy */ *(op - copy - 1) = (uint8_t)(copy - 1); else /* back, to overwrite the copy count */ op--; /* reset literal counter */ copy = 0; /* encode the match */ if (distance < MAX_DISTANCE) { if (len < 7) { MATCH_SHORT(op, op_limit, len, distance) } else { MATCH_LONG(op, op_limit, len, distance) } } else { /* far away, but not yet in the another galaxy... */ distance -= MAX_DISTANCE; if (len < 7) { MATCH_SHORT_FAR(op, op_limit, len, distance) } else { MATCH_LONG_FAR(op, op_limit, len, distance) } } /* update the hash at match boundary */ seq = BLOSCLZ_READU32(ip); HASH_FUNCTION(hval, seq, hashlog) htab[hval] = (uint32_t) (ip++ - ibase); if (clevel == 9) { // In some situations, including a second hash proves to be useful, // but not in others. Activating here in max clevel only. seq >>= 8U; HASH_FUNCTION(hval, seq, hashlog) htab[hval] = (uint32_t) (ip++ - ibase); } else { ip++; } if (BLOSCLZ_UNLIKELY(op + 1 > op_limit)) goto out; /* assuming literal copy */ *op++ = MAX_COPY - 1; } /* left-over as literal copy */ while (BLOSCLZ_UNLIKELY(ip <= ip_bound)) { if (BLOSCLZ_UNLIKELY(op + 2 > op_limit)) goto out; *op++ = *ip++; copy++; if (BLOSCLZ_UNLIKELY(copy == MAX_COPY)) { copy = 0; *op++ = MAX_COPY - 1; } } /* if we have copied something, adjust the copy length */ if (copy) *(op - copy - 1) = (uint8_t)(copy - 1); else op--; /* marker for blosclz */ *(uint8_t*)output |= (1U << 5U); return (int)(op - (uint8_t*)output); out: return 0; } // See https://habr.com/en/company/yandex/blog/457612/ #if defined(__AVX2__) #if defined(_MSC_VER) #define ALIGNED_(x) __declspec(align(x)) #else #if defined(__GNUC__) #define ALIGNED_(x) __attribute__ ((aligned(x))) #endif #endif #define ALIGNED_TYPE_(t, x) t ALIGNED_(x) static unsigned char* copy_match_16(unsigned char *op, const unsigned char *match, int32_t len) { size_t offset = op - match; while (len >= 16) { static const ALIGNED_TYPE_(uint8_t, 16) masks[] = { 0, 1, 2, 1, 4, 1, 4, 2, 8, 7, 6, 5, 4, 3, 2, 1, // offset = 0, not used as mask, but for shift 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // offset = 1 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 0, 1, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 8, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 0, 1, 2, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 0, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, // offset = 16 }; _mm_storeu_si128((__m128i *)(op), _mm_shuffle_epi8(_mm_loadu_si128((const __m128i *)(match)), _mm_load_si128((const __m128i *)(masks) + offset))); match += masks[offset]; op += 16; len -= 16; } // Deal with remainders for (; len > 0; len--) { *op++ = *match++; } return op; } #endif // LZ4 wildCopy which can reach excellent copy bandwidth (even if insecure) static inline void wild_copy(uint8_t *out, const uint8_t* from, uint8_t* end) { uint8_t* d = out; const uint8_t* s = from; uint8_t* const e = end; do { memcpy(d,s,8); d+=8; s+=8; } while (d= 32) { // match int32_t len = (int32_t)(ctrl >> 5U) - 1 ; int32_t ofs = (int32_t)(ctrl & 31U) << 8U; uint8_t code; const uint8_t* ref = op - ofs; if (len == 7 - 1) { do { if (BLOSCLZ_UNLIKELY(ip + 1 >= ip_limit)) { return 0; } code = *ip++; len += code; } while (code == 255); } else { if (BLOSCLZ_UNLIKELY(ip + 1 >= ip_limit)) { return 0; } } code = *ip++; len += 3; ref -= code; /* match from 16-bit distance */ if (BLOSCLZ_UNLIKELY(code == 255)) { if (ofs == (31U << 8U)) { if (ip + 1 >= ip_limit) { return 0; } ofs = (*ip++) << 8U; ofs += *ip++; ref = op - ofs - MAX_DISTANCE; } } if (BLOSCLZ_UNLIKELY(op + len > op_limit)) { return 0; } if (BLOSCLZ_UNLIKELY(ref - 1 < (uint8_t*)output)) { return 0; } if (BLOSCLZ_UNLIKELY(ip >= ip_limit)) break; ctrl = *ip++; ref--; if (ref == op - 1) { /* optimized copy for a run */ memset(op, *ref, len); op += len; } else if ((op - ref >= 8) && (op_limit - op >= len + 8)) { // copy with an overlap not larger than 8 wild_copy(op, ref, op + len); op += len; } else { // general copy with any overlap #if 0 && defined(__AVX2__) if (op - ref <= 16) { // This is not faster on a combination of compilers (clang, gcc, icc) or machines, but // it is not too slower either. op = copy_match_16(op, ref, len); } else { #endif op = copy_match(op, ref, (unsigned) len); #if 0 && defined(__AVX2__) } #endif } } else { // literal ctrl++; if (BLOSCLZ_UNLIKELY(op + ctrl > op_limit)) { return 0; } if (BLOSCLZ_UNLIKELY(ip + ctrl > ip_limit)) { return 0; } memcpy(op, ip, ctrl); op += ctrl; ip += ctrl; // On GCC-6, fastcopy this is still faster than plain memcpy // However, using recent CLANG/LLVM 9.0, there is almost no difference // in performance. // And starting on CLANG/LLVM 10 and GCC 9, memcpy is generally faster. // op = fastcopy(op, ip, (unsigned) ctrl); ip += ctrl; if (BLOSCLZ_UNLIKELY(ip >= ip_limit)) break; ctrl = *ip++; } } return (int)(op - (uint8_t*)output); } zmat-0.9.9/src/blosc2/blosc/sframe.c0000644000175200007730000001000014515254731017424 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include #include #include #include "blosc2.h" #include "frame.h" /* If C11 is supported, use it's built-in aligned allocation. */ #if __STDC_VERSION__ >= 201112L #include #endif /* Open sparse frame index chunk */ void* sframe_open_index(const char* urlpath, const char* mode, const blosc2_io *io) { void* fp = NULL; char* index_path = malloc(strlen(urlpath) + strlen("/chunks.b2frame") + 1); if (index_path) { sprintf(index_path, "%s/chunks.b2frame", urlpath); blosc2_io_cb *io_cb = blosc2_get_io_cb(io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return NULL; } fp = io_cb->open(index_path, mode, io->params); free(index_path); if (fp == NULL) { BLOSC_TRACE_ERROR("Error creating index path in: %s", index_path); return NULL; } } return fp; } /* Open directory/nchunk.chunk with 8 zeros of padding */ void* sframe_open_chunk(const char* urlpath, int64_t nchunk, const char* mode, const blosc2_io *io) { void* fp = NULL; char* chunk_path = malloc(strlen(urlpath) + 1 + 8 + strlen(".chunk") + 1); if (chunk_path) { sprintf(chunk_path, "%s/%08X.chunk", urlpath, (unsigned int)nchunk); blosc2_io_cb *io_cb = blosc2_get_io_cb(io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return NULL; } fp = io_cb->open(chunk_path, mode, io->params); free(chunk_path); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening chunk path in: %s", chunk_path); return NULL; } } return fp; } /* Append an existing chunk into a sparse frame. */ void* sframe_create_chunk(blosc2_frame_s* frame, uint8_t* chunk, int64_t nchunk, int64_t cbytes) { void* fpc = sframe_open_chunk(frame->urlpath, nchunk, "wb", frame->schunk->storage->io); if (fpc == NULL) { BLOSC_TRACE_ERROR("Cannot open the chunkfile."); return NULL; } blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return NULL; } int64_t wbytes = io_cb->write(chunk, 1, cbytes, fpc); io_cb->close(fpc); if (wbytes != cbytes) { BLOSC_TRACE_ERROR("Cannot write the full chunk."); return NULL; } return frame; } /* Append an existing chunk into a sparse frame. */ int sframe_delete_chunk(const char *urlpath, int64_t nchunk) { char* chunk_path = malloc(strlen(urlpath) + 1 + 8 + strlen(".chunk") + 1); if (chunk_path) { sprintf(chunk_path, "%s/%08X.chunk", urlpath, (unsigned int)nchunk); int rc = remove(chunk_path); free(chunk_path); return rc; } return BLOSC2_ERROR_FILE_REMOVE; } /* Get chunk from sparse frame. */ int32_t sframe_get_chunk(blosc2_frame_s* frame, int64_t nchunk, uint8_t** chunk, bool* needs_free){ void *fpc = sframe_open_chunk(frame->urlpath, nchunk, "rb", frame->schunk->storage->io); if(fpc == NULL){ BLOSC_TRACE_ERROR("Cannot open the chunkfile."); return BLOSC2_ERROR_FILE_OPEN; } blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } io_cb->seek(fpc, 0L, SEEK_END); int64_t chunk_cbytes = io_cb->tell(fpc); *chunk = malloc((size_t)chunk_cbytes); io_cb->seek(fpc, 0L, SEEK_SET); int64_t rbytes = io_cb->read(*chunk, 1, chunk_cbytes, fpc); io_cb->close(fpc); if (rbytes != chunk_cbytes) { BLOSC_TRACE_ERROR("Cannot read the chunk out of the chunkfile."); return BLOSC2_ERROR_FILE_READ; } *needs_free = true; return (int32_t)chunk_cbytes; } zmat-0.9.9/src/blosc2/blosc/bitshuffle-generic.c0000644000175200007730000001525014515254731021730 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include "bitshuffle-generic.h" #ifdef _MSC_VER #pragma warning (push) #pragma warning (disable: 4146) #endif /* Transpose bytes within elements, starting partway through input. */ int64_t bshuf_trans_byte_elem_remainder(const void* in, void* out, const size_t size, const size_t elem_size, const size_t start) { char* in_b = (char*) in; char* out_b = (char*) out; size_t ii, jj, kk; CHECK_MULT_EIGHT(start); if (size > start) { /* ii loop separated into 2 loops so the compiler can unroll */ /* the inner one. */ for (ii = start; ii + 7 < size; ii += 8) { for (jj = 0; jj < elem_size; jj++) { for (kk = 0; kk < 8; kk++) { out_b[jj * size + ii + kk] = in_b[ii * elem_size + kk * elem_size + jj]; } } } for (ii = size - size % 8; ii < size; ii ++) { for (jj = 0; jj < elem_size; jj++) { out_b[jj * size + ii] = in_b[ii * elem_size + jj]; } } } return (int64_t)size * (int64_t)elem_size; } /* Transpose bytes within elements. */ int64_t bshuf_trans_byte_elem_scal(const void* in, void* out, const size_t size, const size_t elem_size) { return bshuf_trans_byte_elem_remainder(in, out, size, elem_size, 0); } /* Transpose bits within bytes. */ int64_t bshuf_trans_bit_byte_remainder(const void* in, void* out, const size_t size, const size_t elem_size, const size_t start_byte) { const uint64_t* in_b = (const uint64_t*) in; uint8_t* out_b = (uint8_t*) out; uint64_t x, t; size_t ii, kk; size_t nbyte = elem_size * size; size_t nbyte_bitrow = nbyte / 8; uint64_t e=1; const int little_endian = *(uint8_t *) &e == 1; const size_t bit_row_skip = little_endian ? nbyte_bitrow : -nbyte_bitrow; const size_t bit_row_offset = little_endian ? 0 : 7 * nbyte_bitrow; CHECK_MULT_EIGHT(nbyte); CHECK_MULT_EIGHT(start_byte); for (ii = start_byte / 8; ii < nbyte_bitrow; ii ++) { x = in_b[ii]; if (little_endian) { TRANS_BIT_8X8(x, t); } else { TRANS_BIT_8X8_BE(x, t); } for (kk = 0; kk < 8; kk ++) { out_b[bit_row_offset + kk * bit_row_skip + ii] = (uint8_t)x; x = x >> 8; } } return (int64_t)size * (int64_t)elem_size; } /* Transpose bits within bytes. */ int64_t bshuf_trans_bit_byte_scal(const void* in, void* out, const size_t size, const size_t elem_size) { return bshuf_trans_bit_byte_remainder(in, out, size, elem_size, 0); } /* General transpose of an array, optimized for large element sizes. */ int64_t bshuf_trans_elem(const void* in, void* out, const size_t lda, const size_t ldb, const size_t elem_size) { char* in_b = (char*) in; char* out_b = (char*) out; size_t ii, jj; for (ii = 0; ii < lda; ii++) { for (jj = 0; jj < ldb; jj++) { memcpy(&out_b[(jj*lda + ii) * elem_size], &in_b[(ii*ldb + jj) * elem_size], elem_size); } } return (int64_t)lda * (int64_t)ldb * (int64_t)elem_size; } /* Transpose rows of shuffled bits (size / 8 bytes) within groups of 8. */ int64_t bshuf_trans_bitrow_eight(const void* in, void* out, const size_t size, const size_t elem_size) { size_t nbyte_bitrow = size / 8; CHECK_MULT_EIGHT(size); return bshuf_trans_elem(in, out, 8, elem_size, nbyte_bitrow); } /* Transpose bits within elements. */ int64_t bshuf_trans_bit_elem_scal(const void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf) { int64_t count; CHECK_MULT_EIGHT(size); count = bshuf_trans_byte_elem_scal(in, out, size, elem_size); CHECK_ERR(count); count = bshuf_trans_bit_byte_scal(out, tmp_buf, size, elem_size); CHECK_ERR(count); count = bshuf_trans_bitrow_eight(tmp_buf, out, size, elem_size); return count; } /* For data organized into a row for each bit (8 * elem_size rows), transpose * the bytes. */ int64_t bshuf_trans_byte_bitrow_scal(const void* in, void* out, const size_t size, const size_t elem_size) { char* in_b = (char*) in; char* out_b = (char*) out; size_t nbyte_row = size / 8; size_t ii, jj, kk; CHECK_MULT_EIGHT(size); for (jj = 0; jj < elem_size; jj++) { for (ii = 0; ii < nbyte_row; ii++) { for (kk = 0; kk < 8; kk++) { out_b[ii * 8 * elem_size + jj * 8 + kk] = \ in_b[(jj * 8 + kk) * nbyte_row + ii]; } } } return (int64_t)size * (int64_t)elem_size; } /* Shuffle bits within the bytes of eight element blocks. */ int64_t bshuf_shuffle_bit_eightelem_scal(const void* in, void* out, \ const size_t size, const size_t elem_size) { const char *in_b; char *out_b; uint64_t x, t; size_t ii, jj, kk; size_t nbyte, out_index; uint64_t e=1; const int little_endian = *(uint8_t *) &e == 1; const size_t elem_skip = little_endian ? elem_size : -elem_size; const size_t elem_offset = little_endian ? 0 : 7 * elem_size; CHECK_MULT_EIGHT(size); in_b = (const char*) in; out_b = (char*) out; nbyte = elem_size * size; for (jj = 0; jj < 8 * elem_size; jj += 8) { for (ii = 0; ii + 8 * elem_size - 1 < nbyte; ii += 8 * elem_size) { x = *((uint64_t*) &in_b[ii + jj]); if (little_endian) { TRANS_BIT_8X8(x, t); } else { TRANS_BIT_8X8_BE(x, t); } for (kk = 0; kk < 8; kk++) { out_index = ii + jj / 8 + elem_offset + kk * elem_skip; *((uint8_t*) &out_b[out_index]) = (uint8_t)x; x = x >> 8; } } } return (int64_t)size * (int64_t)elem_size; } /* Untranspose bits within elements. */ int64_t bshuf_untrans_bit_elem_scal(const void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf) { int64_t count; CHECK_MULT_EIGHT(size); count = bshuf_trans_byte_bitrow_scal(in, tmp_buf, size, elem_size); CHECK_ERR(count); count = bshuf_shuffle_bit_eightelem_scal(tmp_buf, out, size, elem_size); return count; } #ifdef _MSC_VER #pragma warning (pop) #endif zmat-0.9.9/src/blosc2/blosc/bitshuffle-avx2.c0000644000175200007730000001740014515254731021173 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /********************************************************************* Bitshuffle - Filter for improving compression of typed binary data. Author: Kiyoshi Masui Website: https://github.com/kiyo-masui/bitshuffle Note: Adapted for c-blosc by Francesc Alted. See LICENSES/BITSHUFFLE.txt file for details about copyright and rights to use. **********************************************************************/ #include "bitshuffle-generic.h" #include "bitshuffle-sse2.h" #include "bitshuffle-avx2.h" /* Make sure AVX2 is available for the compilation target and compiler. */ #if defined(__AVX2__) #include /* The next is useful for debugging purposes */ #if 0 #include #include static void printymm(__m256i ymm0) { uint8_t buf[32]; ((__m256i *)buf)[0] = ymm0; printf("%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x\n", buf[0], buf[1], buf[2], buf[3], buf[4], buf[5], buf[6], buf[7], buf[8], buf[9], buf[10], buf[11], buf[12], buf[13], buf[14], buf[15], buf[16], buf[17], buf[18], buf[19], buf[20], buf[21], buf[22], buf[23], buf[24], buf[25], buf[26], buf[27], buf[28], buf[29], buf[30], buf[31]); } #endif /* ---- Code that requires AVX2. Intel Haswell (2013) and later. ---- */ /* Transpose bits within bytes. */ int64_t bshuf_trans_bit_byte_avx2(void* in, void* out, const size_t size, const size_t elem_size) { char* in_b = (char*)in; char* out_b = (char*)out; int32_t* out_i32; size_t nbyte = elem_size * size; int64_t count; __m256i ymm; int32_t bt; size_t ii, kk; for (ii = 0; ii + 31 < nbyte; ii += 32) { ymm = _mm256_loadu_si256((__m256i*)&in_b[ii]); for (kk = 0; kk < 8; kk++) { bt = _mm256_movemask_epi8(ymm); ymm = _mm256_slli_epi16(ymm, 1); out_i32 = (int32_t*)&out_b[((7 - kk) * nbyte + ii) / 8]; *out_i32 = bt; } } count = bshuf_trans_bit_byte_remainder(in, out, size, elem_size, nbyte - nbyte % 32); return count; } /* Transpose bits within elements. */ int64_t bshuf_trans_bit_elem_avx2(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf) { int64_t count; CHECK_MULT_EIGHT(size); count = bshuf_trans_byte_elem_sse2(in, out, size, elem_size, tmp_buf); CHECK_ERR(count); count = bshuf_trans_bit_byte_avx2(out, tmp_buf, size, elem_size); CHECK_ERR(count); count = bshuf_trans_bitrow_eight(tmp_buf, out, size, elem_size); return count; } /* For data organized into a row for each bit (8 * elem_size rows), transpose * the bytes. */ int64_t bshuf_trans_byte_bitrow_avx2(void* in, void* out, const size_t size, const size_t elem_size) { char* in_b = (char*)in; char* out_b = (char*)out; size_t nrows = 8 * elem_size; size_t nbyte_row = size / 8; size_t ii, jj, kk, hh, mm; CHECK_MULT_EIGHT(size); if (elem_size % 4) return bshuf_trans_byte_bitrow_sse2(in, out, size, elem_size); __m256i ymm_0[8]; __m256i ymm_1[8]; __m256i ymm_storeage[8][4]; for (jj = 0; jj + 31 < nbyte_row; jj += 32) { for (ii = 0; ii + 3 < elem_size; ii += 4) { for (hh = 0; hh < 4; hh++) { for (kk = 0; kk < 8; kk++) { ymm_0[kk] = _mm256_loadu_si256((__m256i*)&in_b[ (ii * 8 + hh * 8 + kk) * nbyte_row + jj]); } for (kk = 0; kk < 4; kk++) { ymm_1[kk] = _mm256_unpacklo_epi8(ymm_0[kk * 2], ymm_0[kk * 2 + 1]); ymm_1[kk + 4] = _mm256_unpackhi_epi8(ymm_0[kk * 2], ymm_0[kk * 2 + 1]); } for (kk = 0; kk < 2; kk++) { for (mm = 0; mm < 2; mm++) { ymm_0[kk * 4 + mm] = _mm256_unpacklo_epi16( ymm_1[kk * 4 + mm * 2], ymm_1[kk * 4 + mm * 2 + 1]); ymm_0[kk * 4 + mm + 2] = _mm256_unpackhi_epi16( ymm_1[kk * 4 + mm * 2], ymm_1[kk * 4 + mm * 2 + 1]); } } for (kk = 0; kk < 4; kk++) { ymm_1[kk * 2] = _mm256_unpacklo_epi32(ymm_0[kk * 2], ymm_0[kk * 2 + 1]); ymm_1[kk * 2 + 1] = _mm256_unpackhi_epi32(ymm_0[kk * 2], ymm_0[kk * 2 + 1]); } for (kk = 0; kk < 8; kk++) { ymm_storeage[kk][hh] = ymm_1[kk]; } } for (mm = 0; mm < 8; mm++) { for (kk = 0; kk < 4; kk++) { ymm_0[kk] = ymm_storeage[mm][kk]; } ymm_1[0] = _mm256_unpacklo_epi64(ymm_0[0], ymm_0[1]); ymm_1[1] = _mm256_unpacklo_epi64(ymm_0[2], ymm_0[3]); ymm_1[2] = _mm256_unpackhi_epi64(ymm_0[0], ymm_0[1]); ymm_1[3] = _mm256_unpackhi_epi64(ymm_0[2], ymm_0[3]); ymm_0[0] = _mm256_permute2x128_si256(ymm_1[0], ymm_1[1], 32); ymm_0[1] = _mm256_permute2x128_si256(ymm_1[2], ymm_1[3], 32); ymm_0[2] = _mm256_permute2x128_si256(ymm_1[0], ymm_1[1], 49); ymm_0[3] = _mm256_permute2x128_si256(ymm_1[2], ymm_1[3], 49); _mm256_storeu_si256((__m256i*)&out_b[ (jj + mm * 2 + 0 * 16) * nrows + ii * 8], ymm_0[0]); _mm256_storeu_si256((__m256i*)&out_b[ (jj + mm * 2 + 0 * 16 + 1) * nrows + ii * 8], ymm_0[1]); _mm256_storeu_si256((__m256i*)&out_b[ (jj + mm * 2 + 1 * 16) * nrows + ii * 8], ymm_0[2]); _mm256_storeu_si256((__m256i*)&out_b[ (jj + mm * 2 + 1 * 16 + 1) * nrows + ii * 8], ymm_0[3]); } } } for (ii = 0; ii < nrows; ii++) { for (jj = nbyte_row - nbyte_row % 32; jj < nbyte_row; jj++) { out_b[jj * nrows + ii] = in_b[ii * nbyte_row + jj]; } } return (int64_t)size * (int64_t)elem_size; } /* Shuffle bits within the bytes of eight element blocks. */ int64_t bshuf_shuffle_bit_eightelem_avx2(void* in, void* out, const size_t size, const size_t elem_size) { CHECK_MULT_EIGHT(size); /* With a bit of care, this could be written such that such that it is */ /* in_buf = out_buf safe. */ char* in_b = (char*)in; char* out_b = (char*)out; size_t nbyte = elem_size * size; size_t ii, jj, kk, ind; __m256i ymm; int32_t bt; if (elem_size % 4) { return bshuf_shuffle_bit_eightelem_sse2(in, out, size, elem_size); } else { for (jj = 0; jj + 31 < 8 * elem_size; jj += 32) { for (ii = 0; ii + 8 * elem_size - 1 < nbyte; ii += 8 * elem_size) { ymm = _mm256_loadu_si256((__m256i*)&in_b[ii + jj]); for (kk = 0; kk < 8; kk++) { bt = _mm256_movemask_epi8(ymm); ymm = _mm256_slli_epi16(ymm, 1); ind = (ii + jj / 8 + (7 - kk) * elem_size); *(int32_t*)&out_b[ind] = bt; } } } } return (int64_t)size * (int64_t)elem_size; } /* Untranspose bits within elements. */ int64_t bshuf_untrans_bit_elem_avx2(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf) { int64_t count; CHECK_MULT_EIGHT(size); count = bshuf_trans_byte_bitrow_avx2(in, tmp_buf, size, elem_size); CHECK_ERR(count); count = bshuf_shuffle_bit_eightelem_avx2(tmp_buf, out, size, elem_size); return count; } #endif /* defined(__AVX2__) */ zmat-0.9.9/src/blosc2/blosc/b2nd_utils.h0000644000175200007730000000160714515254731020236 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef B2ND_B2ND_UTILS_H_ #define B2ND_B2ND_UTILS_H_ #include #include <../plugins/plugin_utils.h> #ifdef __cplusplus extern "C" { #endif int b2nd_copy_buffer(int8_t ndim, uint8_t itemsize, void *src, const int64_t *src_pad_shape, int64_t *src_start, const int64_t *src_stop, void *dst, const int64_t *dst_pad_shape, int64_t *dst_start); #ifdef __cplusplus } #endif #endif // B2ND_B2ND_UTILS_H_ zmat-0.9.9/src/blosc2/blosc/bitshuffle-altivec.h0000644000175200007730000000305614515254731021751 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /* ALTIVEC-accelerated shuffle/unshuffle routines. */ #ifndef BITSHUFFLE_ALTIVEC_H #define BITSHUFFLE_ALTIVEC_H #include "blosc2/blosc2-common.h" #ifdef __cplusplus extern "C" { #endif BLOSC_NO_EXPORT int64_t bshuf_trans_byte_elem_altivec(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf); BLOSC_NO_EXPORT int64_t bshuf_trans_byte_bitrow_altivec(void* in, void* out, const size_t size, const size_t elem_size); BLOSC_NO_EXPORT int64_t bshuf_shuffle_bit_eightelem_altivec(void* in, void* out, const size_t size, const size_t elem_size); /** ALTIVEC-accelerated bitshuffle routine. */ BLOSC_NO_EXPORT int64_t bshuf_trans_bit_elem_altivec(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf); /** ALTIVEC-accelerated bitunshuffle routine. */ BLOSC_NO_EXPORT int64_t bshuf_untrans_bit_elem_altivec(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf); #ifdef __cplusplus } #endif #endif /* BITSHUFFLE_ALTIVEC_H */ zmat-0.9.9/src/blosc2/blosc/shuffle-altivec.c0000644000175200007730000003650714515254731021254 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc developers and Jerome Kieffer https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include "shuffle-generic.h" #include "shuffle-altivec.h" /* Make sure ALTIVEC is available for the compilation target and compiler. */ #if defined(__ALTIVEC__) #include #include "transpose-altivec.h" /* Routine optimized for shuffling a buffer for a type size of 2 bytes. */ static void shuffle2_altivec(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements){ static const int32_t bytesoftype = 2; uint32_t i, j; __vector uint8_t xmm0[2]; for (j = 0; j < vectorizable_elements; j += 16){ /* Fetch 16 elements (32 bytes) */ for (i = 0; i < bytesoftype; i++) xmm0[i] = vec_xl(bytesoftype * j + 16 * i, src); /* Transpose vectors */ transpose2x16(xmm0); /* Store the result vectors */ for (i = 0; i < bytesoftype; i++) vec_xst(xmm0[i], j + i * total_elements, dest); } } /* Routine optimized for shuffling a buffer for a type size of 4 bytes. */ static void shuffle4_altivec(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements){ static const int32_t bytesoftype = 4; int32_t i, j; __vector uint8_t xmm0[4]; for (j = 0; j < vectorizable_elements; j += 16) { /* Fetch 16 elements (64 bytes, 4 vectors) */ for (i = 0; i < bytesoftype; i++) xmm0[i] = vec_xl(bytesoftype * j + 16 * i, src); /* Transpose vectors */ transpose4x16(xmm0); /* Store the result vectors */ for (i = 0; i < bytesoftype; i ++){ vec_xst(xmm0[i], j + i*total_elements, dest); } } } /* Routine optimized for shuffling a buffer for a type size of 8 bytes. */ static void shuffle8_altivec(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const uint8_t bytesoftype = 8; int32_t i, j; __vector uint8_t xmm0[8]; for (j = 0; j < vectorizable_elements; j += 16) { /* Fetch 16 elements (128 bytes, 8 vectors) */ for (i = 0; i < bytesoftype; i++) xmm0[i] = vec_xl(bytesoftype * j + 16 * i, src); /* Transpose vectors */ transpose8x16(xmm0); /* Store the result vectors */ for (i = 0; i < bytesoftype; i++) vec_xst(xmm0[i], j + i*total_elements, dest); } } /* Routine optimized for shuffling a buffer for a type size of 16 bytes. */ static void shuffle16_altivec(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 16; int32_t i, j; __vector uint8_t xmm0[16]; for (j = 0; j < vectorizable_elements; j += 16) { /* Fetch 16 elements (256 bytes, 16 vectors) */ for (i = 0; i < bytesoftype; i++) xmm0[i] = vec_xl(bytesoftype * j + 16 * i, src); // Do the job ! transpose16x16(xmm0); /* Store the result vectors */ for (i = 0; i < bytesoftype; i ++) vec_xst(xmm0[i], j + i * total_elements, dest); } } /* Routine optimized for shuffling a buffer for a type size larger than 16 bytes. */ static void shuffle16_tiled_altivec(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements, const int32_t bytesoftype) { int32_t j, k; const int32_t vecs_per_el_rem = bytesoftype & 0xF; __vector uint8_t xmm[16]; for (j = 0; j < vectorizable_elements; j += 16) { /* Advance the offset into the type by the vector size (in bytes), unless this is the initial iteration and the type size is not a multiple of the vector size. In that case, only advance by the number of bytes necessary so that the number of remaining bytes in the type will be a multiple of the vector size. */ int32_t offset_into_type; for (offset_into_type = 0; offset_into_type < bytesoftype; offset_into_type += (offset_into_type == 0 && vecs_per_el_rem > 0 ? vecs_per_el_rem : 16)) { /* Fetch elements in groups of 256 bytes */ const uint8_t* const src_with_offset = src + offset_into_type; for (k = 0; k < 16; k++) xmm[k] = vec_xl((j + k) * bytesoftype, src_with_offset); // Do the Job! transpose16x16(xmm); /* Store the result vectors */ for (k = 0; k < 16; k++) { vec_xst(xmm[k], j + total_elements * (offset_into_type + k), dest); } } } } /* Routine optimized for unshuffling a buffer for a type size of 2 bytes. */ static void unshuffle2_altivec(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 2; uint32_t i, j; __vector uint8_t xmm0[2], xmm1[2]; for (j = 0; j < vectorizable_elements; j += 16) { /* Load 16 elements (32 bytes) into 2 vectors registers. */ for (i = 0; i < bytesoftype; i++) xmm0[i] = vec_xl(j + i * total_elements, src); /* Shuffle bytes */ /* Note the shuffling is different from intel's SSE2 */ xmm1[0] = vec_vmrghb(xmm0[0], xmm0[1]); xmm1[1] = vec_vmrglb(xmm0[0], xmm0[1]); /* Store the result vectors*/ for (i = 0; i < bytesoftype; i++) vec_xst(xmm1[i], bytesoftype * j + 16 * i, dest); } } /* Routine optimized for unshuffling a buffer for a type size of 4 bytes. */ static void unshuffle4_altivec(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 4; uint32_t i, j; __vector uint8_t xmm0[4], xmm1[4]; for (j = 0; j < vectorizable_elements; j += 16) { /* Load 16 elements (64 bytes) into 4 vectors registers. */ for (i = 0; i < bytesoftype; i++) xmm0[i] = vec_xl(j + i * total_elements, src); /* Shuffle bytes */ for (i = 0; i < 2; i++) { xmm1[i ] = vec_vmrghb(xmm0[i * 2], xmm0[i * 2 + 1]); xmm1[i+2] = vec_vmrglb(xmm0[i * 2], xmm0[i * 2 + 1]); } /* Shuffle 2-byte words */ for (i = 0; i < 2; i++) { /* Compute the low 32 bytes */ xmm0[i] = (__vector uint8_t) vec_vmrghh((__vector uint16_t)xmm1[i * 2], (__vector uint16_t) xmm1[i * 2 + 1]); /* Compute the hi 32 bytes */ xmm0[i+2] = (__vector uint8_t) vec_vmrglh((__vector uint16_t)xmm1[i * 2], (__vector uint16_t)xmm1[i * 2 + 1]); } /* Store the result vectors in proper order */ vec_xst(xmm0[0], bytesoftype * j, dest); vec_xst(xmm0[2], bytesoftype * j + 16, dest); vec_xst(xmm0[1], bytesoftype * j + 32, dest); vec_xst(xmm0[3], bytesoftype * j + 48, dest); } } /* Routine optimized for unshuffling a buffer for a type size of 8 bytes. */ static void unshuffle8_altivec(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const uint8_t bytesoftype = 8; uint32_t i, j; __vector uint8_t xmm0[8], xmm1[8]; // Initialize permutations for writing for (j = 0; j < vectorizable_elements; j += 16) { /* Load 16 elements (64 bytes) into 4 vectors registers. */ for (i = 0; i < bytesoftype; i++) xmm0[i] = vec_xl(j + i * total_elements, src); /* Shuffle bytes */ for (i = 0; i < 4; i++) { xmm1[i] = vec_vmrghb(xmm0[i * 2], xmm0[i * 2 + 1]); xmm1[4 + i] = vec_vmrglb(xmm0[i * 2], xmm0[i * 2 + 1]); } /* Shuffle 2-byte words */ for (i = 0; i < 4; i++) { xmm0[i] = (__vector uint8_t)vec_vmrghh((__vector uint16_t)xmm1[i * 2], (__vector uint16_t)xmm1[i * 2 + 1]); xmm0[4 + i] = (__vector uint8_t)vec_vmrglh((__vector uint16_t)xmm1[i * 2], (__vector uint16_t)xmm1[i * 2 + 1]); } /* Shuffle 4-byte dwords */ for (i = 0; i < 4; i++) { xmm1[i] = (__vector uint8_t)vec_vmrghw((__vector uint32_t)xmm0[i * 2], (__vector uint32_t)xmm0[i * 2 + 1]); xmm1[4 + i] = (__vector uint8_t)vec_vmrglw((__vector uint32_t)xmm0[i * 2], (__vector uint32_t)xmm0[i * 2 + 1]); } /* Store the result vectors in proper order */ vec_xst(xmm1[0], bytesoftype * j, dest); vec_xst(xmm1[4], bytesoftype * j + 16, dest); vec_xst(xmm1[2], bytesoftype * j + 32, dest); vec_xst(xmm1[6], bytesoftype * j + 48, dest); vec_xst(xmm1[1], bytesoftype * j + 64, dest); vec_xst(xmm1[5], bytesoftype * j + 80, dest); vec_xst(xmm1[3], bytesoftype * j + 96, dest); vec_xst(xmm1[7], bytesoftype * j + 112, dest); } } /* Routine optimized for unshuffling a buffer for a type size of 16 bytes. */ static void unshuffle16_altivec(uint8_t* const dest, const uint8_t* const src, const int32_t vectorizable_elements, const int32_t total_elements) { static const int32_t bytesoftype = 16; uint32_t i, j; __vector uint8_t xmm0[16]; for (j = 0; j < vectorizable_elements; j += 16) { /* Load 16 elements (64 bytes) into 4 vectors registers. */ for (i = 0; i < bytesoftype; i++) xmm0[i] = vec_xl(j + i * total_elements, src); // Do the Job! transpose16x16(xmm0); /* Store the result vectors*/ for (i = 0; i < 16; i++) vec_st(xmm0[i], bytesoftype * (i+j), dest); } } /* Routine optimized for unshuffling a buffer for a type size larger than 16 bytes. */ static void unshuffle16_tiled_altivec(uint8_t* const dest, const uint8_t* const orig, const int32_t vectorizable_elements, const int32_t total_elements, const int32_t bytesoftype) { int32_t i, j, offset_into_type; const int32_t vecs_per_el_rem = bytesoftype & 0xF; __vector uint8_t xmm[16]; /* Advance the offset into the type by the vector size (in bytes), unless this is the initial iteration and the type size is not a multiple of the vector size. In that case, only advance by the number of bytes necessary so that the number of remaining bytes in the type will be a multiple of the vector size. */ for (offset_into_type = 0; offset_into_type < bytesoftype; offset_into_type += (offset_into_type == 0 && vecs_per_el_rem > 0 ? vecs_per_el_rem : 16)) { for (i = 0; i < vectorizable_elements; i += 16) { /* Load the first 128 bytes in 16 XMM registers */ for (j = 0; j < 16; j++) xmm[j] = vec_xl(total_elements * (offset_into_type + j) + i, orig); // Do the Job ! transpose16x16(xmm); /* Store the result vectors in proper order */ for (j = 0; j < 16; j++) vec_xst(xmm[j], (i + j) * bytesoftype + offset_into_type, dest); } } } /* Shuffle a block. This can never fail. */ void shuffle_altivec(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest) { int32_t vectorized_chunk_size; vectorized_chunk_size = bytesoftype * 16; /* If the blocksize is not a multiple of both the typesize and the vector size, round the blocksize down to the next value which is a multiple of both. The vectorized shuffle can be used for that portion of the data, and the naive implementation can be used for the remaining portion. */ const int32_t vectorizable_bytes = blocksize - (blocksize % vectorized_chunk_size); const int32_t vectorizable_elements = vectorizable_bytes / bytesoftype; const int32_t total_elements = blocksize / bytesoftype; /* If the block size is too small to be vectorized, use the generic implementation. */ if (blocksize < vectorized_chunk_size) { shuffle_generic(bytesoftype, blocksize, _src, _dest); return; } /* Optimized shuffle implementations */ switch (bytesoftype) { case 2: shuffle2_altivec(_dest, _src, vectorizable_elements, total_elements); break; case 4: shuffle4_altivec(_dest, _src, vectorizable_elements, total_elements); break; case 8: shuffle8_altivec(_dest, _src, vectorizable_elements, total_elements); break; case 16: shuffle16_altivec(_dest, _src, vectorizable_elements, total_elements); break; default: if (bytesoftype > 16) { shuffle16_tiled_altivec(_dest, _src, vectorizable_elements, total_elements, bytesoftype); } else { /* Non-optimized shuffle */ shuffle_generic(bytesoftype, blocksize, _src, _dest); /* The non-optimized function covers the whole buffer, so we're done processing here. */ return; } } /* If the buffer had any bytes at the end which couldn't be handled by the vectorized implementations, use the non-optimized version to finish them up. */ if (vectorizable_bytes < blocksize) { shuffle_generic_inline(bytesoftype, vectorizable_bytes, blocksize, _src, _dest); } } /* Unshuffle a block. This can never fail. */ void unshuffle_altivec(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest) { const int32_t vectorized_chunk_size = bytesoftype * 16; /* If the blocksize is not a multiple of both the typesize and the vector size, round the blocksize down to the next value which is a multiple of both. The vectorized unshuffle can be used for that portion of the data, and the naive implementation can be used for the remaining portion. */ const int32_t vectorizable_bytes = blocksize - (blocksize % vectorized_chunk_size); const int32_t vectorizable_elements = vectorizable_bytes / bytesoftype; const int32_t total_elements = blocksize / bytesoftype; /* If the block size is too small to be vectorized, use the generic implementation. */ if (blocksize < vectorized_chunk_size) { unshuffle_generic(bytesoftype, blocksize, _src, _dest); return; } /* Optimized unshuffle implementations */ switch (bytesoftype) { case 2: unshuffle2_altivec(_dest, _src, vectorizable_elements, total_elements); break; case 4: unshuffle4_altivec(_dest, _src, vectorizable_elements, total_elements); break; case 8: unshuffle8_altivec(_dest, _src, vectorizable_elements, total_elements); break; case 16: unshuffle16_altivec(_dest, _src, vectorizable_elements, total_elements); break; default: if (bytesoftype > 16) { unshuffle16_tiled_altivec(_dest, _src, vectorizable_elements, total_elements, bytesoftype); } else { /* Non-optimized unshuffle */ unshuffle_generic(bytesoftype, blocksize, _src, _dest); /* The non-optimized function covers the whole buffer, so we're done processing here. */ return; } } /* If the buffer had any bytes at the end which couldn't be handled by the vectorized implementations, use the non-optimized version to finish them up. */ if (vectorizable_bytes < blocksize) { unshuffle_generic_inline(bytesoftype, vectorizable_bytes, blocksize, _src, _dest); } } #endif /* defined(__ALTIVEC__) */ zmat-0.9.9/src/blosc2/blosc/stune.c0000644000175200007730000001202114515254731017312 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include #include #include "stune.h" /* Whether a codec is meant for High Compression Ratios Includes LZ4 + BITSHUFFLE here, but not BloscLZ + BITSHUFFLE because, for some reason, the latter does not work too well */ static bool is_HCR(blosc2_context * context) { switch (context->compcode) { case BLOSC_BLOSCLZ : return false; case BLOSC_LZ4 : // return (context->filter_flags & BLOSC_DOBITSHUFFLE) ? true : false; // Do not treat LZ4 differently than BloscLZ here return false; case BLOSC_LZ4HC : case BLOSC_ZLIB : case BLOSC_ZSTD : return true; default : return false; } } void blosc_stune_init(void * config, blosc2_context* cctx, blosc2_context* dctx) { BLOSC_UNUSED_PARAM(config); BLOSC_UNUSED_PARAM(cctx); BLOSC_UNUSED_PARAM(dctx); } // Set the automatic blocksize 0 to its real value void blosc_stune_next_blocksize(blosc2_context *context) { int32_t clevel = context->clevel; int32_t typesize = context->typesize; int32_t nbytes = context->sourcesize; int32_t user_blocksize = context->blocksize; int32_t blocksize = nbytes; // Protection against very small buffers if (nbytes < typesize) { context->blocksize = 1; return; } if (user_blocksize) { blocksize = user_blocksize; goto last; } if (nbytes >= L1) { blocksize = L1; /* For HCR codecs, increase the block sizes by a factor of 2 because they are meant for compressing large blocks (i.e. they show a big overhead when compressing small ones). */ if (is_HCR(context)) { blocksize *= 2; } // Choose a different blocksize depending on the compression level switch (clevel) { case 0: // Case of plain copy blocksize /= 4; break; case 1: blocksize /= 2; break; case 2: blocksize *= 1; break; case 3: blocksize *= 2; break; case 4: case 5: blocksize *= 4; break; case 6: case 7: case 8: blocksize *= 8; break; case 9: // Do not exceed 256 KB for non HCR codecs blocksize *= 8; if (is_HCR(context)) { blocksize *= 2; } break; default: break; } } /* Now the blocksize for splittable codecs */ if (clevel > 0 && split_block(context, typesize, blocksize)) { // For performance reasons, do not exceed 256 KB (it must fit in L2 cache) switch (clevel) { case 1: case 2: case 3: blocksize = 32 * 1024; break; case 4: case 5: case 6: blocksize = 64 * 1024; break; case 7: case 8: blocksize = 256 * 1024; break; case 9: default: blocksize = 512 * 1024; break; } // Multiply by typesize to get proper split sizes blocksize *= typesize; // But do not exceed 4 MB per thread (having this capacity in L3 is normal in modern CPUs) if (blocksize > 4 * 1024 * 1024) { blocksize = 4 * 1024 * 1024; } if (blocksize < 32 * 1024) { /* Do not use a too small blocksize (< 32 KB) when typesize is small */ blocksize = 32 * 1024; } } last: /* Check that blocksize is not too large */ if (blocksize > nbytes) { blocksize = nbytes; } // blocksize *must absolutely* be a multiple of the typesize if (blocksize > typesize) { blocksize = blocksize / typesize * typesize; } context->blocksize = blocksize; } void blosc_stune_next_cparams(blosc2_context * context) { BLOSC_UNUSED_PARAM(context); } void blosc_stune_update(blosc2_context * context, double ctime) { BLOSC_UNUSED_PARAM(context); BLOSC_UNUSED_PARAM(ctime); } void blosc_stune_free(blosc2_context * context) { BLOSC_UNUSED_PARAM(context); } int split_block(blosc2_context *context, int32_t typesize, int32_t blocksize) { switch (context->splitmode) { case BLOSC_ALWAYS_SPLIT: return 1; case BLOSC_NEVER_SPLIT: return 0; case BLOSC_FORWARD_COMPAT_SPLIT: case BLOSC_AUTO_SPLIT: // These cases will be handled later break; default: BLOSC_TRACE_WARNING("Unrecognized split mode. Default to BLOSC_FORWARD_COMPAT_SPLIT"); } int compcode = context->compcode; return ( // Fast codecs like blosclz and lz4 always prefer to always split ((compcode == BLOSC_BLOSCLZ) || (compcode == BLOSC_LZ4)) && // ...but split seems to harm cratio too much when not using shuffle (context->filter_flags & BLOSC_DOSHUFFLE) && (typesize <= MAX_STREAMS) && (blocksize / typesize) >= BLOSC_MIN_BUFFERSIZE); } zmat-0.9.9/src/blosc2/blosc/bitshuffle-avx2.h0000644000175200007730000000203614515254731021177 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /* AVX2-accelerated shuffle/unshuffle routines. */ #ifndef BITSHUFFLE_AVX2_H #define BITSHUFFLE_AVX2_H #include #ifdef __cplusplus extern "C" { #endif /** AVX2-accelerated bitshuffle routine. */ BLOSC_NO_EXPORT int64_t bshuf_trans_bit_elem_avx2(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf); /** AVX2-accelerated bitunshuffle routine. */ BLOSC_NO_EXPORT int64_t bshuf_untrans_bit_elem_avx2(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf); #ifdef __cplusplus } #endif #endif /* BITSHUFFLE_AVX2_H */ zmat-0.9.9/src/blosc2/blosc/context.h0000644000175200007730000001345714515254731017663 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef CONTEXT_H #define CONTEXT_H #if defined(_WIN32) && !defined(__GNUC__) #include "win32/pthread.h" #else #include #endif /* Have problems using posix barriers when symbol value is 200112L */ /* Requires more investigation, but this will work for the moment */ #if defined(_POSIX_BARRIERS) && ( (_POSIX_BARRIERS - 20012L) >= 0 && _POSIX_BARRIERS != 200112L) #define BLOSC_POSIX_BARRIERS #endif #include "blosc2.h" #include "b2nd.h" #if defined(HAVE_ZSTD) #include "zstd.h" #endif /* HAVE_ZSTD */ #ifdef HAVE_IPP #include #endif /* HAVE_IPP */ struct blosc2_context_s { const uint8_t* src; /* The source buffer */ uint8_t* dest; /* The destination buffer */ uint8_t header_flags; /* Flags for header */ uint8_t blosc2_flags; /* Flags specific for blosc2 */ int32_t sourcesize; /* Number of bytes in source buffer */ int32_t header_overhead; /* The number of bytes in chunk header */ int32_t nblocks; /* Number of total blocks in buffer */ int32_t leftover; /* Extra bytes at end of buffer */ int32_t blocksize; /* Length of the block in bytes */ int32_t splitmode; /* Whether the blocks should be split or not */ int32_t output_bytes; /* Counter for the number of input bytes */ int32_t srcsize; /* Counter for the number of output bytes */ int32_t destsize; /* Maximum size for destination buffer */ int32_t typesize; /* Type size */ int32_t* bstarts; /* Starts for every block inside the compressed buffer */ int32_t special_type; /* Special type for chunk. 0 if not special. */ int compcode; /* Compressor code to use */ uint8_t compcode_meta; /* The metainfo for the compressor code */ int clevel; /* Compression level (1-9) */ int use_dict; /* Whether to use dicts or not */ void* dict_buffer; /* The buffer to keep the trained dictionary */ int32_t dict_size; /* The size of the trained dictionary */ void* dict_cdict; /* The dictionary in digested form for compression */ void* dict_ddict; /* The dictionary in digested form for decompression */ uint8_t filter_flags; /* The filter flags in the filter pipeline */ uint8_t filters[BLOSC2_MAX_FILTERS]; /* The (sequence of) filters */ uint8_t filters_meta[BLOSC2_MAX_FILTERS]; /* The metainfo for filters */ blosc2_filter urfilters[BLOSC2_MAX_UDFILTERS]; /* The user-defined filters */ blosc2_prefilter_fn prefilter; /* prefilter function */ blosc2_postfilter_fn postfilter; /* postfilter function */ blosc2_prefilter_params *preparams; /* prefilter params */ blosc2_postfilter_params *postparams; /* postfilter params */ bool* block_maskout; /* The blocks that are not meant to be decompressed. * If NULL (default), all blocks in a chunk should be read. */ int block_maskout_nitems; /* The number of items in block_maskout array (must match * the number of blocks in chunk) */ blosc2_schunk* schunk; /* Associated super-chunk (if available) */ struct thread_context* serial_context; /* Cache for temporaries for serial operation */ int do_compress; /* 1 if we are compressing, 0 if decompressing */ void *btune; /* Entry point for BTune persistence between runs */ blosc2_btune *udbtune; /* User-defined BTune parameters */ void *codec_params; /* User defined parameters for the codec */ void *filter_params[BLOSC2_MAX_FILTERS]; /* User defined parameters for the filters */ /* Threading */ int16_t nthreads; int16_t new_nthreads; int16_t threads_started; int16_t end_threads; pthread_t *threads; struct thread_context *thread_contexts; /* Only for user-managed threads */ pthread_mutex_t count_mutex; pthread_mutex_t nchunk_mutex; #ifdef BLOSC_POSIX_BARRIERS pthread_barrier_t barr_init; pthread_barrier_t barr_finish; #else int count_threads; pthread_mutex_t count_threads_mutex; pthread_cond_t count_threads_cv; #endif #if !defined(_WIN32) pthread_attr_t ct_attr; /* creation time attrs for threads */ #endif int thread_giveup_code; /* error code when give up */ int thread_nblock; /* block counter */ int dref_not_init; /* data ref in delta not initialized */ pthread_mutex_t delta_mutex; pthread_cond_t delta_cv; }; struct b2nd_context_s { int8_t ndim; //!< The array dimensions. int64_t shape[B2ND_MAX_DIM]; //!< The array shape. int32_t chunkshape[B2ND_MAX_DIM]; //!< The shape of each chunk of Blosc. int32_t blockshape[B2ND_MAX_DIM]; //!< The shape of each block of Blosc. char *dtype; //!< Data type. Different formats can be supported (see dtype_format). int8_t dtype_format; //!< The format of the data type. Default is 0 (NumPy). blosc2_storage *b2_storage; //!< The Blosc storage properties blosc2_metalayer metalayers[B2ND_MAX_METALAYERS]; //!< List with the metalayers desired. int32_t nmetalayers; //!< The number of metalayers. }; struct thread_context { blosc2_context* parent_context; int tid; uint8_t* tmp; uint8_t* tmp2; uint8_t* tmp3; uint8_t* tmp4; int32_t tmp_blocksize; /* the blocksize for different temporaries */ size_t tmp_nbytes; /* keep track of how big the temporary buffers are */ int32_t zfp_cell_start; /* cell starter index for ZFP fixed-rate mode */ int32_t zfp_cell_nitems; /* number of items to get for ZFP fixed-rate mode */ #if defined(HAVE_ZSTD) /* The contexts for ZSTD */ ZSTD_CCtx* zstd_cctx; ZSTD_DCtx* zstd_dctx; #endif /* HAVE_ZSTD */ #ifdef HAVE_IPP Ipp8u* lz4_hash_table; #endif }; #endif /* CONTEXT_H */ zmat-0.9.9/src/blosc2/blosc/blosc-private.h0000644000175200007730000001022114515254731020733 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef IARRAY_BLOSC_PRIVATE_H #define IARRAY_BLOSC_PRIVATE_H #ifdef __cplusplus extern "C" { #endif #include "stdbool.h" #include "blosc2.h" #include "blosc2/blosc2-common.h" /********************************************************************* Utility functions meant to be used internally. *********************************************************************/ #define to_little(dest, src, itemsize) endian_handler(true, dest, src, itemsize) #define from_little(dest, src, itemsize) endian_handler(true, dest, src, itemsize) #define to_big(dest, src, itemsize) endian_handler(false, dest, src, itemsize) #define from_big(dest, src, itemsize) endian_handler(false, dest, src, itemsize) // Return true if platform is little endian; else false static bool is_little_endian(void) { static const int i = 1; char* p = (char*)&i; if (p[0] == 1) { return true; } else { return false; } } static inline void endian_handler(bool little, void *dest, const void *pa, int size) { bool little_endian = is_little_endian(); if (little_endian == little) { memcpy(dest, pa, size); } else { uint8_t* pa_ = (uint8_t*)pa; uint8_t pa2_[8]; switch (size) { case 8: pa2_[0] = pa_[7]; pa2_[1] = pa_[6]; pa2_[2] = pa_[5]; pa2_[3] = pa_[4]; pa2_[4] = pa_[3]; pa2_[5] = pa_[2]; pa2_[6] = pa_[1]; pa2_[7] = pa_[0]; break; case 4: pa2_[0] = pa_[3]; pa2_[1] = pa_[2]; pa2_[2] = pa_[1]; pa2_[3] = pa_[0]; break; case 2: pa2_[0] = pa_[1]; pa2_[1] = pa_[0]; break; case 1: pa2_[0] = pa_[0]; break; default: BLOSC_TRACE_ERROR("Unhandled size: %d.", size); } memcpy(dest, pa2_, size); } } /* Copy 4 bytes from @p *pa to int32_t, changing endianness if necessary. */ static inline int32_t sw32_(const void* pa) { int32_t idest; bool little_endian = is_little_endian(); if (little_endian) { idest = *(int32_t *)pa; } else { #if defined (__GNUC__) return __builtin_bswap32(*(unsigned int *)pa); #elif defined (_MSC_VER) /* Visual Studio */ return _byteswap_ulong(*(unsigned int *)pa); #else uint8_t *dest = (uint8_t *)&idest; dest[0] = pa_[3]; dest[1] = pa_[2]; dest[2] = pa_[1]; dest[3] = pa_[0]; #endif } return idest; } /* Copy 4 bytes from int32_t to @p *dest, changing endianness if necessary. */ static inline void _sw32(void* dest, int32_t a) { uint8_t* dest_ = (uint8_t*)dest; uint8_t* pa = (uint8_t*)&a; bool little_endian = is_little_endian(); if (little_endian) { *(int32_t *)dest_ = a; } else { #if defined (__GNUC__) *(int32_t *)dest_ = __builtin_bswap32(*(unsigned int *)pa); #elif defined (_MSC_VER) /* Visual Studio */ *(int32_t *)dest_ = _byteswap_ulong(*(unsigned int *)pa); #else dest_[0] = pa[3]; dest_[1] = pa[2]; dest_[2] = pa[1]; dest_[3] = pa[0]; #endif } } /* Reverse swap bits in a 32-bit integer */ static inline int32_t bswap32_(int32_t a) { #if defined (__GNUC__) return __builtin_bswap32(a); #elif defined (_MSC_VER) /* Visual Studio */ return _byteswap_ulong(a); #else a = ((a & 0x000000FF) << 24) | ((a & 0x0000FF00) << 8) | ((a & 0x00FF0000) >> 8) | ((a & 0xFF000000) >> 24); return a; #endif } /** * @brief Register a filter in Blosc. * * @param filter The filter to register. * * @return 0 if succeeds. Else a negative code is returned. */ int register_filter_private(blosc2_filter *filter); /** * @brief Register a codec in Blosc. * * @param codec The codec to register. * * @return 0 if succeeds. Else a negative code is returned. */ int register_codec_private(blosc2_codec *codec); #ifdef __cplusplus } #endif #endif //IARRAY_BLOSC_PRIVATE_H zmat-0.9.9/src/blosc2/blosc/bitshuffle-sse2.h0000644000175200007730000000300714515254731021172 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /* SSE2-accelerated shuffle/unshuffle routines. */ #ifndef BITSHUFFLE_SSE2_H #define BITSHUFFLE_SSE2_H #include "blosc2/blosc2-common.h" #ifdef __cplusplus extern "C" { #endif BLOSC_NO_EXPORT int64_t bshuf_trans_byte_elem_sse2(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf); BLOSC_NO_EXPORT int64_t bshuf_trans_byte_bitrow_sse2(void* in, void* out, const size_t size, const size_t elem_size); BLOSC_NO_EXPORT int64_t bshuf_shuffle_bit_eightelem_sse2(void* in, void* out, const size_t size, const size_t elem_size); /** SSE2-accelerated bitshuffle routine. */ BLOSC_NO_EXPORT int64_t bshuf_trans_bit_elem_sse2(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf); /** SSE2-accelerated bitunshuffle routine. */ BLOSC_NO_EXPORT int64_t bshuf_untrans_bit_elem_sse2(void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf); #ifdef __cplusplus } #endif #endif /* BITSHUFFLE_SSE2_H */ zmat-0.9.9/src/blosc2/blosc/blosc2-stdio.c0000644000175200007730000000431314515254731020465 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include "blosc2/blosc2-stdio.h" #include "blosc2.h" void *blosc2_stdio_open(const char *urlpath, const char *mode, void *params) { BLOSC_UNUSED_PARAM(params); FILE *file = fopen(urlpath, mode); if (file == NULL) return NULL; blosc2_stdio_file *my_fp = malloc(sizeof(blosc2_stdio_file)); my_fp->file = file; return my_fp; } int blosc2_stdio_close(void *stream) { blosc2_stdio_file *my_fp = (blosc2_stdio_file *) stream; int err = fclose(my_fp->file); free(my_fp); return err; } int64_t blosc2_stdio_tell(void *stream) { blosc2_stdio_file *my_fp = (blosc2_stdio_file *) stream; int64_t pos; #if defined(_MSC_VER) && (_MSC_VER >= 1400) pos = _ftelli64(my_fp->file); #else pos = (int64_t)ftell(my_fp->file); #endif return pos; } int blosc2_stdio_seek(void *stream, int64_t offset, int whence) { blosc2_stdio_file *my_fp = (blosc2_stdio_file *) stream; int rc; #if defined(_MSC_VER) && (_MSC_VER >= 1400) rc = _fseeki64(my_fp->file, offset, whence); #else rc = fseek(my_fp->file, (long) offset, whence); #endif return rc; } int64_t blosc2_stdio_write(const void *ptr, int64_t size, int64_t nitems, void *stream) { blosc2_stdio_file *my_fp = (blosc2_stdio_file *) stream; size_t nitems_ = fwrite(ptr, (size_t) size, (size_t) nitems, my_fp->file); return (int64_t) nitems_; } int64_t blosc2_stdio_read(void *ptr, int64_t size, int64_t nitems, void *stream) { blosc2_stdio_file *my_fp = (blosc2_stdio_file *) stream; size_t nitems_ = fread(ptr, (size_t) size, (size_t) nitems, my_fp->file); return (int64_t) nitems_; } int blosc2_stdio_truncate(void *stream, int64_t size) { blosc2_stdio_file *my_fp = (blosc2_stdio_file *) stream; int rc; #if defined(_MSC_VER) && (_MSC_VER >= 1400) rc = _chsize_s(_fileno(my_fp->file), size); #else rc = ftruncate(fileno(my_fp->file), size); #endif return rc; } zmat-0.9.9/src/blosc2/blosc/frame.h0000644000175200007730000001541614515254731017266 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef BLOSC_FRAME_H #define BLOSC_FRAME_H #include #include // Different types of frames #define FRAME_CONTIGUOUS_TYPE 0 #define FRAME_DIRECTORY_TYPE 1 // Constants for metadata placement in header #define FRAME_HEADER_MAGIC 2 #define FRAME_HEADER_LEN (FRAME_HEADER_MAGIC + 8 + 1) // 11 #define FRAME_LEN (FRAME_HEADER_LEN + 4 + 1) // 16 #define FRAME_FLAGS (FRAME_LEN + 8 + 1) // 25 #define FRAME_TYPE (FRAME_FLAGS + 1) // 26 #define FRAME_CODECS (FRAME_FLAGS + 2) // 27 #define FRAME_OTHER_FLAGS (FRAME_FLAGS + 3) // 28 #define FRAME_NBYTES (FRAME_FLAGS + 4 + 1) // 30 #define FRAME_CBYTES (FRAME_NBYTES + 8 + 1) // 39 #define FRAME_TYPESIZE (FRAME_CBYTES + 8 + 1) // 48 #define FRAME_BLOCKSIZE (FRAME_TYPESIZE + 4 + 1) // 53 #define FRAME_CHUNKSIZE (FRAME_BLOCKSIZE + 4 + 1) // 58 #define FRAME_NTHREADS_C (FRAME_CHUNKSIZE + 4 + 1) // 63 #define FRAME_NTHREADS_D (FRAME_NTHREADS_C + 2 + 1) // 66 #define FRAME_HAS_VLMETALAYERS (FRAME_NTHREADS_D + 2) // 68 #define FRAME_FILTER_PIPELINE (FRAME_HAS_VLMETALAYERS + 1 + 1) // 70 #define FRAME_UDCODEC (FRAME_FILTER_PIPELINE + 1 + 6) // 77 #define FRAME_CODEC_META (FRAME_FILTER_PIPELINE + 1 + 7) // 78 #define FRAME_HEADER_MINLEN (FRAME_FILTER_PIPELINE + 1 + 16) // 87 <- minimum length #define FRAME_METALAYERS (FRAME_HEADER_MINLEN) // 87 #define FRAME_IDX_SIZE (FRAME_METALAYERS + 1 + 1) // 89 #define FRAME_FILTER_PIPELINE_MAX (8) // the maximum number of filters that can be stored in header #define FRAME_TRAILER_VERSION_BETA2 (0U) // for beta.2 and former #define FRAME_TRAILER_VERSION (1U) // can be up to 127 #define FRAME_TRAILER_MINLEN (25) // minimum length for the trailer (msgpack overhead) #define FRAME_TRAILER_LEN_OFFSET (22) // offset to trailer length (counting from the end) #define FRAME_TRAILER_VLMETALAYERS (2) typedef struct { char* urlpath; //!< The name of the file or directory if it's an sframe; if NULL, this is in-memory uint8_t* cframe; //!< The in-memory, contiguous frame buffer bool avoid_cframe_free; //!< Whether the cframe can be freed (false) or not (true). uint8_t* coffsets; //!< Pointers to the (compressed, on-disk) chunk offsets int64_t len; //!< The current length of the frame in (compressed) bytes int64_t maxlen; //!< The maximum length of the frame; if 0, there is no maximum uint32_t trailer_len; //!< The current length of the trailer in (compressed) bytes bool sframe; //!< Whether the frame is sparse (true) or not blosc2_schunk *schunk; //!< The schunk associated int64_t file_offset; //!< The offset where the frame starts inside the file } blosc2_frame_s; /********************************************************************* Frame struct related functions. These are rather low-level and the blosc2_schunk interface is recommended instead. *********************************************************************/ /** * @brief Create a new frame. * * @param urlpath The filename of the frame. If not persistent, pass NULL. * * @return The new frame. */ blosc2_frame_s* frame_new(const char* urlpath); /** * @brief Create a frame from a super-chunk. * * @param schunk The super-chunk from where the frame will be created. * @param frame The pointer for the frame that will be populated. * * @note If frame->urlpath is NULL, a frame is created in-memory; else it is created * on-disk. * * @return The size in bytes of the frame. If an error occurs it returns a negative value. */ int64_t frame_from_schunk(blosc2_schunk* schunk, blosc2_frame_s* frame); /** * @brief Set `avoid_cframe_free` from @param frame to @param avoid_cframe_free. * * @param frame The frame to set the property to. * @param avoid_cframe_free The value to set in @param frame. * @warning If you set it to `true` you will be responsible of freeing it. */ void frame_avoid_cframe_free(blosc2_frame_s* frame, bool avoid_cframe_free); /** * @brief Free all memory from a frame. * * @param frame The frame to be freed. * * @return 0 if succeeds. */ int frame_free(blosc2_frame_s *frame); /** * @brief Initialize a frame out of a file. * * @param urlpath The file name. * * @return The frame created from the file. */ blosc2_frame_s* frame_from_file_offset(const char *urlpath, const blosc2_io *io_cb, int64_t offset); /** * @brief Initialize a frame out of a frame buffer. * * @param buffer The buffer for the frame. * @param len The length of buffer for the frame. * @param copy Whether the frame buffer should be copied internally or not. * * @return The frame created from the frame buffer. */ blosc2_frame_s* frame_from_cframe(uint8_t *cframe, int64_t len, bool copy); /** * @brief Create a super-chunk from a frame. * * @param frame The frame from which the super-chunk will be created. * @param copy If true, a new frame buffer is created * internally to serve as storage for the super-chunk. Else, the * super-chunk will be backed by @p frame (i.e. no copies are made). * * @return The super-chunk corresponding to the frame. */ blosc2_schunk* frame_to_schunk(blosc2_frame_s* frame, bool copy, const blosc2_io *udio); blosc2_storage * get_new_storage(const blosc2_storage *storage, const blosc2_cparams *cdefaults, const blosc2_dparams *ddefaults, const blosc2_io *iodefaults); void* frame_append_chunk(blosc2_frame_s* frame, void* chunk, blosc2_schunk* schunk); void* frame_insert_chunk(blosc2_frame_s* frame, int64_t nchunk, void* chunk, blosc2_schunk* schunk); void* frame_update_chunk(blosc2_frame_s* frame, int64_t nchunk, void* chunk, blosc2_schunk* schunk); void* frame_delete_chunk(blosc2_frame_s* frame, int64_t nchunk, blosc2_schunk* schunk); int frame_reorder_offsets(blosc2_frame_s *frame, const int64_t *offsets_order, blosc2_schunk* schunk); int frame_get_chunk(blosc2_frame_s* frame, int64_t nchunk, uint8_t **chunk, bool *needs_free); int frame_get_lazychunk(blosc2_frame_s* frame, int64_t nchunk, uint8_t **chunk, bool *needs_free); int frame_decompress_chunk(blosc2_context* dctx, blosc2_frame_s* frame, int64_t nchunk, void *dest, int32_t nbytes); int frame_update_header(blosc2_frame_s* frame, blosc2_schunk* schunk, bool new); int frame_update_trailer(blosc2_frame_s* frame, blosc2_schunk* schunk); int64_t frame_fill_special(blosc2_frame_s* frame, int64_t nitems, int special_value, int32_t chunksize, blosc2_schunk* schunk); #endif //BLOSC_FRAME_H zmat-0.9.9/src/blosc2/blosc/frame.c0000644000175200007730000034237414515254731017267 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include #include #include #include #include #include "blosc2.h" #include "blosc-private.h" #include "context.h" #include "frame.h" #include "sframe.h" #include #if defined(_WIN32) #include #include /* stdint.h only available in VS2010 (VC++ 16.0) and newer */ #if defined(_MSC_VER) && _MSC_VER < 1600 #include "win32/stdint-windows.h" #else #include #endif #endif /* _WIN32 */ /* If C11 is supported, use it's built-in aligned allocation. */ #if __STDC_VERSION__ >= 201112L #include #endif /* Create a new (empty) frame */ blosc2_frame_s* frame_new(const char* urlpath) { blosc2_frame_s* new_frame = calloc(1, sizeof(blosc2_frame_s)); if (urlpath != NULL) { char* new_urlpath = malloc(strlen(urlpath) + 1); // + 1 for the trailing NULL new_frame->urlpath = strcpy(new_urlpath, urlpath); new_frame->file_offset = 0; } return new_frame; } /* Free memory from a frame. */ int frame_free(blosc2_frame_s* frame) { if (frame->cframe != NULL && !frame->avoid_cframe_free) { free(frame->cframe); } if (frame->coffsets != NULL) { free(frame->coffsets); } if (frame->urlpath != NULL) { free(frame->urlpath); } free(frame); return 0; } void *new_header_frame(blosc2_schunk *schunk, blosc2_frame_s *frame) { if (frame == NULL) { return NULL; } uint8_t* h2 = calloc(FRAME_HEADER_MINLEN, 1); uint8_t* h2p = h2; // The msgpack header starts here *h2p = 0x90; // fixarray... *h2p += 14; // ...with 13 elements h2p += 1; // Magic number *h2p = 0xa0 + 8; // str with 8 elements h2p += 1; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } strcpy((char*)h2p, "b2frame"); h2p += 8; // Header size *h2p = 0xd2; // int32 h2p += 1 + 4; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // Total frame size *h2p = 0xcf; // uint64 // Fill it with frame->len which is known *after* the creation of the frame (e.g. when updating the header) int64_t flen = frame->len; to_big(h2 + FRAME_LEN, &flen, sizeof(flen)); h2p += 1 + 8; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // Flags *h2p = 0xa0 + 4; // str with 4 elements h2p += 1; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // General flags *h2p = BLOSC2_VERSION_FRAME_FORMAT; // version *h2p += 0x10; // 64-bit offsets. We only support this for now. h2p += 1; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // Frame type // We only support contiguous and sparse directories frames currently *h2p = frame->sframe ? 1 : 0; h2p += 1; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // Codec flags *h2p = schunk->compcode; if (schunk->compcode >= BLOSC_LAST_CODEC) { *h2p = BLOSC_UDCODEC_FORMAT; } *h2p += (schunk->clevel) << 4u; // clevel h2p += 1; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // Other flags *h2p = schunk->splitmode - 1; h2p += 1; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // Uncompressed size *h2p = 0xd3; // int64 h2p += 1; int64_t nbytes = schunk->nbytes; to_big(h2p, &nbytes, sizeof(nbytes)); h2p += 8; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // Compressed size *h2p = 0xd3; // int64 h2p += 1; int64_t cbytes = schunk->cbytes; to_big(h2p, &cbytes, sizeof(cbytes)); h2p += 8; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // Type size *h2p = 0xd2; // int32 h2p += 1; int32_t typesize = schunk->typesize; to_big(h2p, &typesize, sizeof(typesize)); h2p += 4; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // Block size *h2p = 0xd2; // int32 h2p += 1; int32_t blocksize = schunk->blocksize; to_big(h2p, &blocksize, sizeof(blocksize)); h2p += 4; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // Chunk size *h2p = 0xd2; // int32 h2p += 1; int32_t chunksize = schunk->chunksize; to_big(h2p, &chunksize, sizeof(chunksize)); h2p += 4; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // Number of threads for compression *h2p = 0xd1; // int16 h2p += 1; int16_t nthreads = (int16_t)schunk->cctx->nthreads; to_big(h2p, &nthreads, sizeof(nthreads)); h2p += 2; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // Number of threads for decompression *h2p = 0xd1; // int16 h2p += 1; nthreads = (int16_t)schunk->dctx->nthreads; to_big(h2p, &nthreads, sizeof(nthreads)); h2p += 2; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // The boolean for variable-length metalayers *h2p = (schunk->nvlmetalayers > 0) ? (uint8_t)0xc3 : (uint8_t)0xc2; h2p += 1; if (h2p - h2 >= FRAME_HEADER_MINLEN) { return NULL; } // The space for FRAME_FILTER_PIPELINE *h2p = 0xd8; // fixext 16 h2p += 1; if (BLOSC2_MAX_FILTERS > FRAME_FILTER_PIPELINE_MAX) { return NULL; } // Store the filter pipeline in header uint8_t* mp_filters = h2 + FRAME_FILTER_PIPELINE + 1; uint8_t* mp_meta = h2 + FRAME_FILTER_PIPELINE + 1 + FRAME_FILTER_PIPELINE_MAX; for (int i = 0; i < BLOSC2_MAX_FILTERS; i++) { mp_filters[i] = schunk->filters[i]; mp_meta[i] = schunk->filters_meta[i]; } *h2p = (uint8_t) BLOSC2_MAX_FILTERS; h2p += 1; h2p += 16; // User-defined codec and codec metadata uint8_t* udcodec = h2 + FRAME_UDCODEC; *udcodec = schunk->compcode; uint8_t* codec_meta = h2 + FRAME_CODEC_META; *codec_meta = schunk->compcode_meta; if (h2p - h2 != FRAME_HEADER_MINLEN) { return NULL; } int32_t hsize = FRAME_HEADER_MINLEN; // Now, deal with metalayers uint16_t nmetalayers = schunk->nmetalayers; if (nmetalayers > BLOSC2_MAX_METALAYERS) { return NULL; } // Make space for the header of metalayers (array marker, size, map of offsets) h2 = realloc(h2, (size_t)hsize + 1 + 1 + 2 + 1 + 2); h2p = h2 + hsize; // The msgpack header for the metalayers (array_marker, size, map of offsets, list of metalayers) *h2p = 0x90 + 3; // array with 3 elements h2p += 1; // Size for the map (index) of offsets, including this uint16 size (to be filled out later on) *h2p = 0xcd; // uint16 h2p += 1 + 2; // Map (index) of offsets for optional metalayers *h2p = 0xde; // map 16 with N keys h2p += 1; to_big(h2p, &nmetalayers, sizeof(nmetalayers)); h2p += sizeof(nmetalayers); int32_t current_header_len = (int32_t)(h2p - h2); int32_t *offtooff = malloc(nmetalayers * sizeof(int32_t)); for (int nmetalayer = 0; nmetalayer < nmetalayers; nmetalayer++) { if (frame == NULL) { return NULL; } blosc2_metalayer *metalayer = schunk->metalayers[nmetalayer]; uint8_t namelen = (uint8_t) strlen(metalayer->name); h2 = realloc(h2, (size_t)current_header_len + 1 + namelen + 1 + 4); h2p = h2 + current_header_len; // Store the metalayer if (namelen >= (1U << 5U)) { // metalayer strings cannot be longer than 32 bytes free(offtooff); return NULL; } *h2p = (uint8_t)0xa0 + namelen; // str h2p += 1; memcpy(h2p, metalayer->name, namelen); h2p += namelen; // Space for storing the offset for the value of this metalayer *h2p = 0xd2; // int32 h2p += 1; offtooff[nmetalayer] = (int32_t)(h2p - h2); h2p += 4; current_header_len += 1 + namelen + 1 + 4; } int32_t hsize2 = (int32_t)(h2p - h2); if (hsize2 != current_header_len) { // sanity check return NULL; } // Map size + int16 size if ((uint32_t) (hsize2 - hsize) >= (1U << 16U)) { return NULL; } uint16_t map_size = (uint16_t) (hsize2 - hsize); to_big(h2 + FRAME_IDX_SIZE, &map_size, sizeof(map_size)); // Make space for an (empty) array hsize = (int32_t)(h2p - h2); h2 = realloc(h2, (size_t)hsize + 2 + 1 + 2); h2p = h2 + hsize; // Now, store the values in an array *h2p = 0xdc; // array 16 with N elements h2p += 1; to_big(h2p, &nmetalayers, sizeof(nmetalayers)); h2p += sizeof(nmetalayers); current_header_len = (int32_t)(h2p - h2); for (int nmetalayer = 0; nmetalayer < nmetalayers; nmetalayer++) { if (frame == NULL) { return NULL; } blosc2_metalayer *metalayer = schunk->metalayers[nmetalayer]; h2 = realloc(h2, (size_t)current_header_len + 1 + 4 + metalayer->content_len); h2p = h2 + current_header_len; // Store the serialized contents for this metalayer *h2p = 0xc6; // bin32 h2p += 1; to_big(h2p, &(metalayer->content_len), sizeof(metalayer->content_len)); h2p += 4; memcpy(h2p, metalayer->content, metalayer->content_len); // buffer, no need to swap h2p += metalayer->content_len; // Update the offset now that we know it to_big(h2 + offtooff[nmetalayer], ¤t_header_len, sizeof(current_header_len)); current_header_len += 1 + 4 + metalayer->content_len; } free(offtooff); hsize = (int32_t)(h2p - h2); if (hsize != current_header_len) { // sanity check return NULL; } // Set the length of the whole header now that we know it to_big(h2 + FRAME_HEADER_LEN, &hsize, sizeof(hsize)); return h2; } int get_header_info(blosc2_frame_s *frame, int32_t *header_len, int64_t *frame_len, int64_t *nbytes, int64_t *cbytes, int32_t *blocksize, int32_t *chunksize, int64_t *nchunks, int32_t *typesize, uint8_t *compcode, uint8_t *compcode_meta, uint8_t *clevel, uint8_t *filters, uint8_t *filters_meta, uint8_t *splitmode, const blosc2_io *io) { uint8_t* framep = frame->cframe; uint8_t header[FRAME_HEADER_MINLEN]; blosc2_io_cb *io_cb = blosc2_get_io_cb(io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } if (frame->len <= 0) { return BLOSC2_ERROR_READ_BUFFER; } if (frame->cframe == NULL) { int64_t rbytes = 0; void* fp = NULL; if (frame->sframe) { fp = sframe_open_index(frame->urlpath, "rb", io); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } } else { fp = io_cb->open(frame->urlpath, "rb", io->params); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, frame->file_offset, SEEK_SET); } rbytes = io_cb->read(header, 1, FRAME_HEADER_MINLEN, fp); io_cb->close(fp); if (rbytes != FRAME_HEADER_MINLEN) { return BLOSC2_ERROR_FILE_READ; } framep = header; } // Consistency check for frame type uint8_t frame_type = framep[FRAME_TYPE]; if (frame->sframe) { if (frame_type != FRAME_DIRECTORY_TYPE) { return BLOSC2_ERROR_FRAME_TYPE; } } else { if (frame_type != FRAME_CONTIGUOUS_TYPE) { return BLOSC2_ERROR_FRAME_TYPE; } } // Fetch some internal lengths from_big(header_len, framep + FRAME_HEADER_LEN, sizeof(*header_len)); if (*header_len < FRAME_HEADER_MINLEN) { BLOSC_TRACE_ERROR("Header length is zero or smaller than min allowed."); return BLOSC2_ERROR_INVALID_HEADER; } from_big(frame_len, framep + FRAME_LEN, sizeof(*frame_len)); if (*header_len > *frame_len) { BLOSC_TRACE_ERROR("Header length exceeds length of the frame."); return BLOSC2_ERROR_INVALID_HEADER; } from_big(nbytes, framep + FRAME_NBYTES, sizeof(*nbytes)); from_big(cbytes, framep + FRAME_CBYTES, sizeof(*cbytes)); from_big(blocksize, framep + FRAME_BLOCKSIZE, sizeof(*blocksize)); if (chunksize != NULL) { from_big(chunksize, framep + FRAME_CHUNKSIZE, sizeof(*chunksize)); } if (typesize != NULL) { from_big(typesize, framep + FRAME_TYPESIZE, sizeof(*typesize)); if (*typesize <= 0 || *typesize > BLOSC_MAX_TYPESIZE) { BLOSC_TRACE_ERROR("`typesize` is zero or greater than max allowed."); return BLOSC2_ERROR_INVALID_HEADER; } } // Codecs uint8_t frame_codecs = framep[FRAME_CODECS]; if (clevel != NULL) { *clevel = frame_codecs >> 4u; } if (compcode != NULL) { *compcode = frame_codecs & 0xFu; if (*compcode == BLOSC_UDCODEC_FORMAT) { from_big(compcode, framep + FRAME_UDCODEC, sizeof(*compcode)); } } // Other flags uint8_t other_flags = framep[FRAME_OTHER_FLAGS]; if (splitmode != NULL) { *splitmode = other_flags & 0x4u; from_big(splitmode, framep + FRAME_OTHER_FLAGS, sizeof(*splitmode)); *splitmode += 1; } if (compcode_meta != NULL) { from_big(compcode_meta, framep + FRAME_CODEC_META, sizeof(*compcode_meta)); } // Filters if (filters != NULL && filters_meta != NULL) { uint8_t nfilters = framep[FRAME_FILTER_PIPELINE]; if (nfilters > BLOSC2_MAX_FILTERS) { BLOSC_TRACE_ERROR("The number of filters in frame header are too large for Blosc2."); return BLOSC2_ERROR_INVALID_HEADER; } uint8_t *filters_ = framep + FRAME_FILTER_PIPELINE + 1; uint8_t *filters_meta_ = framep + FRAME_FILTER_PIPELINE + 1 + FRAME_FILTER_PIPELINE_MAX; for (int i = 0; i < nfilters; i++) { filters[i] = filters_[i]; filters_meta[i] = filters_meta_[i]; } } if (*nbytes > 0 && *chunksize > 0) { // We can compute the number of chunks only when the frame has actual data *nchunks = *nbytes / *chunksize; if (*nbytes % *chunksize > 0) { if (*nchunks == INT32_MAX) { BLOSC_TRACE_ERROR("Number of chunks exceeds maximum allowed."); return BLOSC2_ERROR_INVALID_HEADER; } *nchunks += 1; } // Sanity check for compressed sizes if ((*cbytes < 0) || ((int64_t)*nchunks * *chunksize < *nbytes)) { BLOSC_TRACE_ERROR("Invalid compressed size in frame header."); return BLOSC2_ERROR_INVALID_HEADER; } } else { *nchunks = 0; } return 0; } int64_t get_trailer_offset(blosc2_frame_s *frame, int32_t header_len, bool has_coffsets) { if (!has_coffsets) { // No data chunks yet return header_len; } return frame->len - frame->trailer_len; } // Update the length in the header int update_frame_len(blosc2_frame_s* frame, int64_t len) { int rc = 1; blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } if (frame->cframe != NULL) { to_big(frame->cframe + FRAME_LEN, &len, sizeof(int64_t)); } else { void* fp = NULL; if (frame->sframe) { fp = sframe_open_index(frame->urlpath, "rb+", frame->schunk->storage->io); } else { fp = io_cb->open(frame->urlpath, "rb+", frame->schunk->storage->io->params); } if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, frame->file_offset + FRAME_LEN, SEEK_SET); int64_t swap_len; to_big(&swap_len, &len, sizeof(int64_t)); int64_t wbytes = io_cb->write(&swap_len, 1, sizeof(int64_t), fp); io_cb->close(fp); if (wbytes != sizeof(int64_t)) { BLOSC_TRACE_ERROR("Cannot write the frame length in header."); return BLOSC2_ERROR_FILE_WRITE; } } return rc; } int frame_update_trailer(blosc2_frame_s* frame, blosc2_schunk* schunk) { if (frame != NULL && frame->len == 0) { BLOSC_TRACE_ERROR("The trailer cannot be updated on empty frames."); } // Create the trailer in msgpack (see the frame format document) int64_t trailer_len = FRAME_TRAILER_MINLEN; uint8_t* trailer = (uint8_t*)calloc((size_t)trailer_len, 1); uint8_t* ptrailer = trailer; *ptrailer = 0x90 + 4; // fixarray with 4 elements ptrailer += 1; // Trailer format version *ptrailer = FRAME_TRAILER_VERSION; ptrailer += 1; int32_t current_trailer_len = (int32_t)(ptrailer - trailer); // Now, deal with variable-length metalayers int16_t nvlmetalayers = schunk->nvlmetalayers; if (nvlmetalayers < 0 || nvlmetalayers > BLOSC2_MAX_METALAYERS) { return -1; } // Make space for the header of metalayers (array marker, size, map of offsets) trailer = realloc(trailer, (size_t) current_trailer_len + 1 + 1 + 2 + 1 + 2); ptrailer = trailer + current_trailer_len; // The msgpack header for the metalayers (array_marker, size, map of offsets, list of metalayers) *ptrailer = 0x90 + 3; // array with 3 elements ptrailer += 1; int32_t tsize = (int32_t)(ptrailer - trailer); // Size for the map (index) of metalayer offsets, including this uint16 size (to be filled out later on) *ptrailer = 0xcd; // uint16 ptrailer += 1 + 2; // Map (index) of offsets for optional metalayers *ptrailer = 0xde; // map 16 with N keys ptrailer += 1; to_big(ptrailer, &nvlmetalayers, sizeof(nvlmetalayers)); ptrailer += sizeof(nvlmetalayers); current_trailer_len = (int32_t)(ptrailer - trailer); int32_t *offtodata = malloc(nvlmetalayers * sizeof(int32_t)); for (int nvlmetalayer = 0; nvlmetalayer < nvlmetalayers; nvlmetalayer++) { if (frame == NULL) { return -1; } blosc2_metalayer *vlmetalayer = schunk->vlmetalayers[nvlmetalayer]; uint8_t name_len = (uint8_t) strlen(vlmetalayer->name); trailer = realloc(trailer, (size_t)current_trailer_len + 1 + name_len + 1 + 4); ptrailer = trailer + current_trailer_len; // Store the vlmetalayer if (name_len >= (1U << 5U)) { // metalayer strings cannot be longer than 32 bytes free(offtodata); return -1; } *ptrailer = (uint8_t)0xa0 + name_len; // str ptrailer += 1; memcpy(ptrailer, vlmetalayer->name, name_len); ptrailer += name_len; // Space for storing the offset for the value of this vlmetalayer *ptrailer = 0xd2; // int32 ptrailer += 1; offtodata[nvlmetalayer] = (int32_t)(ptrailer - trailer); ptrailer += 4; current_trailer_len += 1 + name_len + 1 + 4; } int32_t tsize2 = (int32_t)(ptrailer - trailer); if (tsize2 != current_trailer_len) { // sanity check return -1; } // Map size + int16 size if ((uint32_t) (tsize2 - tsize) >= (1U << 16U)) { return -1; } uint16_t map_size = (uint16_t) (tsize2 - tsize); to_big(trailer + 4, &map_size, sizeof(map_size)); // Make space for an (empty) array tsize = (int32_t)(ptrailer - trailer); trailer = realloc(trailer, (size_t) tsize + 2 + 1 + 2); ptrailer = trailer + tsize; // Now, store the values in an array *ptrailer = 0xdc; // array 16 with N elements ptrailer += 1; to_big(ptrailer, &nvlmetalayers, sizeof(nvlmetalayers)); ptrailer += sizeof(nvlmetalayers); current_trailer_len = (int32_t)(ptrailer - trailer); for (int nvlmetalayer = 0; nvlmetalayer < nvlmetalayers; nvlmetalayer++) { if (frame == NULL) { return -1; } blosc2_metalayer *vlmetalayer = schunk->vlmetalayers[nvlmetalayer]; trailer = realloc(trailer, (size_t)current_trailer_len + 1 + 4 + vlmetalayer->content_len); ptrailer = trailer + current_trailer_len; // Store the serialized contents for this vlmetalayer *ptrailer = 0xc6; // bin32 ptrailer += 1; to_big(ptrailer, &(vlmetalayer->content_len), sizeof(vlmetalayer->content_len)); ptrailer += 4; memcpy(ptrailer, vlmetalayer->content, vlmetalayer->content_len); // buffer, no need to swap ptrailer += vlmetalayer->content_len; // Update the offset now that we know it to_big(trailer + offtodata[nvlmetalayer], ¤t_trailer_len, sizeof(current_trailer_len)); current_trailer_len += 1 + 4 + vlmetalayer->content_len; } free(offtodata); tsize = (int32_t)(ptrailer - trailer); if (tsize != current_trailer_len) { // sanity check return -1; } trailer = realloc(trailer, (size_t)current_trailer_len + 23); ptrailer = trailer + current_trailer_len; trailer_len = (ptrailer - trailer) + 23; // Trailer length *ptrailer = 0xce; // uint32 ptrailer += 1; to_big(ptrailer, &trailer_len, sizeof(uint32_t)); ptrailer += sizeof(uint32_t); // Up to 16 bytes for frame fingerprint (using XXH3 included in https://github.com/Cyan4973/xxHash) // Maybe someone would need 256-bit in the future, but for the time being 128-bit seems like a good tradeoff *ptrailer = 0xd8; // fixext 16 ptrailer += 1; *ptrailer = 0; // fingerprint type: 0 -> no fp; 1 -> 32-bit; 2 -> 64-bit; 3 -> 128-bit ptrailer += 1; // Remove call to memset when we compute an actual fingerprint memset(ptrailer, 0, 16); // Uncomment call to memcpy when we compute an actual fingerprint // memcpy(ptrailer, xxh3_fingerprint, sizeof(xxh3_fingerprint)); ptrailer += 16; // Sanity check if (ptrailer - trailer != trailer_len) { return BLOSC2_ERROR_DATA; } int32_t header_len; int64_t frame_len; int64_t nbytes; int64_t cbytes; int32_t blocksize; int32_t chunksize; int64_t nchunks; int ret = get_header_info(frame, &header_len, &frame_len, &nbytes, &cbytes, &blocksize, &chunksize, &nchunks, NULL, NULL, NULL, NULL, NULL, NULL, NULL, frame->schunk->storage->io); if (ret < 0) { BLOSC_TRACE_ERROR("Unable to get meta info from frame."); return ret; } int64_t trailer_offset = get_trailer_offset(frame, header_len, nbytes > 0); if (trailer_offset < BLOSC_EXTENDED_HEADER_LENGTH) { BLOSC_TRACE_ERROR("Unable to get trailer offset in frame."); return BLOSC2_ERROR_READ_BUFFER; } blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } // Update the trailer. As there are no internal offsets to the trailer section, // and it is always at the end of the frame, we can just write (or overwrite) it // at the end of the frame. if (frame->cframe != NULL) { frame->cframe = realloc(frame->cframe, (size_t)(trailer_offset + trailer_len)); if (frame->cframe == NULL) { BLOSC_TRACE_ERROR("Cannot realloc space for the frame."); return BLOSC2_ERROR_MEMORY_ALLOC; } memcpy(frame->cframe + trailer_offset, trailer, trailer_len); } else { void* fp = NULL; if (frame->sframe) { fp = sframe_open_index(frame->urlpath, "rb+", frame->schunk->storage->io); } else { fp = io_cb->open(frame->urlpath, "rb+", frame->schunk->storage->io->params); } if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, frame->file_offset + trailer_offset, SEEK_SET); int64_t wbytes = io_cb->write(trailer, 1, trailer_len, fp); if (wbytes != trailer_len) { BLOSC_TRACE_ERROR("Cannot write the trailer length in trailer."); return BLOSC2_ERROR_FILE_WRITE; } if (io_cb->truncate(fp, trailer_offset + trailer_len) != 0) { BLOSC_TRACE_ERROR("Cannot truncate the frame."); return BLOSC2_ERROR_FILE_TRUNCATE; } io_cb->close(fp); } free(trailer); int rc = update_frame_len(frame, trailer_offset + trailer_len); if (rc < 0) { return rc; } frame->len = trailer_offset + trailer_len; frame->trailer_len = trailer_len; return 1; } // Remove a file:/// prefix // This is a temporary workaround for allowing to use proper URLs for local files/dirs static char* normalize_urlpath(const char* urlpath) { char* localpath = strstr(urlpath, "file:///"); if (localpath == urlpath) { // There is a file:/// prefix. Get rid of it. localpath += strlen("file:///"); } else { localpath = (char*)urlpath; } return localpath; } /* Initialize a frame out of a file */ blosc2_frame_s* frame_from_file_offset(const char* urlpath, const blosc2_io *io, int64_t offset) { // Get the length of the frame uint8_t header[FRAME_HEADER_MINLEN]; uint8_t trailer[FRAME_TRAILER_MINLEN]; void* fp = NULL; bool sframe = false; struct stat path_stat; urlpath = normalize_urlpath(urlpath); if(stat(urlpath, &path_stat) < 0) { BLOSC_TRACE_ERROR("Cannot get information about the path %s.", urlpath); return NULL; } blosc2_io_cb *io_cb = blosc2_get_io_cb(io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return NULL; } char* urlpath_cpy; if (path_stat.st_mode & S_IFDIR) { urlpath_cpy = malloc(strlen(urlpath) + 1); strcpy(urlpath_cpy, urlpath); char last_char = urlpath[strlen(urlpath) - 1]; if (last_char == '\\' || last_char == '/') { urlpath_cpy[strlen(urlpath) - 1] = '\0'; } else { } fp = sframe_open_index(urlpath_cpy, "rb", io); sframe = true; } else { urlpath_cpy = malloc(strlen(urlpath) + 1); strcpy(urlpath_cpy, urlpath); fp = io_cb->open(urlpath, "rb", io->params); } if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", urlpath); return NULL; } io_cb->seek(fp, offset, SEEK_SET); int64_t rbytes = io_cb->read(header, 1, FRAME_HEADER_MINLEN, fp); if (rbytes != FRAME_HEADER_MINLEN) { BLOSC_TRACE_ERROR("Cannot read from file '%s'.", urlpath); io_cb->close(fp); free(urlpath_cpy); return NULL; } int64_t frame_len; to_big(&frame_len, header + FRAME_LEN, sizeof(frame_len)); blosc2_frame_s* frame = calloc(1, sizeof(blosc2_frame_s)); frame->urlpath = urlpath_cpy; frame->len = frame_len; frame->sframe = sframe; frame->file_offset = offset; // Now, the trailer length io_cb->seek(fp, offset + frame_len - FRAME_TRAILER_MINLEN, SEEK_SET); rbytes = io_cb->read(trailer, 1, FRAME_TRAILER_MINLEN, fp); io_cb->close(fp); if (rbytes != FRAME_TRAILER_MINLEN) { BLOSC_TRACE_ERROR("Cannot read from file '%s'.", urlpath); free(urlpath_cpy); free(frame); return NULL; } int trailer_offset = FRAME_TRAILER_MINLEN - FRAME_TRAILER_LEN_OFFSET; if (trailer[trailer_offset - 1] != 0xce) { free(urlpath_cpy); free(frame); return NULL; } uint32_t trailer_len; to_big(&trailer_len, trailer + trailer_offset, sizeof(trailer_len)); frame->trailer_len = trailer_len; return frame; } /* Initialize a frame out of a contiguous frame buffer */ blosc2_frame_s* frame_from_cframe(uint8_t *cframe, int64_t len, bool copy) { // Get the length of the frame const uint8_t* header = cframe; int64_t frame_len; if (len < FRAME_HEADER_MINLEN) { return NULL; } from_big(&frame_len, header + FRAME_LEN, sizeof(frame_len)); if (frame_len != len) { // sanity check return NULL; } blosc2_frame_s* frame = calloc(1, sizeof(blosc2_frame_s)); frame->len = frame_len; frame->file_offset = 0; // Now, the trailer length const uint8_t* trailer = cframe + frame_len - FRAME_TRAILER_MINLEN; int trailer_offset = FRAME_TRAILER_MINLEN - FRAME_TRAILER_LEN_OFFSET; if (trailer[trailer_offset - 1] != 0xce) { free(frame); return NULL; } uint32_t trailer_len; from_big(&trailer_len, trailer + trailer_offset, sizeof(trailer_len)); frame->trailer_len = trailer_len; if (copy) { frame->cframe = malloc((size_t)len); memcpy(frame->cframe, cframe, (size_t)len); } else { frame->cframe = cframe; frame->avoid_cframe_free = true; } return frame; } /* Create a frame out of a super-chunk. */ int64_t frame_from_schunk(blosc2_schunk *schunk, blosc2_frame_s *frame) { frame->file_offset = 0; int64_t nchunks = schunk->nchunks; int64_t cbytes = schunk->cbytes; int32_t chunk_cbytes; int32_t chunk_nbytes; void* fp = NULL; int rc; uint8_t* h2 = new_header_frame(schunk, frame); if (h2 == NULL) { return BLOSC2_ERROR_DATA; } uint32_t h2len; from_big(&h2len, h2 + FRAME_HEADER_LEN, sizeof(h2len)); // Build the offsets chunk int32_t chunksize = -1; int32_t off_cbytes = 0; uint64_t coffset = 0; int32_t off_nbytes = (int32_t) (nchunks * sizeof(int64_t)); uint64_t* data_tmp = malloc(off_nbytes); bool needs_free = false; for (int i = 0; i < nchunks; i++) { uint8_t* data_chunk; data_chunk = schunk->data[i]; rc = blosc2_cbuffer_sizes(data_chunk, &chunk_nbytes, &chunk_cbytes, NULL); if (rc < 0) { return rc; } data_tmp[i] = coffset; coffset += chunk_cbytes; int32_t chunksize_ = chunk_nbytes; if (i == 0) { chunksize = chunksize_; } else if (chunksize != chunksize_) { // Variable size // TODO: update flags for this (or do not use them at all) chunksize = 0; } if (needs_free) { free(data_chunk); } } if ((int64_t)coffset != cbytes) { free(data_tmp); return BLOSC2_ERROR_DATA; } uint8_t *off_chunk = NULL; if (nchunks > 0) { // Compress the chunk of offsets off_chunk = malloc(off_nbytes + BLOSC2_MAX_OVERHEAD); blosc2_context *cctx = blosc2_create_cctx(BLOSC2_CPARAMS_DEFAULTS); cctx->typesize = sizeof(int64_t); off_cbytes = blosc2_compress_ctx(cctx, data_tmp, off_nbytes, off_chunk, off_nbytes + BLOSC2_MAX_OVERHEAD); blosc2_free_ctx(cctx); if (off_cbytes < 0) { free(off_chunk); free(h2); return off_cbytes; } } else { off_cbytes = 0; } free(data_tmp); // Now that we know them, fill the chunksize and frame length in header to_big(h2 + FRAME_CHUNKSIZE, &chunksize, sizeof(chunksize)); frame->len = h2len + cbytes + off_cbytes + FRAME_TRAILER_MINLEN; if (frame->sframe) { frame->len = h2len + off_cbytes + FRAME_TRAILER_MINLEN; } int64_t tbytes = frame->len; to_big(h2 + FRAME_LEN, &tbytes, sizeof(tbytes)); blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } // Create the frame and put the header at the beginning if (frame->urlpath == NULL) { frame->cframe = malloc((size_t)frame->len); memcpy(frame->cframe, h2, h2len); } else { if (frame->sframe) { fp = sframe_open_index(frame->urlpath, "wb", frame->schunk->storage->io); } else { fp = io_cb->open(frame->urlpath, "wb", frame->schunk->storage->io->params); } if (fp == NULL) { BLOSC_TRACE_ERROR("Error creating file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->write(h2, h2len, 1, fp); } free(h2); // Fill the frame with the actual data chunks if (!frame->sframe) { coffset = 0; for (int i = 0; i < nchunks; i++) { uint8_t* data_chunk = schunk->data[i]; rc = blosc2_cbuffer_sizes(data_chunk, NULL, &chunk_cbytes, NULL); if (rc < 0) { return rc; } if (frame->urlpath == NULL) { memcpy(frame->cframe + h2len + coffset, data_chunk, (size_t)chunk_cbytes); } else { io_cb->write(data_chunk, chunk_cbytes, 1, fp); } coffset += chunk_cbytes; } if ((int64_t)coffset != cbytes) { return BLOSC2_ERROR_FAILURE; } } // Copy the offsets chunk at the end of the frame if (frame->urlpath == NULL) { memcpy(frame->cframe + h2len + cbytes, off_chunk, off_cbytes); } else { io_cb->write(off_chunk, off_cbytes, 1, fp); io_cb->close(fp); } free(off_chunk); rc = frame_update_trailer(frame, schunk); if (rc < 0) { return rc; } return frame->len; } // Get the compressed data offsets uint8_t* get_coffsets(blosc2_frame_s *frame, int32_t header_len, int64_t cbytes, int64_t nchunks, int32_t *off_cbytes) { int32_t chunk_cbytes; int rc; if (frame->coffsets != NULL) { if (off_cbytes != NULL) { rc = blosc2_cbuffer_sizes(frame->coffsets, NULL, &chunk_cbytes, NULL); if (rc < 0) { return NULL; } *off_cbytes = (int32_t)chunk_cbytes; } return frame->coffsets; } if (frame->cframe != NULL) { int64_t off_pos = header_len; if (cbytes < INT64_MAX - header_len) { off_pos += cbytes; } // Check that there is enough room to read Blosc header if (off_pos < 0 || off_pos > INT64_MAX - BLOSC_EXTENDED_HEADER_LENGTH || off_pos + BLOSC_EXTENDED_HEADER_LENGTH > frame->len) { BLOSC_TRACE_ERROR("Cannot read the offsets outside of frame boundary."); return NULL; } // For in-memory frames, the coffset is just one pointer away uint8_t* off_start = frame->cframe + off_pos; if (off_cbytes != NULL) { int32_t chunk_nbytes; int32_t chunk_blocksize; rc = blosc2_cbuffer_sizes(off_start, &chunk_nbytes, &chunk_cbytes, &chunk_blocksize); if (rc < 0) { return NULL; } *off_cbytes = (int32_t)chunk_cbytes; if (*off_cbytes < 0 || off_pos + *off_cbytes > frame->len) { BLOSC_TRACE_ERROR("Cannot read the cbytes outside of frame boundary."); return NULL; } if ((uint64_t)chunk_nbytes != nchunks * sizeof(int64_t)) { BLOSC_TRACE_ERROR("The number of chunks in offset idx " "does not match the ones in the header frame."); return NULL; } } return off_start; } int64_t trailer_offset = get_trailer_offset(frame, header_len, true); if (trailer_offset < BLOSC_EXTENDED_HEADER_LENGTH || trailer_offset + FRAME_TRAILER_MINLEN > frame->len) { BLOSC_TRACE_ERROR("Cannot read the trailer out of the frame."); return NULL; } int32_t coffsets_cbytes; if (frame->sframe) { coffsets_cbytes = (int32_t)(trailer_offset - (header_len + 0)); } else { coffsets_cbytes = (int32_t)(trailer_offset - (header_len + cbytes)); } if (off_cbytes != NULL) { *off_cbytes = coffsets_cbytes; } blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return NULL; } void* fp = NULL; uint8_t* coffsets = malloc((size_t)coffsets_cbytes); if (frame->sframe) { fp = sframe_open_index(frame->urlpath, "rb", frame->schunk->storage->io); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return NULL; } io_cb->seek(fp, header_len + 0, SEEK_SET); } else { fp = io_cb->open(frame->urlpath, "rb", frame->schunk->storage->io->params); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return NULL; } io_cb->seek(fp, frame->file_offset + header_len + cbytes, SEEK_SET); } int64_t rbytes = io_cb->read(coffsets, 1, coffsets_cbytes, fp); io_cb->close(fp); if (rbytes != coffsets_cbytes) { BLOSC_TRACE_ERROR("Cannot read the offsets out of the frame."); free(coffsets); return NULL; } frame->coffsets = coffsets; return coffsets; } // Get the data offsets from a frame int64_t* blosc2_frame_get_offsets(blosc2_schunk *schunk) { if (schunk->frame == NULL) { BLOSC_TRACE_ERROR("This function needs a frame."); return NULL; } blosc2_frame_s* frame = (blosc2_frame_s*)schunk->frame; // Get header info int32_t header_len; int64_t frame_len; int64_t nbytes; int64_t cbytes; int32_t blocksize; int32_t chunksize; int64_t nchunks; int ret = get_header_info(frame, &header_len, &frame_len, &nbytes, &cbytes, &blocksize, &chunksize, &nchunks, NULL, NULL, NULL, NULL, NULL, NULL, NULL, frame->schunk->storage->io); if (ret < 0) { BLOSC_TRACE_ERROR("Cannot get the header info for the frame."); return NULL; } int32_t off_nbytes = (int32_t) (nchunks * sizeof(int64_t)); int64_t* offsets = (int64_t *) malloc((size_t)off_nbytes); int32_t coffsets_cbytes = 0; uint8_t *coffsets = get_coffsets(frame, header_len, cbytes, nchunks, &coffsets_cbytes); // Decompress offsets blosc2_dparams off_dparams = BLOSC2_DPARAMS_DEFAULTS; blosc2_context *dctx = blosc2_create_dctx(off_dparams); int32_t prev_nbytes = blosc2_decompress_ctx(dctx, coffsets, coffsets_cbytes, offsets, off_nbytes); blosc2_free_ctx(dctx); if (prev_nbytes < 0) { free(offsets); BLOSC_TRACE_ERROR("Cannot decompress the offsets chunk."); return NULL; } return offsets; } int frame_update_header(blosc2_frame_s* frame, blosc2_schunk* schunk, bool new) { uint8_t* framep = frame->cframe; uint8_t header[FRAME_HEADER_MINLEN]; if (frame->len <= 0) { return BLOSC2_ERROR_INVALID_PARAM; } if (new && schunk->cbytes > 0) { BLOSC_TRACE_ERROR("New metalayers cannot be added after actual data " "has been appended."); return BLOSC2_ERROR_INVALID_PARAM; } blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } if (frame->cframe == NULL) { int64_t rbytes = 0; void* fp = NULL; if (frame->sframe) { fp = sframe_open_index(frame->urlpath, "rb+", frame->schunk->storage->io); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } } else { fp = io_cb->open(frame->urlpath, "rb", frame->schunk->storage->io->params); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, frame->file_offset, SEEK_SET); } if (fp != NULL) { rbytes = io_cb->read(header, 1, FRAME_HEADER_MINLEN, fp); io_cb->close(fp); } (void) rbytes; if (rbytes != FRAME_HEADER_MINLEN) { return BLOSC2_ERROR_FILE_WRITE; } framep = header; } uint32_t prev_h2len; from_big(&prev_h2len, framep + FRAME_HEADER_LEN, sizeof(prev_h2len)); // Build a new header uint8_t* h2 = new_header_frame(schunk, frame); uint32_t h2len; from_big(&h2len, h2 + FRAME_HEADER_LEN, sizeof(h2len)); // The frame length is outdated when adding a new metalayer, so update it if (new) { int64_t frame_len = h2len; // at adding time, we only have to worry of the header for now to_big(h2 + FRAME_LEN, &frame_len, sizeof(frame_len)); frame->len = frame_len; } if (!new && prev_h2len != h2len) { BLOSC_TRACE_ERROR("The new metalayer sizes should be equal the existing ones."); return BLOSC2_ERROR_DATA; } void* fp = NULL; if (frame->cframe == NULL) { // Write updated header down to file if (frame->sframe) { fp = sframe_open_index(frame->urlpath, "rb+", frame->schunk->storage->io); } else { fp = io_cb->open(frame->urlpath, "rb+", frame->schunk->storage->io->params); } if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, frame->file_offset, SEEK_SET); io_cb->write(h2, h2len, 1, fp); io_cb->close(fp); } else { if (new) { frame->cframe = realloc(frame->cframe, h2len); } memcpy(frame->cframe, h2, h2len); } free(h2); return 1; } static int get_meta_from_header(blosc2_frame_s* frame, blosc2_schunk* schunk, uint8_t* header, int32_t header_len) { BLOSC_UNUSED_PARAM(frame); int64_t header_pos = FRAME_IDX_SIZE; // Get the size for the index of metalayers uint16_t idx_size; header_pos += sizeof(idx_size); if (header_len < header_pos) { return BLOSC2_ERROR_READ_BUFFER; } from_big(&idx_size, header + FRAME_IDX_SIZE, sizeof(idx_size)); // Get the actual index of metalayers uint8_t* metalayers_idx = header + FRAME_IDX_SIZE + 2; header_pos += 1; if (header_len < header_pos) { return BLOSC2_ERROR_READ_BUFFER; } if (metalayers_idx[0] != 0xde) { // sanity check return BLOSC2_ERROR_DATA; } uint8_t* idxp = metalayers_idx + 1; uint16_t nmetalayers; header_pos += sizeof(nmetalayers); if (header_len < header_pos) { return BLOSC2_ERROR_READ_BUFFER; } from_big(&nmetalayers, idxp, sizeof(uint16_t)); idxp += 2; if (nmetalayers > BLOSC2_MAX_METALAYERS) { return BLOSC2_ERROR_DATA; } schunk->nmetalayers = nmetalayers; // Populate the metalayers and its serialized values for (int nmetalayer = 0; nmetalayer < nmetalayers; nmetalayer++) { header_pos += 1; if (header_len < header_pos) { return BLOSC2_ERROR_READ_BUFFER; } if ((*idxp & 0xe0u) != 0xa0u) { // sanity check return BLOSC2_ERROR_DATA; } blosc2_metalayer* metalayer = calloc(sizeof(blosc2_metalayer), 1); schunk->metalayers[nmetalayer] = metalayer; // Populate the metalayer string int8_t nslen = *idxp & (uint8_t)0x1F; idxp += 1; header_pos += nslen; if (header_len < header_pos) { return BLOSC2_ERROR_READ_BUFFER; } char* ns = malloc((size_t)nslen + 1); memcpy(ns, idxp, nslen); ns[nslen] = '\0'; idxp += nslen; metalayer->name = ns; // Populate the serialized value for this metalayer // Get the offset header_pos += 1; if (header_len < header_pos) { return BLOSC2_ERROR_READ_BUFFER; } if ((*idxp & 0xffu) != 0xd2u) { // sanity check return BLOSC2_ERROR_DATA; } idxp += 1; int32_t offset; header_pos += sizeof(offset); if (header_len < header_pos) { return BLOSC2_ERROR_READ_BUFFER; } from_big(&offset, idxp, sizeof(offset)); idxp += 4; if (offset < 0 || offset >= header_len) { // Offset is less than zero or exceeds header length return BLOSC2_ERROR_DATA; } // Go to offset and see if we have the correct marker uint8_t* content_marker = header + offset; if (header_len < offset + 1 + 4) { return BLOSC2_ERROR_READ_BUFFER; } if (*content_marker != 0xc6) { return BLOSC2_ERROR_DATA; } // Read the size of the content int32_t content_len; from_big(&content_len, content_marker + 1, sizeof(content_len)); if (content_len < 0) { return BLOSC2_ERROR_DATA; } metalayer->content_len = content_len; // Finally, read the content if (header_len < offset + 1 + 4 + content_len) { return BLOSC2_ERROR_READ_BUFFER; } char* content = malloc((size_t)content_len); memcpy(content, content_marker + 1 + 4, (size_t)content_len); metalayer->content = (uint8_t*)content; } return 1; } int frame_get_metalayers(blosc2_frame_s* frame, blosc2_schunk* schunk) { int32_t header_len; int64_t frame_len; int64_t nbytes; int64_t cbytes; int32_t blocksize; int32_t chunksize; int64_t nchunks; int ret = get_header_info(frame, &header_len, &frame_len, &nbytes, &cbytes, &blocksize, &chunksize, &nchunks, NULL, NULL, NULL, NULL, NULL, NULL, NULL, schunk->storage->io); if (ret < 0) { BLOSC_TRACE_ERROR("Unable to get the header info from frame."); return ret; } // Get the header uint8_t* header = NULL; if (frame->cframe != NULL) { header = frame->cframe; } else { int64_t rbytes = 0; header = malloc(header_len); blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } void* fp = NULL; if (frame->sframe) { fp = sframe_open_index(frame->urlpath, "rb", frame->schunk->storage->io); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } } else { fp = io_cb->open(frame->urlpath, "rb", frame->schunk->storage->io->params); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, frame->file_offset, SEEK_SET); } if (fp != NULL) { rbytes = io_cb->read(header, 1, header_len, fp); io_cb->close(fp); } if (rbytes != header_len) { BLOSC_TRACE_ERROR("Cannot access the header out of the frame."); free(header); return BLOSC2_ERROR_FILE_READ; } } ret = get_meta_from_header(frame, schunk, header, header_len); if (frame->cframe == NULL) { free(header); } return ret; } static int get_vlmeta_from_trailer(blosc2_frame_s* frame, blosc2_schunk* schunk, uint8_t* trailer, int32_t trailer_len) { BLOSC_UNUSED_PARAM(frame); int64_t trailer_pos = FRAME_TRAILER_VLMETALAYERS + 2; uint8_t* idxp = trailer + trailer_pos; // Get the size for the index of metalayers trailer_pos += 2; if (trailer_len < trailer_pos) { return BLOSC2_ERROR_READ_BUFFER; } uint16_t idx_size; from_big(&idx_size, idxp, sizeof(idx_size)); idxp += 2; trailer_pos += 1; // Get the actual index of metalayers if (trailer_len < trailer_pos) { return BLOSC2_ERROR_READ_BUFFER; } if (idxp[0] != 0xde) { // sanity check return BLOSC2_ERROR_DATA; } idxp += 1; int16_t nmetalayers; trailer_pos += sizeof(nmetalayers); if (trailer_len < trailer_pos) { return BLOSC2_ERROR_READ_BUFFER; } from_big(&nmetalayers, idxp, sizeof(uint16_t)); idxp += 2; if (nmetalayers > BLOSC2_MAX_VLMETALAYERS) { return BLOSC2_ERROR_DATA; } schunk->nvlmetalayers = nmetalayers; // Populate the metalayers and its serialized values for (int nmetalayer = 0; nmetalayer < nmetalayers; nmetalayer++) { trailer_pos += 1; if (trailer_len < trailer_pos) { return BLOSC2_ERROR_READ_BUFFER; } if ((*idxp & 0xe0u) != 0xa0u) { // sanity check return BLOSC2_ERROR_DATA; } blosc2_metalayer* metalayer = calloc(sizeof(blosc2_metalayer), 1); schunk->vlmetalayers[nmetalayer] = metalayer; // Populate the metalayer string int8_t nslen = *idxp & (uint8_t)0x1F; idxp += 1; trailer_pos += nslen; if (trailer_len < trailer_pos) { return BLOSC2_ERROR_READ_BUFFER; } char* ns = malloc((size_t)nslen + 1); memcpy(ns, idxp, nslen); ns[nslen] = '\0'; idxp += nslen; metalayer->name = ns; // Populate the serialized value for this metalayer // Get the offset trailer_pos += 1; if (trailer_len < trailer_pos) { return BLOSC2_ERROR_READ_BUFFER; } if ((*idxp & 0xffu) != 0xd2u) { // sanity check return BLOSC2_ERROR_DATA; } idxp += 1; int32_t offset; trailer_pos += sizeof(offset); if (trailer_len < trailer_pos) { return BLOSC2_ERROR_READ_BUFFER; } from_big(&offset, idxp, sizeof(offset)); idxp += 4; if (offset < 0 || offset >= trailer_len) { // Offset is less than zero or exceeds trailer length return BLOSC2_ERROR_DATA; } // Go to offset and see if we have the correct marker uint8_t* content_marker = trailer + offset; if (trailer_len < offset + 1 + 4) { return BLOSC2_ERROR_READ_BUFFER; } if (*content_marker != 0xc6) { return BLOSC2_ERROR_DATA; } // Read the size of the content int32_t content_len; from_big(&content_len, content_marker + 1, sizeof(content_len)); if (content_len < 0) { return BLOSC2_ERROR_DATA; } metalayer->content_len = content_len; // Finally, read the content if (trailer_len < offset + 1 + 4 + content_len) { return BLOSC2_ERROR_READ_BUFFER; } char* content = malloc((size_t)content_len); memcpy(content, content_marker + 1 + 4, (size_t)content_len); metalayer->content = (uint8_t*)content; } return 1; } int frame_get_vlmetalayers(blosc2_frame_s* frame, blosc2_schunk* schunk) { int32_t header_len; int64_t frame_len; int64_t nbytes; int64_t cbytes; int32_t blocksize; int32_t chunksize; int64_t nchunks; int ret = get_header_info(frame, &header_len, &frame_len, &nbytes, &cbytes, &blocksize, &chunksize, &nchunks, NULL, NULL, NULL, NULL, NULL, NULL, NULL, schunk->storage->io); if (ret < 0) { BLOSC_TRACE_ERROR("Unable to get the trailer info from frame."); return ret; } int64_t trailer_offset = get_trailer_offset(frame, header_len, nbytes > 0); int32_t trailer_len = (int32_t) frame->trailer_len; if (trailer_offset < BLOSC_EXTENDED_HEADER_LENGTH || trailer_offset + trailer_len > frame->len) { BLOSC_TRACE_ERROR("Cannot access the trailer out of the frame."); return BLOSC2_ERROR_READ_BUFFER; } // Get the trailer uint8_t* trailer = NULL; if (frame->cframe != NULL) { trailer = frame->cframe + trailer_offset; } else { int64_t rbytes = 0; trailer = malloc(trailer_len); blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } void* fp = NULL; if (frame->sframe) { char* eframe_name = malloc(strlen(frame->urlpath) + strlen("/chunks.b2frame") + 1); sprintf(eframe_name, "%s/chunks.b2frame", frame->urlpath); fp = io_cb->open(eframe_name, "rb", frame->schunk->storage->io->params); free(eframe_name); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", eframe_name); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, trailer_offset, SEEK_SET); } else { fp = io_cb->open(frame->urlpath, "rb", frame->schunk->storage->io->params); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, frame->file_offset + trailer_offset, SEEK_SET); } if (fp != NULL) { rbytes = io_cb->read(trailer, 1, trailer_len, fp); io_cb->close(fp); } if (rbytes != trailer_len) { BLOSC_TRACE_ERROR("Cannot access the trailer out of the fileframe."); free(trailer); return BLOSC2_ERROR_FILE_READ; } } ret = get_vlmeta_from_trailer(frame, schunk, trailer, trailer_len); if (frame->cframe == NULL) { free(trailer); } return ret; } blosc2_storage* get_new_storage(const blosc2_storage* storage, const blosc2_cparams* cdefaults, const blosc2_dparams* ddefaults, const blosc2_io* iodefaults) { blosc2_storage* new_storage = (blosc2_storage*)calloc(1, sizeof(blosc2_storage)); memcpy(new_storage, storage, sizeof(blosc2_storage)); if (storage->urlpath != NULL) { char* urlpath = normalize_urlpath(storage->urlpath); new_storage->urlpath = malloc(strlen(urlpath) + 1); strcpy(new_storage->urlpath, urlpath); } // cparams blosc2_cparams* cparams = malloc(sizeof(blosc2_cparams)); if (storage->cparams != NULL) { memcpy(cparams, storage->cparams, sizeof(blosc2_cparams)); } else { memcpy(cparams, cdefaults, sizeof(blosc2_cparams)); } new_storage->cparams = cparams; // dparams blosc2_dparams* dparams = malloc(sizeof(blosc2_dparams)); if (storage->dparams != NULL) { memcpy(dparams, storage->dparams, sizeof(blosc2_dparams)); } else { memcpy(dparams, ddefaults, sizeof(blosc2_dparams)); } new_storage->dparams = dparams; // iodefaults blosc2_io* udio = malloc(sizeof(blosc2_io)); if (storage->io != NULL) { memcpy(udio, storage->io, sizeof(blosc2_io)); } else { memcpy(udio, iodefaults, sizeof(blosc2_io)); } new_storage->io = udio; return new_storage; } /* Get a super-chunk out of a frame */ blosc2_schunk* frame_to_schunk(blosc2_frame_s* frame, bool copy, const blosc2_io *udio) { int32_t header_len; int64_t frame_len; int rc; blosc2_schunk* schunk = calloc(1, sizeof(blosc2_schunk)); schunk->frame = (blosc2_frame*)frame; frame->schunk = schunk; rc = get_header_info(frame, &header_len, &frame_len, &schunk->nbytes, &schunk->cbytes, &schunk->blocksize, &schunk->chunksize, &schunk->nchunks, &schunk->typesize, &schunk->compcode, &schunk->compcode_meta, &schunk->clevel, schunk->filters, schunk->filters_meta, &schunk->splitmode, udio); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to get meta info from frame."); blosc2_schunk_free(schunk); return NULL; } int64_t nchunks = schunk->nchunks; int64_t nbytes = schunk->nbytes; (void) nbytes; int64_t cbytes = schunk->cbytes; // Compression and decompression contexts blosc2_cparams *cparams; blosc2_schunk_get_cparams(schunk, &cparams); schunk->cctx = blosc2_create_cctx(*cparams); blosc2_dparams *dparams; blosc2_schunk_get_dparams(schunk, &dparams); schunk->dctx = blosc2_create_dctx(*dparams); blosc2_storage storage = {.contiguous = copy ? false : true}; schunk->storage = get_new_storage(&storage, cparams, dparams, udio); free(cparams); free(dparams); if (!copy) { goto out; } // We are not attached to a frame anymore schunk->frame = NULL; if (nchunks == 0) { frame->schunk = NULL; goto out; } // Get the compressed offsets int32_t coffsets_cbytes = 0; uint8_t* coffsets = get_coffsets(frame, header_len, cbytes, nchunks, &coffsets_cbytes); if (coffsets == NULL) { blosc2_schunk_free(schunk); BLOSC_TRACE_ERROR("Cannot get the offsets for the frame."); return NULL; } // Decompress offsets blosc2_dparams off_dparams = BLOSC2_DPARAMS_DEFAULTS; blosc2_context *dctx = blosc2_create_dctx(off_dparams); int64_t* offsets = (int64_t *) malloc((size_t)nchunks * sizeof(int64_t)); int32_t off_nbytes = blosc2_decompress_ctx(dctx, coffsets, coffsets_cbytes, offsets, (int32_t)(nchunks * sizeof(int64_t))); blosc2_free_ctx(dctx); if (off_nbytes < 0) { free(offsets); blosc2_schunk_free(schunk); BLOSC_TRACE_ERROR("Cannot decompress the offsets chunk."); return NULL; } // We want the contiguous schunk, so create the actual data chunks (and, while doing this, // get a guess at the blocksize used in this frame) int64_t acc_nbytes = 0; int64_t acc_cbytes = 0; int32_t blocksize = 0; int32_t chunk_nbytes; int32_t chunk_cbytes; int32_t chunk_blocksize; size_t prev_alloc = BLOSC_EXTENDED_HEADER_LENGTH; uint8_t* data_chunk = NULL; bool needs_free = false; const blosc2_io_cb *io_cb = blosc2_get_io_cb(udio->id); if (io_cb == NULL) { blosc2_schunk_free(schunk); BLOSC_TRACE_ERROR("Error getting the input/output API"); return NULL; } void* fp = NULL; if (frame->cframe == NULL) { data_chunk = malloc((size_t)prev_alloc); needs_free = true; if (!frame->sframe) { // If not the chunks won't be in the frame fp = io_cb->open(frame->urlpath, "rb", udio->params); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); rc = BLOSC2_ERROR_FILE_OPEN; goto end; } } } schunk->data = malloc(nchunks * sizeof(void*)); for (int i = 0; i < nchunks; i++) { if (frame->cframe != NULL) { if (needs_free) { free(data_chunk); } if (offsets[i] < 0) { int64_t rbytes = frame_get_chunk(frame, i, &data_chunk, &needs_free); if (rbytes < 0) { break; } } else { data_chunk = frame->cframe + header_len + offsets[i]; } rc = blosc2_cbuffer_sizes(data_chunk, NULL, &chunk_cbytes, NULL); if (rc < 0) { break; } } else { int64_t rbytes; if (frame->sframe) { if (needs_free) { free(data_chunk); } rbytes = frame_get_chunk(frame, i, &data_chunk, &needs_free); if (rbytes < 0) { break; } } else { io_cb->seek(fp, frame->file_offset + header_len + offsets[i], SEEK_SET); rbytes = io_cb->read(data_chunk, 1, BLOSC_EXTENDED_HEADER_LENGTH, fp); } if (rbytes != BLOSC_EXTENDED_HEADER_LENGTH) { rc = BLOSC2_ERROR_READ_BUFFER; break; } rc = blosc2_cbuffer_sizes(data_chunk, NULL, &chunk_cbytes, NULL); if (rc < 0) { break; } if (chunk_cbytes > (int32_t)prev_alloc) { data_chunk = realloc(data_chunk, chunk_cbytes); prev_alloc = chunk_cbytes; } if (!frame->sframe) { io_cb->seek(fp, frame->file_offset + header_len + offsets[i], SEEK_SET); rbytes = io_cb->read(data_chunk, 1, chunk_cbytes, fp); if (rbytes != chunk_cbytes) { rc = BLOSC2_ERROR_READ_BUFFER; break; } } } uint8_t* new_chunk = malloc(chunk_cbytes); memcpy(new_chunk, data_chunk, chunk_cbytes); schunk->data[i] = new_chunk; rc = blosc2_cbuffer_sizes(data_chunk, &chunk_nbytes, NULL, &chunk_blocksize); if (rc < 0) { break; } acc_nbytes += chunk_nbytes; acc_cbytes += chunk_cbytes; if (i == 0) { blocksize = chunk_blocksize; } else if (blocksize != chunk_blocksize) { // Blocksize varies blocksize = 0; } } // We are not attached to a schunk anymore frame->schunk = NULL; end: if (needs_free) { free(data_chunk); } if (frame->cframe == NULL) { if (!frame->sframe) { io_cb->close(fp); } } free(offsets); // cframes and sframes have different ways to store chunks with special values: // 1) cframes represent special chunks as negative offsets // 2) sframes does not have the concept of offsets, but rather of data pointers (.data) // so they always have a pointer to a special chunk // This is why cframes and sframes have different cbytes and hence, we cannot enforce acc_bytes == schunk->cbytes // In the future, maybe we could provide special meanings for .data[i] > 0x7FFFFFFF, but not there yet // if (rc < 0 || acc_nbytes != nbytes || acc_cbytes != cbytes) { if (rc < 0 || acc_nbytes != nbytes) { blosc2_schunk_free(schunk); return NULL; } // Update counters schunk->cbytes = acc_cbytes; schunk->blocksize = blocksize; out: rc = frame_get_metalayers(frame, schunk); if (rc < 0) { blosc2_schunk_free(schunk); BLOSC_TRACE_ERROR("Cannot access the metalayers."); return NULL; } rc = frame_get_vlmetalayers(frame, schunk); if (rc < 0) { blosc2_schunk_free(schunk); BLOSC_TRACE_ERROR("Cannot access the vlmetalayers."); return NULL; } return schunk; } void frame_avoid_cframe_free(blosc2_frame_s* frame, bool avoid_cframe_free) { frame->avoid_cframe_free = avoid_cframe_free; } struct csize_idx { int32_t val; int32_t idx; }; // Helper function for qsorting block offsets int sort_offset(const void* a, const void* b) { int32_t a_ = ((struct csize_idx*)a)->val; int32_t b_ = ((struct csize_idx*)b)->val; return a_ - b_; } int get_coffset(blosc2_frame_s* frame, int32_t header_len, int64_t cbytes, int64_t nchunk, int64_t nchunks, int64_t *offset) { int32_t off_cbytes; // Get the offset to nchunk uint8_t *coffsets = get_coffsets(frame, header_len, cbytes, nchunks, &off_cbytes); if (coffsets == NULL) { BLOSC_TRACE_ERROR("Cannot get the offset for chunk %" PRId64 " for the frame.", nchunk); return BLOSC2_ERROR_DATA; } // Get the 64-bit offset int rc = blosc2_getitem(coffsets, off_cbytes, (int32_t)nchunk, 1, offset, (int32_t)sizeof(int64_t)); if (rc < 0) { BLOSC_TRACE_ERROR("Problems retrieving a chunk offset."); } else if (!frame->sframe && *offset > frame->len) { BLOSC_TRACE_ERROR("Cannot read chunk %" PRId64 " outside of frame boundary.", nchunk); rc = BLOSC2_ERROR_READ_BUFFER; } return rc; } // Detect and return a chunk with special values in offsets (only zeros, NaNs and non initialized) int frame_special_chunk(int64_t special_value, int32_t nbytes, int32_t typesize, int32_t blocksize, uint8_t** chunk, int32_t cbytes, bool *needs_free) { int rc = 0; *chunk = malloc(cbytes); *needs_free = true; // Detect the kind of special value uint64_t zeros_mask = (uint64_t) BLOSC2_SPECIAL_ZERO << (8 * 7); // chunk of zeros uint64_t nans_mask = (uint64_t) BLOSC2_SPECIAL_NAN << (8 * 7); // chunk of NaNs uint64_t uninit_mask = (uint64_t) BLOSC2_SPECIAL_UNINIT << (8 * 7); // chunk of uninit values blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.typesize = typesize; cparams.blocksize = blocksize; if (special_value & zeros_mask) { rc = blosc2_chunk_zeros(cparams, nbytes, *chunk, cbytes); if (rc < 0) { BLOSC_TRACE_ERROR("Error creating a zero chunk"); } } else if (special_value & uninit_mask) { rc = blosc2_chunk_uninit(cparams, nbytes, *chunk, cbytes); if (rc < 0) { BLOSC_TRACE_ERROR("Error creating a non initialized chunk"); } } else if (special_value & nans_mask) { rc = blosc2_chunk_nans(cparams, nbytes, *chunk, cbytes); if (rc < 0) { BLOSC_TRACE_ERROR("Error creating a nan chunk"); } } else { BLOSC_TRACE_ERROR("Special value not recognized: %" PRId64 "", special_value); rc = BLOSC2_ERROR_DATA; } if (rc < 0) { free(*chunk); *needs_free = false; *chunk = NULL; } return rc; } /* Return a compressed chunk that is part of a frame in the `chunk` parameter. * If the frame is disk-based, a buffer is allocated for the (compressed) chunk, * and hence a free is needed. You can check if the chunk requires a free with the `needs_free` * parameter. * If the chunk does not need a free, it means that a pointer to the location in frame is returned * in the `chunk` parameter. * * The size of the (compressed) chunk is returned. If some problem is detected, a negative code * is returned instead. */ int frame_get_chunk(blosc2_frame_s *frame, int64_t nchunk, uint8_t **chunk, bool *needs_free) { int32_t header_len; int64_t frame_len; int64_t nbytes; int64_t cbytes; int32_t blocksize; int32_t chunksize; int64_t nchunks; int32_t typesize; int64_t offset; int32_t chunk_cbytes; int rc; *chunk = NULL; *needs_free = false; rc = get_header_info(frame, &header_len, &frame_len, &nbytes, &cbytes, &blocksize, &chunksize, &nchunks, &typesize, NULL, NULL, NULL, NULL, NULL, NULL, frame->schunk->storage->io); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to get meta info from frame."); return rc; } if (nchunk >= nchunks) { BLOSC_TRACE_ERROR("nchunk ('%" PRId64 "') exceeds the number of chunks " "('%" PRId64 "') in frame.", nchunk, nchunks); return BLOSC2_ERROR_INVALID_PARAM; } // Get the offset to nchunk rc = get_coffset(frame, header_len, cbytes, nchunk, nchunks, &offset); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to get offset to chunk %" PRId64 ".", nchunk); return rc; } if (offset < 0) { // Special value chunk_cbytes = BLOSC_EXTENDED_HEADER_LENGTH; int32_t chunksize_ = chunksize; if ((nchunk == nchunks - 1) && (nbytes % chunksize)) { // Last chunk is incomplete. Compute its actual size. chunksize_ = (int32_t) (nbytes % chunksize); } rc = frame_special_chunk(offset, chunksize_, typesize, blocksize, chunk, chunk_cbytes, needs_free); if (rc < 0) { return rc; } goto end; } if (frame->sframe) { // Sparse on-disk nchunk = offset; return sframe_get_chunk(frame, nchunk, chunk, needs_free); } blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } if (frame->cframe == NULL) { uint8_t header[BLOSC_EXTENDED_HEADER_LENGTH]; void* fp = io_cb->open(frame->urlpath, "rb", frame->schunk->storage->io->params); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, frame->file_offset + header_len + offset, SEEK_SET); int64_t rbytes = io_cb->read(header, 1, sizeof(header), fp); if (rbytes != sizeof(header)) { BLOSC_TRACE_ERROR("Cannot read the cbytes for chunk in the frame."); io_cb->close(fp); return BLOSC2_ERROR_FILE_READ; } rc = blosc2_cbuffer_sizes(header, NULL, &chunk_cbytes, NULL); if (rc < 0) { BLOSC_TRACE_ERROR("Cannot read the cbytes for chunk in the frame."); io_cb->close(fp); return rc; } *chunk = malloc(chunk_cbytes); io_cb->seek(fp, frame->file_offset + header_len + offset, SEEK_SET); rbytes = io_cb->read(*chunk, 1, chunk_cbytes, fp); io_cb->close(fp); if (rbytes != chunk_cbytes) { BLOSC_TRACE_ERROR("Cannot read the chunk out of the frame."); return BLOSC2_ERROR_FILE_READ; } *needs_free = true; } else { // The chunk is in memory and just one pointer away *chunk = frame->cframe + header_len + offset; rc = blosc2_cbuffer_sizes(*chunk, NULL, &chunk_cbytes, NULL); if (rc < 0) { return rc; } } end: return (int32_t)chunk_cbytes; } /* Return a compressed chunk that is part of a frame in the `chunk` parameter. * If the frame is disk-based, a buffer is allocated for the (lazy) chunk, * and hence a free is needed. You can check if the chunk requires a free with the `needs_free` * parameter. * If the chunk does not need a free, it means that the frame is in memory and that just a * pointer to the location of the chunk in memory is returned. * * The size of the (compressed, potentially lazy) chunk is returned. If some problem is detected, * a negative code is returned instead. */ int frame_get_lazychunk(blosc2_frame_s *frame, int64_t nchunk, uint8_t **chunk, bool *needs_free) { int32_t header_len; int64_t frame_len; int64_t nbytes; int64_t cbytes; int32_t blocksize; int32_t chunksize; int64_t nchunks; int32_t typesize; int32_t lazychunk_cbytes; int64_t offset; void* fp = NULL; *chunk = NULL; *needs_free = false; int rc = get_header_info(frame, &header_len, &frame_len, &nbytes, &cbytes, &blocksize, &chunksize, &nchunks, &typesize, NULL, NULL, NULL, NULL, NULL, NULL, frame->schunk->storage->io); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to get meta info from frame."); return rc; } if (nchunk >= nchunks) { BLOSC_TRACE_ERROR("nchunk ('%" PRId64 "') exceeds the number of chunks " "('%" PRId64 "') in frame.", nchunk, nchunks); return BLOSC2_ERROR_INVALID_PARAM; } // Get the offset to nchunk rc = get_coffset(frame, header_len, cbytes, nchunk, nchunks, &offset); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to get offset to chunk %" PRId64 ".", nchunk); return rc; } if (offset < 0) { // Special value lazychunk_cbytes = BLOSC_EXTENDED_HEADER_LENGTH; int32_t chunksize_ = chunksize; if ((nchunk == nchunks - 1) && (nbytes % chunksize)) { // Last chunk is incomplete. Compute its actual size. chunksize_ = (int32_t) (nbytes % chunksize); } rc = frame_special_chunk(offset, chunksize_, typesize, blocksize, chunk, (int32_t)lazychunk_cbytes, needs_free); goto end; } blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); rc = BLOSC2_ERROR_PLUGIN_IO; goto end; } if (frame->cframe == NULL) { // TODO: make this portable across different endianness // Get info for building a lazy chunk int32_t chunk_nbytes; int32_t chunk_cbytes; int32_t chunk_blocksize; uint8_t header[BLOSC_EXTENDED_HEADER_LENGTH]; if (frame->sframe) { // The chunk is not in the frame fp = sframe_open_chunk(frame->urlpath, offset, "rb", frame->schunk->storage->io); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } } else { fp = io_cb->open(frame->urlpath, "rb", frame->schunk->storage->io->params); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, frame->file_offset + header_len + offset, SEEK_SET); } int64_t rbytes = io_cb->read(header, 1, BLOSC_EXTENDED_HEADER_LENGTH, fp); if (rbytes != BLOSC_EXTENDED_HEADER_LENGTH) { BLOSC_TRACE_ERROR("Cannot read the header for chunk in the frame."); rc = BLOSC2_ERROR_FILE_READ; goto end; } rc = blosc2_cbuffer_sizes(header, &chunk_nbytes, &chunk_cbytes, &chunk_blocksize); if (rc < 0) { goto end; } size_t nblocks = chunk_nbytes / chunk_blocksize; size_t leftover_block = chunk_nbytes % chunk_blocksize; nblocks = leftover_block ? nblocks + 1 : nblocks; // Allocate space for the lazy chunk int32_t trailer_len; int32_t special_type = (header[BLOSC2_CHUNK_BLOSC2_FLAGS] >> 4) & BLOSC2_SPECIAL_MASK; int memcpyed = header[BLOSC2_CHUNK_FLAGS] & (uint8_t) BLOSC_MEMCPYED; int32_t trailer_offset = BLOSC_EXTENDED_HEADER_LENGTH; size_t streams_offset = BLOSC_EXTENDED_HEADER_LENGTH; if (special_type == 0) { // Regular values have offsets for blocks trailer_offset += (int32_t) (nblocks * sizeof(int32_t)); if (memcpyed) { streams_offset += 0; } else { streams_offset += nblocks * sizeof(int32_t); } trailer_len = (int32_t) (sizeof(int32_t) + sizeof(int64_t) + nblocks * sizeof(int32_t)); lazychunk_cbytes = trailer_offset + trailer_len; } else if (special_type == BLOSC2_SPECIAL_VALUE) { trailer_offset += typesize; streams_offset += typesize; trailer_len = 0; lazychunk_cbytes = trailer_offset + trailer_len; } else { rc = BLOSC2_ERROR_INVALID_HEADER; goto end; } *chunk = malloc(lazychunk_cbytes); *needs_free = true; // Read just the full header and bstarts section too (lazy partial length) if (frame->sframe) { io_cb->seek(fp, 0, SEEK_SET); } else { io_cb->seek(fp, frame->file_offset + header_len + offset, SEEK_SET); } rbytes = io_cb->read(*chunk, 1, (int64_t)streams_offset, fp); if (rbytes != (int64_t)streams_offset) { BLOSC_TRACE_ERROR("Cannot read the (lazy) chunk out of the frame."); rc = BLOSC2_ERROR_FILE_READ; goto end; } if (special_type == BLOSC2_SPECIAL_VALUE) { // Value runlen is not returning a lazy chunk. We are done. goto end; } // Mark chunk as lazy uint8_t* blosc2_flags = *chunk + BLOSC2_CHUNK_BLOSC2_FLAGS; *blosc2_flags |= 0x08U; // Add the trailer (currently, nchunk + offset + block_csizes) if (frame->sframe) { *(int32_t*)(*chunk + trailer_offset) = (int32_t)offset; // offset is nchunk for sframes *(int64_t*)(*chunk + trailer_offset + sizeof(int32_t)) = offset; } else { *(int32_t*)(*chunk + trailer_offset) = (int32_t)nchunk; *(int64_t*)(*chunk + trailer_offset + sizeof(int32_t)) = header_len + offset; } int32_t* block_csizes = malloc(nblocks * sizeof(int32_t)); if (memcpyed) { // When memcpyed the blocksizes are trivial to compute for (int i = 0; i < (int)nblocks - 1; i++) { block_csizes[i] = (int)chunk_blocksize; } // The last block could be incomplete, mainly due to the fact that the block size is not divisible // by the typesize block_csizes[nblocks - 1] = (int32_t)leftover_block ? (int32_t)leftover_block : chunk_blocksize; } else { // In regular, compressed chunks, we need to sort the bstarts (they can be out // of order because of multi-threading), and get a reverse index too. memcpy(block_csizes, *chunk + BLOSC_EXTENDED_HEADER_LENGTH, nblocks * sizeof(int32_t)); // Helper structure to keep track of original indexes struct csize_idx *csize_idx = malloc(nblocks * sizeof(struct csize_idx)); for (int n = 0; n < (int)nblocks; n++) { csize_idx[n].val = block_csizes[n]; csize_idx[n].idx = n; } qsort(csize_idx, nblocks, sizeof(struct csize_idx), &sort_offset); // Compute the actual csizes int idx; for (int n = 0; n < (int)nblocks - 1; n++) { idx = csize_idx[n].idx; block_csizes[idx] = csize_idx[n + 1].val - csize_idx[n].val; } idx = csize_idx[nblocks - 1].idx; block_csizes[idx] = (int)chunk_cbytes - csize_idx[nblocks - 1].val; free(csize_idx); } // Copy the csizes at the end of the trailer void *trailer_csizes = *chunk + lazychunk_cbytes - nblocks * sizeof(int32_t); memcpy(trailer_csizes, block_csizes, nblocks * sizeof(int32_t)); free(block_csizes); } else { // The chunk is in memory and just one pointer away int64_t chunk_header_offset = header_len + offset; int64_t chunk_cbytes_offset = chunk_header_offset + BLOSC_MIN_HEADER_LENGTH; *chunk = frame->cframe + chunk_header_offset; if (chunk_cbytes_offset > frame->len) { BLOSC_TRACE_ERROR("Cannot read the header for chunk in the (contiguous) frame."); rc = BLOSC2_ERROR_READ_BUFFER; } else { rc = blosc2_cbuffer_sizes(*chunk, NULL, &lazychunk_cbytes, NULL); if (rc && chunk_cbytes_offset + lazychunk_cbytes > frame_len) { BLOSC_TRACE_ERROR("Compressed bytes exceed beyond frame length."); rc = BLOSC2_ERROR_READ_BUFFER; } } } end: if (fp != NULL) { io_cb->close(fp); } if (rc < 0) { if (*needs_free) { free(*chunk); *chunk = NULL; *needs_free = false; } return rc; } return (int)lazychunk_cbytes; } /* Fill an empty frame with special values (fast path). */ int64_t frame_fill_special(blosc2_frame_s* frame, int64_t nitems, int special_value, int32_t chunksize, blosc2_schunk* schunk) { int32_t header_len; int64_t frame_len; int64_t nbytes; int64_t cbytes; int32_t blocksize; int32_t typesize; int64_t nchunks; int rc = get_header_info(frame, &header_len, &frame_len, &nbytes, &cbytes, &blocksize, NULL, &nchunks, &typesize, NULL, NULL, NULL, NULL, NULL, NULL, schunk->storage->io); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to get meta info from frame."); return BLOSC2_ERROR_DATA; } if (nitems == 0) { return frame_len; } if ((nitems / chunksize) > INT_MAX) { BLOSC_TRACE_ERROR("nitems is too large. Try increasing the chunksize."); return BLOSC2_ERROR_FRAME_SPECIAL; } if ((nbytes > 0) || (cbytes > 0)) { BLOSC_TRACE_ERROR("Filling with special values only works on empty frames"); return BLOSC2_ERROR_FRAME_SPECIAL; } // Compute the number of chunks and the length of the offsets chunk int32_t chunkitems = chunksize / typesize; nchunks = nitems / chunkitems; int32_t leftover_items = (int32_t)(nitems % chunkitems); if (leftover_items) { nchunks += 1; } blosc2_cparams* cparams; blosc2_schunk_get_cparams(schunk, &cparams); // Build the offsets with a special chunk int new_off_cbytes = BLOSC_EXTENDED_HEADER_LENGTH + sizeof(int64_t); uint8_t* off_chunk = malloc(new_off_cbytes); uint64_t offset_value = ((uint64_t)1 << 63); uint8_t* sample_chunk = malloc(BLOSC_EXTENDED_HEADER_LENGTH); int csize; switch (special_value) { case BLOSC2_SPECIAL_ZERO: offset_value += (uint64_t) BLOSC2_SPECIAL_ZERO << (8 * 7); csize = blosc2_chunk_zeros(*cparams, chunksize, sample_chunk, BLOSC_EXTENDED_HEADER_LENGTH); break; case BLOSC2_SPECIAL_UNINIT: offset_value += (uint64_t) BLOSC2_SPECIAL_UNINIT << (8 * 7); csize = blosc2_chunk_uninit(*cparams, chunksize, sample_chunk, BLOSC_EXTENDED_HEADER_LENGTH); break; case BLOSC2_SPECIAL_NAN: offset_value += (uint64_t)BLOSC2_SPECIAL_NAN << (8 * 7); csize = blosc2_chunk_nans(*cparams, chunksize, sample_chunk, BLOSC_EXTENDED_HEADER_LENGTH); break; default: BLOSC_TRACE_ERROR("Only zeros, NaNs or non-initialized values are supported."); return BLOSC2_ERROR_FRAME_SPECIAL; } if (csize < 0) { BLOSC_TRACE_ERROR("Error creating sample chunk"); return BLOSC2_ERROR_FRAME_SPECIAL; } cparams->typesize = sizeof(int64_t); // change it to offsets typesize // cparams->blocksize = 0; // automatic blocksize cparams->blocksize = 8 * 2 * 1024; // based on experiments with create_frame.c bench cparams->clevel = 5; cparams->compcode = BLOSC_BLOSCLZ; int32_t special_nbytes = (int32_t) (nchunks * sizeof(int64_t)); rc = blosc2_chunk_repeatval(*cparams, special_nbytes, off_chunk, new_off_cbytes, &offset_value); free(cparams); if (rc < 0) { BLOSC_TRACE_ERROR("Error creating a special offsets chunk"); return BLOSC2_ERROR_DATA; } // Get the blocksize associated to the sample chunk blosc2_cbuffer_sizes(sample_chunk, NULL, NULL, &blocksize); free(sample_chunk); // and use it for the super-chunk schunk->blocksize = blocksize; // schunk->blocksize = 0; // for experimenting with automatic blocksize // We have the new offsets; update the frame. blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } int64_t new_frame_len = header_len + new_off_cbytes + frame->trailer_len; void* fp = NULL; if (frame->cframe != NULL) { uint8_t* framep = frame->cframe; /* Make space for the new chunk and copy it */ frame->cframe = framep = realloc(framep, (size_t)new_frame_len); if (framep == NULL) { BLOSC_TRACE_ERROR("Cannot realloc space for the frame."); return BLOSC2_ERROR_FRAME_SPECIAL; } /* Copy the offsets */ memcpy(framep + header_len, off_chunk, (size_t)new_off_cbytes); } else { size_t wbytes; if (frame->sframe) { // Update the offsets chunk in the chunks frame fp = sframe_open_index(frame->urlpath, "rb+", frame->schunk->storage->io); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, frame->file_offset + header_len, SEEK_SET); } else { // Regular frame fp = io_cb->open(frame->urlpath, "rb+", schunk->storage->io->params); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, frame->file_offset + header_len + cbytes, SEEK_SET); } wbytes = io_cb->write(off_chunk, 1, new_off_cbytes, fp); // the new offsets io_cb->close(fp); if (wbytes != (size_t)new_off_cbytes) { BLOSC_TRACE_ERROR("Cannot write the offsets to frame."); return BLOSC2_ERROR_FRAME_SPECIAL; } } // Invalidate the cache for chunk offsets if (frame->coffsets != NULL) { free(frame->coffsets); frame->coffsets = NULL; } free(off_chunk); frame->len = new_frame_len; rc = frame_update_header(frame, schunk, false); if (rc < 0) { return BLOSC2_ERROR_FRAME_SPECIAL; } rc = frame_update_trailer(frame, schunk); if (rc < 0) { return BLOSC2_ERROR_FRAME_SPECIAL; } return frame->len; } /* Append an existing chunk into a frame. */ void* frame_append_chunk(blosc2_frame_s* frame, void* chunk, blosc2_schunk* schunk) { int8_t* chunk_ = chunk; int32_t header_len; int64_t frame_len; int64_t nbytes; int64_t cbytes; int32_t blocksize; int32_t chunksize; int64_t nchunks; int rc = get_header_info(frame, &header_len, &frame_len, &nbytes, &cbytes, &blocksize, &chunksize, &nchunks, NULL, NULL, NULL, NULL, NULL, NULL, NULL, frame->schunk->storage->io); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to get meta info from frame."); return NULL; } /* The uncompressed and compressed sizes start at byte 4 and 12 */ int32_t chunk_nbytes; int32_t chunk_cbytes; rc = blosc2_cbuffer_sizes(chunk, &chunk_nbytes, &chunk_cbytes, NULL); if (rc < 0) { return NULL; } if ((nchunks > 0) && (chunk_nbytes > chunksize)) { BLOSC_TRACE_ERROR("Appending chunks with a larger chunksize than frame is " "not allowed yet %d != %d.", chunk_nbytes, chunksize); return NULL; } // Check that we are not appending a small chunk after another small chunk int32_t chunk_nbytes_last; if (chunksize == 0 && (nchunks > 0) && (chunk_nbytes < chunksize)) { uint8_t* last_chunk; bool needs_free; rc = frame_get_lazychunk(frame, nchunks - 1, &last_chunk, &needs_free); if (rc < 0) { BLOSC_TRACE_ERROR("Cannot get the last chunk (in position %" PRId64 ").", nchunks - 1); } else { rc = blosc2_cbuffer_sizes(last_chunk, &chunk_nbytes_last, NULL, NULL); } if (needs_free) { free(last_chunk); } if (rc < 0) { return NULL; } if ((chunk_nbytes_last < chunksize) && (nbytes < chunksize)) { BLOSC_TRACE_ERROR("Appending two consecutive chunks with a chunksize smaller " "than the frame chunksize is not allowed yet: %d != %d.", chunk_nbytes, chunksize); return NULL; } } // Get the current offsets and add one more int32_t off_nbytes = (int32_t) ((nchunks + 1) * sizeof(int64_t)); int64_t* offsets = (int64_t *) malloc((size_t)off_nbytes); if (nchunks > 0) { int32_t coffsets_cbytes; uint8_t *coffsets = get_coffsets(frame, header_len, cbytes, nchunks, &coffsets_cbytes); if (coffsets == NULL) { BLOSC_TRACE_ERROR("Cannot get the offsets for the frame."); free(offsets); return NULL; } if (coffsets_cbytes == 0) { coffsets_cbytes = (int32_t)cbytes; } // Decompress offsets blosc2_dparams off_dparams = BLOSC2_DPARAMS_DEFAULTS; blosc2_context *dctx = blosc2_create_dctx(off_dparams); int32_t prev_nbytes = blosc2_decompress_ctx(dctx, coffsets, coffsets_cbytes, offsets, off_nbytes); blosc2_free_ctx(dctx); if (prev_nbytes < 0) { free(offsets); BLOSC_TRACE_ERROR("Cannot decompress the offsets chunk."); return NULL; } } // Add the new offset int64_t sframe_chunk_id = -1; int special_value = (chunk_[BLOSC2_CHUNK_BLOSC2_FLAGS] >> 4) & BLOSC2_SPECIAL_MASK; uint64_t offset_value = ((uint64_t)1 << 63); switch (special_value) { case BLOSC2_SPECIAL_ZERO: // Zero chunk. Code it in a special way. offset_value += (uint64_t) BLOSC2_SPECIAL_ZERO << (8 * 7); // chunk of zeros to_little(offsets + nchunks, &offset_value, sizeof(uint64_t)); chunk_cbytes = 0; // we don't need to store the chunk break; case BLOSC2_SPECIAL_UNINIT: // Non initizalized values chunk. Code it in a special way. offset_value += (uint64_t) BLOSC2_SPECIAL_UNINIT << (8 * 7); // chunk of uninit values to_little(offsets + nchunks, &offset_value, sizeof(uint64_t)); chunk_cbytes = 0; // we don't need to store the chunk break; case BLOSC2_SPECIAL_NAN: // NaN chunk. Code it in a special way. offset_value += (uint64_t)BLOSC2_SPECIAL_NAN << (8 * 7); // chunk of NANs to_little(offsets + nchunks, &offset_value, sizeof(uint64_t)); chunk_cbytes = 0; // we don't need to store the chunk break; default: if (frame->sframe) { // Compute the sframe_chunk_id value for (int i = 0; i < nchunks; ++i) { if (offsets[i] > sframe_chunk_id) { sframe_chunk_id = offsets[i]; } } offsets[nchunks] = ++sframe_chunk_id; } else { offsets[nchunks] = cbytes; } } // Re-compress the offsets again blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = sizeof(int64_t); cparams.blocksize = 16 * 1024; // based on experiments with create_frame.c bench cparams.nthreads = 4; // 4 threads seems a decent default for nowadays CPUs cparams.compcode = BLOSC_BLOSCLZ; blosc2_context* cctx = blosc2_create_cctx(cparams); cctx->typesize = sizeof(int64_t); // override a possible BLOSC_TYPESIZE env variable (or chaos may appear) void* off_chunk = malloc((size_t)off_nbytes + BLOSC2_MAX_OVERHEAD); int32_t new_off_cbytes = blosc2_compress_ctx(cctx, offsets, off_nbytes, off_chunk, off_nbytes + BLOSC2_MAX_OVERHEAD); blosc2_free_ctx(cctx); free(offsets); if (new_off_cbytes < 0) { free(off_chunk); return NULL; } // printf("%f\n", (double) off_nbytes / new_off_cbytes); int64_t new_cbytes = cbytes + chunk_cbytes; int64_t new_frame_len; if (frame->sframe) { new_frame_len = header_len + 0 + new_off_cbytes + frame->trailer_len; } else { new_frame_len = header_len + new_cbytes + new_off_cbytes + frame->trailer_len; } void* fp = NULL; if (frame->cframe != NULL) { uint8_t* framep = frame->cframe; /* Make space for the new chunk and copy it */ frame->cframe = framep = realloc(framep, (size_t)new_frame_len); if (framep == NULL) { BLOSC_TRACE_ERROR("Cannot realloc space for the frame."); return NULL; } /* Copy the chunk */ memcpy(framep + header_len + cbytes, chunk, (size_t)chunk_cbytes); /* Copy the offsets */ memcpy(framep + header_len + new_cbytes, off_chunk, (size_t)new_off_cbytes); } else { int64_t wbytes; blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return NULL; } if (frame->sframe) { // Update the offsets chunk in the chunks frame if (chunk_cbytes != 0) { if (sframe_chunk_id < 0) { BLOSC_TRACE_ERROR("The chunk id (%" PRId64 ") is not correct", sframe_chunk_id); return NULL; } if (sframe_create_chunk(frame, chunk, sframe_chunk_id, chunk_cbytes) == NULL) { BLOSC_TRACE_ERROR("Cannot write the full chunk."); return NULL; } } fp = sframe_open_index(frame->urlpath, "rb+", frame->schunk->storage->io); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return NULL; } io_cb->seek(fp, frame->file_offset + header_len, SEEK_SET); } else { // Regular frame fp = io_cb->open(frame->urlpath, "rb+", frame->schunk->storage->io->params); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return NULL; } io_cb->seek(fp, frame->file_offset + header_len + cbytes, SEEK_SET); wbytes = io_cb->write(chunk, 1, chunk_cbytes, fp); // the new chunk if (wbytes != chunk_cbytes) { BLOSC_TRACE_ERROR("Cannot write the full chunk to frame."); io_cb->close(fp); return NULL; } } wbytes = io_cb->write(off_chunk, 1, new_off_cbytes, fp); // the new offsets io_cb->close(fp); if (wbytes != new_off_cbytes) { BLOSC_TRACE_ERROR("Cannot write the offsets to frame."); return NULL; } } // Invalidate the cache for chunk offsets if (frame->coffsets != NULL) { free(frame->coffsets); frame->coffsets = NULL; } free(chunk); // chunk has always to be a copy when reaching here... free(off_chunk); frame->len = new_frame_len; rc = frame_update_header(frame, schunk, false); if (rc < 0) { return NULL; } rc = frame_update_trailer(frame, schunk); if (rc < 0) { return NULL; } return frame; } void* frame_insert_chunk(blosc2_frame_s* frame, int64_t nchunk, void* chunk, blosc2_schunk* schunk) { uint8_t* chunk_ = chunk; int32_t header_len; int64_t frame_len; int64_t nbytes; int64_t cbytes; int32_t blocksize; int32_t chunksize; int64_t nchunks; int rc = get_header_info(frame, &header_len, &frame_len, &nbytes, &cbytes, &blocksize, &chunksize, &nchunks, NULL, NULL, NULL, NULL, NULL, NULL, NULL, frame->schunk->storage->io); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to get meta info from frame."); return NULL; } int32_t chunk_cbytes; rc = blosc2_cbuffer_sizes(chunk_, NULL, &chunk_cbytes, NULL); if (rc < 0) { return NULL; } // Get the current offsets int32_t off_nbytes = (int32_t) ((nchunks + 1) * sizeof(int64_t)); int64_t* offsets = (int64_t *) malloc((size_t)off_nbytes); if (nchunks > 0) { int32_t coffsets_cbytes = 0; uint8_t *coffsets = get_coffsets(frame, header_len, cbytes, nchunks, &coffsets_cbytes); if (coffsets == NULL) { BLOSC_TRACE_ERROR("Cannot get the offsets for the frame."); return NULL; } if (coffsets_cbytes == 0) { coffsets_cbytes = (int32_t)cbytes; } // Decompress offsets blosc2_dparams off_dparams = BLOSC2_DPARAMS_DEFAULTS; blosc2_context *dctx = blosc2_create_dctx(off_dparams); int32_t prev_nbytes = blosc2_decompress_ctx(dctx, coffsets, coffsets_cbytes, offsets, off_nbytes); blosc2_free_ctx(dctx); if (prev_nbytes < 0) { free(offsets); BLOSC_TRACE_ERROR("Cannot decompress the offsets chunk."); return NULL; } } // TODO: Improvement: Check if new chunk is smaller than previous one // Move offsets for (int64_t i = nchunks; i > nchunk; i--) { offsets[i] = offsets[i - 1]; } // Add the new offset int64_t sframe_chunk_id = -1; int special_value = (chunk_[BLOSC2_CHUNK_BLOSC2_FLAGS] >> 4) & BLOSC2_SPECIAL_MASK; uint64_t offset_value = ((uint64_t)1 << 63); switch (special_value) { case BLOSC2_SPECIAL_ZERO: // Zero chunk. Code it in a special way. offset_value += (uint64_t)BLOSC2_SPECIAL_ZERO << (8 * 7); // indicate a chunk of zeros to_little(offsets + nchunk, &offset_value, sizeof(uint64_t)); chunk_cbytes = 0; // we don't need to store the chunk break; case BLOSC2_SPECIAL_UNINIT: // Non initizalized values chunk. Code it in a special way. offset_value += (uint64_t) BLOSC2_SPECIAL_UNINIT << (8 * 7); // chunk of uninit values to_little(offsets + nchunk, &offset_value, sizeof(uint64_t)); chunk_cbytes = 0; // we don't need to store the chunk break; case BLOSC2_SPECIAL_NAN: // NaN chunk. Code it in a special way. offset_value += (uint64_t)BLOSC2_SPECIAL_NAN << (8 * 7); // indicate a chunk of NANs to_little(offsets + nchunk, &offset_value, sizeof(uint64_t)); chunk_cbytes = 0; // we don't need to store the chunk break; default: if (frame->sframe) { for (int i = 0; i <= nchunks; ++i) { // offsets[nchunk] is still uninitialized here if (i != nchunk && offsets[i] > sframe_chunk_id) { sframe_chunk_id = offsets[i]; } } offsets[nchunk] = ++sframe_chunk_id; } else { offsets[nchunk] = cbytes; } } // Re-compress the offsets again blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = sizeof(int64_t); cparams.blocksize = 16 * 1024; // based on experiments with create_frame.c bench cparams.nthreads = 4; // 4 threads seems a decent default for nowadays CPUs cparams.compcode = BLOSC_BLOSCLZ; blosc2_context* cctx = blosc2_create_cctx(cparams); void* off_chunk = malloc((size_t)off_nbytes + BLOSC2_MAX_OVERHEAD); int32_t new_off_cbytes = blosc2_compress_ctx(cctx, offsets, off_nbytes, off_chunk, off_nbytes + BLOSC2_MAX_OVERHEAD); blosc2_free_ctx(cctx); free(offsets); if (new_off_cbytes < 0) { free(off_chunk); return NULL; } int64_t new_cbytes = cbytes + chunk_cbytes; int64_t new_frame_len; if (frame->sframe) { new_frame_len = header_len + 0 + new_off_cbytes + frame->trailer_len; } else { new_frame_len = header_len + new_cbytes + new_off_cbytes + frame->trailer_len; } // Add the chunk and update meta void* fp = NULL; if (frame->cframe != NULL) { uint8_t* framep = frame->cframe; /* Make space for the new chunk and copy it */ frame->cframe = framep = realloc(framep, (size_t)new_frame_len); if (framep == NULL) { BLOSC_TRACE_ERROR("Cannot realloc space for the frame."); return NULL; } /* Copy the chunk */ memcpy(framep + header_len + cbytes, chunk, (size_t)chunk_cbytes); /* Copy the offsets */ memcpy(framep + header_len + new_cbytes, off_chunk, (size_t)new_off_cbytes); } else { int64_t wbytes; blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return NULL; } if (frame->sframe) { if (chunk_cbytes != 0) { if (sframe_chunk_id < 0) { BLOSC_TRACE_ERROR("The chunk id (%" PRId64 ") is not correct", sframe_chunk_id); return NULL; } if (sframe_create_chunk(frame, chunk, sframe_chunk_id, chunk_cbytes) == NULL) { BLOSC_TRACE_ERROR("Cannot write the full chunk."); return NULL; } } // Update the offsets chunk in the chunks frame fp = sframe_open_index(frame->urlpath, "rb+", frame->schunk->storage->io); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return NULL; } io_cb->seek(fp, frame->file_offset + header_len + 0, SEEK_SET); } else { // Regular frame fp = io_cb->open(frame->urlpath, "rb+", frame->schunk->storage->io->params); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return NULL; } io_cb->seek(fp, frame->file_offset + header_len + cbytes, SEEK_SET); wbytes = io_cb->write(chunk, 1, chunk_cbytes, fp); // the new chunk if (wbytes != chunk_cbytes) { BLOSC_TRACE_ERROR("Cannot write the full chunk to frame."); io_cb->close(fp); return NULL; } } wbytes = io_cb->write(off_chunk, 1, new_off_cbytes, fp); // the new offsets io_cb->close(fp); if (wbytes != new_off_cbytes) { BLOSC_TRACE_ERROR("Cannot write the offsets to frame."); return NULL; } // Invalidate the cache for chunk offsets if (frame->coffsets != NULL) { free(frame->coffsets); frame->coffsets = NULL; } } free(chunk); // chunk has always to be a copy when reaching here... free(off_chunk); frame->len = new_frame_len; rc = frame_update_header(frame, schunk, false); if (rc < 0) { return NULL; } rc = frame_update_trailer(frame, schunk); if (rc < 0) { return NULL; } return frame; } void* frame_update_chunk(blosc2_frame_s* frame, int64_t nchunk, void* chunk, blosc2_schunk* schunk) { uint8_t *chunk_ = (uint8_t *) chunk; int32_t header_len; int64_t frame_len; int64_t nbytes; int64_t cbytes; int32_t blocksize; int32_t chunksize; int64_t nchunks; int rc = get_header_info(frame, &header_len, &frame_len, &nbytes, &cbytes, &blocksize, &chunksize, &nchunks, NULL, NULL, NULL, NULL, NULL, NULL, NULL, frame->schunk->storage->io); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to get meta info from frame."); return NULL; } if (nchunk >= nchunks) { BLOSC_TRACE_ERROR("The chunk must already exist."); return NULL; } int32_t chunk_cbytes; rc = blosc2_cbuffer_sizes(chunk, NULL, &chunk_cbytes, NULL); if (rc < 0) { return NULL; } // Get the current offsets int32_t off_nbytes = (int32_t) (nchunks * sizeof(int64_t)); int64_t* offsets = (int64_t *) malloc((size_t)off_nbytes); if (nchunks > 0) { int32_t coffsets_cbytes = 0; uint8_t *coffsets = get_coffsets(frame, header_len, cbytes, nchunks, &coffsets_cbytes); if (coffsets == NULL) { BLOSC_TRACE_ERROR("Cannot get the offsets for the frame."); return NULL; } if (coffsets_cbytes == 0) { coffsets_cbytes = (int32_t)cbytes; } // Decompress offsets blosc2_dparams off_dparams = BLOSC2_DPARAMS_DEFAULTS; blosc2_context *dctx = blosc2_create_dctx(off_dparams); int32_t prev_nbytes = blosc2_decompress_ctx(dctx, coffsets, coffsets_cbytes, offsets, off_nbytes); blosc2_free_ctx(dctx); if (prev_nbytes < 0) { free(offsets); BLOSC_TRACE_ERROR("Cannot decompress the offsets chunk."); return NULL; } } int32_t cbytes_old; int64_t old_offset; if (!frame->sframe) { // See how big would be the space old_offset = offsets[nchunk]; bool needs_free; uint8_t *chunk_old; int err = blosc2_schunk_get_chunk(schunk, nchunk, &chunk_old, &needs_free); if (err < 0) { BLOSC_TRACE_ERROR("%" PRId64 " chunk can not be obtained from schunk.", nchunk); return NULL; } if (chunk_old == NULL) { cbytes_old = 0; } else { cbytes_old = sw32_(chunk_old + BLOSC2_CHUNK_CBYTES); if (cbytes_old == BLOSC2_MAX_OVERHEAD) { cbytes_old = 0; } } if (needs_free) { free(chunk_old); } } // Add the new offset int64_t sframe_chunk_id; if (frame->sframe) { if (offsets[nchunk] < 0) { sframe_chunk_id = -1; } else { // In case there was a reorder in a sframe sframe_chunk_id = offsets[nchunk]; } } int special_value = (chunk_[BLOSC2_CHUNK_BLOSC2_FLAGS] >> 4) & BLOSC2_SPECIAL_MASK; uint64_t offset_value = ((uint64_t)1 << 63); switch (special_value) { case BLOSC2_SPECIAL_ZERO: // Zero chunk. Code it in a special way. offset_value += (uint64_t)BLOSC2_SPECIAL_ZERO << (8 * 7); // indicate a chunk of zeros to_little(offsets + nchunk, &offset_value, sizeof(uint64_t)); chunk_cbytes = 0; // we don't need to store the chunk break; case BLOSC2_SPECIAL_UNINIT: // Non initizalized values chunk. Code it in a special way. offset_value += (uint64_t)BLOSC2_SPECIAL_UNINIT << (8 * 7); // indicate a chunk of uninit values to_little(offsets + nchunk, &offset_value, sizeof(uint64_t)); chunk_cbytes = 0; // we don't need to store the chunk break; case BLOSC2_SPECIAL_NAN: // NaN chunk. Code it in a special way. offset_value += (uint64_t)BLOSC2_SPECIAL_NAN << (8 * 7); // indicate a chunk of NANs to_little(offsets + nchunk, &offset_value, sizeof(uint64_t)); chunk_cbytes = 0; // we don't need to store the chunk break; default: if (frame->sframe) { if (sframe_chunk_id < 0) { for (int i = 0; i < nchunks; ++i) { if (offsets[i] > sframe_chunk_id) { sframe_chunk_id = offsets[i]; } } offsets[nchunk] = ++sframe_chunk_id; } } else { // Add the new offset offsets[nchunk] = cbytes; } } if (!frame->sframe && chunk_cbytes != 0 && cbytes_old >= chunk_cbytes) { offsets[nchunk] = old_offset; cbytes = old_offset; } // Re-compress the offsets again blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = sizeof(int64_t); cparams.blocksize = 16 * 1024; // based on experiments with create_frame.c bench cparams.nthreads = 4; // 4 threads seems a decent default for nowadays CPUs cparams.compcode = BLOSC_BLOSCLZ; blosc2_context* cctx = blosc2_create_cctx(cparams); void* off_chunk = malloc((size_t)off_nbytes + BLOSC2_MAX_OVERHEAD); int32_t new_off_cbytes = blosc2_compress_ctx(cctx, offsets, off_nbytes, off_chunk, off_nbytes + BLOSC2_MAX_OVERHEAD); blosc2_free_ctx(cctx); free(offsets); if (new_off_cbytes < 0) { free(off_chunk); return NULL; } int64_t new_cbytes = schunk->cbytes; int64_t new_frame_len; if (frame->sframe) { // The chunk is not stored in the frame new_frame_len = header_len + 0 + new_off_cbytes + frame->trailer_len; } else { new_frame_len = header_len + new_cbytes + new_off_cbytes + frame->trailer_len; } void* fp = NULL; if (frame->cframe != NULL) { uint8_t* framep = frame->cframe; /* Make space for the new chunk and copy it */ frame->cframe = framep = realloc(framep, (size_t)new_frame_len); if (framep == NULL) { BLOSC_TRACE_ERROR("Cannot realloc space for the frame."); return NULL; } /* Copy the chunk */ memcpy(framep + header_len + cbytes, chunk, (size_t)chunk_cbytes); /* Copy the offsets */ memcpy(framep + header_len + new_cbytes, off_chunk, (size_t)new_off_cbytes); } else { int64_t wbytes; blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return NULL; } if (frame->sframe) { // Create the chunks file, if it's a special value this will delete its old content if (sframe_chunk_id >= 0) { if (sframe_create_chunk(frame, chunk, sframe_chunk_id, chunk_cbytes) == NULL) { BLOSC_TRACE_ERROR("Cannot write the full chunk."); return NULL; } } // Update the offsets chunk in the chunks frame fp = sframe_open_index(frame->urlpath, "rb+", frame->schunk->storage->io); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return NULL; } io_cb->seek(fp, frame->file_offset + header_len + 0, SEEK_SET); } else { // Regular frame fp = io_cb->open(frame->urlpath, "rb+", frame->schunk->storage->io->params); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return NULL; } io_cb->seek(fp, frame->file_offset + header_len + cbytes, SEEK_SET); wbytes = io_cb->write(chunk, 1, chunk_cbytes, fp); // the new chunk if (wbytes != chunk_cbytes) { BLOSC_TRACE_ERROR("Cannot write the full chunk to frame."); io_cb->close(fp); return NULL; } io_cb->seek(fp, frame->file_offset + header_len + new_cbytes, SEEK_SET); } wbytes = io_cb->write(off_chunk, 1, new_off_cbytes, fp); // the new offsets io_cb->close(fp); if (wbytes != new_off_cbytes) { BLOSC_TRACE_ERROR("Cannot write the offsets to frame."); return NULL; } // Invalidate the cache for chunk offsets if (frame->coffsets != NULL) { free(frame->coffsets); frame->coffsets = NULL; } } free(chunk); // chunk has always to be a copy when reaching here... free(off_chunk); frame->len = new_frame_len; rc = frame_update_header(frame, schunk, false); if (rc < 0) { return NULL; } rc = frame_update_trailer(frame, schunk); if (rc < 0) { return NULL; } return frame; } void* frame_delete_chunk(blosc2_frame_s* frame, int64_t nchunk, blosc2_schunk* schunk) { int32_t header_len; int64_t frame_len; int64_t nbytes; int64_t cbytes; int32_t blocksize; int32_t chunksize; int64_t nchunks; int rc = get_header_info(frame, &header_len, &frame_len, &nbytes, &cbytes, &blocksize, &chunksize, &nchunks, NULL, NULL, NULL, NULL, NULL, NULL, NULL, frame->schunk->storage->io); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to get meta info from frame."); return NULL; } // Get the current offsets int32_t off_nbytes = (int32_t) (nchunks * sizeof(int64_t)); int64_t* offsets = (int64_t *) malloc((size_t)off_nbytes); if (nchunks > 0) { int32_t coffsets_cbytes = 0; uint8_t *coffsets = get_coffsets(frame, header_len, cbytes, nchunks, &coffsets_cbytes); if (coffsets == NULL) { BLOSC_TRACE_ERROR("Cannot get the offsets for the frame."); return NULL; } if (coffsets_cbytes == 0) { coffsets_cbytes = (int32_t)cbytes; } // Decompress offsets blosc2_dparams off_dparams = BLOSC2_DPARAMS_DEFAULTS; blosc2_context *dctx = blosc2_create_dctx(off_dparams); int32_t prev_nbytes = blosc2_decompress_ctx(dctx, coffsets, coffsets_cbytes, offsets, off_nbytes); blosc2_free_ctx(dctx); if (prev_nbytes < 0) { free(offsets); BLOSC_TRACE_ERROR("Cannot decompress the offsets chunk."); return NULL; } } // Delete the new offset for (int64_t i = nchunk; i < nchunks - 1; i++) { offsets[i] = offsets[i + 1]; } offsets[nchunks - 1] = 0; // Re-compress the offsets again blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = sizeof(int64_t); cparams.blocksize = 16 * 1024; // based on experiments with create_frame.c bench cparams.nthreads = 4; // 4 threads seems a decent default for nowadays CPUs cparams.compcode = BLOSC_BLOSCLZ; blosc2_context* cctx = blosc2_create_cctx(cparams); void* off_chunk = malloc((size_t)off_nbytes + BLOSC2_MAX_OVERHEAD); int32_t new_off_cbytes = blosc2_compress_ctx(cctx, offsets, off_nbytes - (int32_t)sizeof(int64_t), off_chunk, off_nbytes + BLOSC2_MAX_OVERHEAD); blosc2_free_ctx(cctx); free(offsets); if (new_off_cbytes < 0) { free(off_chunk); return NULL; } int64_t new_cbytes = cbytes; int64_t new_frame_len; if (frame->sframe) { new_frame_len = header_len + 0 + new_off_cbytes + frame->trailer_len; } else { new_frame_len = header_len + new_cbytes + new_off_cbytes + frame->trailer_len; } // Add the chunk and update meta FILE* fp = NULL; if (frame->cframe != NULL) { uint8_t* framep = frame->cframe; /* Make space for the new chunk and copy it */ frame->cframe = framep = realloc(framep, (size_t)new_frame_len); if (framep == NULL) { BLOSC_TRACE_ERROR("Cannot realloc space for the frame."); return NULL; } /* Copy the offsets */ memcpy(framep + header_len + new_cbytes, off_chunk, (size_t)new_off_cbytes); } else { blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return NULL; } size_t wbytes; if (frame->sframe) { int64_t offset; rc = get_coffset(frame, header_len, cbytes, nchunk, nchunks, &offset); if (rc < 0) { BLOSC_TRACE_ERROR("Unable to get offset to chunk %" PRId64 ".", nchunk); return NULL; } if (offset >= 0){ // Remove the chunk file only if it is not a special value chunk int err = sframe_delete_chunk(frame->urlpath, offset); if (err != 0) { BLOSC_TRACE_ERROR("Unable to delete chunk!"); return NULL; } } // Update the offsets chunk in the chunks frame fp = sframe_open_index(frame->urlpath, "rb+", frame->schunk->storage->io); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return NULL; } io_cb->seek(fp, frame->file_offset + header_len + 0, SEEK_SET); } else { // Regular frame fp = io_cb->open(frame->urlpath, "rb+", frame->schunk->storage->io); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return NULL; } io_cb->seek(fp, frame->file_offset + header_len + cbytes, SEEK_SET); } wbytes = io_cb->write(off_chunk, 1, new_off_cbytes, fp); // the new offsets io_cb->close(fp); if (wbytes != (size_t)new_off_cbytes) { BLOSC_TRACE_ERROR("Cannot write the offsets to frame."); return NULL; } // Invalidate the cache for chunk offsets if (frame->coffsets != NULL) { free(frame->coffsets); frame->coffsets = NULL; } } free(off_chunk); frame->len = new_frame_len; rc = frame_update_header(frame, schunk, false); if (rc < 0) { return NULL; } rc = frame_update_trailer(frame, schunk); if (rc < 0) { return NULL; } return frame; } int frame_reorder_offsets(blosc2_frame_s* frame, const int64_t* offsets_order, blosc2_schunk* schunk) { // Get header info int32_t header_len; int64_t frame_len; int64_t nbytes; int64_t cbytes; int32_t blocksize; int32_t chunksize; int64_t nchunks; int ret = get_header_info(frame, &header_len, &frame_len, &nbytes, &cbytes, &blocksize, &chunksize, &nchunks, NULL, NULL, NULL, NULL, NULL, NULL, NULL, frame->schunk->storage->io); if (ret < 0) { BLOSC_TRACE_ERROR("Cannot get the header info for the frame."); return ret; } // Get the current offsets and add one more int32_t off_nbytes = (int32_t) (nchunks * sizeof(int64_t)); int64_t* offsets = (int64_t *) malloc((size_t)off_nbytes); int32_t coffsets_cbytes = 0; uint8_t *coffsets = get_coffsets(frame, header_len, cbytes, nchunks, &coffsets_cbytes); if (coffsets == NULL) { BLOSC_TRACE_ERROR("Cannot get the offsets for the frame."); free(offsets); return BLOSC2_ERROR_DATA; } // Decompress offsets blosc2_dparams off_dparams = BLOSC2_DPARAMS_DEFAULTS; blosc2_context *dctx = blosc2_create_dctx(off_dparams); int32_t prev_nbytes = blosc2_decompress_ctx(dctx, coffsets, coffsets_cbytes, offsets, off_nbytes); blosc2_free_ctx(dctx); if (prev_nbytes < 0) { free(offsets); BLOSC_TRACE_ERROR("Cannot decompress the offsets chunk."); return prev_nbytes; } // Make a copy of the chunk offsets and reorder it int64_t *offsets_copy = malloc(prev_nbytes); memcpy(offsets_copy, offsets, prev_nbytes); for (int i = 0; i < nchunks; ++i) { offsets[i] = offsets_copy[offsets_order[i]]; } free(offsets_copy); // Re-compress the offsets again blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = sizeof(int64_t); cparams.blocksize = 16 * 1024; // based on experiments with create_frame.c bench cparams.nthreads = 4; // 4 threads seems a decent default for nowadays CPUs cparams.compcode = BLOSC_BLOSCLZ; blosc2_context* cctx = blosc2_create_cctx(cparams); void* off_chunk = malloc((size_t)off_nbytes + BLOSC2_MAX_OVERHEAD); int32_t new_off_cbytes = blosc2_compress_ctx(cctx, offsets, off_nbytes, off_chunk, off_nbytes + BLOSC2_MAX_OVERHEAD); blosc2_free_ctx(cctx); if (new_off_cbytes < 0) { free(offsets); free(off_chunk); return new_off_cbytes; } free(offsets); int64_t new_frame_len; if (frame->sframe) { // The chunks are not in the frame new_frame_len = header_len + 0 + new_off_cbytes + frame->trailer_len; } else { new_frame_len = header_len + cbytes + new_off_cbytes + frame->trailer_len; } if (frame->cframe != NULL) { uint8_t* framep = frame->cframe; /* Make space for the new chunk and copy it */ frame->cframe = framep = realloc(framep, (size_t)new_frame_len); if (framep == NULL) { BLOSC_TRACE_ERROR("Cannot realloc space for the frame."); return BLOSC2_ERROR_MEMORY_ALLOC; } /* Copy the offsets */ memcpy(framep + header_len + cbytes, off_chunk, (size_t)new_off_cbytes); } else { void* fp = NULL; blosc2_io_cb *io_cb = blosc2_get_io_cb(frame->schunk->storage->io->id); if (io_cb == NULL) { BLOSC_TRACE_ERROR("Error getting the input/output API"); return BLOSC2_ERROR_PLUGIN_IO; } if (frame->sframe) { // Update the offsets chunk in the chunks frame fp = sframe_open_index(frame->urlpath, "rb+", frame->schunk->storage->io); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, frame->file_offset + header_len + 0, SEEK_SET); } else { // Regular frame fp = io_cb->open(frame->urlpath, "rb+", frame->schunk->storage->io->params); if (fp == NULL) { BLOSC_TRACE_ERROR("Error opening file in: %s", frame->urlpath); return BLOSC2_ERROR_FILE_OPEN; } io_cb->seek(fp, frame->file_offset + header_len + cbytes, SEEK_SET); } int64_t wbytes = io_cb->write(off_chunk, 1, new_off_cbytes, fp); // the new offsets io_cb->close(fp); if (wbytes != new_off_cbytes) { BLOSC_TRACE_ERROR("Cannot write the offsets to frame."); return BLOSC2_ERROR_FILE_WRITE; } } // Invalidate the cache for chunk offsets if (frame->coffsets != NULL) { free(frame->coffsets); frame->coffsets = NULL; } free(off_chunk); frame->len = new_frame_len; int rc = frame_update_header(frame, schunk, false); if (rc < 0) { return rc; } rc = frame_update_trailer(frame, schunk); if (rc < 0) { return rc; } return 0; } /* Decompress and return a chunk that is part of a frame. */ int frame_decompress_chunk(blosc2_context *dctx, blosc2_frame_s* frame, int64_t nchunk, void *dest, int32_t nbytes) { uint8_t* src; bool needs_free; int32_t chunk_nbytes; int32_t chunk_cbytes; int rc; // Use a lazychunk here in order to do a potential parallel read. rc = frame_get_lazychunk(frame, nchunk, &src, &needs_free); if (rc < 0) { BLOSC_TRACE_ERROR("Cannot get the chunk in position %" PRId64 ".", nchunk); goto end; } chunk_cbytes = rc; if (chunk_cbytes < (signed)sizeof(int32_t)) { /* Not enough input to read `nbytes` */ rc = BLOSC2_ERROR_READ_BUFFER; } rc = blosc2_cbuffer_sizes(src, &chunk_nbytes, &chunk_cbytes, NULL); if (rc < 0) { goto end; } /* Create a buffer for destination */ if (chunk_nbytes > nbytes) { BLOSC_TRACE_ERROR("Not enough space for decompressing in dest."); rc = BLOSC2_ERROR_WRITE_BUFFER; goto end; } /* And decompress it */ dctx->header_overhead = BLOSC_EXTENDED_HEADER_LENGTH; int chunksize = rc = blosc2_decompress_ctx(dctx, src, chunk_cbytes, dest, nbytes); if (chunksize < 0 || chunksize != chunk_nbytes) { BLOSC_TRACE_ERROR("Error in decompressing chunk."); if (chunksize >= 0) rc = BLOSC2_ERROR_FAILURE; } end: if (needs_free) { free(src); } return rc; } zmat-0.9.9/src/blosc2/blosc/stune.h0000644000175200007730000000277114515254731017332 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef STUNE_H #define STUNE_H #include "context.h" /* The size of L1 cache. 32 KB is quite common nowadays. */ #define L1 (32 * 1024) /* The size of L2 cache. 256 KB is quite common nowadays. */ #define L2 (256 * 1024) /* The maximum number of compressed data streams in a block for compression */ #define MAX_STREAMS 16 /* Cannot be larger than 128 */ void blosc_stune_init(void * config, blosc2_context* cctx, blosc2_context* dctx); void blosc_stune_next_blocksize(blosc2_context * context); void blosc_stune_next_cparams(blosc2_context * context); void blosc_stune_update(blosc2_context * context, double ctime); void blosc_stune_free(blosc2_context * context); static blosc2_btune BTUNE_DEFAULTS = { .btune_init = blosc_stune_init, .btune_free = blosc_stune_free, .btune_update = blosc_stune_update, .btune_next_cparams = blosc_stune_next_cparams, .btune_next_blocksize = blosc_stune_next_blocksize, .btune_config = NULL, }; /* Conditions for splitting a block before compressing with a codec. */ int split_block(blosc2_context *context, int32_t typesize, int32_t blocksize); #endif /* STUNE_H */ zmat-0.9.9/src/blosc2/blosc/trunc-prec.h0000644000175200007730000000120414515254731020244 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef BLOSC_TRUNC_PREC_H #define BLOSC_TRUNC_PREC_H #include #include int truncate_precision(int8_t prec_bits, int32_t typesize, int32_t nbytes, const uint8_t* src, uint8_t* dest); #endif //BLOSC_TRUNC_PREC_H zmat-0.9.9/src/blosc2/blosc/shuffle.h0000644000175200007730000000625614515254731017632 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /********************************************************************* Shuffle/unshuffle routines which dynamically dispatch to hardware- accelerated routines based on the processor's architecture. Consumers should almost always prefer to call these routines instead of directly calling one of the hardware-accelerated routines, since these are cross-platform and future-proof. **********************************************************************/ #ifndef SHUFFLE_H #define SHUFFLE_H #include "blosc2/blosc2-common.h" /* Toggle hardware-accelerated routines based on SHUFFLE_*_ENABLED macros and availability on the target architecture. */ #if defined(SHUFFLE_AVX2_ENABLED) && defined(__AVX2__) #define SHUFFLE_USE_AVX2 #endif #if defined(SHUFFLE_SSE2_ENABLED) && defined(__SSE2__) #define SHUFFLE_USE_SSE2 #endif #if defined(SHUFFLE_ALTIVEC_ENABLED) && defined(__ALTIVEC__) #define SHUFFLE_USE_ALTIVEC #endif #if defined(SHUFFLE_NEON_ENABLED) && defined(__ARM_NEON) #define SHUFFLE_USE_NEON #endif #ifdef __cplusplus extern "C" { #endif /** Primary shuffle and bitshuffle routines. This function dynamically dispatches to the appropriate hardware-accelerated routine based on the host processor's architecture. If the host processor is not supported by any of the hardware-accelerated routines, the generic (non-accelerated) implementation is used instead. Consumers should almost always prefer to call this routine instead of directly calling the hardware-accelerated routines because this method is both cross- platform and future-proof. */ BLOSC_NO_EXPORT void shuffle(const int32_t bytesoftype, const int32_t blocksize, const uint8_t* _src, const uint8_t* _dest); BLOSC_NO_EXPORT int32_t bitshuffle(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, const uint8_t *_dest, const uint8_t *_tmp); /** Primary unshuffle and bitunshuffle routine. This function dynamically dispatches to the appropriate hardware-accelerated routine based on the host processor's architecture. If the host processor is not supported by any of the hardware-accelerated routines, the generic (non-accelerated) implementation is used instead. Consumers should almost always prefer to call this routine instead of directly calling the hardware-accelerated routines because this method is both cross- platform and future-proof. */ BLOSC_NO_EXPORT void unshuffle(const int32_t bytesoftype, const int32_t blocksize, const uint8_t* _src, const uint8_t* _dest); BLOSC_NO_EXPORT int32_t bitunshuffle(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, const uint8_t *_dest, const uint8_t *_tmp, const uint8_t format_version); #ifdef __cplusplus } #endif #endif /* SHUFFLE_H */ zmat-0.9.9/src/blosc2/blosc/directories.c0000644000175200007730000001016714515254731020501 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include #include "blosc2.h" #include #if defined(_WIN32) || defined(__MINGW32__) #include #include #include int blosc2_remove_dir(const char* dir_path) { char* path; char last_char = dir_path[strlen(dir_path) - 1]; if (last_char != '\\' || last_char != '/') { path = malloc(strlen(dir_path) + 2 + 1); sprintf(path, "%s\\*", dir_path); } else { path = malloc(strlen(dir_path) + 1 + 1); strcpy(path, dir_path); strcat(path, "*"); } char* fname; struct _finddata_t cfile; intptr_t file = _findfirst(path, &cfile); free(path); if (file == -1) { BLOSC_TRACE_ERROR("Could not open the file."); return BLOSC2_ERROR_FILE_OPEN; } int ret; while ( _findnext(file, &cfile) == 0) { if (strcmp(".", cfile.name) == 0 || strcmp("..", cfile.name) == 0) { continue; } fname = malloc(strlen(dir_path) + 1 + strlen(cfile.name) + 1); sprintf(fname, "%s\\%s", dir_path, cfile.name); ret = remove(fname); free(fname); if (ret < 0) { BLOSC_TRACE_ERROR("Could not remove file %s", fname); _findclose(file); return BLOSC2_ERROR_FAILURE; } } rmdir(dir_path); _findclose(file); return BLOSC2_ERROR_SUCCESS; } #else #include #include /* Return the directory path with the '/' at the end */ char* blosc2_normalize_dirpath(const char* dir_path) { char last_char = dir_path[strlen(dir_path) - 1]; char* path; if (last_char != '\\' || last_char != '/'){ path = malloc(strlen(dir_path) + 1 + 1); sprintf(path, "%s/", dir_path); } else { path = malloc(strlen(dir_path) + 1); strcpy(path, dir_path); } return path; } /* Function needed for removing each time the directory */ int blosc2_remove_dir(const char* dir_path) { char* path = blosc2_normalize_dirpath(dir_path); DIR* dr = opendir(path); struct stat statbuf; if (dr == NULL) { BLOSC_TRACE_ERROR("No file or directory found."); free(path); return BLOSC2_ERROR_NOT_FOUND; } struct dirent *de; int ret; char* fname; while ((de = readdir(dr)) != NULL) { fname = malloc(strlen(path) + strlen(de->d_name) + 1); sprintf(fname, "%s%s", path, de->d_name); if (!strcmp(de->d_name, ".") || !strcmp(de->d_name, "..")) { free(fname); continue; } if (!stat(fname, &statbuf)) { ret = unlink(fname); if (ret < 0) { BLOSC_TRACE_ERROR("Could not remove file %s", fname); free(fname); closedir(dr); free(path); return BLOSC2_ERROR_FAILURE; } } free(fname); } closedir(dr); rmdir(path); free(path); return BLOSC2_ERROR_SUCCESS; } #endif /* _WIN32 */ int blosc2_remove_urlpath(const char* urlpath){ if (urlpath != NULL) { struct stat statbuf; if (stat(urlpath, &statbuf) != 0){ BLOSC_TRACE_ERROR("Could not access %s", urlpath); return BLOSC2_ERROR_FAILURE; } if ((statbuf.st_mode & S_IFDIR) != 0) { return blosc2_remove_dir(urlpath); } if (remove(urlpath) < 0) { BLOSC_TRACE_ERROR("Could not remove %s", urlpath); return BLOSC2_ERROR_FILE_REMOVE; } } return BLOSC2_ERROR_SUCCESS; } int blosc2_rename_urlpath(char* old_urlpath, char* new_urlpath){ if (old_urlpath != NULL && new_urlpath != NULL) { struct stat statbuf; if (stat(old_urlpath, &statbuf) != 0) { BLOSC_TRACE_ERROR("Could not access %s", old_urlpath); return BLOSC2_ERROR_FAILURE; } int ret = rename(old_urlpath, new_urlpath); if (ret < 0) { BLOSC_TRACE_ERROR("Could not rename %s to %s", old_urlpath, new_urlpath); return BLOSC2_ERROR_FAILURE; } } return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/blosc/shuffle-neon.c0000644000175200007730000005273414515254731020564 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 Lucian Marc https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #include "shuffle-generic.h" #include "shuffle-neon.h" /* Make sure NEON is available for the compilation target and compiler. */ #if defined(__ARM_NEON) #include /* The next is useful for debugging purposes */ #if 0 #include #include static void printmem(uint8_t* buf) { printf("%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,%x,\n", buf[0], buf[1], buf[2], buf[3], buf[4], buf[5], buf[6], buf[7], buf[8], buf[9], buf[10], buf[11], buf[12], buf[13], buf[14], buf[15]); } #endif /* Routine optimized for shuffling a buffer for a type size of 2 bytes. */ static void shuffle2_neon(uint8_t *const dest, const uint8_t *const src, const size_t vectorizable_elements, const size_t total_elements) { size_t i, k; static const size_t bytesoftype = 2; uint8x16x2_t r0; for (i = 0, k = 0; i < vectorizable_elements * bytesoftype; i += 32, k++) { /* Load (and permute) 32 bytes to the structure r0 */ r0 = vld2q_u8(src + i); /* Store the results in the destination vector */ vst1q_u8(dest + total_elements * 0 + k * 16, r0.val[0]); vst1q_u8(dest + total_elements * 1 + k * 16, r0.val[1]); } } /* Routine optimized for shuffling a buffer for a type size of 4 bytes. */ static void shuffle4_neon(uint8_t *const dest, const uint8_t *const src, const size_t vectorizable_elements, const size_t total_elements) { size_t i, k; static const size_t bytesoftype = 4; uint8x16x4_t r0; for (i = 0, k = 0; i < vectorizable_elements * bytesoftype; i += 64, k++) { /* Load (and permute) 64 bytes to the structure r0 */ r0 = vld4q_u8(src + i); /* Store the results in the destination vector */ vst1q_u8(dest + total_elements * 0 + k * 16, r0.val[0]); vst1q_u8(dest + total_elements * 1 + k * 16, r0.val[1]); vst1q_u8(dest + total_elements * 2 + k * 16, r0.val[2]); vst1q_u8(dest + total_elements * 3 + k * 16, r0.val[3]); } } /* Routine optimized for shuffling a buffer for a type size of 8 bytes. */ static void shuffle8_neon(uint8_t *const dest, const uint8_t *const src, const size_t vectorizable_elements, const size_t total_elements) { size_t i, k; static const size_t bytesoftype = 8; uint8x8x2_t r0[4]; uint16x4x2_t r1[4]; uint32x2x2_t r2[4]; for (i = 0, k = 0; i < vectorizable_elements * bytesoftype; i += 64, k++) { /* Load and interleave groups of 8 bytes (64 bytes) to the structure r0 */ r0[0] = vzip_u8(vld1_u8(src + i + 0 * 8), vld1_u8(src + i + 1 * 8)); r0[1] = vzip_u8(vld1_u8(src + i + 2 * 8), vld1_u8(src + i + 3 * 8)); r0[2] = vzip_u8(vld1_u8(src + i + 4 * 8), vld1_u8(src + i + 5 * 8)); r0[3] = vzip_u8(vld1_u8(src + i + 6 * 8), vld1_u8(src + i + 7 * 8)); /* Interleave 16 bytes */ r1[0] = vzip_u16(vreinterpret_u16_u8(r0[0].val[0]), vreinterpret_u16_u8(r0[1].val[0])); r1[1] = vzip_u16(vreinterpret_u16_u8(r0[0].val[1]), vreinterpret_u16_u8(r0[1].val[1])); r1[2] = vzip_u16(vreinterpret_u16_u8(r0[2].val[0]), vreinterpret_u16_u8(r0[3].val[0])); r1[3] = vzip_u16(vreinterpret_u16_u8(r0[2].val[1]), vreinterpret_u16_u8(r0[3].val[1])); /* Interleave 32 bytes */ r2[0] = vzip_u32(vreinterpret_u32_u16(r1[0].val[0]), vreinterpret_u32_u16(r1[2].val[0])); r2[1] = vzip_u32(vreinterpret_u32_u16(r1[0].val[1]), vreinterpret_u32_u16(r1[2].val[1])); r2[2] = vzip_u32(vreinterpret_u32_u16(r1[1].val[0]), vreinterpret_u32_u16(r1[3].val[0])); r2[3] = vzip_u32(vreinterpret_u32_u16(r1[1].val[1]), vreinterpret_u32_u16(r1[3].val[1])); /* Store the results in the destination vector */ vst1_u8(dest + k * 8 + 0 * total_elements, vreinterpret_u8_u32(r2[0].val[0])); vst1_u8(dest + k * 8 + 1 * total_elements, vreinterpret_u8_u32(r2[0].val[1])); vst1_u8(dest + k * 8 + 2 * total_elements, vreinterpret_u8_u32(r2[1].val[0])); vst1_u8(dest + k * 8 + 3 * total_elements, vreinterpret_u8_u32(r2[1].val[1])); vst1_u8(dest + k * 8 + 4 * total_elements, vreinterpret_u8_u32(r2[2].val[0])); vst1_u8(dest + k * 8 + 5 * total_elements, vreinterpret_u8_u32(r2[2].val[1])); vst1_u8(dest + k * 8 + 6 * total_elements, vreinterpret_u8_u32(r2[3].val[0])); vst1_u8(dest + k * 8 + 7 * total_elements, vreinterpret_u8_u32(r2[3].val[1])); } } /* Routine optimized for shuffling a buffer for a type size of 16 bytes. */ static void shuffle16_neon(uint8_t *const dest, const uint8_t *const src, const size_t vectorizable_elements, const size_t total_elements) { size_t i, k; static const size_t bytesoftype = 16; uint8x8x2_t r0[8]; uint16x4x2_t r1[8]; uint32x2x2_t r2[8]; for (i = 0, k = 0; i < vectorizable_elements * bytesoftype; i += 128, k++) { /* Load and interleave groups of 16 bytes (128 bytes) to the structure r0 */ r0[0] = vzip_u8(vld1_u8(src + i + 0 * 8), vld1_u8(src + i + 2 * 8)); r0[1] = vzip_u8(vld1_u8(src + i + 1 * 8), vld1_u8(src + i + 3 * 8)); r0[2] = vzip_u8(vld1_u8(src + i + 4 * 8), vld1_u8(src + i + 6 * 8)); r0[3] = vzip_u8(vld1_u8(src + i + 5 * 8), vld1_u8(src + i + 7 * 8)); r0[4] = vzip_u8(vld1_u8(src + i + 8 * 8), vld1_u8(src + i + 10 * 8)); r0[5] = vzip_u8(vld1_u8(src + i + 9 * 8), vld1_u8(src + i + 11 * 8)); r0[6] = vzip_u8(vld1_u8(src + i + 12 * 8), vld1_u8(src + i + 14 * 8)); r0[7] = vzip_u8(vld1_u8(src + i + 13 * 8), vld1_u8(src + i + 15 * 8)); /* Interleave 16 bytes */ r1[0] = vzip_u16(vreinterpret_u16_u8(r0[0].val[0]), vreinterpret_u16_u8(r0[2].val[0])); r1[1] = vzip_u16(vreinterpret_u16_u8(r0[0].val[1]), vreinterpret_u16_u8(r0[2].val[1])); r1[2] = vzip_u16(vreinterpret_u16_u8(r0[1].val[0]), vreinterpret_u16_u8(r0[3].val[0])); r1[3] = vzip_u16(vreinterpret_u16_u8(r0[1].val[1]), vreinterpret_u16_u8(r0[3].val[1])); r1[4] = vzip_u16(vreinterpret_u16_u8(r0[4].val[0]), vreinterpret_u16_u8(r0[6].val[0])); r1[5] = vzip_u16(vreinterpret_u16_u8(r0[4].val[1]), vreinterpret_u16_u8(r0[6].val[1])); r1[6] = vzip_u16(vreinterpret_u16_u8(r0[5].val[0]), vreinterpret_u16_u8(r0[7].val[0])); r1[7] = vzip_u16(vreinterpret_u16_u8(r0[5].val[1]), vreinterpret_u16_u8(r0[7].val[1])); /* Interleave 32 bytes */ r2[0] = vzip_u32(vreinterpret_u32_u16(r1[0].val[0]), vreinterpret_u32_u16(r1[4].val[0])); r2[1] = vzip_u32(vreinterpret_u32_u16(r1[0].val[1]), vreinterpret_u32_u16(r1[4].val[1])); r2[2] = vzip_u32(vreinterpret_u32_u16(r1[1].val[0]), vreinterpret_u32_u16(r1[5].val[0])); r2[3] = vzip_u32(vreinterpret_u32_u16(r1[1].val[1]), vreinterpret_u32_u16(r1[5].val[1])); r2[4] = vzip_u32(vreinterpret_u32_u16(r1[2].val[0]), vreinterpret_u32_u16(r1[6].val[0])); r2[5] = vzip_u32(vreinterpret_u32_u16(r1[2].val[1]), vreinterpret_u32_u16(r1[6].val[1])); r2[6] = vzip_u32(vreinterpret_u32_u16(r1[3].val[0]), vreinterpret_u32_u16(r1[7].val[0])); r2[7] = vzip_u32(vreinterpret_u32_u16(r1[3].val[1]), vreinterpret_u32_u16(r1[7].val[1])); /* Store the results to the destination vector */ vst1_u8(dest + k * 8 + 0 * total_elements, vreinterpret_u8_u32(r2[0].val[0])); vst1_u8(dest + k * 8 + 1 * total_elements, vreinterpret_u8_u32(r2[0].val[1])); vst1_u8(dest + k * 8 + 2 * total_elements, vreinterpret_u8_u32(r2[1].val[0])); vst1_u8(dest + k * 8 + 3 * total_elements, vreinterpret_u8_u32(r2[1].val[1])); vst1_u8(dest + k * 8 + 4 * total_elements, vreinterpret_u8_u32(r2[2].val[0])); vst1_u8(dest + k * 8 + 5 * total_elements, vreinterpret_u8_u32(r2[2].val[1])); vst1_u8(dest + k * 8 + 6 * total_elements, vreinterpret_u8_u32(r2[3].val[0])); vst1_u8(dest + k * 8 + 7 * total_elements, vreinterpret_u8_u32(r2[3].val[1])); vst1_u8(dest + k * 8 + 8 * total_elements, vreinterpret_u8_u32(r2[4].val[0])); vst1_u8(dest + k * 8 + 9 * total_elements, vreinterpret_u8_u32(r2[4].val[1])); vst1_u8(dest + k * 8 + 10 * total_elements, vreinterpret_u8_u32(r2[5].val[0])); vst1_u8(dest + k * 8 + 11 * total_elements, vreinterpret_u8_u32(r2[5].val[1])); vst1_u8(dest + k * 8 + 12 * total_elements, vreinterpret_u8_u32(r2[6].val[0])); vst1_u8(dest + k * 8 + 13 * total_elements, vreinterpret_u8_u32(r2[6].val[1])); vst1_u8(dest + k * 8 + 14 * total_elements, vreinterpret_u8_u32(r2[7].val[0])); vst1_u8(dest + k * 8 + 15 * total_elements, vreinterpret_u8_u32(r2[7].val[1])); } } /* Routine optimized for unshuffling a buffer for a type size of 2 bytes. */ static void unshuffle2_neon(uint8_t *const dest, const uint8_t *const src, const size_t vectorizable_elements, const size_t total_elements) { size_t i, k; static const size_t bytesoftype = 2; uint8x16x2_t r0; for (i = 0, k = 0; i < vectorizable_elements * bytesoftype; i += 32, k++) { /* Load 32 bytes to the structure r0 */ r0.val[0] = vld1q_u8(src + total_elements * 0 + k * 16); r0.val[1] = vld1q_u8(src + total_elements * 1 + k * 16); /* Store (with permutation) the results in the destination vector */ vst2q_u8(dest + k * 32, r0); } } /* Routine optimized for unshuffling a buffer for a type size of 4 bytes. */ static void unshuffle4_neon(uint8_t *const dest, const uint8_t *const src, const size_t vectorizable_elements, const size_t total_elements) { size_t i, k; static const size_t bytesoftype = 4; uint8x16x4_t r0; for (i = 0, k = 0; i < vectorizable_elements * bytesoftype; i += 64, k++) { /* load 64 bytes to the structure r0 */ r0.val[0] = vld1q_u8(src + total_elements * 0 + k * 16); r0.val[1] = vld1q_u8(src + total_elements * 1 + k * 16); r0.val[2] = vld1q_u8(src + total_elements * 2 + k * 16); r0.val[3] = vld1q_u8(src + total_elements * 3 + k * 16); /* Store (with permutation) the results in the destination vector */ vst4q_u8(dest + k * 64, r0); } } /* Routine optimized for unshuffling a buffer for a type size of 8 bytes. */ static void unshuffle8_neon(uint8_t *const dest, const uint8_t *const src, const size_t vectorizable_elements, const size_t total_elements) { size_t i, k; static const size_t bytesoftype = 8; uint8x8x2_t r0[4]; uint16x4x2_t r1[4]; uint32x2x2_t r2[4]; for (i = 0, k = 0; i < vectorizable_elements * bytesoftype; i += 64, k++) { /* Load and interleave groups of 8 bytes (64 bytes) to the structure r0 */ r0[0] = vzip_u8(vld1_u8(src + 0 * total_elements + k * 8), vld1_u8(src + 1 * total_elements + k * 8)); r0[1] = vzip_u8(vld1_u8(src + 2 * total_elements + k * 8), vld1_u8(src + 3 * total_elements + k * 8)); r0[2] = vzip_u8(vld1_u8(src + 4 * total_elements + k * 8), vld1_u8(src + 5 * total_elements + k * 8)); r0[3] = vzip_u8(vld1_u8(src + 6 * total_elements + k * 8), vld1_u8(src + 7 * total_elements + k * 8)); /* Interleave 16 bytes */ r1[0] = vzip_u16(vreinterpret_u16_u8(r0[0].val[0]), vreinterpret_u16_u8(r0[1].val[0])); r1[1] = vzip_u16(vreinterpret_u16_u8(r0[0].val[1]), vreinterpret_u16_u8(r0[1].val[1])); r1[2] = vzip_u16(vreinterpret_u16_u8(r0[2].val[0]), vreinterpret_u16_u8(r0[3].val[0])); r1[3] = vzip_u16(vreinterpret_u16_u8(r0[2].val[1]), vreinterpret_u16_u8(r0[3].val[1])); /* Interleave 32 bytes */ r2[0] = vzip_u32(vreinterpret_u32_u16(r1[0].val[0]), vreinterpret_u32_u16(r1[2].val[0])); r2[1] = vzip_u32(vreinterpret_u32_u16(r1[0].val[1]), vreinterpret_u32_u16(r1[2].val[1])); r2[2] = vzip_u32(vreinterpret_u32_u16(r1[1].val[0]), vreinterpret_u32_u16(r1[3].val[0])); r2[3] = vzip_u32(vreinterpret_u32_u16(r1[1].val[1]), vreinterpret_u32_u16(r1[3].val[1])); /* Store the results in the destination vector */ vst1_u8(dest + i + 0 * 8, vreinterpret_u8_u32(r2[0].val[0])); vst1_u8(dest + i + 1 * 8, vreinterpret_u8_u32(r2[0].val[1])); vst1_u8(dest + i + 2 * 8, vreinterpret_u8_u32(r2[1].val[0])); vst1_u8(dest + i + 3 * 8, vreinterpret_u8_u32(r2[1].val[1])); vst1_u8(dest + i + 4 * 8, vreinterpret_u8_u32(r2[2].val[0])); vst1_u8(dest + i + 5 * 8, vreinterpret_u8_u32(r2[2].val[1])); vst1_u8(dest + i + 6 * 8, vreinterpret_u8_u32(r2[3].val[0])); vst1_u8(dest + i + 7 * 8, vreinterpret_u8_u32(r2[3].val[1])); } } /* Routine optimized for unshuffling a buffer for a type size of 16 bytes. */ static void unshuffle16_neon(uint8_t *const dest, const uint8_t *const src, const size_t vectorizable_elements, const size_t total_elements) { size_t i, k; static const size_t bytesoftype = 16; uint8x8x2_t r0[8]; uint16x4x2_t r1[8]; uint32x2x2_t r2[8]; for (i = 0, k = 0; i < vectorizable_elements * bytesoftype; i += 128, k++) { /* Load and interleave groups of 16 bytes (128 bytes) to the structure r0*/ r0[0] = vzip_u8(vld1_u8(src + k * 8 + 0 * total_elements), vld1_u8(src + k * 8 + 1 * total_elements)); r0[1] = vzip_u8(vld1_u8(src + k * 8 + 2 * total_elements), vld1_u8(src + k * 8 + 3 * total_elements)); r0[2] = vzip_u8(vld1_u8(src + k * 8 + 4 * total_elements), vld1_u8(src + k * 8 + 5 * total_elements)); r0[3] = vzip_u8(vld1_u8(src + k * 8 + 6 * total_elements), vld1_u8(src + k * 8 + 7 * total_elements)); r0[4] = vzip_u8(vld1_u8(src + k * 8 + 8 * total_elements), vld1_u8(src + k * 8 + 9 * total_elements)); r0[5] = vzip_u8(vld1_u8(src + k * 8 + 10 * total_elements), vld1_u8(src + k * 8 + 11 * total_elements)); r0[6] = vzip_u8(vld1_u8(src + k * 8 + 12 * total_elements), vld1_u8(src + k * 8 + 13 * total_elements)); r0[7] = vzip_u8(vld1_u8(src + k * 8 + 14 * total_elements), vld1_u8(src + k * 8 + 15 * total_elements)); /* Interleave 16 bytes */ r1[0] = vzip_u16(vreinterpret_u16_u8(r0[0].val[0]), vreinterpret_u16_u8(r0[1].val[0])); r1[1] = vzip_u16(vreinterpret_u16_u8(r0[0].val[1]), vreinterpret_u16_u8(r0[1].val[1])); r1[2] = vzip_u16(vreinterpret_u16_u8(r0[2].val[0]), vreinterpret_u16_u8(r0[3].val[0])); r1[3] = vzip_u16(vreinterpret_u16_u8(r0[2].val[1]), vreinterpret_u16_u8(r0[3].val[1])); r1[4] = vzip_u16(vreinterpret_u16_u8(r0[4].val[0]), vreinterpret_u16_u8(r0[5].val[0])); r1[5] = vzip_u16(vreinterpret_u16_u8(r0[4].val[1]), vreinterpret_u16_u8(r0[5].val[1])); r1[6] = vzip_u16(vreinterpret_u16_u8(r0[6].val[0]), vreinterpret_u16_u8(r0[7].val[0])); r1[7] = vzip_u16(vreinterpret_u16_u8(r0[6].val[1]), vreinterpret_u16_u8(r0[7].val[1])); /* Interleave 32 bytes */ r2[0] = vzip_u32(vreinterpret_u32_u16(r1[0].val[0]), vreinterpret_u32_u16(r1[2].val[0])); r2[1] = vzip_u32(vreinterpret_u32_u16(r1[0].val[1]), vreinterpret_u32_u16(r1[2].val[1])); r2[2] = vzip_u32(vreinterpret_u32_u16(r1[1].val[0]), vreinterpret_u32_u16(r1[3].val[0])); r2[3] = vzip_u32(vreinterpret_u32_u16(r1[1].val[1]), vreinterpret_u32_u16(r1[3].val[1])); r2[4] = vzip_u32(vreinterpret_u32_u16(r1[4].val[0]), vreinterpret_u32_u16(r1[6].val[0])); r2[5] = vzip_u32(vreinterpret_u32_u16(r1[4].val[1]), vreinterpret_u32_u16(r1[6].val[1])); r2[6] = vzip_u32(vreinterpret_u32_u16(r1[5].val[0]), vreinterpret_u32_u16(r1[7].val[0])); r2[7] = vzip_u32(vreinterpret_u32_u16(r1[5].val[1]), vreinterpret_u32_u16(r1[7].val[1])); /* Store the results in the destination vector */ vst1_u8(dest + i + 0 * 8, vreinterpret_u8_u32(r2[0].val[0])); vst1_u8(dest + i + 1 * 8, vreinterpret_u8_u32(r2[4].val[0])); vst1_u8(dest + i + 2 * 8, vreinterpret_u8_u32(r2[0].val[1])); vst1_u8(dest + i + 3 * 8, vreinterpret_u8_u32(r2[4].val[1])); vst1_u8(dest + i + 4 * 8, vreinterpret_u8_u32(r2[1].val[0])); vst1_u8(dest + i + 5 * 8, vreinterpret_u8_u32(r2[5].val[0])); vst1_u8(dest + i + 6 * 8, vreinterpret_u8_u32(r2[1].val[1])); vst1_u8(dest + i + 7 * 8, vreinterpret_u8_u32(r2[5].val[1])); vst1_u8(dest + i + 8 * 8, vreinterpret_u8_u32(r2[2].val[0])); vst1_u8(dest + i + 9 * 8, vreinterpret_u8_u32(r2[6].val[0])); vst1_u8(dest + i + 10 * 8, vreinterpret_u8_u32(r2[2].val[1])); vst1_u8(dest + i + 11 * 8, vreinterpret_u8_u32(r2[6].val[1])); vst1_u8(dest + i + 12 * 8, vreinterpret_u8_u32(r2[3].val[0])); vst1_u8(dest + i + 13 * 8, vreinterpret_u8_u32(r2[7].val[0])); vst1_u8(dest + i + 14 * 8, vreinterpret_u8_u32(r2[3].val[1])); vst1_u8(dest + i + 15 * 8, vreinterpret_u8_u32(r2[7].val[1])); } } /* Shuffle a block. This can never fail. */ void shuffle_neon(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *const _src, uint8_t *_dest) { int32_t vectorized_chunk_size = 1; if (bytesoftype == 2 || bytesoftype == 4) { vectorized_chunk_size = bytesoftype * 16; } else if (bytesoftype == 8 || bytesoftype == 16) { vectorized_chunk_size = bytesoftype * 8; } /* If the blocksize is not a multiple of both the typesize and the vector size, round the blocksize down to the next value which is a multiple of both. The vectorized shuffle can be used for that portion of the data, and the naive implementation can be used for the remaining portion. */ const int32_t vectorizable_bytes = blocksize - (blocksize % vectorized_chunk_size); const int32_t vectorizable_elements = vectorizable_bytes / bytesoftype; const int32_t total_elements = blocksize / bytesoftype; /* If the block size is too small to be vectorized, use the generic implementation. */ if (blocksize < vectorized_chunk_size) { shuffle_generic(bytesoftype, blocksize, _src, _dest); return; } /* Optimized shuffle implementations */ switch (bytesoftype) { case 2: shuffle2_neon(_dest, _src, vectorizable_elements, total_elements); break; case 4: shuffle4_neon(_dest, _src, vectorizable_elements, total_elements); break; case 8: shuffle8_neon(_dest, _src, vectorizable_elements, total_elements); break; case 16: shuffle16_neon(_dest, _src, vectorizable_elements, total_elements); break; default: /* Non-optimized shuffle */ shuffle_generic(bytesoftype, blocksize, _src, _dest); /* The non-optimized function covers the whole buffer, so we're done processing here. */ return; } /* If the buffer had any bytes at the end which couldn't be handled by the vectorized implementations, use the non-optimized version to finish them up. */ if (vectorizable_bytes < blocksize) { shuffle_generic_inline(bytesoftype, vectorizable_bytes, blocksize, _src, _dest); } } /* Unshuffle a block. This can never fail. */ void unshuffle_neon(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *const _src, uint8_t *_dest) { int32_t vectorized_chunk_size = 1; if (bytesoftype == 2 || bytesoftype == 4) { vectorized_chunk_size = bytesoftype * 16; } else if (bytesoftype == 8 || bytesoftype == 16) { vectorized_chunk_size = bytesoftype * 8; } /* If the blocksize is not a multiple of both the typesize and the vector size, round the blocksize down to the next value which is a multiple of both. The vectorized unshuffle can be used for that portion of the data, and the naive implementation can be used for the remaining portion. */ const int32_t vectorizable_bytes = blocksize - (blocksize % vectorized_chunk_size); const int32_t vectorizable_elements = vectorizable_bytes / bytesoftype; const int32_t total_elements = blocksize / bytesoftype; /* If the block size is too small to be vectorized, use the generic implementation. */ if (blocksize < vectorized_chunk_size) { unshuffle_generic(bytesoftype, blocksize, _src, _dest); return; } /* Optimized unshuffle implementations */ switch (bytesoftype) { case 2: unshuffle2_neon(_dest, _src, vectorizable_elements, total_elements); break; case 4: unshuffle4_neon(_dest, _src, vectorizable_elements, total_elements); break; case 8: unshuffle8_neon(_dest, _src, vectorizable_elements, total_elements); break; case 16: unshuffle16_neon(_dest, _src, vectorizable_elements, total_elements); break; default: /* Non-optimized unshuffle */ unshuffle_generic(bytesoftype, blocksize, _src, _dest); /* The non-optimized function covers the whole buffer, so we're done processing here. */ return; } /* If the buffer had any bytes at the end which couldn't be handled by the vectorized implementations, use the non-optimized version to finish them up. */ if (vectorizable_bytes < blocksize) { unshuffle_generic_inline(bytesoftype, vectorizable_bytes, blocksize, _src, _dest); } } #endif /* defined(__ARM_NEON) */ zmat-0.9.9/src/blosc2/blosc/shuffle-neon.h0000644000175200007730000000207114515254731020556 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) Note: Adapted for NEON by Lucian Marc. See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /* NEON-accelerated shuffle/unshuffle routines. */ #ifndef SHUFFLE_NEON_H #define SHUFFLE_NEON_H #include "blosc2/blosc2-common.h" #ifdef __cplusplus extern "C" { #endif /** NEON-accelerated shuffle routine. */ BLOSC_NO_EXPORT void shuffle_neon(const int32_t bytesoftype, const int32_t blocksize, const uint8_t* const _src, uint8_t* const _dest); /** NEON-accelerated unshuffle routine. */ BLOSC_NO_EXPORT void unshuffle_neon(const int32_t bytesoftype, const int32_t blocksize, const uint8_t *_src, uint8_t *_dest); #ifdef __cplusplus } #endif #endif /* SHUFFLE_NEON_H */ zmat-0.9.9/src/blosc2/blosc/bitshuffle-generic.h0000644000175200007730000001453014515254731021735 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /* Generic (non-hardware-accelerated) shuffle/unshuffle routines. These are used when hardware-accelerated functions aren't available for a particular platform; they are also used by the hardware- accelerated functions to handle any remaining elements in a block which isn't a multiple of the hardware's vector size. */ #ifndef BITSHUFFLE_GENERIC_H #define BITSHUFFLE_GENERIC_H #include #include #ifdef __cplusplus extern "C" { #endif /* Macros. */ #define CHECK_MULT_EIGHT(n) if ((n) % 8) return -80; #define MIN(X,Y) ((X) < (Y) ? (X) : (Y)) #define MAX(X,Y) ((X) > (Y) ? (X) : (Y)) #define CHECK_ERR(count) if ((count) < 0) { return count; } /* ---- Worker code not requiring special instruction sets. ---- * * The following code does not use any x86 specific vectorized instructions * and should compile on any machine * */ /* Transpose 8x8 bit array packed into a single quadword *x*. * *t* is workspace. */ #define TRANS_BIT_8X8(x, t) { \ (t) = ((x) ^ ((x) >> 7)) & 0x00AA00AA00AA00AALL; \ (x) = (x) ^ (t) ^ ((t) << 7); \ (t) = ((x) ^ ((x) >> 14)) & 0x0000CCCC0000CCCCLL; \ (x) = (x) ^ (t) ^ ((t) << 14); \ (t) = ((x) ^ ((x) >> 28)) & 0x00000000F0F0F0F0LL; \ (x) = (x) ^ (t) ^ ((t) << 28); \ } /* Transpose 8x8 bit array along the diagonal from upper right to lower left */ #define TRANS_BIT_8X8_BE(x, t) { \ (t) = ((x) ^ ((x) >> 9)) & 0x0055005500550055LL; \ (x) = (x) ^ (t) ^ ((t) << 9); \ (t) = ((x) ^ ((x) >> 18)) & 0x0000333300003333LL; \ (x) = (x) ^ (t) ^ ((t) << 18); \ (t) = ((x) ^ ((x) >> 36)) & 0x000000000F0F0F0FLL; \ (x) = (x) ^ (t) ^ ((t) << 36); \ } /* Transpose of an array of arbitrarily typed elements. */ #define TRANS_ELEM_TYPE(in, out, lda, ldb, type_t) { \ type_t* in_type = (type_t*) (in); \ type_t* out_type = (type_t*) (out); \ size_t ii, jj, kk; \ for (ii = 0; ii + 7 < (lda); ii += 8) { \ for (jj = 0; jj < (ldb); jj++) { \ for (kk = 0; kk < 8; kk++) { \ out_type[jj*(lda) + ii + kk] = \ in_type[ii*(ldb) + kk * (ldb) + jj]; \ } \ } \ } \ for (ii = (lda) - (lda) % 8; ii < (lda); ii ++) { \ for (jj = 0; jj < (ldb); jj++) { \ out_type[jj*(lda) + ii] = in_type[ii*(ldb) + jj]; \ } \ } \ } /* Private functions */ BLOSC_NO_EXPORT int64_t bshuf_trans_byte_elem_remainder(const void* in, void* out, const size_t size, const size_t elem_size, const size_t start); BLOSC_NO_EXPORT int64_t bshuf_trans_byte_elem_scal(const void* in, void* out, const size_t size, const size_t elem_size); BLOSC_NO_EXPORT int64_t bshuf_trans_bit_byte_remainder(const void* in, void* out, const size_t size, const size_t elem_size, const size_t start_byte); BLOSC_NO_EXPORT int64_t bshuf_trans_elem(const void* in, void* out, const size_t lda, const size_t ldb, const size_t elem_size); BLOSC_NO_EXPORT int64_t bshuf_trans_bitrow_eight(const void* in, void* out, const size_t size, const size_t elem_size); BLOSC_NO_EXPORT int64_t bshuf_shuffle_bit_eightelem_scal(const void* in, void* out, const size_t size, const size_t elem_size); BLOSC_NO_EXPORT int64_t bshuf_trans_byte_bitrow_scal(const void* in, void* out, const size_t size, const size_t elem_size); /* Bitshuffle the data. * * Transpose the bits within elements. * * Parameters * ---------- * in : input buffer, must be of size * elem_size bytes * out : output buffer, must be of size * elem_size bytes * size : number of elements in input * elem_size : element size of typed data * tmp_buffer : temporary buffer with the same `size` than `in` and `out` * * Returns * ------- * nothing -- this cannot fail * */ BLOSC_NO_EXPORT int64_t bshuf_trans_bit_elem_scal(const void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf); /* Unshuffle bitshuffled data. * * Untranspose the bits within elements. * * To properly unshuffle bitshuffled data, *size* and *elem_size* must * match the parameters used to shuffle the data. * * Parameters * ---------- * in : input buffer, must be of size * elem_size bytes * out : output buffer, must be of size * elem_size bytes * size : number of elements in input * elem_size : element size of typed data * tmp_buffer : temporary buffer with the same `size` than `in` and `out` * * Returns * ------- * nothing -- this cannot fail * */ BLOSC_NO_EXPORT int64_t bshuf_untrans_bit_elem_scal(const void* in, void* out, const size_t size, const size_t elem_size, void* tmp_buf); #ifdef __cplusplus } #endif #endif /* BITSHUFFLE_GENERIC_H */ zmat-0.9.9/src/blosc2/plugins/0000755000175200007730000000000014515254731016373 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/plugins/plugin_utils.c0000644000175200007730000000515514515254731021263 0ustar rlaboissrlaboiss/* Copyright (C) 2021 The Blosc Developers http://blosc.org License: BSD 3-Clause (see LICENSE.txt) */ #include #include "blosc2.h" #include "plugin_utils.h" #define BLOSC_PLUGINS_MAX_DIM 8 void swap_store(void *dest, const void *pa, int size) { uint8_t *pa_ = (uint8_t *) pa; uint8_t *pa2_ = malloc((size_t) size); int i = 1; /* for big/little endian detection */ char *p = (char *) &i; if (p[0] == 1) { /* little endian */ switch (size) { case 8: pa2_[0] = pa_[7]; pa2_[1] = pa_[6]; pa2_[2] = pa_[5]; pa2_[3] = pa_[4]; pa2_[4] = pa_[3]; pa2_[5] = pa_[2]; pa2_[6] = pa_[1]; pa2_[7] = pa_[0]; break; case 4: pa2_[0] = pa_[3]; pa2_[1] = pa_[2]; pa2_[2] = pa_[1]; pa2_[3] = pa_[0]; break; case 2: pa2_[0] = pa_[1]; pa2_[1] = pa_[0]; break; case 1: pa2_[0] = pa_[0]; break; default: fprintf(stderr, "Unhandled nitems: %d\n", size); } } memcpy(dest, pa2_, size); free(pa2_); } int32_t deserialize_meta(uint8_t *smeta, int32_t smeta_len, int8_t *ndim, int64_t *shape, int32_t *chunkshape, int32_t *blockshape) { BLOSC_UNUSED_PARAM(smeta_len); uint8_t *pmeta = smeta; // Check that we have an array with 5 entries (version, ndim, shape, chunkshape, blockshape) pmeta += 1; // version entry int8_t version = (int8_t)pmeta[0]; // positive fixnum (7-bit positive integer) BLOSC_UNUSED_PARAM(version); pmeta += 1; // ndim entry *ndim = (int8_t)pmeta[0]; int8_t ndim_aux = *ndim; // positive fixnum (7-bit positive integer) pmeta += 1; // shape entry // Initialize to ones, as required by Caterva for (int i = 0; i < BLOSC_PLUGINS_MAX_DIM; i++) shape[i] = 1; pmeta += 1; for (int8_t i = 0; i < ndim_aux; i++) { pmeta += 1; swap_store(shape + i, pmeta, sizeof(int64_t)); pmeta += sizeof(int64_t); } // chunkshape entry // Initialize to ones, as required by Caterva for (int i = 0; i < BLOSC_PLUGINS_MAX_DIM; i++) chunkshape[i] = 1; pmeta += 1; for (int8_t i = 0; i < ndim_aux; i++) { pmeta += 1; swap_store(chunkshape + i, pmeta, sizeof(int32_t)); pmeta += sizeof(int32_t); } // blockshape entry // Initialize to ones, as required by Caterva for (int i = 0; i < BLOSC_PLUGINS_MAX_DIM; i++) blockshape[i] = 1; pmeta += 1; for (int8_t i = 0; i < ndim_aux; i++) { pmeta += 1; swap_store(blockshape + i, pmeta, sizeof(int32_t)); pmeta += sizeof(int32_t); } int32_t slen = (int32_t)(pmeta - smeta); return slen; } zmat-0.9.9/src/blosc2/plugins/plugin_utils.h0000644000175200007730000000050414515254731021261 0ustar rlaboissrlaboiss/* Copyright (C) 2021 The Blosc Developers http://blosc.org License: BSD 3-Clause (see LICENSE.txt) */ void swap_store(void *dest, const void *pa, int size); int32_t deserialize_meta(uint8_t *smeta, int32_t smeta_len, int8_t *ndim, int64_t *shape, int32_t *chunkshape, int32_t *blockshape); zmat-0.9.9/src/blosc2/plugins/README.md0000644000175200007730000001505314515254731017656 0ustar rlaboissrlaboissPlugins registry for Blosc users ================================ Blosc has a tradition of supporting different filters and codecs for compressing data, and it was up to the user to choose one or another depending on his needs. However, it is clear that there will always be scenarios where a more richer variety of them could be useful. So the Blosc Development Team has set new goals: 1) Implement a way for users to locally register filters and codecs so that they can use them in their setup at will. 2) Setup a central registry so that *other* users can make use of these filters and codecs without intefering with other ones that have been created by other users. As a bonus, those codecs and filters accepted in the central registry and meeting the quality standards defined in these guidelines will be distributed *inside* the C-Blosc2 library, allowing a much easier way for others to use them: install C-Blosc2 library and you are all set. Of course, to achieve such a status, plugins will require a careful testing process described below. Plugin types -------------- The plugins that are registered in the repository can be codecs or filters. A **codec** is a program able to compress and decompress a digital data stream with the objective of reduce dataset size to enable a faster transmission of data. Some of the codecs used by Blosc are e.g. *BLOSCLZ*, *LZ4* and *ZSTANDARD*. A **filter** is a program that reorders the data without changing its size, so that the initial and final size are equal. A filter consists of encoder and decoder. Filter encoder is applied before using the codec compressor (or codec encoder) in order to make data easier to compress and filter decoder is used after codec decompressor (or codec decoder) to recover the original data arrangement. Some filters actually used by Blosc are e.g. *SHUFFLE*, which rearranges data based on the typesize, or *TRUNC*, which zeroes mantissa bits so as to reduce the precision of (floating point) data, and hence, increase the compression ratio. Here it is an example on how the compression process goes: -------------------- filter encoder ------------------- codec encoder ------- | src | -----------> | tmp | ----------> | c_src | -------------------- ------------------- ------- And the decompression process: -------- codec decoder ------------------- filter decoder ------------------- | c_src | -----------> | tmp | ----------> | src | -------- ------------------- ------------------- Moreover, during the pipeline process you can use even 6 different filters ordered as you prefer. Blosc global registered plugins vs user registered plugins ---------------------------------------------------------- **Blosc global registered plugins** are official Blosc plugins that have passed through a selection process and have been recognised by the Blosc Development Team. These plugins are available for everybody in the C-Blosc2 GitHub repository and users can install them anytime. **User registered plugins** are plugins that users register locally and they can use them in the same way as in the examples `urcodecs.c` and `urfilters.c`. If you only want to use a plugin on your own devices you can just register it as a user registered plugin with an ID between *BLOSC2_USER_REGISTERED_FILTERS_START* and *BLOSC2_USER_REGISTERED_FILTERS_STOP*. Otherwise, if you think that your plugin could be useful for the community you can apply for registering it as an official Blosc plugin following the next steps. Requirements for registering plugins ------------------------------------ For users wanting to register a new codec or filter, there are some requirements that their code must satisfy: - First, the plugin code must be **developed in C**, have a relatively small footprint and meet decent quality code standards. - Second, users must develop a test suite which prove that the plugin works correctly. Finally, even if these requirements are completely satisfied, it is not guaranteed that the plugin will be useful or contribute something different than the existing ones, so the Blosc development team has the final say and will decide if a plugin is to be accepted or not. Steps ----- 1. First, tests must be provided and be passing. **It is completely mandatory and necessary to add these lines to `main()` in each test to make plugins machinery work:** - `blosc2_init()` at the beginning - `blosc2_destroy()` in the end 2. Then, the user must make a fork of the C-Blosc2 Github repository, adding a new folder within the plugin sources to the path `plugins/codecs` or `plugins/filters` depending on the plugin type. 3. Furthermore, a text file named `README.rst` must be provided where it is explained: * The plugin motivation, why and for what purpose was the plugin created. * How to use the plugin. * What does the plugin do and how it works. * The advantages and disadvantages of the plugin compared to the rest. 4. To register a plugin the user must choose a plugin ID between *BLOSC2_GLOBAL_REGISTERED_FILTERS_START* and *BLOSC2_GLOBAL_REGISTERED_FILTERS_STOP* and write it at `include/blosc2/codecs-registry.h` or `include/blosc2/filters-registry.h` depending on the plugin type. Then, you have to edit `include/blosc2/codecs-registry.c`or `include/blosc2/filters-registry.c` in the next way: At the top it must be added `#include "plugin_folder/plugin_header.h"`, and into the register function you must follow the same steps that were done for the existing plugins. 5. Finally, the Blosc development team will carry out the evaluation process (probably via a votation process, with the BDFL having the last say in case of the team is undecided) so as to decide whether the plugin is useful and hence, candidate to be integrated into the C-Blosc2 source code distribution. In case of a negative decision, the original author will be informed, together with a series of advices for starting a new iteration if desired. Examples -------- In the `plugins/` directory there can be found different examples of codecs and filters available as plugins that can be used in the compression process, and that can be used as an example on how to implement plugins that can make into C-Blosc2. Some of these examples are `ndlz`, `ndcell` or `ndmean`. Thanks ------ We would like to express our gratitude to the NumFOCUS Foundation so as to provide the funds to implement this functionality. zmat-0.9.9/src/blosc2/plugins/CMakeLists.txt0000644000175200007730000000021514515254731021131 0ustar rlaboissrlaboissadd_subdirectory(codecs) add_subdirectory(filters) set(SOURCES ${SOURCES} ../plugins/plugin_utils.c ../plugins/plugin_utils.h PARENT_SCOPE) zmat-0.9.9/src/blosc2/plugins/filters/0000755000175200007730000000000014515254731020043 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/plugins/filters/ndmean/0000755000175200007730000000000014515254731021305 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/plugins/filters/ndmean/README.md0000644000175200007730000000532414515254731022570 0ustar rlaboissrlaboissNDMEAN: a multidimensional filter for lossy compression ============================================================================= Given an n-dim array or matrix, *NDMEAN* is a filter based on *NDCELL* (https://github.com/Blosc/c-blosc2/tree/main/plugins/filters/ndcell/README.rst) that not only divides data into multidimensional cells so that a codec compress cell by cell, but also replaces each cell elements by the cell mean. Plugin motivation -------------------- *NDMEAN* was created in order to add another depth level to *NDCELL*, taking advantage of multidimensionality and making each cell items equal to earn better compression ratios for the codecs. Plugin usage ------------------- The codec consists of an encoder called *ndmean_encoder()* to reorder data and a decoder called *ndmean_decoder()* to recover the original order of data but not the original data (lossy compression). The parameters used by *NDMEAN* are the ones specified in the *blosc2_filter* structure of *blosc2.h*. Furthermore, since *NDMEAN* goes through dataset blocks dividing them into fixed size cells, user must specify the parameter meta as the cellshape, so if in a 3-dim dataset user specifies meta = 4, then cellshape will be 4x4x4. If user tries to use other value for meta, the codec will return an error value. NDMEAN only works for float or double datasets (depending on the typesize), so if it is used for a different data type it may return an error code. Plugin behaviour ------------------- This filter is meant first to order the data so that when the dataset is traversed the elements of a cell are all in a row. Furthermore, *NDMEAN* replaces cell elements by the cell mean. ------------------------ ----------------------------- | 0 1 | 1 2 | 2 3 | 3 5 | | 1 1 1 1 1 1 | 2 2 2 2 2 2 | | 2 0 | 3 1 | 4 2 | 3 5 | ----------------------------- | 1 2 | 2 3 | 3 4 | 5 3 | NDMEAN encoder | 3 3 3 3 3 3 | 4 4 4 4 4 4 | ------------------------- ------> ----------------------------- | 1 3 | 1 2 | 2 4 | 4 8 | | 2 2 2 2 2 2 | 1 1 1 1 1 1 | | 3 1 | 0 1 | 1 5 | 7 5 | ----------------------------- | 1 3 | 0 2 | 0 6 | 4 8 | | 3 3 3 3 3 3 | 6 6 6 6 6 6 | ------------------------- ----------------------------- Advantages and disadvantages ------------------------------ The particularity of *NDMEAN* is that this filter not only considers datasets multidimensionality and takes advantage of it instead of processing all data as serial, but also makes cell data easier to compress. The main disadvantage of *NDMEAN* is that it changes the original data and can not recover it (lossy compression). zmat-0.9.9/src/blosc2/plugins/filters/ndmean/CMakeLists.txt0000644000175200007730000000227414515254731024052 0ustar rlaboissrlaboiss# sources set(SOURCES ${SOURCES} ../plugins/filters/ndmean/ndmean.c ../plugins/filters/ndmean/ndmean.h PARENT_SCOPE) if(BUILD_TESTS) # targets add_executable(test_ndmean_repart test_ndmean_repart.c) add_executable(test_ndmean_mean test_ndmean_mean.c) # Define the BLOSC_TESTING symbol so normally-hidden functions # aren't hidden from the view of the test programs. set_property( TARGET test_ndmean_mean APPEND PROPERTY COMPILE_DEFINITIONS BLOSC_TESTING) set_property( TARGET test_ndmean_repart APPEND PROPERTY COMPILE_DEFINITIONS BLOSC_TESTING) target_link_libraries(test_ndmean_repart blosc_testing) target_link_libraries(test_ndmean_mean blosc_testing) # tests add_test(NAME test_plugin_ndmean_repart COMMAND ${CMAKE_CROSSCOMPILING_EMULATOR} $) add_test(NAME test_plugin_ndmean_mean COMMAND ${CMAKE_CROSSCOMPILING_EMULATOR} $) # Copy test files file(GLOB TESTS_DATA *.caterva) foreach (data ${TESTS_DATA}) file(COPY ${data} DESTINATION ${CMAKE_CURRENT_BINARY_DIR}/) endforeach(data) endif() zmat-0.9.9/src/blosc2/plugins/filters/ndmean/test_ndmean_repart.c0000644000175200007730000001377214515254731025341 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. Test program demonstrating use of the Blosc filter from C code. To compile this program: $ gcc -O test_ndmean_repart.c -o test_ndmean_repart -lblosc2 To run: $ ./test_ndmean_repart Blosc version info: 2.0.0a6.dev ($Date:: 2018-05-18 #$) Using 1 thread Using ZSTD compressor Successful roundtrip! Compression: 41472 -> 1015 (40.9x) rand: 40457 obtained Successful roundtrip! Compression: 1792 -> 323 (5.5x) same_cells: 1469 obtained Successful roundtrip! Compression: 16128 -> 767 (21.0x) some_matches: 15361 obtained **********************************************************************/ #include #include "ndmean.h" #include #include #include "blosc2/filters-registry.h" #define EPSILON (float) (1e-5) static bool is_close(double d1, double d2) { double aux = 1; if (fabs(d1) < fabs(d2)) { if (fabs(d1) > 0) { aux = fabs(d2); } } else { if (fabs(d2) > 0) { aux = fabs(d1); } } return fabs(d1 - d2) < aux * EPSILON; } static int test_ndmean(blosc2_schunk* schunk) { int32_t typesize = schunk->typesize; int64_t nchunks = schunk->nchunks; int32_t chunksize = (int32_t) (schunk->chunksize); // int isize = (int) array->extchunknitems * typesize; uint8_t *data_in = malloc(chunksize); int decompressed; int64_t csize; int64_t dsize; int64_t csize_f = 0; uint8_t *data_out = malloc(chunksize + BLOSC2_MAX_OVERHEAD); uint8_t *data_dest = malloc(chunksize); /* Create a context for compression */ blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_ALWAYS_SPLIT; cparams.typesize = typesize; cparams.compcode = BLOSC_ZSTD; cparams.filters[4] = BLOSC_FILTER_NDMEAN; cparams.filters_meta[4] = 4; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_SHUFFLE; cparams.clevel = 9; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; dparams.schunk = schunk; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* printf("\n data \n"); for (int i = 0; i < nbytes; i++) { printf("%u, ", data2[i]); } */ /* Compress with clevel=5 and shuffle active */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, data_out, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } csize_f += csize; /* Decompress */ dsize = blosc2_decompress_ctx(dctx, data_out, chunksize + BLOSC2_MAX_OVERHEAD, data_dest, chunksize); if (dsize <= 0) { printf("Decompression error. Error code: %" PRId64 "\n", dsize); return (int) dsize; } switch (typesize) { case 4: for (int i = 0; i < chunksize / typesize; i++) { if (!is_close(((float *) data_in)[i], ((float *) data_dest)[i])) { printf("i: %d, data %.9f, dest %.9f", i, ((float *) data_in)[i], ((float *) data_dest)[i]); printf("\n Decompressed data differs from original!\n"); return -1; } } break; case 8: for (int i = 0; i < chunksize / typesize; i++) { if (!is_close(((double *) data_in)[i], ((double *) data_dest)[i])) { printf("i: %d, data %.9f, dest %.9f", i, ((double *) data_in)[i], ((double *) data_dest)[i]); printf("\n Decompressed data differs from original!\n"); return -1; } } break; default : break; } } csize_f = csize_f / nchunks; free(data_in); free(data_out); free(data_dest); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); printf("Successful roundtrip!\n"); printf("Compression: %d -> %" PRId64 " (%.1fx)\n", chunksize, csize_f, (1. * chunksize) / (double)csize_f); return (int) (chunksize - csize_f); } int rand_() { blosc2_schunk *schunk = blosc2_schunk_open("example_ndmean_repart_rand.caterva"); /* Run the test. */ int result = test_ndmean(schunk); blosc2_schunk_free(schunk); return result; } int same_cells() { blosc2_schunk *schunk = blosc2_schunk_open("example_ndmean_repart_same_cells.caterva"); /* Run the test. */ int result = test_ndmean(schunk); blosc2_schunk_free(schunk); return result; } int some_matches() { blosc2_schunk *schunk = blosc2_schunk_open("example_ndmean_repart_some_matches.caterva"); /* Run the test. */ int result = test_ndmean(schunk); blosc2_schunk_free(schunk); return result; } int main(void) { int result; blosc2_init(); result = rand_(); printf("rand: %d obtained \n \n", result); result = same_cells(); printf("same_cells: %d obtained \n \n", result); result = some_matches(); printf("some_matches: %d obtained \n \n", result); blosc2_destroy(); return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/plugins/filters/ndmean/ndmean.h0000644000175200007730000000134714515254731022725 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef CATERVA_NDMEAN_H #define CATERVA_NDMEAN_H #include "blosc2.h" #define NDMEAN_MAX_DIM 8 int ndmean_encoder(const uint8_t* input, uint8_t* output, int32_t length, uint8_t meta, blosc2_cparams* cparams); int ndmean_decoder(const uint8_t* input, uint8_t* output, int32_t length, uint8_t meta, blosc2_dparams* dparams); #endif //CATERVA_NDMEAN_H zmat-0.9.9/src/blosc2/plugins/filters/ndmean/test_ndmean_mean.c0000644000175200007730000002055014515254731024754 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. Test program demonstrating use of the Blosc filter from C code. To compile this program: $ gcc -O test_ndmean_mean.c -o test_ndmean_mean -lblosc2 To run: $ ./test_ndmean_mean Blosc version info: 2.0.0a6.dev ($Date:: 2018-05-18 #$) Using 1 thread Using ZSTD compressor Successful roundtrip! Compression: 256 -> 137 (1.9x) 2_rows_matches: 119 obtained Successful roundtrip! Compression: 128 -> 110 (1.2x) same_cells: 18 obtained Successful roundtrip! Compression: 448 -> 259 (1.7x) some_matches: 189 obtained **********************************************************************/ #include #include "ndmean.h" #include #include #include "blosc2/filters-registry.h" #include "../plugins/plugin_utils.h" #define EPSILON (float) (1) static bool is_close(double d1, double d2) { double aux = 1; if (fabs(d1) < fabs(d2)) { if (fabs(d1) > 0) { aux = fabs(d2); } } else { if (fabs(d2) > 0) { aux = fabs(d1); } } return fabs(d1 - d2) < aux * EPSILON; } static int test_ndmean(blosc2_schunk* schunk) { int8_t ndim; int64_t* shape = malloc(8 * sizeof(int64_t)); int32_t* chunkshape = malloc(8 * sizeof(int32_t)); int32_t* blockshape = malloc(8 * sizeof(int32_t)); uint8_t cellshape = 4; uint8_t* smeta; int32_t smeta_len; if (blosc2_meta_get(schunk, "caterva", &smeta, &smeta_len) < 0) { printf("Blosc error"); return 0; } deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape); free(smeta); if (ndim != 1) { fprintf(stderr, "This test only works for ndim = 1"); return -1; } int32_t typesize = schunk->typesize; int64_t nchunks = schunk->nchunks; int32_t chunksize = (int32_t) (schunk->chunksize); // int isize = (int) array->extchunknitems * typesize; uint8_t *data_in = malloc(chunksize); int decompressed; int64_t csize; int64_t dsize; int64_t csize_f = 0; uint8_t *data_out = malloc(chunksize + BLOSC2_MAX_OVERHEAD); uint8_t *data_dest = malloc(chunksize); /* Create a context for compression */ blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_ALWAYS_SPLIT; cparams.typesize = typesize; cparams.compcode = BLOSC_ZSTD; cparams.filters[4] = BLOSC_FILTER_NDMEAN; cparams.filters_meta[4] = cellshape; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_SHUFFLE; cparams.clevel = 9; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; dparams.schunk = schunk; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); double cell_mean; for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* printf("\n data \n"); for (int i = 0; i < nbytes; i++) { printf("%u, ", data2[i]); } */ /* Compress with clevel=5 and shuffle active */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, data_out, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } csize_f += csize; /* Decompress */ dsize = blosc2_decompress_ctx(dctx, data_out, chunksize + BLOSC2_MAX_OVERHEAD, data_dest, chunksize); if (dsize <= 0) { printf("Decompression error. Error code: %" PRId64 "\n", dsize); return (int) dsize; } /* printf("\n data_dest: \n"); for (int i = 0; i < chunksize / typesize; i++) { if (typesize == 4) { printf("%f, ", ((float *) data_dest)[i]); } else if (typesize == 8) { printf("%f, ", ((double *) data_dest)[i]); } } */ int chunk_shape = chunkshape[0]; if (ci == nchunks - 1) { chunk_shape = (int) (shape[0] % chunkshape[0]); } int nblocks = (chunk_shape + blockshape[0] - 1) / blockshape[0]; for (int bi = 0; bi < nblocks; bi++) { int block_shape = blockshape[0]; if (bi == nblocks - 1) { block_shape = chunk_shape % blockshape[0]; } int ncells = (block_shape + cellshape - 1) / cellshape; for (int cei = 0; cei < ncells; cei++) { int ind = bi * blockshape[0] + cei * cellshape; cell_mean = 0; int cell_shape = cellshape; if (cei == ncells - 1) { cell_shape = block_shape % cellshape; } switch (typesize) { case 4: for (int i = 0; i < cell_shape; i++) { cell_mean += ((float *) data_in)[ind + i]; } cell_mean /= cell_shape; for (int i = 0; i < cell_shape; i++) { if (!is_close(cell_mean, ((float *) data_dest)[ind + i])) { printf("i: %d, cell_mean %.9f, dest %.9f", ind + i, cell_mean, ((float *) data_dest)[ind + i]); printf("\n Decompressed data differs from original!\n"); return -1; } } break; case 8: for (int i = 0; i < cell_shape; i++) { cell_mean += ((double *) data_in)[ind + i]; } cell_mean /= cell_shape; for (int i = 0; i < cell_shape; i++) { if (!is_close(cell_mean, ((double *) data_dest)[ind + i])) { printf("i: %d, cell_mean %.9f, dest %.9f", ind + i, cell_mean, ((double *) data_dest)[ind + i]); printf("\n Decompressed data differs from original!\n"); return -1; } } break; default : break; } } } } csize_f = csize_f / nchunks; free(data_in); free(data_out); free(data_dest); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); free(shape); free(chunkshape); free(blockshape); printf("Successful roundtrip!\n"); printf("Compression: %d -> %" PRId64 " (%.1fx)\n", chunksize, csize_f, (1. * chunksize) / (double)csize_f); return (int) (chunksize - csize_f); } int rows_matches() { blosc2_schunk *schunk = blosc2_schunk_open("example_ndmean_1dim_2rows.caterva"); /* Run the test. */ int result = test_ndmean(schunk); blosc2_schunk_free(schunk); return result; } int same_cells() { blosc2_schunk *schunk = blosc2_schunk_open("example_ndmean_1dim_same_cells.caterva"); /* Run the test. */ int result = test_ndmean(schunk); blosc2_schunk_free(schunk); return result; } int some_matches() { blosc2_schunk *schunk = blosc2_schunk_open("example_ndmean_1dim_some_matches.caterva"); /* Run the test. */ int result = test_ndmean(schunk); blosc2_schunk_free(schunk); return result; } int main(void) { int result; blosc2_init(); result = rows_matches(); printf("2_rows_matches: %d obtained \n \n", result); result = same_cells(); printf("same_cells: %d obtained \n \n", result); result = some_matches(); printf("some_matches: %d obtained \n \n", result); blosc2_destroy(); return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/plugins/filters/ndmean/ndmean.c0000644000175200007730000002254314515254731022721 0ustar rlaboissrlaboiss/* Copyright (C) 2021 The Blosc Developers http://blosc.org License: BSD 3-Clause (see LICENSE.txt) */ #include "ndmean.h" #include #include #include "../plugins/plugin_utils.h" int ndmean_encoder(const uint8_t* input, uint8_t* output, int32_t length, uint8_t meta, blosc2_cparams* cparams) { int8_t ndim; int64_t* shape = malloc(8 * sizeof(int64_t)); int32_t* chunkshape = malloc(8 * sizeof(int32_t)); int32_t* blockshape = malloc(8 * sizeof(int32_t)); uint8_t* smeta; int32_t smeta_len; if (blosc2_meta_get(cparams->schunk, "caterva", &smeta, &smeta_len) < 0) { free(shape); free(chunkshape); free(blockshape); printf("Blosc error"); return 0; } deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape); free(smeta); int typesize = cparams->typesize; if ((typesize != 4) && (typesize != 8)) { free(shape); free(chunkshape); free(blockshape); fprintf(stderr, "This filter only works for float or double buffers"); return -1; } int8_t cellshape[8]; int cell_size = 1; for (int i = 0; i < 8; ++i) { if (i < ndim) { cellshape[i] = (int8_t)meta; if (cellshape[i] > blockshape[i]) { cellshape[i] = (int8_t) blockshape[i]; } cell_size *= cellshape[i]; } else { cellshape[i] = 1; } } int32_t blocksize = (int32_t) typesize; for (int i = 0; i < ndim; i++){ blocksize *= blockshape[i]; } if (length != blocksize) { free(shape); free(chunkshape); free(blockshape); printf("Length not equal to blocksize %d %d \n", length, blocksize); return -1; } uint8_t* ip = (uint8_t *) input; float* ip_float = (float *) ip; double * ip_double = (double *) ip; uint8_t* op = (uint8_t *) output; uint8_t* op_limit = op + length; int64_t cell_length; float mean_float = 0; double mean_double = 0; /* input and output buffer cannot be less than cell size */ if (length < cell_size * typesize) { free(shape); free(chunkshape); free(blockshape); printf("Incorrect length"); return 0; } uint8_t* obase = op; int64_t i_shape[NDMEAN_MAX_DIM]; for (int i = 0; i < ndim; ++i) { i_shape[i] = (blockshape[i] + cellshape[i] - 1) / cellshape[i]; } int64_t ncells = 1; for (int i = 0; i < ndim; ++i) { ncells *= i_shape[i]; } /* main loop */ int64_t pad_shape[NDMEAN_MAX_DIM]; int64_t ii[NDMEAN_MAX_DIM]; for (int cell_ind = 0; cell_ind < ncells; cell_ind++) { // for each cell blosc2_unidim_to_multidim(ndim, i_shape, cell_ind, ii); uint32_t orig = 0; int64_t nd_aux = (int64_t) (cellshape[0]); for (int i = ndim - 1; i >= 0; i--) { orig += ii[i] * nd_aux; nd_aux *= blockshape[i]; } for (int dim_ind = 0; dim_ind < ndim; dim_ind++) { if ((blockshape[dim_ind] % cellshape[dim_ind] != 0) && (ii[dim_ind] == i_shape[dim_ind] - 1)) { pad_shape[dim_ind] = blockshape[dim_ind] % cellshape[dim_ind]; } else { pad_shape[dim_ind] = (int32_t) cellshape[dim_ind]; } } int64_t ncopies = 1; for (int i = 0; i < ndim - 1; ++i) { ncopies *= pad_shape[i]; } int64_t kk[8]; mean_float = 0; mean_double = 0; for (int copy_ind = 0; copy_ind < ncopies; ++copy_ind) { blosc2_unidim_to_multidim((int8_t) (ndim - 1), pad_shape, copy_ind, kk); nd_aux = blockshape[ndim - 1]; int64_t ind = orig; for (int i = ndim - 2; i >= 0; i--) { ind += kk[i] * nd_aux; nd_aux *= blockshape[i]; } switch (typesize) { case 4: for (int i = 0; i < pad_shape[ndim - 1]; i++) { mean_float += ip_float[ind + i]; } break; case 8: for (int i = 0; i < pad_shape[ndim - 1]; i++) { mean_double += ip_double[ind + i]; } break; default : break; } } cell_length = ncopies * pad_shape[ndim - 1]; switch (typesize) { case 4: mean_float /= (float) cell_length; for (int i = 0; i < cell_length; i++) { memcpy(op, &mean_float, typesize); op += typesize; } break; case 8: mean_double /= (double) cell_length; for (int i = 0; i < cell_length; i++) { memcpy(op, &mean_double, typesize); op += typesize; } break; default : break; } if (op > op_limit) { free(shape); free(chunkshape); free(blockshape); printf("Output too big"); return 0; } } free(shape); free(chunkshape); free(blockshape); if((op - obase) != length) { printf("Output size must be equal to input size \n"); return 0; } return BLOSC2_ERROR_SUCCESS; } int ndmean_decoder(const uint8_t* input, uint8_t* output, int32_t length, uint8_t meta, blosc2_dparams* dparams) { blosc2_schunk *schunk = dparams->schunk; int8_t ndim; int64_t* shape = malloc(8 * sizeof(int64_t)); int32_t* chunkshape = malloc(8 * sizeof(int32_t)); int32_t* blockshape = malloc(8 * sizeof(int32_t)); uint8_t* smeta; int32_t smeta_len; if (blosc2_meta_get(schunk, "caterva", &smeta, &smeta_len) < 0) { free(shape); free(chunkshape); free(blockshape); printf("Blosc error"); return 0; } deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape); free(smeta); int8_t cellshape[8]; int cell_size = 1; for (int i = 0; i < 8; ++i) { if (i < ndim) { cellshape[i] = (int8_t)meta; if (cellshape[i] > blockshape[i]) { cellshape[i] = (int8_t) blockshape[i]; } cell_size *= cellshape[i]; } else { cellshape[i] = 1; } } int8_t typesize = (int8_t) schunk->typesize; uint8_t* ip = (uint8_t*)input; uint8_t* ip_limit = ip + length; uint8_t* op = (uint8_t*)output; int32_t blocksize = (int32_t) typesize; for (int i = 0; i < ndim; i++){ blocksize *= blockshape[i]; } if (length != blocksize) { free(shape); free(chunkshape); free(blockshape); printf("Length not equal to blocksize \n"); return -1; } /* input and output buffer cannot be less than cell size */ if (length < cell_size * typesize) { free(shape); free(chunkshape); free(blockshape); printf("Incorrect length"); return 0; } int64_t i_shape[NDMEAN_MAX_DIM]; for (int i = 0; i < ndim; ++i) { i_shape[i] = (blockshape[i] + cellshape[i] - 1) / cellshape[i]; } int64_t ncells = 1; for (int i = 0; i < ndim; ++i) { ncells *= i_shape[i]; } /* main loop */ int64_t pad_shape[NDMEAN_MAX_DIM] = {0}; int64_t ii[NDMEAN_MAX_DIM]; int32_t ind = 0; for (int cell_ind = 0; cell_ind < ncells; cell_ind++) { // for each cell if (ip > ip_limit) { free(shape); free(chunkshape); free(blockshape); printf("Literal copy \n"); return 0; } blosc2_unidim_to_multidim(ndim, i_shape, cell_ind, ii); uint32_t orig = 0; int64_t nd_aux = (int64_t) (cellshape[0]); for (int i = ndim - 1; i >= 0; i--) { orig += ii[i] * nd_aux; nd_aux *= blockshape[i]; } for (int dim_ind = 0; dim_ind < ndim; dim_ind++) { if ((blockshape[dim_ind] % cellshape[dim_ind] != 0) && (ii[dim_ind] == i_shape[dim_ind] - 1)) { pad_shape[dim_ind] = blockshape[dim_ind] % cellshape[dim_ind]; } else { pad_shape[dim_ind] = (int64_t) cellshape[dim_ind]; } } int64_t ncopies = 1; for (int i = 0; i < ndim - 1; ++i) { ncopies *= pad_shape[i]; } int64_t kk[NDMEAN_MAX_DIM]; for (int copy_ind = 0; copy_ind < ncopies; ++copy_ind) { blosc2_unidim_to_multidim((int8_t) (ndim - 1), pad_shape, copy_ind, kk); nd_aux = blockshape[ndim - 1]; ind = (int32_t) orig; for (int i = ndim - 2; i >= 0; i--) { ind += (int32_t) (kk[i] * nd_aux); nd_aux *= blockshape[i]; } memcpy(&op[ind * typesize], ip, pad_shape[ndim - 1] * typesize); ip += pad_shape[ndim - 1] * typesize; } } ind += (int32_t) pad_shape[ndim - 1]; free(shape); free(chunkshape); free(blockshape); if (ind != (int32_t) (blocksize / typesize)) { printf("Output size is not compatible with embedded blockshape ind %d %d \n", ind, (blocksize / typesize)); return 0; } return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/plugins/filters/CMakeLists.txt0000644000175200007730000000017514515254731022606 0ustar rlaboissrlaboissadd_subdirectory(ndcell) add_subdirectory(ndmean) set(SOURCES ${SOURCES} ../plugins/filters/filters-registry.c PARENT_SCOPE)zmat-0.9.9/src/blosc2/plugins/filters/ndcell/0000755000175200007730000000000014515254731021304 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/plugins/filters/ndcell/ndcell.c0000644000175200007730000001672114515254731022720 0ustar rlaboissrlaboiss/* Copyright (C) 2021 The Blosc Developers http://blosc.org License: BSD 3-Clause (see LICENSE.txt) */ #include #include "ndcell.h" #include #include #include "../plugins/plugin_utils.h" int ndcell_encoder(const uint8_t* input, uint8_t* output, int32_t length, uint8_t meta, blosc2_cparams* cparams) { int8_t ndim; int64_t* shape = malloc(8 * sizeof(int64_t)); int32_t* chunkshape = malloc(8 * sizeof(int32_t)); int32_t* blockshape = malloc(8 * sizeof(int32_t)); uint8_t* smeta; int32_t smeta_len; if (blosc2_meta_get(cparams->schunk, "caterva", &smeta, &smeta_len) < 0) { free(shape); free(chunkshape); free(blockshape); printf("Blosc error"); return 0; } deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape); free(smeta); int typesize = cparams->typesize; int8_t cell_shape = (int8_t)meta; const int cell_size = (int) pow(cell_shape, ndim); int32_t blocksize = (int32_t) typesize; for (int i = 0; i < ndim; i++){ blocksize *= blockshape[i]; } if (length != blocksize) { free(shape); free(chunkshape); free(blockshape); printf("Length not equal to blocksize %d %d \n", length, blocksize); return -1; } uint8_t* ip = (uint8_t *) input; uint8_t* op = (uint8_t *) output; uint8_t* op_limit = op + length; /* input and output buffer cannot be less than cell size */ if (length < cell_size * typesize) { free(shape); free(chunkshape); free(blockshape); printf("Incorrect length"); return 0; } uint8_t* obase = op; int64_t i_shape[NDCELL_MAX_DIM]; for (int i = 0; i < ndim; ++i) { i_shape[i] = (blockshape[i] + cell_shape - 1) / cell_shape; } int64_t ncells = 1; for (int i = 0; i < ndim; ++i) { ncells *= i_shape[i]; } /* main loop */ int64_t pad_shape[NDCELL_MAX_DIM]; int64_t ii[NDCELL_MAX_DIM]; for (int cell_ind = 0; cell_ind < ncells; cell_ind++) { // for each cell blosc2_unidim_to_multidim(ndim, i_shape, cell_ind, ii); uint32_t orig = 0; int64_t nd_aux = (int64_t) cell_shape; for (int i = ndim - 1; i >= 0; i--) { orig += ii[i] * nd_aux; nd_aux *= blockshape[i]; } for (int dim_ind = 0; dim_ind < ndim; dim_ind++) { if ((blockshape[dim_ind] % cell_shape != 0) && (ii[dim_ind] == i_shape[dim_ind] - 1)) { pad_shape[dim_ind] = blockshape[dim_ind] % cell_shape; } else { pad_shape[dim_ind] = (int64_t) cell_shape; } } int64_t ncopies = 1; for (int i = 0; i < ndim - 1; ++i) { ncopies *= pad_shape[i]; } int64_t kk[NDCELL_MAX_DIM]; for (int copy_ind = 0; copy_ind < ncopies; ++copy_ind) { blosc2_unidim_to_multidim(ndim - 1, pad_shape, copy_ind, kk); nd_aux = blockshape[ndim - 1]; int64_t ind = orig; for (int i = ndim - 2; i >= 0; i--) { ind += kk[i] * nd_aux; nd_aux *= blockshape[i]; } memcpy(op, &ip[ind * typesize], pad_shape[ndim - 1] * typesize); op += pad_shape[ndim - 1] * typesize; } if (op > op_limit) { free(shape); free(chunkshape); free(blockshape); printf("Output too big"); return 0; } } if((op - obase) != length) { free(shape); free(chunkshape); free(blockshape); printf("Output size must be equal to input size \n"); return 0; } free(shape); free(chunkshape); free(blockshape); return BLOSC2_ERROR_SUCCESS; } int ndcell_decoder(const uint8_t* input, uint8_t* output, int32_t length, uint8_t meta, blosc2_dparams* dparams) { blosc2_schunk *schunk = dparams->schunk; int8_t ndim; int64_t* shape = malloc(8 * sizeof(int64_t)); int32_t* chunkshape = malloc(8 * sizeof(int32_t)); int32_t* blockshape = malloc(8 * sizeof(int32_t)); uint8_t* smeta; int32_t smeta_len; if (blosc2_meta_get(schunk, "caterva", &smeta, &smeta_len) < 0) { free(shape); free(chunkshape); free(blockshape); printf("Blosc error"); return 0; } deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape); free(smeta); int8_t cell_shape = (int8_t)meta; int cell_size = (int) pow(cell_shape, ndim); int32_t typesize = schunk->typesize; uint8_t* ip = (uint8_t*)input; uint8_t* ip_limit = ip + length; uint8_t* op = (uint8_t*)output; int32_t blocksize = (int32_t) typesize; for (int i = 0; i < ndim; i++){ blocksize *= blockshape[i]; } if (length != blocksize) { free(shape); free(chunkshape); free(blockshape); printf("Length not equal to blocksize \n"); return -1; } /* input and output buffer cannot be less than cell size */ if (length < cell_size * typesize) { free(shape); free(chunkshape); free(blockshape); printf("Incorrect length"); return 0; } int64_t i_shape[NDCELL_MAX_DIM]; for (int i = 0; i < ndim; ++i) { i_shape[i] = (blockshape[i] + cell_shape - 1) / cell_shape; } int64_t ncells = 1; for (int i = 0; i < ndim; ++i) { ncells *= i_shape[i]; } /* main loop */ int64_t pad_shape[NDCELL_MAX_DIM] = {0}; int64_t ii[NDCELL_MAX_DIM]; int32_t ind = 0; for (int cell_ind = 0; cell_ind < ncells; cell_ind++) { // for each cell if (ip > ip_limit) { free(shape); free(chunkshape); free(blockshape); printf("Literal copy \n"); return 0; } blosc2_unidim_to_multidim(ndim, i_shape, cell_ind, ii); uint32_t orig = 0; int64_t nd_aux = (int64_t) cell_shape; for (int i = ndim - 1; i >= 0; i--) { orig += ii[i] * nd_aux; nd_aux *= blockshape[i]; } for (int dim_ind = 0; dim_ind < ndim; dim_ind++) { if ((blockshape[dim_ind] % cell_shape != 0) && (ii[dim_ind] == i_shape[dim_ind] - 1)) { pad_shape[dim_ind] = blockshape[dim_ind] % cell_shape; } else { pad_shape[dim_ind] = (int64_t) cell_shape; } } int64_t ncopies = 1; for (int i = 0; i < ndim - 1; ++i) { ncopies *= pad_shape[i]; } int64_t kk[NDCELL_MAX_DIM]; for (int copy_ind = 0; copy_ind < ncopies; ++copy_ind) { blosc2_unidim_to_multidim(ndim - 1, pad_shape, copy_ind, kk); nd_aux = blockshape[ndim - 1]; ind = (int32_t) orig; for (int i = ndim - 2; i >= 0; i--) { ind += (int32_t) (kk[i] * nd_aux); nd_aux *= blockshape[i]; } memcpy(&op[ind * typesize], ip, pad_shape[ndim - 1] * typesize); ip += pad_shape[ndim - 1] * typesize; } } ind += (int32_t) pad_shape[ndim - 1]; if (ind != (int32_t) (blocksize / typesize)) { free(shape); free(chunkshape); free(blockshape); printf("Output size is not compatible with embedded blockshape ind %d %d \n", ind, (blocksize / typesize)); return 0; } free(shape); free(chunkshape); free(blockshape); return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/plugins/filters/ndcell/README.md0000644000175200007730000000443214515254731022566 0ustar rlaboissrlaboissNDCELL: a multidimensional filter for lossless compression ============================================================================= Given an n-dim array or matrix, *NDCELL* is a filter based on the codec *NDLZ* that divides data into multidimensional cells, reordering the dataset so that the codec compress cell by cell. Plugin motivation -------------------- *NDCELL* was created in order to make easy for codecs to search for patterns repetitions in multidimensional datasets using the Caterva blocking machinery. Plugin usage ------------------- The codec consists of an encoder called *ndcell_encoder()* to reorder data and a decoder called *ndcell_decoder()* to recover the original data. The parameters used by *NDCELL* are the ones specified in the *blosc2_filter* structure of *blosc2.h*. Furthermore, since *NDCELL* goes through dataset blocks dividing them into fixed size cells, user must specify the parameter meta as the cellshape, so if in a 3-dim dataset user specifies meta = 4, then cellshape will be 4x4x4. Plugin behaviour ------------------- This filter is meant to leverage multidimensionality for helping codecs to get better compression ratios. The idea is to order the data so that when the dataset is traversed the elements of a cell are all in a row. ------------------------ ----------------------------- | 0 1 | 2 3 | 4 5 | 6 7 | | 0 1 8 9 | 2 3 10 11 | | 8 9 |10 11|12 13|14 15| NDCELL encoder ----------------------------- ------------------------- ------> | 4 5 12 13 | 6 7 14 15 | |16 17|18 19|20 21|22 23| ----------------------------- |24 25|26 27|28 29|30 31| | 16 17 24 25 | 18 19 26 27 | ------------------------- ----------------------------- | 20 21 28 29 | 22 23 30 31 | ----------------------------- Advantages and disadvantages ------------------------------ The particularity of *NDCELL* is that this filter considers datasets multidimensionality and takes advantage of it instead of processing all data as serial. The main disadvantage of *NDCELL* is that only a few codecs are benefited by it and only for multidimensional datasets. zmat-0.9.9/src/blosc2/plugins/filters/ndcell/CMakeLists.txt0000644000175200007730000000150514515254731024045 0ustar rlaboissrlaboiss# sources set(SOURCES ${SOURCES} ../plugins/filters/ndcell/ndcell.c ../plugins/filters/ndcell/ndcell.h PARENT_SCOPE) if(BUILD_TESTS) # targets add_executable(test_ndcell test_ndcell.c) # Define the BLOSC_TESTING symbol so normally-hidden functions # aren't hidden from the view of the test programs. set_property( TARGET test_ndcell APPEND PROPERTY COMPILE_DEFINITIONS BLOSC_TESTING) target_link_libraries(test_ndcell blosc_testing) # tests add_test(NAME test_plugin_test_ndcell COMMAND ${CMAKE_CROSSCOMPILING_EMULATOR} $) # Copy test files file(GLOB TESTS_DATA ../../test_data/*.caterva) foreach (data ${TESTS_DATA}) file(COPY ${data} DESTINATION ${CMAKE_CURRENT_BINARY_DIR}/) endforeach(data) endif() zmat-0.9.9/src/blosc2/plugins/filters/ndcell/ndcell.h0000644000175200007730000000134714515254731022723 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef CATERVA_NDCELL_H #define CATERVA_NDCELL_H #include #define NDCELL_MAX_DIM 8 int ndcell_encoder(const uint8_t* input, uint8_t* output, int32_t length, uint8_t meta, blosc2_cparams* cparams); int ndcell_decoder(const uint8_t* input, uint8_t* output, int32_t length, uint8_t meta, blosc2_dparams* dparams); #endif //CATERVA_NDCELL_H zmat-0.9.9/src/blosc2/plugins/filters/ndcell/test_ndcell.c0000644000175200007730000001134014515254731023747 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. Test program demonstrating use of the Blosc filter from C code. To compile this program: $ gcc -O test_ndcell.c -o test_ndcell -lblosc2 To run: $ ./test_ndcell Blosc version info: 2.0.0a6.dev ($Date:: 2018-05-18 #$) Using 1 thread Using ZSTD compressor Successful roundtrip! Compression: 41472 -> 3992 (10.4x) rand: 37480 obtained Successful roundtrip! Compression: 1792 -> 979 (1.8x) same_cells: 813 obtained Successful roundtrip! Compression: 16128 -> 1438 (11.2x) some_matches: 14690 obtained **********************************************************************/ #include #include "ndcell.h" #include #include "blosc2/filters-registry.h" static int test_ndcell(blosc2_schunk* schunk) { int64_t nchunks = schunk->nchunks; int32_t chunksize = (int32_t) (schunk->chunksize); // int isize = (int) array->extchunknitems * typesize; uint8_t *data_in = malloc(chunksize); int decompressed; int64_t csize; int64_t dsize; int64_t csize_f = 0; uint8_t *data_out = malloc(chunksize + BLOSC2_MAX_OVERHEAD); uint8_t *data_dest = malloc(chunksize); /* Create a context for compression */ blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_ALWAYS_SPLIT; cparams.typesize = schunk->typesize; cparams.compcode = BLOSC_ZSTD; cparams.filters[4] = BLOSC_FILTER_NDCELL; cparams.filters_meta[4] =4; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_SHUFFLE; cparams.clevel = 9; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; dparams.schunk = schunk; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress with clevel=5 and shuffle active */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, data_out, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } csize_f += csize; /* Decompress */ dsize = blosc2_decompress_ctx(dctx, data_out, chunksize + BLOSC2_MAX_OVERHEAD, data_dest, chunksize); if (dsize <= 0) { printf("Decompression error. Error code: %" PRId64 "\n", dsize); return (int) dsize; } for (int i = 0; i < chunksize; i++) { if (data_in[i] != data_dest[i]) { printf("i: %d, data %u, dest %u", i, data_in[i], data_dest[i]); printf("\n Decompressed data differs from original!\n"); return -1; } } } csize_f = csize_f / nchunks; free(data_in); free(data_out); free(data_dest); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); printf("Successful roundtrip!\n"); printf("Compression: %d -> %" PRId64 " (%.1fx)\n", chunksize, csize_f, (1. * chunksize) / (double)csize_f); return (int) (chunksize - csize_f); } int rand_() { blosc2_schunk *schunk = blosc2_schunk_open("example_rand.caterva"); /* Run the test. */ int result = test_ndcell(schunk); blosc2_schunk_free(schunk); return result; } int same_cells() { blosc2_schunk *schunk = blosc2_schunk_open("example_same_cells.caterva"); /* Run the test. */ int result = test_ndcell(schunk); blosc2_schunk_free(schunk); return result; } int some_matches() { blosc2_schunk *schunk = blosc2_schunk_open("example_some_matches.caterva"); /* Run the test. */ int result = test_ndcell(schunk); blosc2_schunk_free(schunk); return result; } int main(void) { int result; blosc2_init(); result = rand_(); printf("rand: %d obtained \n \n", result); result = same_cells(); printf("same_cells: %d obtained \n \n", result); result = some_matches(); printf("some_matches: %d obtained \n \n", result); blosc2_destroy(); return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/plugins/filters/filters-registry.c0000644000175200007730000000132514515254731023526 0ustar rlaboissrlaboiss/* Copyright (C) 2021 The Blosc Developers http://blosc.org License: BSD 3-Clause (see LICENSE.txt) */ #include #include "blosc2/filters-registry.h" #include "ndmean/ndmean.h" #include "ndcell/ndcell.h" void register_filters(void) { blosc2_filter ndcell; ndcell.id = BLOSC_FILTER_NDCELL; ndcell.forward = (blosc2_filter_forward_cb) ndcell_encoder; ndcell.backward = (blosc2_filter_backward_cb) ndcell_decoder; register_filter_private(&ndcell); blosc2_filter ndmean; ndmean.id = BLOSC_FILTER_NDMEAN; ndmean.forward = (blosc2_filter_forward_cb) ndmean_encoder; ndmean.backward = (blosc2_filter_backward_cb) ndmean_decoder; register_filter_private(&ndmean); }zmat-0.9.9/src/blosc2/plugins/codecs/0000755000175200007730000000000014515254731017633 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/plugins/codecs/CMakeLists.txt0000644000175200007730000000016614515254731022376 0ustar rlaboissrlaboissadd_subdirectory(ndlz) add_subdirectory(zfp) set(SOURCES ${SOURCES} ../plugins/codecs/codecs-registry.c PARENT_SCOPE)zmat-0.9.9/src/blosc2/plugins/codecs/codecs-registry.c0000644000175200007730000000267714515254731023121 0ustar rlaboissrlaboiss/* Copyright (C) 2021 The Blosc Developers http://blosc.org License: BSD 3-Clause (see LICENSE.txt) */ #include #include "blosc2/codecs-registry.h" #include "ndlz/ndlz.h" #include "zfp/blosc2-zfp.h" void register_codecs(void) { blosc2_codec ndlz; ndlz.compcode = BLOSC_CODEC_NDLZ; ndlz.compver = 1; ndlz.complib = BLOSC_CODEC_NDLZ; ndlz.encoder = ndlz_compress; ndlz.decoder = ndlz_decompress; ndlz.compname = "ndlz"; register_codec_private(&ndlz); blosc2_codec zfp_acc; zfp_acc.compcode = BLOSC_CODEC_ZFP_FIXED_ACCURACY; zfp_acc.compver = 1; zfp_acc.complib = BLOSC_CODEC_ZFP_FIXED_ACCURACY; zfp_acc.encoder = zfp_acc_compress; zfp_acc.decoder = zfp_acc_decompress; zfp_acc.compname = "zfp_acc"; register_codec_private(&zfp_acc); blosc2_codec zfp_prec; zfp_prec.compcode = BLOSC_CODEC_ZFP_FIXED_PRECISION; zfp_prec.compver = 1; zfp_prec.complib = BLOSC_CODEC_ZFP_FIXED_PRECISION; zfp_prec.encoder = zfp_prec_compress; zfp_prec.decoder = zfp_prec_decompress; zfp_prec.compname = "zfp_prec"; register_codec_private(&zfp_prec); blosc2_codec zfp_rate; zfp_rate.compcode = BLOSC_CODEC_ZFP_FIXED_RATE; zfp_rate.compver = 1; zfp_rate.complib = BLOSC_CODEC_ZFP_FIXED_RATE; zfp_rate.encoder = zfp_rate_compress; zfp_rate.decoder = zfp_rate_decompress; zfp_rate.compname = "zfp_rate"; register_codec_private(&zfp_rate); } zmat-0.9.9/src/blosc2/plugins/codecs/ndlz/0000755000175200007730000000000014515254731020602 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/plugins/codecs/ndlz/xxhash.c0000644000175200007730000000347714515254731022264 0ustar rlaboissrlaboiss/* * xxHash - Extremely Fast Hash algorithm * Copyright (C) 2012-2020 Yann Collet * * BSD 2-Clause License (https://www.opensource.org/licenses/bsd-license.php) * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are * met: * * * Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above * copyright notice, this list of conditions and the following disclaimer * in the documentation and/or other materials provided with the * distribution. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * You can contact the author at: * - xxHash homepage: https://www.xxhash.com * - xxHash source repository: https://github.com/Cyan4973/xxHash */ /* * xxhash.c instantiates functions defined in xxhash.h */ #define XXH_STATIC_LINKING_ONLY /* access advanced declarations */ #define XXH_IMPLEMENTATION /* access definitions */ #include "xxhash.h" zmat-0.9.9/src/blosc2/plugins/codecs/ndlz/ndlz4x4.c0000644000175200007730000005405114515254731022262 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Author: Francesc Alted Author: Oscar Griñón Author: Aleix Alcacer Creation date: 2020-06-12 See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /********************************************************************* This codec is meant to leverage multidimensionality for getting better compression ratios. The idea is to look for similarities in places that are closer in a euclidean metric, not the typical linear one. **********************************************************************/ #include #include "ndlz4x4.h" #include "ndlz.h" #include "xxhash.h" #include "../plugins/plugin_utils.h" /* * Give hints to the compiler for branch prediction optimization. */ #if defined(__GNUC__) && (__GNUC__ > 2) #define NDLZ_EXPECT_CONDITIONAL(c) (__builtin_expect((c), 1)) #define NDLZ_UNEXPECT_CONDITIONAL(c) (__builtin_expect((c), 0)) #else #define NDLZ_EXPECT_CONDITIONAL(c) (c) #define NDLZ_UNEXPECT_CONDITIONAL(c) (c) #endif /* * Use inlined functions for supported systems. */ #if defined(_MSC_VER) && !defined(__cplusplus) /* Visual Studio */ #define inline __inline /* Visual C is not C99, but supports some kind of inline */ #endif #define MAX_COPY 32U #define MAX_DISTANCE 65535 #ifdef BLOSC_STRICT_ALIGN #define NDLZ_READU16(p) ((p)[0] | (p)[1]<<8) #define NDLZ_READU32(p) ((p)[0] | (p)[1]<<8 | (p)[2]<<16 | (p)[3]<<24) #else #define NDLZ_READU16(p) *((const uint16_t*)(p)) #define NDLZ_READU32(p) *((const uint32_t*)(p)) #endif #define HASH_LOG (12) int ndlz4_compress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_cparams *cparams) { BLOSC_UNUSED_PARAM(meta); int8_t ndim; int64_t* shape = malloc(8 * sizeof(int64_t)); int32_t* chunkshape = malloc(8 * sizeof(int32_t)); int32_t* blockshape = malloc(8 * sizeof(int32_t)); uint8_t* smeta; int32_t smeta_len; if (blosc2_meta_get(cparams->schunk, "caterva", &smeta, &smeta_len) < 0) { printf("Blosc error"); return -1; } deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape); free(smeta); if (ndim != 2) { fprintf(stderr, "This codec only works for ndim = 2"); return -1; } if (input_len != (blockshape[0] * blockshape[1])) { printf("Length not equal to blocksize \n"); return -1; } if (NDLZ_UNEXPECT_CONDITIONAL(output_len < (int) (1 + ndim * sizeof(int32_t)))) { printf("Output too small \n"); return -1; } uint8_t* ip = (uint8_t *) input; uint8_t* op = (uint8_t *) output; uint8_t* op_limit; uint32_t hval, hash_cell; uint32_t hash_triple[2] = {0}; uint32_t hash_pair[3] = {0}; uint8_t bufarea[16]; uint8_t* buf_cell = bufarea; uint8_t buf_triple[12]; uint8_t buf_pair[8]; uint8_t* buf_aux; uint32_t tab_cell[1U << 12U] = {0}; uint32_t tab_triple[1U << 12U] = {0}; uint32_t tab_pair[1U << 12U] = {0}; uint32_t update_triple[2] = {0}; uint32_t update_pair[3] = {0}; // Minimum cratios before issuing and _early giveup_ // Remind that ndlz is not meant for cratios <= 2 (too costly to decompress) op_limit = op + output_len; // Initialize the hash table to distances of 0 for (unsigned i = 0; i < (1U << 12U); i++) { tab_cell[i] = 0; } /* input and output buffer cannot be less than 16 and 66 bytes or we can get into trouble */ int overhead = 17 + (blockshape[0] * blockshape[1] / 16 - 1) * 2; if (input_len < 16 || output_len < overhead) { printf("Incorrect length or maxout"); return 0; } uint8_t* obase = op; /* we start with literal copy */ *op++ = ndim; memcpy(op, &blockshape[0], 4); op += 4; memcpy(op, &blockshape[1], 4); op += 4; uint32_t i_stop[2]; for (int i = 0; i < 2; ++i) { i_stop[i] = (blockshape[i] + 3) / 4; } /* main loop */ uint32_t padding[2]; uint32_t ii[2]; for (ii[0] = 0; ii[0] < i_stop[0]; ++ii[0]) { for (ii[1] = 0; ii[1] < i_stop[1]; ++ii[1]) { // for each cell uint8_t token; for (int h = 0; h < 2; h++) { // new cell -> new possible references update_triple[h] = 0; update_pair[h] = 0; } update_pair[2] = 0; if (NDLZ_UNEXPECT_CONDITIONAL(op + 16 + 1 > op_limit)) { free(shape); free(chunkshape); free(blockshape); return 0; } uint32_t orig = ii[0] * 4 * blockshape[1] + ii[1] * 4; if (((blockshape[0] % 4 != 0) && (ii[0] == i_stop[0] - 1)) || ((blockshape[1] % 4 != 0) && (ii[1] == i_stop[1] - 1))) { token = 0; // padding -> literal copy *op++ = token; if (ii[0] == i_stop[0] - 1) { padding[0] = (blockshape[0] % 4 == 0) ? 4 : blockshape[0] % 4; } else { padding[0] = 4; } if (ii[1] == i_stop[1] - 1) { padding[1] = (blockshape[1] % 4 == 0) ? 4 : blockshape[1] % 4; } else { padding[1] = 4; } for (uint32_t i = 0; i < padding[0]; i++) { memcpy(op, &ip[orig + i * blockshape[1]], padding[1]); op += padding[1]; } } else { for (uint64_t i = 0; i < 4; i++) { // fill cell buffer uint64_t ind = orig + i * blockshape[1]; memcpy(buf_cell, &ip[ind], 4); buf_cell += 4; } buf_cell -= 16; const uint8_t* ref; uint32_t distance; uint8_t* anchor = op; /* comparison starting-point */ /* find potential match */ hash_cell = XXH32(buf_cell, 16, 1); // calculate cell hash hash_cell >>= 32U - 12U; ref = obase + tab_cell[hash_cell]; /* calculate distance to the match */ if (tab_cell[hash_cell] == 0) { distance = 0; } else { bool same = true; buf_aux = obase + tab_cell[hash_cell]; for(int i = 0; i < 16; i++){ if (buf_cell[i] != buf_aux[i]) { same = false; break; } } if (same) { distance = (int32_t) (anchor - ref); } else { distance = 0; } } bool alleq = true; for (int i = 1; i < 16; i++) { if (buf_cell[i] != buf_cell[0]) { alleq = false; break; } } if (alleq) { // all elements of the cell equal token = (uint8_t) (1U << 6U); *op++ = token; *op++ = buf_cell[0]; } else if (distance == 0 || (distance >= MAX_DISTANCE)) { // no cell match bool literal = true; // 2 rows pairs matches for (int j = 1; j < 4; j++) { memcpy(buf_pair, buf_cell, 4); memcpy(&buf_pair[4], &buf_cell[j * 4], 4); hval = XXH32(buf_pair, 8, 1); // calculate rows pair hash hval >>= 32U - 12U; ref = obase + tab_pair[hval]; /* calculate distance to the match */ bool same = true; uint16_t offset; if (tab_pair[hval] != 0) { buf_aux = obase + tab_pair[hval]; for (int k = 0; k < 8; k++) { if (buf_pair[k] != buf_aux[k]) { same = false; break; } } offset = (uint16_t) (anchor - obase - tab_pair[hval]); } else { same = false; } if (same) { distance = (int32_t) (anchor - ref); } else { distance = 0; } if ((distance != 0) && (distance < MAX_DISTANCE)) { /* rows pair match */ int k, m, l = -1; for (k = 1; k < 4; k++) { if (k != j) { if (l == -1) { l = k; } else { m = k; } } } memcpy(buf_pair, &buf_cell[l * 4], 4); memcpy(&buf_pair[4], &buf_cell[m * 4], 4); hval = XXH32(buf_pair, 8, 1); // calculate rows pair hash hval >>= 32U - 12U; ref = obase + tab_pair[hval]; same = true; if (tab_pair[hval] != 0) { buf_aux = obase + tab_pair[hval]; for (k = 0; k < 8; k++) { if (buf_pair[k] != buf_aux[k]) { same = false; break; } } } else { same = false; } if (same) { distance = (int32_t) (anchor + l * 4 - ref); } else { distance = 0; } if ((distance != 0) && (distance < MAX_DISTANCE)) { /* 2 pair matches */ literal = false; token = (uint8_t) ((1U << 5U) | (j << 3U)); *op++ = token; uint16_t offset_2 = (uint16_t) (anchor - obase - tab_pair[hval]); *(uint16_t *) op = offset; op += sizeof(offset); *(uint16_t *) op = offset_2; op += sizeof(offset_2); goto match; } } } // rows triples for(int i = 0; i < 2; i++) { memcpy(buf_triple, &buf_cell[i * 4], 4); for (int j = i + 1; j < 3; j++) { memcpy(&buf_triple[4], &buf_cell[j * 4], 4); for (int k = j + 1; k < 4; k++) { memcpy(&buf_triple[8], &buf_cell[k * 4], 4); hval = XXH32(buf_triple, 12, 1); // calculate triple hash hval >>= 32U - 12U; /* calculate distance to the match */ bool same = true; uint16_t offset; if (tab_triple[hval] != 0) { buf_aux = obase + tab_triple[hval]; for (int l = 0; l < 12; l++) { if (buf_triple[l] != buf_aux[l]) { same = false; break; } } offset = (uint16_t) (anchor - obase - tab_triple[hval]); } else { same = false; if ((j - i == 1) && (k - j == 1)) { update_triple[i] = (uint32_t) (anchor + 1 + i * 4 - obase); /* update hash table */ hash_triple[i] = hval; } } ref = obase + tab_triple[hval]; if (same) { distance = (int32_t) (anchor + i * 4 - ref); } else { distance = 0; } if ((distance != 0) && (distance < MAX_DISTANCE)) { literal = false; if (i == 1) { token = (uint8_t) (7U << 5U); } else { token = (uint8_t) ((7U << 5U) | ((j + k - 2) << 3U)); } *op++ = token; memcpy(op, &offset, 2); op += 2; for (int l = 0; l < 4; l++) { if ((l != i) && (l != j) && (l != k)) { memcpy(op, &buf_cell[4 * l], 4); op += 4; goto match; } } } } } } // rows pairs for(int i = 0; i < 3; i++) { memcpy(buf_pair, &buf_cell[i * 4], 4); for (int j = i + 1; j < 4; j++) { memcpy(&buf_pair[4], &buf_cell[j * 4], 4); hval = XXH32(buf_pair, 8, 1); // calculate rows pair hash hval >>= 32U - 12U; ref = obase + tab_pair[hval]; /* calculate distance to the match */ bool same = true; uint16_t offset; if (tab_pair[hval] != 0) { buf_aux = obase + tab_pair[hval]; for(int k = 0; k < 8; k++){ if(buf_pair[k] != buf_aux[k]) { same = false; break; } } offset = (uint16_t) (anchor - obase - tab_pair[hval]); } else { same = false; if (j - i == 1) { update_pair[i] = (uint32_t) (anchor + 1 + i * 4 - obase); /* update hash table */ hash_pair[i] = hval; } } if (same) { distance = (int32_t) (anchor + i * 4 - ref); } else { distance = 0; } if ((distance != 0) && (distance < MAX_DISTANCE)) { /* rows pair match */ literal = false; if (i == 2) { token = (uint8_t) (1U << 7U); } else { token = (uint8_t) ((1U << 7U) | (i << 5U) | (j << 3U)); } *op++ = token; memcpy(op, &offset, 2); op += 2; for (int k = 0; k < 4; k++) { if ((k != i) && (k != j)) { memcpy(op, &buf_cell[4 * k], 4); op += 4; } } goto match; } } } match: if (literal) { tab_cell[hash_cell] = (uint32_t) (anchor + 1 - obase); /* update hash tables */ if (update_triple[0] != 0) { for (int h = 0; h < 2; h++) { tab_triple[hash_triple[h]] = update_triple[h]; } } if (update_pair[0] != 0) { for (int h = 0; h < 3; h++) { tab_pair[hash_pair[h]] = update_pair[h]; } } token = 0; *op++ = token; memcpy(op, buf_cell, 16); op += 16; } } else { // cell match token = (uint8_t )((1U << 7U) | (1U << 6U)); *op++ = token; uint16_t offset = (uint16_t) (anchor - obase - tab_cell[hash_cell]); memcpy(op, &offset, 2); op += 2; } } if((op - obase) > input_len) { printf("Compressed data is bigger than input! \n"); return 0; } } } free(shape); free(chunkshape); free(blockshape); return (int)(op - obase); } // See https://habr.com/en/company/yandex/blog/457612/ #ifdef __AVX2__ #if defined(_MSC_VER) #define ALIGNED_(x) __declspec(align(x)) #else #if defined(__GNUC__) #define ALIGNED_(x) __attribute__ ((aligned(x))) #endif #endif #define ALIGNED_TYPE_(t, x) t ALIGNED_(x) static unsigned char* copy_match_16(unsigned char *op, const unsigned char *match, int32_t len) { size_t offset = op - match; while (len >= 16) { static const ALIGNED_TYPE_(uint8_t, 16) masks[] = { 0, 1, 2, 1, 4, 1, 4, 2, 8, 7, 6, 5, 4, 3, 2, 1, // offset = 0, not used as mask, but for shift 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // offset = 1 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 0, 1, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 8, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 0, 1, 2, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 0, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, // offset = 16 }; _mm_storeu_si128((__m128i *)(op), _mm_shuffle_epi8(_mm_loadu_si128((const __m128i *)(match)), _mm_load_si128((const __m128i *)(masks) + offset))); match += masks[offset]; op += 16; len -= 16; } // Deal with remainders for (; len > 0; len--) { *op++ = *match++; } return op; } #endif int ndlz4_decompress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_dparams *dparams) { BLOSC_UNUSED_PARAM(meta); BLOSC_UNUSED_PARAM(dparams); uint8_t* ip = (uint8_t*)input; uint8_t* ip_limit = ip + input_len; uint8_t* op = (uint8_t*)output; uint8_t ndim; uint32_t blockshape[2]; uint32_t eshape[2]; uint8_t* buffercpy; uint8_t local_buffer[16]; uint8_t token; if (NDLZ_UNEXPECT_CONDITIONAL(input_len < 8)) { return 0; } /* we start with literal copy */ ndim = *ip; ip ++; if (ndim != 2) { fprintf(stderr, "This codec only works for ndim = 2"); return -1; } memcpy(&blockshape[0], ip, 4); ip += 4; memcpy(&blockshape[1], ip, 4); ip += 4; eshape[0] = ((blockshape[0] + 3) / 4) * 4; eshape[1] = ((blockshape[1] + 3) / 4) * 4; if (NDLZ_UNEXPECT_CONDITIONAL(output_len < (int32_t)(blockshape[0] * blockshape[1]))) { return 0; } memset(op, 0, blockshape[0] * blockshape[1]); uint32_t i_stop[2]; for (int i = 0; i < 2; ++i) { i_stop[i] = eshape[i] / 4; } /* main loop */ uint32_t ii[2]; uint32_t padding[2] = {0}; uint32_t ind = 0; uint8_t cell_aux[16]; for (ii[0] = 0; ii[0] < i_stop[0]; ++ii[0]) { for (ii[1] = 0; ii[1] < i_stop[1]; ++ii[1]) { // for each cell if (NDLZ_UNEXPECT_CONDITIONAL(ip > ip_limit)) { printf("Literal copy \n"); return 0; } if (ii[0] == i_stop[0] - 1) { padding[0] = (blockshape[0] % 4 == 0) ? 4 : blockshape[0] % 4; } else { padding[0] = 4; } if (ii[1] == i_stop[1] - 1) { padding[1] = (blockshape[1] % 4 == 0) ? 4 : blockshape[1] % 4; } else { padding[1] = 4; } token = *ip++; if (token == 0){ // no match buffercpy = ip; ip += padding[0] * padding[1]; } else if (token == (uint8_t)((1U << 7U) | (1U << 6U))) { // cell match uint16_t offset = *((uint16_t*) ip); buffercpy = ip - offset - 1; ip += 2; } else if (token == (uint8_t)(1U << 6U)) { // whole cell of same element buffercpy = cell_aux; memset(buffercpy, *ip, 16); ip++; } else if (token >= 224) { // three rows match buffercpy = local_buffer; uint16_t offset = *((uint16_t*) ip); offset += 3; ip += 2; int i, j, k; if ((token >> 3U) == 28) { i = 1; j = 2; k = 3; } else { i = 0; if ((token >> 3U) < 30) { j = 1; k = 2; } else { k = 3; if ((token >> 3U) == 30) { j = 1; } else { j = 2; } } } memcpy(&buffercpy[i * 4], ip - offset, 4); memcpy(&buffercpy[j * 4], ip - offset + 4, 4); memcpy(&buffercpy[k * 4], ip - offset + 8, 4); for (int l = 0; l < 4; l++) { if ((l != i) && (l != j) && (l != k)) { memcpy(&buffercpy[l * 4], ip, 4); ip += 4; break; } } } else if ((token >= 128) && (token <= 191)){ // rows pair match buffercpy = local_buffer; uint16_t offset = *((uint16_t*) ip); offset += 3; ip += 2; int i, j; if (token == 128) { i = 2; j = 3; } else { i = (token - 128) >> 5U; j = ((token - 128) >> 3U) - (i << 2U); } memcpy(&buffercpy[i * 4], ip - offset, 4); memcpy(&buffercpy[j * 4], ip - offset + 4, 4); for (int k = 0; k < 4; k++) { if ((k != i) && (k != j)) { memcpy(&buffercpy[k * 4], ip, 4); ip += 4; } } } else if ((token >= 40) && (token <= 63)) { // 2 rows pair matches buffercpy = local_buffer; uint16_t offset_1 = *((uint16_t*) ip); offset_1 += 5; ip += 2; uint16_t offset_2 = *((uint16_t*) ip); offset_2 += 5; ip += 2; int i, j, k, l, m; i = 0; j = ((token - 32) >> 3U); l = -1; for (k = 1; k < 4; k++) { if ((k != i) && (k != j)) { if (l == -1) { l = k; } else { m = k; } } } memcpy(&buffercpy[i * 4], ip - offset_1, 4); memcpy(&buffercpy[j * 4], ip - offset_1 + 4, 4); memcpy(&buffercpy[l * 4], ip - offset_2, 4); memcpy(&buffercpy[m * 4], ip - offset_2 + 4, 4); } else { printf("Invalid token: %u at cell [%d, %d]\n", token, ii[0], ii[1]); return 0; } // fill op with buffercpy uint32_t orig = ii[0] * 4 * blockshape[1] + ii[1] * 4; for (uint32_t i = 0; i < 4; i++) { if (i < padding[0]) { ind = orig + i * blockshape[1]; memcpy(&op[ind], buffercpy, padding[1]); } buffercpy += padding[1]; } if (ind > (uint32_t) output_len) { printf("Output size is bigger than max \n"); return 0; } } } ind += padding[1]; if (ind != (blockshape[0] * blockshape[1])) { printf("Output size is not compatible with embedded blockshape \n"); return 0; } if (ind > (uint32_t) output_len) { printf("Output size is bigger than max \n"); return 0; } return (int)ind; } zmat-0.9.9/src/blosc2/plugins/codecs/ndlz/ndlz-private.h0000644000175200007730000000146714515254731023402 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef NDLZ_PRIVATE_H #define NDLZ_PRIVATE_H #include "context.h" #if defined (__cplusplus) extern "C" { #endif #define XXH_INLINE_ALL #define NDLZ_ERROR_NULL(pointer) \ do { \ if ((pointer) == NULL) { \ return 0; \ } \ } while (0) #if defined (__cplusplus) } #endif #endif /* NDLZ_PRIVATE_H */ zmat-0.9.9/src/blosc2/plugins/codecs/ndlz/ndlz.c0000644000175200007730000000417714515254731021726 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Author: Francesc Alted Author: Oscar Griñón Author: Aleix Alcacer Creation date: 2020-06-12 See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /********************************************************************* This codec is meant to leverage multidimensionality for getting better compression ratios. The idea is to look for similarities in places that are closer in a euclidean metric, not the typical linear one. **********************************************************************/ #include #include "ndlz.h" #include "ndlz-private.h" #include "ndlz4x4.h" #include "ndlz8x8.h" int ndlz_compress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_cparams *cparams, const void* chunk) { NDLZ_ERROR_NULL(input); NDLZ_ERROR_NULL(output); NDLZ_ERROR_NULL(cparams); BLOSC_UNUSED_PARAM(chunk); switch (meta) { case 4: return ndlz4_compress(input, input_len, output, output_len, meta, cparams); case 8: return ndlz8_compress(input, input_len, output, output_len, meta, cparams); default: printf("\n NDLZ is not available for this cellsize \n"); return 0; } } int ndlz_decompress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_dparams *dparams, const void* chunk) { NDLZ_ERROR_NULL(input); NDLZ_ERROR_NULL(output); NDLZ_ERROR_NULL(dparams); BLOSC_UNUSED_PARAM(chunk); switch (meta) { case 4: return ndlz4_decompress(input, input_len, output, output_len, meta, dparams); case 8: return ndlz8_decompress(input, input_len, output, output_len, meta, dparams); default: printf("\n NDLZ is not available for this cellsize \n"); return 0; } } zmat-0.9.9/src/blosc2/plugins/codecs/ndlz/ndlz8x8.c0000644000175200007730000004445014515254731022274 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Author: Francesc Alted Author: Oscar Griñón Author: Aleix Alcacer Creation date: 2020-06-12 See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ /********************************************************************* This codec is meant to leverage multidimensionality for getting better compression ratios. The idea is to look for similarities in places that are closer in a euclidean metric, not the typical linear one. **********************************************************************/ #include #include "ndlz8x8.h" #include "ndlz.h" #include "xxhash.h" #include "../plugins/plugin_utils.h" /* * Give hints to the compiler for branch prediction optimization. */ #if defined(__GNUC__) && (__GNUC__ > 2) #define NDLZ_EXPECT_CONDITIONAL(c) (__builtin_expect((c), 1)) #define NDLZ_UNEXPECT_CONDITIONAL(c) (__builtin_expect((c), 0)) #else #define NDLZ_EXPECT_CONDITIONAL(c) (c) #define NDLZ_UNEXPECT_CONDITIONAL(c) (c) #endif /* * Use inlined functions for supported systems. */ #if defined(_MSC_VER) && !defined(__cplusplus) /* Visual Studio */ #define inline __inline /* Visual C is not C99, but supports some kind of inline */ #endif #define MAX_COPY 32U #define MAX_DISTANCE 65535 #ifdef BLOSC_STRICT_ALIGN #define NDLZ_READU16(p) ((p)[0] | (p)[1]<<8) #define NDLZ_READU32(p) ((p)[0] | (p)[1]<<8 | (p)[2]<<16 | (p)[3]<<24) #else #define NDLZ_READU16(p) *((const uint16_t*)(p)) #define NDLZ_READU32(p) *((const uint32_t*)(p)) #endif #define HASH_LOG (12) int ndlz8_compress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_cparams *cparams) { BLOSC_UNUSED_PARAM(meta); const int cell_shape = 8; const int cell_size = 64; int8_t ndim; int64_t* shape = malloc(8 * sizeof(int64_t)); int32_t* chunkshape = malloc(8 * sizeof(int32_t)); int32_t* blockshape = malloc(8 * sizeof(int32_t)); uint8_t* smeta; int32_t smeta_len; if (blosc2_meta_get(cparams->schunk, "caterva", &smeta, &smeta_len) < 0) { printf("Blosc error"); return 0; } deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape); free(smeta); if (ndim != 2) { fprintf(stderr, "This codec only works for ndim = 2"); return -1; } if (input_len != (blockshape[0] * blockshape[1])) { printf("Length not equal to blocksize \n"); return -1; } if (NDLZ_UNEXPECT_CONDITIONAL(output_len < (int) (1 + ndim * sizeof(int32_t)))) { printf("Output too small \n"); return -1; } uint8_t* ip = (uint8_t *) input; uint8_t* op = (uint8_t *) output; uint8_t* op_limit; uint32_t hval, hash_cell; uint32_t hash_triple[6] = {0}; uint32_t hash_pair[7] = {0}; uint8_t* bufarea = malloc(cell_size); uint8_t* buf_cell = bufarea; uint8_t* buf_aux; uint32_t tab_cell[1U << 12U] = {0}; uint32_t tab_triple[1U << 12U] = {0}; uint32_t tab_pair[1U << 12U] = {0}; uint32_t update_triple[6] = {0}; uint32_t update_pair[7] = {0}; // Minimum cratios before issuing and _early giveup_ // Remind that ndlz is not meant for cratios <= 2 (too costly to decompress) op_limit = op + output_len; // Initialize the hash table to distances of 0 for (unsigned i = 0; i < (1U << 12U); i++) { tab_cell[i] = 0; tab_triple[i] = 0; tab_pair[i] = 0; } /* input and output buffer cannot be less than 64 (cells are 8x8) */ int overhead = 17 + (blockshape[0] * blockshape[1] / cell_size - 1) * 2; if (input_len < cell_size || output_len < overhead) { printf("Incorrect length or maxout"); return 0; } uint8_t* obase = op; /* we start with literal copy */ *op++ = ndim; memcpy(op, &blockshape[0], 4); op += 4; memcpy(op, &blockshape[1], 4); op += 4; uint32_t i_stop[2]; for (int i = 0; i < 2; ++i) { i_stop[i] = (blockshape[i] + cell_shape - 1) / cell_shape; } /* main loop */ uint32_t padding[2]; uint32_t ii[2]; for (ii[0] = 0; ii[0] < i_stop[0]; ++ii[0]) { for (ii[1] = 0; ii[1] < i_stop[1]; ++ii[1]) { // for each cell for (int h = 0; h < 7; h++) { // new cell -> new possible references update_pair[h] = 0; if (h != 6) { update_triple[h] = 0; } } if (NDLZ_UNEXPECT_CONDITIONAL(op + cell_size + 1 > op_limit)) { free(shape); free(chunkshape); free(blockshape); free(bufarea); return 0; } uint32_t orig = ii[0] * cell_shape * blockshape[1] + ii[1] * cell_shape; if (((blockshape[0] % cell_shape != 0) && (ii[0] == i_stop[0] - 1)) || ((blockshape[1] % cell_shape != 0) && (ii[1] == i_stop[1] - 1))) { uint8_t token = 0; // padding -> literal copy *op++ = token; if (ii[0] == i_stop[0] - 1) { padding[0] = (blockshape[0] % cell_shape == 0) ? cell_shape : blockshape[0] % cell_shape; } else { padding[0] = cell_shape; } if (ii[1] == i_stop[1] - 1) { padding[1] = (blockshape[1] % cell_shape == 0) ? cell_shape : blockshape[1] % cell_shape; } else { padding[1] = cell_shape; } for (uint32_t i = 0; i < padding[0]; i++) { memcpy(op, &ip[orig + i * blockshape[1]], padding[1]); op += padding[1]; } } else { for (uint64_t i = 0; i < (uint64_t) cell_shape; i++) { // fill cell buffer uint64_t ind = orig + i * blockshape[1]; memcpy(buf_cell, &ip[ind], cell_shape); buf_cell += cell_shape; } buf_cell -= cell_size; const uint8_t* ref; uint32_t distance; uint8_t* anchor = op; /* comparison starting-point */ /* find potential match */ hash_cell = XXH32(buf_cell, cell_size, 1); // calculate cell hash hash_cell >>= 32U - 12U; ref = obase + tab_cell[hash_cell]; /* calculate distance to the match */ if (tab_cell[hash_cell] == 0) { distance = 0; } else { bool same = true; buf_aux = obase + tab_cell[hash_cell]; for(int i = 0; i < cell_size; i++){ if (buf_cell[i] != buf_aux[i]) { same = false; break; } } if (same) { distance = (int32_t) (anchor - ref); } else { distance = 0; } } bool alleq = true; for (int i = 1; i < cell_size; i++) { if (buf_cell[i] != buf_cell[0]) { alleq = false; break; } } if (alleq) { // all elements of the cell equal uint8_t token = (uint8_t) (1U << 6U); *op++ = token; *op++ = buf_cell[0]; } else if (distance == 0 || (distance >= MAX_DISTANCE)) { // no cell match bool literal = true; // rows triples matches for (int i = 0; i < 6; i++) { int triple_start = i * cell_shape; hval = XXH32(&buf_cell[triple_start], 24, 1); // calculate triple hash hval >>= 32U - 12U; /* calculate distance to the match */ bool same = true; uint16_t offset; if (tab_triple[hval] != 0) { buf_aux = obase + tab_triple[hval]; for (int l = 0; l < 24; l++) { if (buf_cell[triple_start + l] != buf_aux[l]) { same = false; break; } } offset = (uint16_t) (anchor - obase - tab_triple[hval]); } else { same = false; update_triple[i] = (uint32_t) (anchor + 1 + triple_start - obase); /* update hash table */ hash_triple[i] = hval; } ref = obase + tab_triple[hval]; if (same) { distance = (int32_t) (anchor + triple_start - ref); } else { distance = 0; } if ((distance != 0) && (distance < MAX_DISTANCE)) { // 3 rows match literal = false; uint8_t token = (uint8_t) ((21 << 3U) | i); *op++ = token; memcpy(op, &offset, 2); op += 2; for (int l = 0; l < 8; l++) { if ((l < i) || (l > i + 2)) { memcpy(op, &buf_cell[l * cell_shape], cell_shape); op += cell_shape; } } goto match; } } // rows pairs matches for (int i = 0; i < 7; i++) { int pair_start = i * cell_shape; hval = XXH32(&buf_cell[pair_start], 16, 1); // calculate rows pair hash hval >>= 32U - 12U; ref = obase + tab_pair[hval]; /* calculate distance to the match */ bool same = true; uint16_t offset; if (tab_pair[hval] != 0) { buf_aux = obase + tab_pair[hval]; for (int k = 0; k < 16; k++) { if (buf_cell[pair_start + k] != buf_aux[k]) { same = false; break; } } offset = (uint16_t) (anchor - obase - tab_pair[hval]); } else { same = false; update_pair[i] = (uint32_t) (anchor + 1 + pair_start - obase); /* update hash table */ hash_pair[i] = hval; } if (same) { distance = (int32_t) (anchor + pair_start - ref); } else { distance = 0; } if ((distance != 0) && (distance < MAX_DISTANCE)) { /* 1 rows pair match */ literal = false; uint8_t token = (uint8_t) ((17 << 3U) | i); *op++ = token; offset = (uint16_t) (anchor - obase - tab_pair[hval]); memcpy(op, &offset, 2); op += 2; for (int l = 0; l < 8; l++) { if ((l < i) || (l > i + 1)) { memcpy(op, &buf_cell[l * cell_shape], cell_shape); op += cell_shape; } } goto match; } } match: if (literal) { tab_cell[hash_cell] = (uint32_t) (anchor + 1 - obase); /* update hash tables */ if (update_triple[0] != 0) { for (int h = 0; h < 6; h++) { tab_triple[hash_triple[h]] = update_triple[h]; } } if (update_pair[0] != 0) { for (int h = 0; h < 7; h++) { tab_pair[hash_pair[h]] = update_pair[h]; } } uint8_t token = 0; *op++ = token; memcpy(op, buf_cell, cell_size); op += cell_size; } } else { // cell match uint8_t token = (uint8_t)((1U << 7U) | (1U << 6U)); *op++ = token; uint16_t offset = (uint16_t) (anchor - obase - tab_cell[hash_cell]); memcpy(op, &offset, 2); op += 2; } } if((op - obase) > input_len) { free(shape); free(chunkshape); free(blockshape); free(bufarea); return 0; } } } free(shape); free(chunkshape); free(blockshape); free(bufarea); return (int)(op - obase); } // See https://habr.com/en/company/yandex/blog/457612/ #ifdef __AVX2__ #if defined(_MSC_VER) #define ALIGNED_(x) __declspec(align(x)) #else #if defined(__GNUC__) #define ALIGNED_(x) __attribute__ ((aligned(x))) #endif #endif #define ALIGNED_TYPE_(t, x) t ALIGNED_(x) static unsigned char* copy_match_16(unsigned char *op, const unsigned char *match, int32_t len) { size_t offset = op - match; while (len >= 16) { static const ALIGNED_TYPE_(uint8_t, 16) masks[] = { 0, 1, 2, 1, 4, 1, 4, 2, 8, 7, 6, 5, 4, 3, 2, 1, // offset = 0, not used as mask, but for shift 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // offset = 1 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 0, 1, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 8, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 0, 1, 2, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 0, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, // offset = 16 }; _mm_storeu_si128((__m128i *)(op), _mm_shuffle_epi8(_mm_loadu_si128((const __m128i *)(match)), _mm_load_si128((const __m128i *)(masks) + offset))); match += masks[offset]; op += 16; len -= 16; } // Deal with remainders for (; len > 0; len--) { *op++ = *match++; } return op; } #endif int ndlz8_decompress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_dparams *dparams) { BLOSC_UNUSED_PARAM(meta); BLOSC_UNUSED_PARAM(dparams); const int cell_shape = 8; const int cell_size = 64; uint8_t* ip = (uint8_t*)input; uint8_t* ip_limit = ip + input_len; uint8_t* op = (uint8_t*)output; uint8_t ndim; int32_t blockshape[2]; int32_t eshape[2]; uint8_t* buffercpy; uint8_t token; if (NDLZ_UNEXPECT_CONDITIONAL(input_len < 8)) { return 0; } /* we start with literal copy */ ndim = *ip; ip ++; if (ndim != 2) { fprintf(stderr, "This codec only works for ndim = 2"); return -1; } memcpy(&blockshape[0], ip, 4); ip += 4; memcpy(&blockshape[1], ip, 4); ip += 4; eshape[0] = ((blockshape[0] + 7) / cell_shape) * cell_shape; eshape[1] = ((blockshape[1] + 7) / cell_shape) * cell_shape; if (NDLZ_UNEXPECT_CONDITIONAL(output_len < blockshape[0] * blockshape[1])) { return 0; } memset(op, 0, blockshape[0] * blockshape[1]); int32_t i_stop[2]; for (int i = 0; i < 2; ++i) { i_stop[i] = eshape[i] / cell_shape; } /* main loop */ int32_t ii[2]; int32_t padding[2] = {0}; int32_t ind = 0; uint8_t* local_buffer = malloc(cell_size); uint8_t* cell_aux = malloc(cell_size); for (ii[0] = 0; ii[0] < i_stop[0]; ++ii[0]) { for (ii[1] = 0; ii[1] < i_stop[1]; ++ii[1]) { // for each cell if (NDLZ_UNEXPECT_CONDITIONAL(ip > ip_limit)) { printf("Literal copy \n"); free(local_buffer); free(cell_aux); return 0; } if (ii[0] == i_stop[0] - 1) { padding[0] = (blockshape[0] % cell_shape == 0) ? cell_shape : blockshape[0] % cell_shape; } else { padding[0] = cell_shape; } if (ii[1] == i_stop[1] - 1) { padding[1] = (blockshape[1] % cell_shape == 0) ? cell_shape : blockshape[1] % cell_shape; } else { padding[1] = cell_shape; } token = *ip++; uint8_t match_type = (token >> 3U); if (token == 0){ // no match buffercpy = ip; ip += padding[0] * padding[1]; } else if (token == (uint8_t)((1U << 7U) | (1U << 6U))) { // cell match uint16_t offset = *((uint16_t*) ip); buffercpy = ip - offset - 1; ip += 2; } else if (token == (uint8_t)(1U << 6U)) { // whole cell of same element buffercpy = cell_aux; memset(buffercpy, *ip, cell_size); ip++; } else if (match_type == 21) { // triple match buffercpy = local_buffer; int row = (int) (token & 7); uint16_t offset = *((uint16_t*) ip); ip += 2; for (int l = 0; l < 3; l++) { memcpy(&buffercpy[(row + l) * cell_shape], ip - sizeof(token) - sizeof(offset) - offset + l * cell_shape, cell_shape); } for (int l = 0; l < cell_shape; l++) { if ((l < row) || (l > row + 2)) { memcpy(&buffercpy[l * cell_shape], ip, cell_shape); ip += cell_shape; } } } else if (match_type == 17) { // pair match buffercpy = local_buffer; int row = (int) (token & 7); uint16_t offset = *((uint16_t*) ip); ip += 2; for (int l = 0; l < 2; l++) { memcpy(&buffercpy[(row + l) * cell_shape], ip - sizeof(token) - sizeof(offset) - offset + l * cell_shape, cell_shape); } for (int l = 0; l < cell_shape; l++) { if ((l < row) || (l > row + 1)) { memcpy(&buffercpy[l * cell_shape], ip, cell_shape); ip += cell_shape; } } } else { printf("Invalid token: %u at cell [%d, %d]\n", token, ii[0], ii[1]); free(local_buffer); free(cell_aux); return 0; } uint32_t orig = ii[0] * cell_shape * blockshape[1] + ii[1] * cell_shape; for (int32_t i = 0; i < (int32_t) cell_shape; i++) { if (i < padding[0]) { ind = orig + i * blockshape[1]; memcpy(&op[ind], buffercpy, padding[1]); } buffercpy += padding[1]; } if (ind > output_len) { printf("Output size is bigger than max \n"); free(local_buffer); free(cell_aux); return 0; } } } ind += padding[1]; free(cell_aux); free(local_buffer); if (ind != (blockshape[0] * blockshape[1])) { printf("Output size is not compatible with embedded blockshape \n"); return 0; } if (ind > output_len) { printf("Output size is bigger than max \n"); return 0; } return (int)ind; } zmat-0.9.9/src/blosc2/plugins/codecs/ndlz/README.md0000644000175200007730000000625014515254731022064 0ustar rlaboissrlaboissNDLZ: a multidimensional lossless codec ============================================================================= Given a 2-dim array or matrix, *NDLZ* is a compressor based on the Lempel-Ziv algorithm of lossless data compression. Plugin motivation -------------------- *NDLZ* was created in order to search for patterns repetitions in multidimensional cells using the Caterva blocking machinery. Plugin usage ------------------- The codec consists of an encoder called *ndlz_compress()* to codify data and a decoder called *ndlz_decompress()* to recover the original data. The parameters used by *NDLZ* are the ones specified in the *blosc2_codec* structure of *blosc2.h*. Furthermore, since *NDLZ* goes through dataset blocks dividing them into fixed size cells, user must specify the parameter meta as 4 to use cells of size 4x4 or 8 to use 8x8 cells. If user tries to use other value for meta, the codec will return an error value. NDLZ only works for 2-dim datasets of 1 byte items (typesize = 1), so if you want to use it for a dataset with bigger typesize then you must activate SHUFFLE filter and splitting mode. Plugin behaviour ------------------- This codec is meant to leverage multidimensionality for getting better compression ratios. The idea is to look for similarities in places that are closer in a euclidean metric, not the typical linear one. First *NDLZ* goes through dataset blocks dividing them into fixed size cells. Then, for each cell the codec searches for data coincidences with previous cells in order to copy only references to those cells instead of copying the full current cell. To understand how the compressor and decompressor work it is important to learn about the compressed block format. An *NDLZ* compressed block is composed of a not-compressed byte called token and some 2 bytes values called offsets. The token is divided in two fields. The first field is composed of the 2 first bits of the token and gives important information about the cells and rows couples matches. The high-bit of the field is activated when there exists repeated information (there are matches) and there exists offset. If it is activated, the other field indicates special patterns of matches, and if not we have to look at the second bit. If it is activated (token = 01000000), this means that the whole cell is composed of the same element, and if not (token = 00000000) there is not repeated information and the whole cell is literally copied. The offsets are references to previous literal copies that match with the data that is being evaluated at the moment. Otherwise, it is important to know that there exist different hash tables which store the references to the literal copies of cells and rows in their hash position. Advantages and disadvantages ------------------------------ The main advantage of *NDLZ* in front of most of the codecs is that this one considers dataset multidimensionality and takes advantage of it instead of processing all data as serial. The main disadvantage of *NDLZ* is that it is only useful for 2-dim datasets and at the moment other more developed codecs that do not consider multidimensionality obtain better results (times and ratios) for 2-dim datasets. zmat-0.9.9/src/blosc2/plugins/codecs/ndlz/test_ndlz.c0000644000175200007730000001640414515254731022761 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. Test program demonstrating use of the Blosc filter from C code. To compile this program: $ gcc -O test_ndlz.c -o test_ndlz -lblosc2 To run: $ ./test_ndlz Blosc version info: 2.0.0a6.dev ($Date:: 2018-05-18 #$) Successful roundtrip! Compression: 1792 -> 1630 (1.1x) Successful roundtrip! Compression: 1792 -> 1749 (1.0x) same_cells: 43 obtained Successful roundtrip! Compression: 16128 -> 2579 (6.3x) Successful roundtrip! Compression: 16128 -> 3829 (4.2x) some_matches: 12299 obtained **********************************************************************/ #include #include "blosc2.h" #include "blosc2/codecs-registry.h" #include static int test_ndlz_4(blosc2_schunk* schunk) { int64_t nchunks = schunk->nchunks; int32_t chunksize = schunk->chunksize; uint8_t *data_in = malloc(chunksize); int decompressed; int64_t csize; int64_t dsize; int64_t csize_f = 0; uint8_t *data_out = malloc(chunksize + BLOSC2_MAX_OVERHEAD); uint8_t *data_dest = malloc(chunksize); /* Create a context for compression */ blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_ALWAYS_SPLIT; cparams.typesize = schunk->typesize; cparams.compcode = BLOSC_CODEC_NDLZ; cparams.compcode_meta = 4; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_SHUFFLE; cparams.clevel = 5; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress with clevel=5 and shuffle active */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, data_out, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } csize_f += csize; /* Decompress */ dsize = blosc2_decompress_ctx(dctx, data_out, chunksize + BLOSC2_MAX_OVERHEAD, data_dest, chunksize); if (dsize <= 0) { printf("Decompression error. Error code: %" PRId64 "\n", dsize); return (int) dsize; } for (int i = 0; i < chunksize; i++) { if (data_in[i] != data_dest[i]) { printf("i: %d, data %u, dest %u", i, data_in[i], data_dest[i]); printf("\n Decompressed data differs from original!\n"); return -1; } } } csize_f = csize_f / nchunks; free(data_in); free(data_out); free(data_dest); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); printf("Successful roundtrip!\n"); printf("Compression: %d -> %" PRId64 " (%.1fx)\n", chunksize, csize_f, (1. * chunksize) / (double)csize_f); return (int) (chunksize - csize_f); } static int test_ndlz_8(blosc2_schunk* schunk) { int64_t nchunks = schunk->nchunks; int32_t chunksize = (int32_t) (schunk->chunksize); // int isize = (int) array->extchunknitems * typesize; uint8_t *data_in = malloc(chunksize); int decompressed; int64_t csize; int64_t dsize; int64_t csize_f = 0; uint8_t *data_out = malloc(chunksize + BLOSC2_MAX_OVERHEAD); uint8_t *data_dest = malloc(chunksize); /* Create a context for compression */ blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_ALWAYS_SPLIT; cparams.typesize = schunk->typesize; cparams.compcode = BLOSC_CODEC_NDLZ; cparams.compcode_meta = 8; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_SHUFFLE; cparams.clevel = 5; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress with clevel=5 and shuffle active */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, data_out, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } csize_f += csize; /* Decompress */ dsize = blosc2_decompress_ctx(dctx, data_out, chunksize + BLOSC2_MAX_OVERHEAD, data_dest, chunksize); if (dsize <= 0) { printf("Decompression error. Error code: %" PRId64 "\n", dsize); return (int) dsize; } for (int i = 0; i < chunksize; i++) { if (data_in[i] != data_dest[i]) { printf("i: %d, data %u, dest %u", i, data_in[i], data_dest[i]); printf("\n Decompressed data differs from original!\n"); return -1; } } } csize_f = csize_f / nchunks; free(data_in); free(data_out); free(data_dest); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); printf("Successful roundtrip!\n"); printf("Compression: %d -> %" PRId64 " (%.1fx)\n", chunksize, csize_f, (1. * chunksize) / (double)csize_f); return (int) (chunksize - csize_f); } int same_cells() { blosc2_schunk *schunk = blosc2_schunk_open("example_same_cells.caterva"); /* Run the test. */ int result = test_ndlz_4(schunk); if (result < 0) { blosc2_schunk_free(schunk); return result; } result = test_ndlz_8(schunk); blosc2_schunk_free(schunk); return result; } int some_matches() { blosc2_schunk *schunk = blosc2_schunk_open("example_some_matches.caterva"); /* Run the test. */ int result = test_ndlz_4(schunk); if (result < 0) { blosc2_schunk_free(schunk); return result; } result = test_ndlz_8(schunk); blosc2_schunk_free(schunk); return result; } int main(void) { int result; blosc2_init(); // this is mandatory for initiallizing the plugin mechanism result = same_cells(); printf("same_cells: %d obtained \n \n", result); result = some_matches(); printf("some_matches: %d obtained \n \n", result); blosc2_destroy(); return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/plugins/codecs/ndlz/ndlz.h0000644000175200007730000000160114515254731021720 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef NDLZ_H #define NDLZ_H #include "context.h" #if defined (__cplusplus) extern "C" { #endif int ndlz_compress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_cparams *cparams, const void* chunk); int ndlz_decompress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_dparams *dparams, const void* chunk); #if defined (__cplusplus) } #endif #endif /* NDLZ_H */ zmat-0.9.9/src/blosc2/plugins/codecs/ndlz/CMakeLists.txt0000644000175200007730000000205514515254731023344 0ustar rlaboissrlaboiss# sources set(SOURCES ${SOURCES} ../plugins/codecs/ndlz/ndlz.c ../plugins/codecs/ndlz/ndlz.h ../plugins/codecs/ndlz/ndlz-private.h ../plugins/codecs/ndlz/ndlz4x4.c ../plugins/codecs/ndlz/ndlz4x4.h ../plugins/codecs/ndlz/ndlz8x8.c ../plugins/codecs/ndlz/ndlz8x8.h ../plugins/codecs/ndlz/xxhash.c ../plugins/codecs/ndlz/xxhash.h PARENT_SCOPE) # targets if(BUILD_TESTS) add_executable(test_ndlz test_ndlz.c) # Define the BLOSC_TESTING symbol so normally-hidden functions # aren't hidden from the view of the test programs. set_property( TARGET test_ndlz APPEND PROPERTY COMPILE_DEFINITIONS BLOSC_TESTING) target_link_libraries(test_ndlz blosc_testing) # tests add_test(NAME test_plugin_test_ndlz COMMAND ${CMAKE_CROSSCOMPILING_EMULATOR} $) # Copy test files file(GLOB TESTS_DATA ../../test_data/example_s*.caterva) foreach (data ${TESTS_DATA}) file(COPY ${data} DESTINATION ${CMAKE_CURRENT_BINARY_DIR}/) endforeach(data) endif() zmat-0.9.9/src/blosc2/plugins/codecs/ndlz/ndlz8x8.h0000644000175200007730000000402714515254731022275 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef NDLZ8_H #define NDLZ8_H #include "context.h" #if defined (__cplusplus) extern "C" { #endif #include "ndlz.h" #include "ndlz-private.h" /** Compress a block of data in the input buffer and returns the size of compressed block. The size of input buffer is specified by length. The minimum input buffer size is 16. The output buffer must be at least 5% larger than the input buffer and can not be smaller than 66 bytes. If the input is not compressible, or output does not fit in maxout bytes, the return value will be 0 and you will have to discard the output buffer. The acceleration parameter is related with the frequency for updating the internal hash. An acceleration of 1 means that the internal hash is updated at full rate. A value < 1 is not allowed and will be silently set to 1. The input buffer and the output buffer can not overlap. */ int ndlz8_compress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_cparams *cparams); /** Decompress a block of compressed data and returns the size of the decompressed block. If error occurs, e.g. the compressed data is corrupted or the output buffer is not large enough, then 0 (zero) will be returned instead. The input buffer and the output buffer can not overlap. Decompression is memory safe and guaranteed not to write the output buffer more than what is specified in maxout. */ int ndlz8_decompress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_dparams *dparams); #if defined (__cplusplus) } #endif #endif /* NDLZ8_H */ zmat-0.9.9/src/blosc2/plugins/codecs/ndlz/xxhash.h0000644000175200007730000060460514515254731022271 0ustar rlaboissrlaboiss/* * xxHash - Extremely Fast Hash algorithm * Header File * Copyright (C) 2012-2020 Yann Collet * * BSD 2-Clause License (https://www.opensource.org/licenses/bsd-license.php) * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are * met: * * * Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above * copyright notice, this list of conditions and the following disclaimer * in the documentation and/or other materials provided with the * distribution. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * You can contact the author at: * - xxHash homepage: https://www.xxhash.com * - xxHash source repository: https://github.com/Cyan4973/xxHash */ /*! * @mainpage xxHash * * @file xxhash.h * xxHash prototypes and implementation */ /* TODO: update */ /* Notice extracted from xxHash homepage: xxHash is an extremely fast hash algorithm, running at RAM speed limits. It also successfully passes all tests from the SMHasher suite. Comparison (single thread, Windows Seven 32 bits, using SMHasher on a Core 2 Duo @3GHz) Name Speed Q.Score Author xxHash 5.4 GB/s 10 CrapWow 3.2 GB/s 2 Andrew MurmurHash 3a 2.7 GB/s 10 Austin Appleby SpookyHash 2.0 GB/s 10 Bob Jenkins SBox 1.4 GB/s 9 Bret Mulvey Lookup3 1.2 GB/s 9 Bob Jenkins SuperFastHash 1.2 GB/s 1 Paul Hsieh CityHash64 1.05 GB/s 10 Pike & Alakuijala FNV 0.55 GB/s 5 Fowler, Noll, Vo CRC32 0.43 GB/s 9 MD5-32 0.33 GB/s 10 Ronald L. Rivest SHA1-32 0.28 GB/s 10 Q.Score is a measure of quality of the hash function. It depends on successfully passing SMHasher test set. 10 is a perfect score. Note: SMHasher's CRC32 implementation is not the fastest one. Other speed-oriented implementations can be faster, especially in combination with PCLMUL instruction: https://fastcompression.blogspot.com/2019/03/presenting-xxh3.html?showComment=1552696407071#c3490092340461170735 A 64-bit version, named XXH64, is available since r35. It offers much better speed, but for 64-bit applications only. Name Speed on 64 bits Speed on 32 bits XXH64 13.8 GB/s 1.9 GB/s XXH32 6.8 GB/s 6.0 GB/s */ #if defined (__cplusplus) extern "C" { #endif /* **************************** * INLINE mode ******************************/ /*! * XXH_INLINE_ALL (and XXH_PRIVATE_API) * Use these build macros to inline xxhash into the target unit. * Inlining improves performance on small inputs, especially when the length is * expressed as a compile-time constant: * * https://fastcompression.blogspot.com/2018/03/xxhash-for-small-keys-impressive-power.html * * It also keeps xxHash symbols private to the unit, so they are not exported. * * Usage: * #define XXH_INLINE_ALL * #include "xxhash.h" * * Do not compile and link xxhash.o as a separate object, as it is not useful. */ #if (defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)) \ && !defined(XXH_INLINE_ALL_31684351384) /* this section should be traversed only once */ # define XXH_INLINE_ALL_31684351384 /* give access to the advanced API, required to compile implementations */ # undef XXH_STATIC_LINKING_ONLY /* avoid macro redef */ # define XXH_STATIC_LINKING_ONLY /* make all functions private */ # undef XXH_PUBLIC_API # if defined(__GNUC__) # define XXH_PUBLIC_API static __inline __attribute__((unused)) # elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) # define XXH_PUBLIC_API static inline # elif defined(_MSC_VER) # define XXH_PUBLIC_API static __inline # else /* note: this version may generate warnings for unused static functions */ # define XXH_PUBLIC_API static # endif /* * This part deals with the special case where a unit wants to inline xxHash, * but "xxhash.h" has previously been included without XXH_INLINE_ALL, such * as part of some previously included *.h header file. * Without further action, the new include would just be ignored, * and functions would effectively _not_ be inlined (silent failure). * The following macros solve this situation by prefixing all inlined names, * avoiding naming collision with previous inclusions. */ # ifdef XXH_NAMESPACE # error "XXH_INLINE_ALL with XXH_NAMESPACE is not supported" /* * Note: Alternative: #undef all symbols (it's a pretty large list). * Without #error: it compiles, but functions are actually not inlined. */ # endif # define XXH_NAMESPACE XXH_INLINE_ /* * Some identifiers (enums, type names) are not symbols, but they must * still be renamed to avoid redeclaration. * Alternative solution: do not redeclare them. * However, this requires some #ifdefs, and is a more dispersed action. * Meanwhile, renaming can be achieved in a single block */ # define XXH_IPREF(Id) XXH_INLINE_ ## Id # define XXH_OK XXH_IPREF(XXH_OK) # define XXH_ERROR XXH_IPREF(XXH_ERROR) # define XXH_errorcode XXH_IPREF(XXH_errorcode) # define XXH32_canonical_t XXH_IPREF(XXH32_canonical_t) # define XXH64_canonical_t XXH_IPREF(XXH64_canonical_t) # define XXH128_canonical_t XXH_IPREF(XXH128_canonical_t) # define XXH32_state_s XXH_IPREF(XXH32_state_s) # define XXH32_state_t XXH_IPREF(XXH32_state_t) # define XXH64_state_s XXH_IPREF(XXH64_state_s) # define XXH64_state_t XXH_IPREF(XXH64_state_t) # define XXH3_state_s XXH_IPREF(XXH3_state_s) # define XXH3_state_t XXH_IPREF(XXH3_state_t) # define XXH128_hash_t XXH_IPREF(XXH128_hash_t) /* Ensure the header is parsed again, even if it was previously included */ # undef XXHASH_H_5627135585666179 # undef XXHASH_H_STATIC_13879238742 #endif /* XXH_INLINE_ALL || XXH_PRIVATE_API */ /* **************************************************************** * Stable API *****************************************************************/ #ifndef XXHASH_H_5627135585666179 #define XXHASH_H_5627135585666179 1 /*! * @defgroup public Public API * Contains details on the public xxHash functions. * @{ */ /* specific declaration modes for Windows */ #if !defined(XXH_INLINE_ALL) && !defined(XXH_PRIVATE_API) # if defined(WIN32) && defined(_MSC_VER) && (defined(XXH_IMPORT) || defined(XXH_EXPORT)) # ifdef XXH_EXPORT # define XXH_PUBLIC_API __declspec(dllexport) # elif XXH_IMPORT # define XXH_PUBLIC_API __declspec(dllimport) # endif # else # define XXH_PUBLIC_API /* do nothing */ # endif #endif #ifdef XXH_DOXYGEN /*! * @brief Emulate a namespace by transparently prefixing all symbols. * * If you want to include _and expose_ xxHash functions from within your own * library, but also want to avoid symbol collisions with other libraries which * may also include xxHash, you can use XXH_NAMESPACE to automatically prefix * any public symbol from xxhash library with the value of XXH_NAMESPACE * (therefore, avoid empty or numeric values). * * Note that no change is required within the calling program as long as it * includes `xxhash.h`: Regular symbol names will be automatically translated * by this header. */ # define XXH_NAMESPACE /* YOUR NAME HERE */ # undef XXH_NAMESPACE #endif #undef XXH_NAMESPACE # define XXH_NAMESPACE NDLZ #ifdef XXH_NAMESPACE # define XXH_CAT(A,B) A##B # define XXH_NAME2(A,B) XXH_CAT(A,B) # define XXH_versionNumber XXH_NAME2(XXH_NAMESPACE, XXH_versionNumber) /* XXH32 */ # define XXH32 XXH_NAME2(XXH_NAMESPACE, XXH32) # define XXH32_createState XXH_NAME2(XXH_NAMESPACE, XXH32_createState) # define XXH32_freeState XXH_NAME2(XXH_NAMESPACE, XXH32_freeState) # define XXH32_reset XXH_NAME2(XXH_NAMESPACE, XXH32_reset) # define XXH32_update XXH_NAME2(XXH_NAMESPACE, XXH32_update) # define XXH32_digest XXH_NAME2(XXH_NAMESPACE, XXH32_digest) # define XXH32_copyState XXH_NAME2(XXH_NAMESPACE, XXH32_copyState) # define XXH32_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH32_canonicalFromHash) # define XXH32_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH32_hashFromCanonical) /* XXH64 */ # define XXH64 XXH_NAME2(XXH_NAMESPACE, XXH64) # define XXH64_createState XXH_NAME2(XXH_NAMESPACE, XXH64_createState) # define XXH64_freeState XXH_NAME2(XXH_NAMESPACE, XXH64_freeState) # define XXH64_reset XXH_NAME2(XXH_NAMESPACE, XXH64_reset) # define XXH64_update XXH_NAME2(XXH_NAMESPACE, XXH64_update) # define XXH64_digest XXH_NAME2(XXH_NAMESPACE, XXH64_digest) # define XXH64_copyState XXH_NAME2(XXH_NAMESPACE, XXH64_copyState) # define XXH64_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH64_canonicalFromHash) # define XXH64_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH64_hashFromCanonical) /* XXH3_64bits */ # define XXH3_64bits XXH_NAME2(XXH_NAMESPACE, XXH3_64bits) # define XXH3_64bits_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSecret) # define XXH3_64bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSeed) # define XXH3_createState XXH_NAME2(XXH_NAMESPACE, XXH3_createState) # define XXH3_freeState XXH_NAME2(XXH_NAMESPACE, XXH3_freeState) # define XXH3_copyState XXH_NAME2(XXH_NAMESPACE, XXH3_copyState) # define XXH3_64bits_reset XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset) # define XXH3_64bits_reset_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset_withSeed) # define XXH3_64bits_reset_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset_withSecret) # define XXH3_64bits_update XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_update) # define XXH3_64bits_digest XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_digest) # define XXH3_generateSecret XXH_NAME2(XXH_NAMESPACE, XXH3_generateSecret) /* XXH3_128bits */ # define XXH128 XXH_NAME2(XXH_NAMESPACE, XXH128) # define XXH3_128bits XXH_NAME2(XXH_NAMESPACE, XXH3_128bits) # define XXH3_128bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSeed) # define XXH3_128bits_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSecret) # define XXH3_128bits_reset XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset) # define XXH3_128bits_reset_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset_withSeed) # define XXH3_128bits_reset_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset_withSecret) # define XXH3_128bits_update XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_update) # define XXH3_128bits_digest XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_digest) # define XXH128_isEqual XXH_NAME2(XXH_NAMESPACE, XXH128_isEqual) # define XXH128_cmp XXH_NAME2(XXH_NAMESPACE, XXH128_cmp) # define XXH128_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH128_canonicalFromHash) # define XXH128_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH128_hashFromCanonical) #endif /* ************************************* * Version ***************************************/ #define XXH_VERSION_MAJOR 0 #define XXH_VERSION_MINOR 8 #define XXH_VERSION_RELEASE 0 #define XXH_VERSION_NUMBER (XXH_VERSION_MAJOR *100*100 + XXH_VERSION_MINOR *100 + XXH_VERSION_RELEASE) /*! * @brief Obtains the xxHash version. * * This is only useful when xxHash is compiled as a shared library, as it is * independent of the version defined in the header. * * @return `XXH_VERSION_NUMBER` as of when the function was compiled. */ XXH_PUBLIC_API unsigned XXH_versionNumber (void); /* **************************** * Definitions ******************************/ #include /* size_t */ typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode; /*-********************************************************************** * 32-bit hash ************************************************************************/ #if defined(XXH_DOXYGEN) /* Don't show include */ /*! * @brief An unsigned 32-bit integer. * * Not necessarily defined to `uint32_t` but functionally equivalent. */ typedef uint32_t XXH32_hash_t; #elif !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) # include typedef uint32_t XXH32_hash_t; #else # include # if UINT_MAX == 0xFFFFFFFFUL typedef unsigned int XXH32_hash_t; # else # if ULONG_MAX == 0xFFFFFFFFUL typedef unsigned long XXH32_hash_t; # else # error "unsupported platform: need a 32-bit type" # endif # endif #endif /*! * @} * * @defgroup xxh32_family XXH32 family * @ingroup public * Contains functions used in the classic 32-bit xxHash algorithm. * * @note * XXH32 is considered rather weak by today's standards. * The @ref xxh3_family provides competitive speed for both 32-bit and 64-bit * systems, and offers true 64/128 bit hash results. It provides a superior * level of dispersion, and greatly reduces the risks of collisions. * * @see @ref xxh64_family, @ref xxh3_family : Other xxHash families * @see @ref xxh32_impl for implementation details * @{ */ /*! * @brief Calculates the 32-bit hash of @p input using xxHash32. * * Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark): 5.4 GB/s * * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * @param seed The 32-bit seed to alter the hash's output predictably. * * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return The calculated 32-bit hash value. * * @see * XXH64(), XXH3_64bits_withSeed(), XXH3_128bits_withSeed(), XXH128(): * Direct equivalents for the other variants of xxHash. * @see * XXH32_createState(), XXH32_update(), XXH32_digest(): Streaming version. */ XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t length, XXH32_hash_t seed); /*! * Streaming functions generate the xxHash value from an incremental input. * This method is slower than single-call functions, due to state management. * For small inputs, prefer `XXH32()` and `XXH64()`, which are better optimized. * * An XXH state must first be allocated using `XXH*_createState()`. * * Start a new hash by initializing the state with a seed using `XXH*_reset()`. * * Then, feed the hash state by calling `XXH*_update()` as many times as necessary. * * The function returns an error code, with 0 meaning OK, and any other value * meaning there is an error. * * Finally, a hash value can be produced anytime, by using `XXH*_digest()`. * This function returns the nn-bits hash as an int or long long. * * It's still possible to continue inserting input into the hash state after a * digest, and generate new hash values later on by invoking `XXH*_digest()`. * * When done, release the state using `XXH*_freeState()`. * * Example code for incrementally hashing a file: * @code{.c} * #include * #include * #define BUFFER_SIZE 256 * * // Note: XXH64 and XXH3 use the same interface. * XXH32_hash_t * hashFile(FILE* stream) * { * XXH32_state_t* state; * unsigned char buf[BUFFER_SIZE]; * size_t amt; * XXH32_hash_t hash; * * state = XXH32_createState(); // Create a state * assert(state != NULL); // Error check here * XXH32_reset(state, 0xbaad5eed); // Reset state with our seed * while ((amt = fread(buf, 1, sizeof(buf), stream)) != 0) { * XXH32_update(state, buf, amt); // Hash the file in chunks * } * hash = XXH32_digest(state); // Finalize the hash * XXH32_freeState(state); // Clean up * return hash; * } * @endcode */ /*! * @typedef struct XXH32_state_s XXH32_state_t * @brief The opaque state struct for the XXH32 streaming API. * * @see XXH32_state_s for details. */ typedef struct XXH32_state_s XXH32_state_t; /*! * @brief Allocates an @ref XXH32_state_t. * * Must be freed with XXH32_freeState(). * @return An allocated XXH32_state_t on success, `NULL` on failure. */ XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void); /*! * @brief Frees an @ref XXH32_state_t. * * Must be allocated with XXH32_createState(). * @param statePtr A pointer to an @ref XXH32_state_t allocated with @ref XXH32_createState(). * @return XXH_OK. */ XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr); /*! * @brief Copies one @ref XXH32_state_t to another. * * @param dst_state The state to copy to. * @param src_state The state to copy from. * @pre * @p dst_state and @p src_state must not be `NULL` and must not overlap. */ XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dst_state, const XXH32_state_t* src_state); /*! * @brief Resets an @ref XXH32_state_t to begin a new hash. * * This function resets and seeds a state. Call it before @ref XXH32_update(). * * @param statePtr The state struct to reset. * @param seed The 32-bit seed to alter the hash result predictably. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH32_reset (XXH32_state_t* statePtr, XXH32_hash_t seed); /*! * @brief Consumes a block of @p input to an @ref XXH32_state_t. * * Call this to incrementally consume blocks of data. * * @param statePtr The state struct to update. * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * * @pre * @p statePtr must not be `NULL`. * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length); /*! * @brief Returns the calculated hash value from an @ref XXH32_state_t. * * @note * Calling XXH32_digest() will not affect @p statePtr, so you can update, * digest, and update again. * * @param statePtr The state struct to calculate the hash from. * * @pre * @p statePtr must not be `NULL`. * * @return The calculated xxHash32 value from that state. */ XXH_PUBLIC_API XXH32_hash_t XXH32_digest (const XXH32_state_t* statePtr); /******* Canonical representation *******/ /* * The default return values from XXH functions are unsigned 32 and 64 bit * integers. * This the simplest and fastest format for further post-processing. * * However, this leaves open the question of what is the order on the byte level, * since little and big endian conventions will store the same number differently. * * The canonical representation settles this issue by mandating big-endian * convention, the same convention as human-readable numbers (large digits first). * * When writing hash values to storage, sending them over a network, or printing * them, it's highly recommended to use the canonical representation to ensure * portability across a wider range of systems, present and future. * * The following functions allow transformation of hash values to and from * canonical format. */ /*! * @brief Canonical (big endian) representation of @ref XXH32_hash_t. */ typedef struct { unsigned char digest[4]; /*!< Hash bytes, big endian */ } XXH32_canonical_t; /*! * @brief Converts an @ref XXH32_hash_t to a big endian @ref XXH32_canonical_t. * * @param dst The @ref XXH32_canonical_t pointer to be stored to. * @param hash The @ref XXH32_hash_t to be converted. * * @pre * @p dst must not be `NULL`. */ XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash); /*! * @brief Converts an @ref XXH32_canonical_t to a native @ref XXH32_hash_t. * * @param src The @ref XXH32_canonical_t to convert. * * @pre * @p src must not be `NULL`. * * @return The converted hash. */ XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src); /*! * @} * @ingroup public * @{ */ #ifndef XXH_NO_LONG_LONG /*-********************************************************************** * 64-bit hash ************************************************************************/ #if defined(XXH_DOXYGEN) /* don't include */ /*! * @brief An unsigned 64-bit integer. * * Not necessarily defined to `uint64_t` but functionally equivalent. */ typedef uint64_t XXH64_hash_t; #elif !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) # include typedef uint64_t XXH64_hash_t; #else # include # if defined(__LP64__) && ULONG_MAX == 0xFFFFFFFFFFFFFFFFULL /* LP64 ABI says uint64_t is unsigned long */ typedef unsigned long XXH64_hash_t; # else /* the following type must have a width of 64-bit */ typedef unsigned long long XXH64_hash_t; # endif #endif /*! * @} * * @defgroup xxh64_family XXH64 family * @ingroup public * @{ * Contains functions used in the classic 64-bit xxHash algorithm. * * @note * XXH3 provides competitive speed for both 32-bit and 64-bit systems, * and offers true 64/128 bit hash results. It provides a superior level of * dispersion, and greatly reduces the risks of collisions. */ /*! * @brief Calculates the 64-bit hash of @p input using xxHash64. * * This function usually runs faster on 64-bit systems, but slower on 32-bit * systems (see benchmark). * * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * @param seed The 64-bit seed to alter the hash's output predictably. * * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return The calculated 64-bit hash. * * @see * XXH32(), XXH3_64bits_withSeed(), XXH3_128bits_withSeed(), XXH128(): * Direct equivalents for the other variants of xxHash. * @see * XXH64_createState(), XXH64_update(), XXH64_digest(): Streaming version. */ XXH_PUBLIC_API XXH64_hash_t XXH64(const void* input, size_t length, XXH64_hash_t seed); /******* Streaming *******/ /*! * @brief The opaque state struct for the XXH64 streaming API. * * @see XXH64_state_s for details. */ typedef struct XXH64_state_s XXH64_state_t; /* incomplete type */ XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void); XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr); XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dst_state, const XXH64_state_t* src_state); XXH_PUBLIC_API XXH_errorcode XXH64_reset (XXH64_state_t* statePtr, XXH64_hash_t seed); XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* statePtr, const void* input, size_t length); XXH_PUBLIC_API XXH64_hash_t XXH64_digest (const XXH64_state_t* statePtr); /******* Canonical representation *******/ typedef struct { unsigned char digest[sizeof(XXH64_hash_t)]; } XXH64_canonical_t; XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash); XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src); /*! * @} * ************************************************************************ * @defgroup xxh3_family XXH3 family * @ingroup public * @{ * * XXH3 is a more recent hash algorithm featuring: * - Improved speed for both small and large inputs * - True 64-bit and 128-bit outputs * - SIMD acceleration * - Improved 32-bit viability * * Speed analysis methodology is explained here: * * https://fastcompression.blogspot.com/2019/03/presenting-xxh3.html * * Compared to XXH64, expect XXH3 to run approximately * ~2x faster on large inputs and >3x faster on small ones, * exact differences vary depending on platform. * * XXH3's speed benefits greatly from SIMD and 64-bit arithmetic, * but does not require it. * Any 32-bit and 64-bit targets that can run XXH32 smoothly * can run XXH3 at competitive speeds, even without vector support. * Further details are explained in the implementation. * * Optimized implementations are provided for AVX512, AVX2, SSE2, NEON, POWER8, * ZVector and scalar targets. This can be controlled via the XXH_VECTOR macro. * * XXH3 implementation is portable: * it has a generic C90 formulation that can be compiled on any platform, * all implementations generage exactly the same hash value on all platforms. * Starting from v0.8.0, it's also labelled "stable", meaning that * any future version will also generate the same hash value. * * XXH3 offers 2 variants, _64bits and _128bits. * * When only 64 bits are needed, prefer invoking the _64bits variant, as it * reduces the amount of mixing, resulting in faster speed on small inputs. * It's also generally simpler to manipulate a scalar return type than a struct. * * The API supports one-shot hashing, streaming mode, and custom secrets. */ /*-********************************************************************** * XXH3 64-bit variant ************************************************************************/ /* XXH3_64bits(): * default 64-bit variant, using default secret and default seed of 0. * It's the fastest variant. */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits(const void* data, size_t len); /* * XXH3_64bits_withSeed(): * This variant generates a custom secret on the fly * based on default secret altered using the `seed` value. * While this operation is decently fast, note that it's not completely free. * Note: seed==0 produces the same results as XXH3_64bits(). */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSeed(const void* data, size_t len, XXH64_hash_t seed); /*! * The bare minimum size for a custom secret. * * @see * XXH3_64bits_withSecret(), XXH3_64bits_reset_withSecret(), * XXH3_128bits_withSecret(), XXH3_128bits_reset_withSecret(). */ #define XXH3_SECRET_SIZE_MIN 136 /* * XXH3_64bits_withSecret(): * It's possible to provide any blob of bytes as a "secret" to generate the hash. * This makes it more difficult for an external actor to prepare an intentional collision. * The main condition is that secretSize *must* be large enough (>= XXH3_SECRET_SIZE_MIN). * However, the quality of produced hash values depends on secret's entropy. * Technically, the secret must look like a bunch of random bytes. * Avoid "trivial" or structured data such as repeated sequences or a text document. * Whenever unsure about the "randomness" of the blob of bytes, * consider relabelling it as a "custom seed" instead, * and employ "XXH3_generateSecret()" (see below) * to generate a high entropy secret derived from the custom seed. */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSecret(const void* data, size_t len, const void* secret, size_t secretSize); /******* Streaming *******/ /* * Streaming requires state maintenance. * This operation costs memory and CPU. * As a consequence, streaming is slower than one-shot hashing. * For better performance, prefer one-shot functions whenever applicable. */ /*! * @brief The state struct for the XXH3 streaming API. * * @see XXH3_state_s for details. */ typedef struct XXH3_state_s XXH3_state_t; XXH_PUBLIC_API XXH3_state_t* XXH3_createState(void); XXH_PUBLIC_API XXH_errorcode XXH3_freeState(XXH3_state_t* statePtr); XXH_PUBLIC_API void XXH3_copyState(XXH3_state_t* dst_state, const XXH3_state_t* src_state); /* * XXH3_64bits_reset(): * Initialize with default parameters. * digest will be equivalent to `XXH3_64bits()`. */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset(XXH3_state_t* statePtr); /* * XXH3_64bits_reset_withSeed(): * Generate a custom secret from `seed`, and store it into `statePtr`. * digest will be equivalent to `XXH3_64bits_withSeed()`. */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSeed(XXH3_state_t* statePtr, XXH64_hash_t seed); /* * XXH3_64bits_reset_withSecret(): * `secret` is referenced, it _must outlive_ the hash streaming session. * Similar to one-shot API, `secretSize` must be >= `XXH3_SECRET_SIZE_MIN`, * and the quality of produced hash values depends on secret's entropy * (secret's content should look like a bunch of random bytes). * When in doubt about the randomness of a candidate `secret`, * consider employing `XXH3_generateSecret()` instead (see below). */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecret(XXH3_state_t* statePtr, const void* secret, size_t secretSize); XXH_PUBLIC_API XXH_errorcode XXH3_64bits_update (XXH3_state_t* statePtr, const void* input, size_t length); XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_digest (const XXH3_state_t* statePtr); /* note : canonical representation of XXH3 is the same as XXH64 * since they both produce XXH64_hash_t values */ /*-********************************************************************** * XXH3 128-bit variant ************************************************************************/ /*! * @brief The return value from 128-bit hashes. * * Stored in little endian order, although the fields themselves are in native * endianness. */ typedef struct { XXH64_hash_t low64; /*!< `value & 0xFFFFFFFFFFFFFFFF` */ XXH64_hash_t high64; /*!< `value >> 64` */ } XXH128_hash_t; XXH_PUBLIC_API XXH128_hash_t XXH3_128bits(const void* data, size_t len); XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSeed(const void* data, size_t len, XXH64_hash_t seed); XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSecret(const void* data, size_t len, const void* secret, size_t secretSize); /******* Streaming *******/ /* * Streaming requires state maintenance. * This operation costs memory and CPU. * As a consequence, streaming is slower than one-shot hashing. * For better performance, prefer one-shot functions whenever applicable. * * XXH3_128bits uses the same XXH3_state_t as XXH3_64bits(). * Use already declared XXH3_createState() and XXH3_freeState(). * * All reset and streaming functions have same meaning as their 64-bit counterpart. */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset(XXH3_state_t* statePtr); XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSeed(XXH3_state_t* statePtr, XXH64_hash_t seed); XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecret(XXH3_state_t* statePtr, const void* secret, size_t secretSize); XXH_PUBLIC_API XXH_errorcode XXH3_128bits_update (XXH3_state_t* statePtr, const void* input, size_t length); XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_digest (const XXH3_state_t* statePtr); /* Following helper functions make it possible to compare XXH128_hast_t values. * Since XXH128_hash_t is a structure, this capability is not offered by the language. * Note: For better performance, these functions can be inlined using XXH_INLINE_ALL */ /*! * XXH128_isEqual(): * Return: 1 if `h1` and `h2` are equal, 0 if they are not. */ XXH_PUBLIC_API int XXH128_isEqual(XXH128_hash_t h1, XXH128_hash_t h2); /*! * XXH128_cmp(): * * This comparator is compatible with stdlib's `qsort()`/`bsearch()`. * * return: >0 if *h128_1 > *h128_2 * =0 if *h128_1 == *h128_2 * <0 if *h128_1 < *h128_2 */ XXH_PUBLIC_API int XXH128_cmp(const void* h128_1, const void* h128_2); /******* Canonical representation *******/ typedef struct { unsigned char digest[sizeof(XXH128_hash_t)]; } XXH128_canonical_t; XXH_PUBLIC_API void XXH128_canonicalFromHash(XXH128_canonical_t* dst, XXH128_hash_t hash); XXH_PUBLIC_API XXH128_hash_t XXH128_hashFromCanonical(const XXH128_canonical_t* src); #endif /* XXH_NO_LONG_LONG */ /*! * @} */ #endif /* XXHASH_H_5627135585666179 */ #if defined(XXH_STATIC_LINKING_ONLY) && !defined(XXHASH_H_STATIC_13879238742) #define XXHASH_H_STATIC_13879238742 /* **************************************************************************** * This section contains declarations which are not guaranteed to remain stable. * They may change in future versions, becoming incompatible with a different * version of the library. * These declarations should only be used with static linking. * Never use them in association with dynamic linking! ***************************************************************************** */ /* * These definitions are only present to allow static allocation * of XXH states, on stack or in a struct, for example. * Never **ever** access their members directly. */ /*! * @internal * @brief Structure for XXH32 streaming API. * * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. Otherwise it is * an opaque type. This allows fields to safely be changed. * * Typedef'd to @ref XXH32_state_t. * Do not access the members of this struct directly. * @see XXH64_state_s, XXH3_state_s */ struct XXH32_state_s { XXH32_hash_t total_len_32; /*!< Total length hashed, modulo 2^32 */ XXH32_hash_t large_len; /*!< Whether the hash is >= 16 (handles @ref total_len_32 overflow) */ XXH32_hash_t v1; /*!< First accumulator lane */ XXH32_hash_t v2; /*!< Second accumulator lane */ XXH32_hash_t v3; /*!< Third accumulator lane */ XXH32_hash_t v4; /*!< Fourth accumulator lane */ XXH32_hash_t mem32[4]; /*!< Internal buffer for partial reads. Treated as unsigned char[16]. */ XXH32_hash_t memsize; /*!< Amount of data in @ref mem32 */ XXH32_hash_t reserved; /*!< Reserved field. Do not read or write to it, it may be removed. */ }; /* typedef'd to XXH32_state_t */ #ifndef XXH_NO_LONG_LONG /* defined when there is no 64-bit support */ /*! * @internal * @brief Structure for XXH64 streaming API. * * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. Otherwise it is * an opaque type. This allows fields to safely be changed. * * Typedef'd to @ref XXH64_state_t. * Do not access the members of this struct directly. * @see XXH32_state_s, XXH3_state_s */ struct XXH64_state_s { XXH64_hash_t total_len; /*!< Total length hashed. This is always 64-bit. */ XXH64_hash_t v1; /*!< First accumulator lane */ XXH64_hash_t v2; /*!< Second accumulator lane */ XXH64_hash_t v3; /*!< Third accumulator lane */ XXH64_hash_t v4; /*!< Fourth accumulator lane */ XXH64_hash_t mem64[4]; /*!< Internal buffer for partial reads. Treated as unsigned char[32]. */ XXH32_hash_t memsize; /*!< Amount of data in @ref mem64 */ XXH32_hash_t reserved32; /*!< Reserved field, needed for padding anyways*/ XXH64_hash_t reserved64; /*!< Reserved field. Do not read or write to it, it may be removed. */ }; /* typedef'd to XXH64_state_t */ #if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L) /* C11+ */ # include # define XXH_ALIGN(n) alignas(n) #elif defined(__GNUC__) # define XXH_ALIGN(n) __attribute__ ((aligned(n))) #elif defined(_MSC_VER) # define XXH_ALIGN(n) __declspec(align(n)) #else # define XXH_ALIGN(n) /* disabled */ #endif /* Old GCC versions only accept the attribute after the type in structures. */ #if !(defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L)) /* C11+ */ \ && defined(__GNUC__) # define XXH_ALIGN_MEMBER(align, type) type XXH_ALIGN(align) #else # define XXH_ALIGN_MEMBER(align, type) XXH_ALIGN(align) type #endif /*! * @brief The size of the internal XXH3 buffer. * * This is the optimal update size for incremental hashing. * * @see XXH3_64b_update(), XXH3_128b_update(). */ #define XXH3_INTERNALBUFFER_SIZE 256 /*! * @brief Default size of the secret buffer (and @ref XXH3_kSecret). * * This is the size used in @ref XXH3_kSecret and the seeded functions. * * Not to be confused with @ref XXH3_SECRET_SIZE_MIN. */ #define XXH3_SECRET_DEFAULT_SIZE 192 /*! * @internal * @brief Structure for XXH3 streaming API. * * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. Otherwise it is * an opaque type. This allows fields to safely be changed. * * @note **This structure has a strict alignment requirement of 64 bytes.** Do * not allocate this with `malloc()` or `new`, it will not be sufficiently * aligned. Use @ref XXH3_createState() and @ref XXH3_freeState(), or stack * allocation. * * Typedef'd to @ref XXH3_state_t. * Do not access the members of this struct directly. * * @see XXH3_INITSTATE() for stack initialization. * @see XXH3_createState(), XXH3_freeState(). * @see XXH32_state_s, XXH64_state_s */ struct XXH3_state_s { XXH_ALIGN_MEMBER(64, XXH64_hash_t acc[8]); /*!< The 8 accumulators. Similar to `vN` in @ref XXH32_state_s::v1 and @ref XXH64_state_s */ XXH_ALIGN_MEMBER(64, unsigned char customSecret[XXH3_SECRET_DEFAULT_SIZE]); /*!< Used to store a custom secret generated from a seed. */ XXH_ALIGN_MEMBER(64, unsigned char buffer[XXH3_INTERNALBUFFER_SIZE]); /*!< The internal buffer. @see XXH32_state_s::mem32 */ XXH32_hash_t bufferedSize; /*!< The amount of memory in @ref buffer, @see XXH32_state_s::memsize */ XXH32_hash_t reserved32; /*!< Reserved field. Needed for padding on 64-bit. */ size_t nbStripesSoFar; /*!< Number or stripes processed. */ XXH64_hash_t totalLen; /*!< Total length hashed. 64-bit even on 32-bit targets. */ size_t nbStripesPerBlock; /*!< Number of stripes per block. */ size_t secretLimit; /*!< Size of @ref customSecret or @ref extSecret */ XXH64_hash_t seed; /*!< Seed for _withSeed variants. Must be zero otherwise, @see XXH3_INITSTATE() */ XXH64_hash_t reserved64; /*!< Reserved field. */ const unsigned char* extSecret; /*!< Reference to an external secret for the _withSecret variants, NULL * for other variants. */ /* note: there may be some padding at the end due to alignment on 64 bytes */ }; /* typedef'd to XXH3_state_t */ #undef XXH_ALIGN_MEMBER /*! * @brief Initializes a stack-allocated `XXH3_state_s`. * * When the @ref XXH3_state_t structure is merely emplaced on stack, * it should be initialized with XXH3_INITSTATE() or a memset() * in case its first reset uses XXH3_NNbits_reset_withSeed(). * This init can be omitted if the first reset uses default or _withSecret mode. * This operation isn't necessary when the state is created with XXH3_createState(). * Note that this doesn't prepare the state for a streaming operation, * it's still necessary to use XXH3_NNbits_reset*() afterwards. */ #define XXH3_INITSTATE(XXH3_state_ptr) { (XXH3_state_ptr)->seed = 0; } /* === Experimental API === */ /* Symbols defined below must be considered tied to a specific library version. */ /* * XXH3_generateSecret(): * * Derive a high-entropy secret from any user-defined content, named customSeed. * The generated secret can be used in combination with `*_withSecret()` functions. * The `_withSecret()` variants are useful to provide a higher level of protection than 64-bit seed, * as it becomes much more difficult for an external actor to guess how to impact the calculation logic. * * The function accepts as input a custom seed of any length and any content, * and derives from it a high-entropy secret of length XXH3_SECRET_DEFAULT_SIZE * into an already allocated buffer secretBuffer. * The generated secret is _always_ XXH_SECRET_DEFAULT_SIZE bytes long. * * The generated secret can then be used with any `*_withSecret()` variant. * Functions `XXH3_128bits_withSecret()`, `XXH3_64bits_withSecret()`, * `XXH3_128bits_reset_withSecret()` and `XXH3_64bits_reset_withSecret()` * are part of this list. They all accept a `secret` parameter * which must be very long for implementation reasons (>= XXH3_SECRET_SIZE_MIN) * _and_ feature very high entropy (consist of random-looking bytes). * These conditions can be a high bar to meet, so * this function can be used to generate a secret of proper quality. * * customSeed can be anything. It can have any size, even small ones, * and its content can be anything, even stupidly "low entropy" source such as a bunch of zeroes. * The resulting `secret` will nonetheless provide all expected qualities. * * Supplying NULL as the customSeed copies the default secret into `secretBuffer`. * When customSeedSize > 0, supplying NULL as customSeed is undefined behavior. */ XXH_PUBLIC_API void XXH3_generateSecret(void* secretBuffer, const void* customSeed, size_t customSeedSize); /* simple short-cut to pre-selected XXH3_128bits variant */ XXH_PUBLIC_API XXH128_hash_t XXH128(const void* data, size_t len, XXH64_hash_t seed); #endif /* XXH_NO_LONG_LONG */ #if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API) # define XXH_IMPLEMENTATION #endif #endif /* defined(XXH_STATIC_LINKING_ONLY) && !defined(XXHASH_H_STATIC_13879238742) */ /* ======================================================================== */ /* ======================================================================== */ /* ======================================================================== */ /*-********************************************************************** * xxHash implementation *-********************************************************************** * xxHash's implementation used to be hosted inside xxhash.c. * * However, inlining requires implementation to be visible to the compiler, * hence be included alongside the header. * Previously, implementation was hosted inside xxhash.c, * which was then #included when inlining was activated. * This construction created issues with a few build and install systems, * as it required xxhash.c to be stored in /include directory. * * xxHash implementation is now directly integrated within xxhash.h. * As a consequence, xxhash.c is no longer needed in /include. * * xxhash.c is still available and is still useful. * In a "normal" setup, when xxhash is not inlined, * xxhash.h only exposes the prototypes and public symbols, * while xxhash.c can be built into an object file xxhash.o * which can then be linked into the final binary. ************************************************************************/ #if ( defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API) \ || defined(XXH_IMPLEMENTATION) ) && !defined(XXH_IMPLEM_13a8737387) # define XXH_IMPLEM_13a8737387 /* ************************************* * Tuning parameters ***************************************/ /*! * @defgroup tuning Tuning parameters * @{ * * Various macros to control xxHash's behavior. */ #ifdef XXH_DOXYGEN /*! * @brief Define this to disable 64-bit code. * * Useful if only using the @ref xxh32_family and you have a strict C90 compiler. */ # define XXH_NO_LONG_LONG # undef XXH_NO_LONG_LONG /* don't actually */ /*! * @brief Controls how unaligned memory is accessed. * * By default, access to unaligned memory is controlled by `memcpy()`, which is * safe and portable. * * Unfortunately, on some target/compiler combinations, the generated assembly * is sub-optimal. * * The below switch allow selection of a different access method * in the search for improved performance. * * @par Possible options: * * - `XXH_FORCE_MEMORY_ACCESS=0` (default): `memcpy` * @par * Use `memcpy()`. Safe and portable. Note that most modern compilers will * eliminate the function call and treat it as an unaligned access. * * - `XXH_FORCE_MEMORY_ACCESS=1`: `__attribute__((packed))` * @par * Depends on compiler extensions and is therefore not portable. * This method is safe _if_ your compiler supports it, * and *generally* as fast or faster than `memcpy`. * * - `XXH_FORCE_MEMORY_ACCESS=2`: Direct cast * @par * Casts directly and dereferences. This method doesn't depend on the * compiler, but it violates the C standard as it directly dereferences an * unaligned pointer. It can generate buggy code on targets which do not * support unaligned memory accesses, but in some circumstances, it's the * only known way to get the most performance. * * - `XXH_FORCE_MEMORY_ACCESS=3`: Byteshift * @par * Also portable. This can generate the best code on old compilers which don't * inline small `memcpy()` calls, and it might also be faster on big-endian * systems which lack a native byteswap instruction. However, some compilers * will emit literal byteshifts even if the target supports unaligned access. * . * * @warning * Methods 1 and 2 rely on implementation-defined behavior. Use these with * care, as what works on one compiler/platform/optimization level may cause * another to read garbage data or even crash. * * See https://stackoverflow.com/a/32095106/646947 for details. * * Prefer these methods in priority order (0 > 3 > 1 > 2) */ # define XXH_FORCE_MEMORY_ACCESS 0 /*! * @def XXH_ACCEPT_NULL_INPUT_POINTER * @brief Whether to add explicit `NULL` checks. * * If the input pointer is `NULL` and the length is non-zero, xxHash's default * behavior is to dereference it, triggering a segfault. * * When this macro is enabled, xxHash actively checks the input for a null pointer. * If it is, the result for null input pointers is the same as a zero-length input. */ # define XXH_ACCEPT_NULL_INPUT_POINTER 0 /*! * @def XXH_FORCE_ALIGN_CHECK * @brief If defined to non-zero, adds a special path for aligned inputs (XXH32() * and XXH64() only). * * This is an important performance trick for architectures without decent * unaligned memory access performance. * * It checks for input alignment, and when conditions are met, uses a "fast * path" employing direct 32-bit/64-bit reads, resulting in _dramatically * faster_ read speed. * * The check costs one initial branch per hash, which is generally negligible, * but not zero. * * Moreover, it's not useful to generate an additional code path if memory * access uses the same instruction for both aligned and unaligned * addresses (e.g. x86 and aarch64). * * In these cases, the alignment check can be removed by setting this macro to 0. * Then the code will always use unaligned memory access. * Align check is automatically disabled on x86, x64 & arm64, * which are platforms known to offer good unaligned memory accesses performance. * * This option does not affect XXH3 (only XXH32 and XXH64). */ # define XXH_FORCE_ALIGN_CHECK 0 /*! * @def XXH_NO_INLINE_HINTS * @brief When non-zero, sets all functions to `static`. * * By default, xxHash tries to force the compiler to inline almost all internal * functions. * * This can usually improve performance due to reduced jumping and improved * constant folding, but significantly increases the size of the binary which * might not be favorable. * * Additionally, sometimes the forced inlining can be detrimental to performance, * depending on the architecture. * * XXH_NO_INLINE_HINTS marks all internal functions as static, giving the * compiler full control on whether to inline or not. * * When not optimizing (-O0), optimizing for size (-Os, -Oz), or using * -fno-inline with GCC or Clang, this will automatically be defined. */ # define XXH_NO_INLINE_HINTS 0 /*! * @def XXH_REROLL * @brief Whether to reroll `XXH32_finalize` and `XXH64_finalize`. * * For performance, `XXH32_finalize` and `XXH64_finalize` use an unrolled loop * in the form of a switch statement. * * This is not always desirable, as it generates larger code, and depending on * the architecture, may even be slower * * This is automatically defined with `-Os`/`-Oz` on GCC and Clang. */ # define XXH_REROLL 0 /*! * @internal * @brief Redefines old internal names. * * For compatibility with code that uses xxHash's internals before the names * were changed to improve namespacing. There is no other reason to use this. */ # define XXH_OLD_NAMES # undef XXH_OLD_NAMES /* don't actually use, it is ugly. */ #endif /* XXH_DOXYGEN */ /*! * @} */ #ifndef XXH_FORCE_MEMORY_ACCESS /* can be defined externally, on command line for example */ /* prefer __packed__ structures (method 1) for gcc on armv7 and armv8 */ # if !defined(__clang__) && ( \ (defined(__INTEL_COMPILER) && !defined(_WIN32)) || \ (defined(__GNUC__) && (defined(__ARM_ARCH) && __ARM_ARCH >= 7)) ) # define XXH_FORCE_MEMORY_ACCESS 1 # endif #endif #ifndef XXH_ACCEPT_NULL_INPUT_POINTER /* can be defined externally */ # define XXH_ACCEPT_NULL_INPUT_POINTER 0 #endif #ifndef XXH_FORCE_ALIGN_CHECK /* can be defined externally */ # if defined(__i386) || defined(__x86_64__) || defined(__aarch64__) \ || defined(_M_IX86) || defined(_M_X64) || defined(_M_ARM64) /* visual */ # define XXH_FORCE_ALIGN_CHECK 0 # else # define XXH_FORCE_ALIGN_CHECK 1 # endif #endif #ifndef XXH_NO_INLINE_HINTS # if defined(__OPTIMIZE_SIZE__) /* -Os, -Oz */ \ || defined(__NO_INLINE__) /* -O0, -fno-inline */ # define XXH_NO_INLINE_HINTS 1 # else # define XXH_NO_INLINE_HINTS 0 # endif #endif #ifndef XXH_REROLL # if defined(__OPTIMIZE_SIZE__) # define XXH_REROLL 1 # else # define XXH_REROLL 0 # endif #endif /*! * @defgroup impl Implementation * @{ */ /* ************************************* * Includes & Memory related functions ***************************************/ /* * Modify the local functions below should you wish to use * different memory routines for malloc() and free() */ #include /*! * @internal * @brief Modify this function to use a different routine than malloc(). */ static void* XXH_malloc(size_t s) { return malloc(s); } /*! * @internal * @brief Modify this function to use a different routine than free(). */ static void XXH_free(void* p) { free(p); } #include /*! * @internal * @brief Modify this function to use a different routine than memcpy(). */ static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); } #include /* ULLONG_MAX */ /* ************************************* * Compiler Specific Options ***************************************/ #ifdef _MSC_VER /* Visual Studio warning fix */ # pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ #endif #if XXH_NO_INLINE_HINTS /* disable inlining hints */ # if defined(__GNUC__) # define XXH_FORCE_INLINE static __attribute__((unused)) # else # define XXH_FORCE_INLINE static # endif # define XXH_NO_INLINE static /* enable inlining hints */ #elif defined(_MSC_VER) /* Visual Studio */ # define XXH_FORCE_INLINE static __forceinline # define XXH_NO_INLINE static __declspec(noinline) #elif defined(__GNUC__) # define XXH_FORCE_INLINE static __inline__ __attribute__((always_inline, unused)) # define XXH_NO_INLINE static __attribute__((noinline)) #elif defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L)) /* C99 */ # define XXH_FORCE_INLINE static inline # define XXH_NO_INLINE static #else # define XXH_FORCE_INLINE static # define XXH_NO_INLINE static #endif /* ************************************* * Debug ***************************************/ /*! * @ingroup tuning * @def XXH_DEBUGLEVEL * @brief Sets the debugging level. * * XXH_DEBUGLEVEL is expected to be defined externally, typically via the * compiler's command line options. The value must be a number. */ #ifndef XXH_DEBUGLEVEL # ifdef DEBUGLEVEL /* backwards compat */ # define XXH_DEBUGLEVEL DEBUGLEVEL # else # define XXH_DEBUGLEVEL 0 # endif #endif #if (XXH_DEBUGLEVEL>=1) # include /* note: can still be disabled with NDEBUG */ # define XXH_ASSERT(c) assert(c) #else # define XXH_ASSERT(c) ((void)0) #endif /* note: use after variable declarations */ #define XXH_STATIC_ASSERT(c) do { enum { XXH_sa = 1/(int)(!!(c)) }; } while (0) /*! * @internal * @def XXH_COMPILER_GUARD(var) * @brief Used to prevent unwanted optimizations for @p var. * * It uses an empty GCC inline assembly statement with a register constraint * which forces @p var into a general purpose register (eg eax, ebx, ecx * on x86) and marks it as modified. * * This is used in a few places to avoid unwanted autovectorization (e.g. * XXH32_round()). All vectorization we want is explicit via intrinsics, * and _usually_ isn't wanted elsewhere. * * We also use it to prevent unwanted constant folding for AArch64 in * XXH3_initCustomSecret_scalar(). */ #ifdef __GNUC__ # define XXH_COMPILER_GUARD(var) __asm__ __volatile__("" : "+r" (var)) #else # define XXH_COMPILER_GUARD(var) ((void)0) #endif /* ************************************* * Basic Types ***************************************/ #if !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) # include typedef uint8_t xxh_u8; #else typedef unsigned char xxh_u8; #endif typedef XXH32_hash_t xxh_u32; #ifdef XXH_OLD_NAMES # define BYTE xxh_u8 # define U8 xxh_u8 # define U32 xxh_u32 #endif /* *** Memory access *** */ /*! * @internal * @fn xxh_u32 XXH_read32(const void* ptr) * @brief Reads an unaligned 32-bit integer from @p ptr in native endianness. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * * @param ptr The pointer to read from. * @return The 32-bit native endian integer from the bytes at @p ptr. */ /*! * @internal * @fn xxh_u32 XXH_readLE32(const void* ptr) * @brief Reads an unaligned 32-bit little endian integer from @p ptr. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * * @param ptr The pointer to read from. * @return The 32-bit little endian integer from the bytes at @p ptr. */ /*! * @internal * @fn xxh_u32 XXH_readBE32(const void* ptr) * @brief Reads an unaligned 32-bit big endian integer from @p ptr. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * * @param ptr The pointer to read from. * @return The 32-bit big endian integer from the bytes at @p ptr. */ /*! * @internal * @fn xxh_u32 XXH_readLE32_align(const void* ptr, XXH_alignment align) * @brief Like @ref XXH_readLE32(), but has an option for aligned reads. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * Note that when @ref XXH_FORCE_ALIGN_CHECK == 0, the @p align parameter is * always @ref XXH_alignment::XXH_unaligned. * * @param ptr The pointer to read from. * @param align Whether @p ptr is aligned. * @pre * If @p align == @ref XXH_alignment::XXH_aligned, @p ptr must be 4 byte * aligned. * @return The 32-bit little endian integer from the bytes at @p ptr. */ #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) /* * Manual byteshift. Best for old compilers which don't inline memcpy. * We actually directly use XXH_readLE32 and XXH_readBE32. */ #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2)) /* * Force direct memory access. Only works on CPU which support unaligned memory * access in hardware. */ static xxh_u32 XXH_read32(const void* memPtr) { return *(const xxh_u32*) memPtr; } #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1)) /* * __pack instructions are safer but compiler specific, hence potentially * problematic for some compilers. * * Currently only defined for GCC and ICC. */ #ifdef XXH_OLD_NAMES typedef union { xxh_u32 u32; } __attribute__((packed)) unalign; #endif static xxh_u32 XXH_read32(const void* ptr) { typedef union { xxh_u32 u32; } __attribute__((packed)) xxh_unalign; return ((const xxh_unalign*)ptr)->u32; } #else /* * Portable and safe solution. Generally efficient. * see: https://stackoverflow.com/a/32095106/646947 */ static xxh_u32 XXH_read32(const void* memPtr) { xxh_u32 val; memcpy(&val, memPtr, sizeof(val)); return val; } #endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */ /* *** Endianness *** */ typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess; /*! * @ingroup tuning * @def XXH_CPU_LITTLE_ENDIAN * @brief Whether the target is little endian. * * Defined to 1 if the target is little endian, or 0 if it is big endian. * It can be defined externally, for example on the compiler command line. * * If it is not defined, a runtime check (which is usually constant folded) * is used instead. * * @note * This is not necessarily defined to an integer constant. * * @see XXH_isLittleEndian() for the runtime check. */ #ifndef XXH_CPU_LITTLE_ENDIAN /* * Try to detect endianness automatically, to avoid the nonstandard behavior * in `XXH_isLittleEndian()` */ # if defined(_WIN32) /* Windows is always little endian */ \ || defined(__LITTLE_ENDIAN__) \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) # define XXH_CPU_LITTLE_ENDIAN 1 # elif defined(__BIG_ENDIAN__) \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) # define XXH_CPU_LITTLE_ENDIAN 0 # else /*! * @internal * @brief Runtime check for @ref XXH_CPU_LITTLE_ENDIAN. * * Most compilers will constant fold this. */ static int XXH_isLittleEndian(void) { /* * Portable and well-defined behavior. * Don't use static: it is detrimental to performance. */ const union { xxh_u32 u; xxh_u8 c[4]; } one = { 1 }; return one.c[0]; } # define XXH_CPU_LITTLE_ENDIAN XXH_isLittleEndian() # endif #endif /* **************************************** * Compiler-specific Functions and Macros ******************************************/ #define XXH_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) #ifdef __has_builtin # define XXH_HAS_BUILTIN(x) __has_builtin(x) #else # define XXH_HAS_BUILTIN(x) 0 #endif /*! * @internal * @def XXH_rotl32(x,r) * @brief 32-bit rotate left. * * @param x The 32-bit integer to be rotated. * @param r The number of bits to rotate. * @pre * @p r > 0 && @p r < 32 * @note * @p x and @p r may be evaluated multiple times. * @return The rotated result. */ #if !defined(NO_CLANG_BUILTIN) && XXH_HAS_BUILTIN(__builtin_rotateleft32) \ && XXH_HAS_BUILTIN(__builtin_rotateleft64) # define XXH_rotl32 __builtin_rotateleft32 # define XXH_rotl64 __builtin_rotateleft64 /* Note: although _rotl exists for minGW (GCC under windows), performance seems poor */ #elif defined(_MSC_VER) # define XXH_rotl32(x,r) _rotl(x,r) # define XXH_rotl64(x,r) _rotl64(x,r) #else # define XXH_rotl32(x,r) (((x) << (r)) | ((x) >> (32 - (r)))) # define XXH_rotl64(x,r) (((x) << (r)) | ((x) >> (64 - (r)))) #endif /*! * @internal * @fn xxh_u32 XXH_swap32(xxh_u32 x) * @brief A 32-bit byteswap. * * @param x The 32-bit integer to byteswap. * @return @p x, byteswapped. */ #if defined(_MSC_VER) /* Visual Studio */ # define XXH_swap32 _byteswap_ulong #elif XXH_GCC_VERSION >= 403 # define XXH_swap32 __builtin_bswap32 #else static xxh_u32 XXH_swap32 (xxh_u32 x) { return ((x << 24) & 0xff000000 ) | ((x << 8) & 0x00ff0000 ) | ((x >> 8) & 0x0000ff00 ) | ((x >> 24) & 0x000000ff ); } #endif /* *************************** * Memory reads *****************************/ /*! * @internal * @brief Enum to indicate whether a pointer is aligned. */ typedef enum { XXH_aligned, /*!< Aligned */ XXH_unaligned /*!< Possibly unaligned */ } XXH_alignment; /* * XXH_FORCE_MEMORY_ACCESS==3 is an endian-independent byteshift load. * * This is ideal for older compilers which don't inline memcpy. */ #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) XXH_FORCE_INLINE xxh_u32 XXH_readLE32(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[0] | ((xxh_u32)bytePtr[1] << 8) | ((xxh_u32)bytePtr[2] << 16) | ((xxh_u32)bytePtr[3] << 24); } XXH_FORCE_INLINE xxh_u32 XXH_readBE32(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[3] | ((xxh_u32)bytePtr[2] << 8) | ((xxh_u32)bytePtr[1] << 16) | ((xxh_u32)bytePtr[0] << 24); } #else XXH_FORCE_INLINE xxh_u32 XXH_readLE32(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr)); } static xxh_u32 XXH_readBE32(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_swap32(XXH_read32(ptr)) : XXH_read32(ptr); } #endif XXH_FORCE_INLINE xxh_u32 XXH_readLE32_align(const void* ptr, XXH_alignment align) { if (align==XXH_unaligned) { return XXH_readLE32(ptr); } else { return XXH_CPU_LITTLE_ENDIAN ? *(const xxh_u32*)ptr : XXH_swap32(*(const xxh_u32*)ptr); } } /* ************************************* * Misc ***************************************/ /*! @ingroup public */ XXH_PUBLIC_API unsigned XXH_versionNumber (void) { return XXH_VERSION_NUMBER; } /* ******************************************************************* * 32-bit hash functions *********************************************************************/ /*! * @} * @defgroup xxh32_impl XXH32 implementation * @ingroup impl * @{ */ /* #define instead of static const, to be used as initializers */ #define XXH_PRIME32_1 0x9E3779B1U /*!< 0b10011110001101110111100110110001 */ #define XXH_PRIME32_2 0x85EBCA77U /*!< 0b10000101111010111100101001110111 */ #define XXH_PRIME32_3 0xC2B2AE3DU /*!< 0b11000010101100101010111000111101 */ #define XXH_PRIME32_4 0x27D4EB2FU /*!< 0b00100111110101001110101100101111 */ #define XXH_PRIME32_5 0x165667B1U /*!< 0b00010110010101100110011110110001 */ #ifdef XXH_OLD_NAMES # define PRIME32_1 XXH_PRIME32_1 # define PRIME32_2 XXH_PRIME32_2 # define PRIME32_3 XXH_PRIME32_3 # define PRIME32_4 XXH_PRIME32_4 # define PRIME32_5 XXH_PRIME32_5 #endif /*! * @internal * @brief Normal stripe processing routine. * * This shuffles the bits so that any bit from @p input impacts several bits in * @p acc. * * @param acc The accumulator lane. * @param input The stripe of input to mix. * @return The mixed accumulator lane. */ static xxh_u32 XXH32_round(xxh_u32 acc, xxh_u32 input) { acc += input * XXH_PRIME32_2; acc = XXH_rotl32(acc, 13); acc *= XXH_PRIME32_1; #if (defined(__SSE4_1__) || defined(__aarch64__)) && !defined(XXH_ENABLE_AUTOVECTORIZE) /* * UGLY HACK: * A compiler fence is the only thing that prevents GCC and Clang from * autovectorizing the XXH32 loop (pragmas and attributes don't work for some * reason) without globally disabling SSE4.1. * * The reason we want to avoid vectorization is because despite working on * 4 integers at a time, there are multiple factors slowing XXH32 down on * SSE4: * - There's a ridiculous amount of lag from pmulld (10 cycles of latency on * newer chips!) making it slightly slower to multiply four integers at * once compared to four integers independently. Even when pmulld was * fastest, Sandy/Ivy Bridge, it is still not worth it to go into SSE * just to multiply unless doing a long operation. * * - Four instructions are required to rotate, * movqda tmp, v // not required with VEX encoding * pslld tmp, 13 // tmp <<= 13 * psrld v, 19 // x >>= 19 * por v, tmp // x |= tmp * compared to one for scalar: * roll v, 13 // reliably fast across the board * shldl v, v, 13 // Sandy Bridge and later prefer this for some reason * * - Instruction level parallelism is actually more beneficial here because * the SIMD actually serializes this operation: While v1 is rotating, v2 * can load data, while v3 can multiply. SSE forces them to operate * together. * * This is also enabled on AArch64, as Clang autovectorizes it incorrectly * and it is pointless writing a NEON implementation that is basically the * same speed as scalar for XXH32. */ XXH_COMPILER_GUARD(acc); #endif return acc; } /*! * @internal * @brief Mixes all bits to finalize the hash. * * The final mix ensures that all input bits have a chance to impact any bit in * the output digest, resulting in an unbiased distribution. * * @param h32 The hash to avalanche. * @return The avalanched hash. */ static xxh_u32 XXH32_avalanche(xxh_u32 h32) { h32 ^= h32 >> 15; h32 *= XXH_PRIME32_2; h32 ^= h32 >> 13; h32 *= XXH_PRIME32_3; h32 ^= h32 >> 16; return(h32); } #define XXH_get32bits(p) XXH_readLE32_align(p, align) /*! * @internal * @brief Processes the last 0-15 bytes of @p ptr. * * There may be up to 15 bytes remaining to consume from the input. * This final stage will digest them to ensure that all input bytes are present * in the final mix. * * @param h32 The hash to finalize. * @param ptr The pointer to the remaining input. * @param len The remaining length, modulo 16. * @param align Whether @p ptr is aligned. * @return The finalized hash. */ static xxh_u32 XXH32_finalize(xxh_u32 h32, const xxh_u8* ptr, size_t len, XXH_alignment align) { #define XXH_PROCESS1 do { \ h32 += (*ptr++) * XXH_PRIME32_5; \ h32 = XXH_rotl32(h32, 11) * XXH_PRIME32_1; \ } while (0) #define XXH_PROCESS4 do { \ h32 += XXH_get32bits(ptr) * XXH_PRIME32_3; \ ptr += 4; \ h32 = XXH_rotl32(h32, 17) * XXH_PRIME32_4; \ } while (0) /* Compact rerolled version */ if (XXH_REROLL) { len &= 15; while (len >= 4) { XXH_PROCESS4; len -= 4; } while (len > 0) { XXH_PROCESS1; --len; } return XXH32_avalanche(h32); } else { switch(len&15) /* or switch(bEnd - p) */ { case 12: XXH_PROCESS4; /* fallthrough */ case 8: XXH_PROCESS4; /* fallthrough */ case 4: XXH_PROCESS4; return XXH32_avalanche(h32); case 13: XXH_PROCESS4; /* fallthrough */ case 9: XXH_PROCESS4; /* fallthrough */ case 5: XXH_PROCESS4; XXH_PROCESS1; return XXH32_avalanche(h32); case 14: XXH_PROCESS4; /* fallthrough */ case 10: XXH_PROCESS4; /* fallthrough */ case 6: XXH_PROCESS4; XXH_PROCESS1; XXH_PROCESS1; return XXH32_avalanche(h32); case 15: XXH_PROCESS4; /* fallthrough */ case 11: XXH_PROCESS4; /* fallthrough */ case 7: XXH_PROCESS4; /* fallthrough */ case 3: XXH_PROCESS1; /* fallthrough */ case 2: XXH_PROCESS1; /* fallthrough */ case 1: XXH_PROCESS1; /* fallthrough */ case 0: return XXH32_avalanche(h32); } XXH_ASSERT(0); return h32; /* reaching this point is deemed impossible */ } } #ifdef XXH_OLD_NAMES # define PROCESS1 XXH_PROCESS1 # define PROCESS4 XXH_PROCESS4 #else # undef XXH_PROCESS1 # undef XXH_PROCESS4 #endif /*! * @internal * @brief The implementation for @ref XXH32(). * * @param input, len, seed Directly passed from @ref XXH32(). * @param align Whether @p input is aligned. * @return The calculated hash. */ XXH_FORCE_INLINE xxh_u32 XXH32_endian_align(const xxh_u8* input, size_t len, xxh_u32 seed, XXH_alignment align) { const xxh_u8* bEnd = input ? input + len : NULL; xxh_u32 h32; #if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1) if (input==NULL) { len=0; bEnd=input=(const xxh_u8*)(size_t)16; } #endif if (len>=16) { const xxh_u8* const limit = bEnd - 15; xxh_u32 v1 = seed + XXH_PRIME32_1 + XXH_PRIME32_2; xxh_u32 v2 = seed + XXH_PRIME32_2; xxh_u32 v3 = seed + 0; xxh_u32 v4 = seed - XXH_PRIME32_1; do { v1 = XXH32_round(v1, XXH_get32bits(input)); input += 4; v2 = XXH32_round(v2, XXH_get32bits(input)); input += 4; v3 = XXH32_round(v3, XXH_get32bits(input)); input += 4; v4 = XXH32_round(v4, XXH_get32bits(input)); input += 4; } while (input < limit); h32 = XXH_rotl32(v1, 1) + XXH_rotl32(v2, 7) + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18); } else { h32 = seed + XXH_PRIME32_5; } h32 += (xxh_u32)len; return XXH32_finalize(h32, input, len&15, align); } /*! @ingroup xxh32_family */ XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t len, XXH32_hash_t seed) { #if 0 /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ XXH32_state_t state; XXH32_reset(&state, seed); XXH32_update(&state, (const xxh_u8*)input, len); return XXH32_digest(&state); #else if (XXH_FORCE_ALIGN_CHECK) { if ((((size_t)input) & 3) == 0) { /* Input is 4-bytes aligned, leverage the speed benefit */ return XXH32_endian_align((const xxh_u8*)input, len, seed, XXH_aligned); } } return XXH32_endian_align((const xxh_u8*)input, len, seed, XXH_unaligned); #endif } /******* Hash streaming *******/ /*! * @ingroup xxh32_family */ XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void) { return (XXH32_state_t*)XXH_malloc(sizeof(XXH32_state_t)); } /*! @ingroup xxh32_family */ XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr) { XXH_free(statePtr); return XXH_OK; } /*! @ingroup xxh32_family */ XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dstState, const XXH32_state_t* srcState) { memcpy(dstState, srcState, sizeof(*dstState)); } /*! @ingroup xxh32_family */ XXH_PUBLIC_API XXH_errorcode XXH32_reset(XXH32_state_t* statePtr, XXH32_hash_t seed) { XXH32_state_t state; /* using a local state to memcpy() in order to avoid strict-aliasing warnings */ memset(&state, 0, sizeof(state)); state.v1 = seed + XXH_PRIME32_1 + XXH_PRIME32_2; state.v2 = seed + XXH_PRIME32_2; state.v3 = seed + 0; state.v4 = seed - XXH_PRIME32_1; /* do not write into reserved, planned to be removed in a future version */ memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved)); return XXH_OK; } /*! @ingroup xxh32_family */ XXH_PUBLIC_API XXH_errorcode XXH32_update(XXH32_state_t* state, const void* input, size_t len) { if (input==NULL) #if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1) return XXH_OK; #else return XXH_ERROR; #endif { const xxh_u8* p = (const xxh_u8*)input; const xxh_u8* const bEnd = p + len; state->total_len_32 += (XXH32_hash_t)len; state->large_len |= (XXH32_hash_t)((len>=16) | (state->total_len_32>=16)); if (state->memsize + len < 16) { /* fill in tmp buffer */ XXH_memcpy((xxh_u8*)(state->mem32) + state->memsize, input, len); state->memsize += (XXH32_hash_t)len; return XXH_OK; } if (state->memsize) { /* some data left from previous update */ XXH_memcpy((xxh_u8*)(state->mem32) + state->memsize, input, 16-state->memsize); { const xxh_u32* p32 = state->mem32; state->v1 = XXH32_round(state->v1, XXH_readLE32(p32)); p32++; state->v2 = XXH32_round(state->v2, XXH_readLE32(p32)); p32++; state->v3 = XXH32_round(state->v3, XXH_readLE32(p32)); p32++; state->v4 = XXH32_round(state->v4, XXH_readLE32(p32)); } p += 16-state->memsize; state->memsize = 0; } if (p <= bEnd-16) { const xxh_u8* const limit = bEnd - 16; xxh_u32 v1 = state->v1; xxh_u32 v2 = state->v2; xxh_u32 v3 = state->v3; xxh_u32 v4 = state->v4; do { v1 = XXH32_round(v1, XXH_readLE32(p)); p+=4; v2 = XXH32_round(v2, XXH_readLE32(p)); p+=4; v3 = XXH32_round(v3, XXH_readLE32(p)); p+=4; v4 = XXH32_round(v4, XXH_readLE32(p)); p+=4; } while (p<=limit); state->v1 = v1; state->v2 = v2; state->v3 = v3; state->v4 = v4; } if (p < bEnd) { XXH_memcpy(state->mem32, p, (size_t)(bEnd-p)); state->memsize = (unsigned)(bEnd-p); } } return XXH_OK; } /*! @ingroup xxh32_family */ XXH_PUBLIC_API XXH32_hash_t XXH32_digest(const XXH32_state_t* state) { xxh_u32 h32; if (state->large_len) { h32 = XXH_rotl32(state->v1, 1) + XXH_rotl32(state->v2, 7) + XXH_rotl32(state->v3, 12) + XXH_rotl32(state->v4, 18); } else { h32 = state->v3 /* == seed */ + XXH_PRIME32_5; } h32 += state->total_len_32; return XXH32_finalize(h32, (const xxh_u8*)state->mem32, state->memsize, XXH_aligned); } /******* Canonical representation *******/ /*! * @ingroup xxh32_family * The default return values from XXH functions are unsigned 32 and 64 bit * integers. * * The canonical representation uses big endian convention, the same convention * as human-readable numbers (large digits first). * * This way, hash values can be written into a file or buffer, remaining * comparable across different systems. * * The following functions allow transformation of hash values to and from their * canonical format. */ XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash) { XXH_STATIC_ASSERT(sizeof(XXH32_canonical_t) == sizeof(XXH32_hash_t)); if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap32(hash); memcpy(dst, &hash, sizeof(*dst)); } /*! @ingroup xxh32_family */ XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src) { return XXH_readBE32(src); } #ifndef XXH_NO_LONG_LONG /* ******************************************************************* * 64-bit hash functions *********************************************************************/ /*! * @} * @ingroup impl * @{ */ /******* Memory access *******/ typedef XXH64_hash_t xxh_u64; #ifdef XXH_OLD_NAMES # define U64 xxh_u64 #endif #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) /* * Manual byteshift. Best for old compilers which don't inline memcpy. * We actually directly use XXH_readLE64 and XXH_readBE64. */ #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2)) /* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */ static xxh_u64 XXH_read64(const void* memPtr) { return *(const xxh_u64*) memPtr; } #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1)) /* * __pack instructions are safer, but compiler specific, hence potentially * problematic for some compilers. * * Currently only defined for GCC and ICC. */ #ifdef XXH_OLD_NAMES typedef union { xxh_u32 u32; xxh_u64 u64; } __attribute__((packed)) unalign64; #endif static xxh_u64 XXH_read64(const void* ptr) { typedef union { xxh_u32 u32; xxh_u64 u64; } __attribute__((packed)) xxh_unalign64; return ((const xxh_unalign64*)ptr)->u64; } #else /* * Portable and safe solution. Generally efficient. * see: https://stackoverflow.com/a/32095106/646947 */ static xxh_u64 XXH_read64(const void* memPtr) { xxh_u64 val; memcpy(&val, memPtr, sizeof(val)); return val; } #endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */ #if defined(_MSC_VER) /* Visual Studio */ # define XXH_swap64 _byteswap_uint64 #elif XXH_GCC_VERSION >= 403 # define XXH_swap64 __builtin_bswap64 #else static xxh_u64 XXH_swap64(xxh_u64 x) { return ((x << 56) & 0xff00000000000000ULL) | ((x << 40) & 0x00ff000000000000ULL) | ((x << 24) & 0x0000ff0000000000ULL) | ((x << 8) & 0x000000ff00000000ULL) | ((x >> 8) & 0x00000000ff000000ULL) | ((x >> 24) & 0x0000000000ff0000ULL) | ((x >> 40) & 0x000000000000ff00ULL) | ((x >> 56) & 0x00000000000000ffULL); } #endif /* XXH_FORCE_MEMORY_ACCESS==3 is an endian-independent byteshift load. */ #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) XXH_FORCE_INLINE xxh_u64 XXH_readLE64(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[0] | ((xxh_u64)bytePtr[1] << 8) | ((xxh_u64)bytePtr[2] << 16) | ((xxh_u64)bytePtr[3] << 24) | ((xxh_u64)bytePtr[4] << 32) | ((xxh_u64)bytePtr[5] << 40) | ((xxh_u64)bytePtr[6] << 48) | ((xxh_u64)bytePtr[7] << 56); } XXH_FORCE_INLINE xxh_u64 XXH_readBE64(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[7] | ((xxh_u64)bytePtr[6] << 8) | ((xxh_u64)bytePtr[5] << 16) | ((xxh_u64)bytePtr[4] << 24) | ((xxh_u64)bytePtr[3] << 32) | ((xxh_u64)bytePtr[2] << 40) | ((xxh_u64)bytePtr[1] << 48) | ((xxh_u64)bytePtr[0] << 56); } #else XXH_FORCE_INLINE xxh_u64 XXH_readLE64(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr)); } static xxh_u64 XXH_readBE64(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_swap64(XXH_read64(ptr)) : XXH_read64(ptr); } #endif XXH_FORCE_INLINE xxh_u64 XXH_readLE64_align(const void* ptr, XXH_alignment align) { if (align==XXH_unaligned) return XXH_readLE64(ptr); else return XXH_CPU_LITTLE_ENDIAN ? *(const xxh_u64*)ptr : XXH_swap64(*(const xxh_u64*)ptr); } /******* xxh64 *******/ /*! * @} * @defgroup xxh64_impl XXH64 implementation * @ingroup impl * @{ */ /* #define rather that static const, to be used as initializers */ #define XXH_PRIME64_1 0x9E3779B185EBCA87ULL /*!< 0b1001111000110111011110011011000110000101111010111100101010000111 */ #define XXH_PRIME64_2 0xC2B2AE3D27D4EB4FULL /*!< 0b1100001010110010101011100011110100100111110101001110101101001111 */ #define XXH_PRIME64_3 0x165667B19E3779F9ULL /*!< 0b0001011001010110011001111011000110011110001101110111100111111001 */ #define XXH_PRIME64_4 0x85EBCA77C2B2AE63ULL /*!< 0b1000010111101011110010100111011111000010101100101010111001100011 */ #define XXH_PRIME64_5 0x27D4EB2F165667C5ULL /*!< 0b0010011111010100111010110010111100010110010101100110011111000101 */ #ifdef XXH_OLD_NAMES # define PRIME64_1 XXH_PRIME64_1 # define PRIME64_2 XXH_PRIME64_2 # define PRIME64_3 XXH_PRIME64_3 # define PRIME64_4 XXH_PRIME64_4 # define PRIME64_5 XXH_PRIME64_5 #endif static xxh_u64 XXH64_round(xxh_u64 acc, xxh_u64 input) { acc += input * XXH_PRIME64_2; acc = XXH_rotl64(acc, 31); acc *= XXH_PRIME64_1; return acc; } static xxh_u64 XXH64_mergeRound(xxh_u64 acc, xxh_u64 val) { val = XXH64_round(0, val); acc ^= val; acc = acc * XXH_PRIME64_1 + XXH_PRIME64_4; return acc; } static xxh_u64 XXH64_avalanche(xxh_u64 h64) { h64 ^= h64 >> 33; h64 *= XXH_PRIME64_2; h64 ^= h64 >> 29; h64 *= XXH_PRIME64_3; h64 ^= h64 >> 32; return h64; } #define XXH_get64bits(p) XXH_readLE64_align(p, align) static xxh_u64 XXH64_finalize(xxh_u64 h64, const xxh_u8* ptr, size_t len, XXH_alignment align) { len &= 31; while (len >= 8) { xxh_u64 const k1 = XXH64_round(0, XXH_get64bits(ptr)); ptr += 8; h64 ^= k1; h64 = XXH_rotl64(h64,27) * XXH_PRIME64_1 + XXH_PRIME64_4; len -= 8; } if (len >= 4) { h64 ^= (xxh_u64)(XXH_get32bits(ptr)) * XXH_PRIME64_1; ptr += 4; h64 = XXH_rotl64(h64, 23) * XXH_PRIME64_2 + XXH_PRIME64_3; len -= 4; } while (len > 0) { h64 ^= (*ptr++) * XXH_PRIME64_5; h64 = XXH_rotl64(h64, 11) * XXH_PRIME64_1; --len; } return XXH64_avalanche(h64); } #ifdef XXH_OLD_NAMES # define PROCESS1_64 XXH_PROCESS1_64 # define PROCESS4_64 XXH_PROCESS4_64 # define PROCESS8_64 XXH_PROCESS8_64 #else # undef XXH_PROCESS1_64 # undef XXH_PROCESS4_64 # undef XXH_PROCESS8_64 #endif XXH_FORCE_INLINE xxh_u64 XXH64_endian_align(const xxh_u8* input, size_t len, xxh_u64 seed, XXH_alignment align) { const xxh_u8* bEnd = input ? input + len : NULL; xxh_u64 h64; #if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1) if (input==NULL) { len=0; bEnd=input=(const xxh_u8*)(size_t)32; } #endif if (len>=32) { const xxh_u8* const limit = bEnd - 32; xxh_u64 v1 = seed + XXH_PRIME64_1 + XXH_PRIME64_2; xxh_u64 v2 = seed + XXH_PRIME64_2; xxh_u64 v3 = seed + 0; xxh_u64 v4 = seed - XXH_PRIME64_1; do { v1 = XXH64_round(v1, XXH_get64bits(input)); input+=8; v2 = XXH64_round(v2, XXH_get64bits(input)); input+=8; v3 = XXH64_round(v3, XXH_get64bits(input)); input+=8; v4 = XXH64_round(v4, XXH_get64bits(input)); input+=8; } while (input<=limit); h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18); h64 = XXH64_mergeRound(h64, v1); h64 = XXH64_mergeRound(h64, v2); h64 = XXH64_mergeRound(h64, v3); h64 = XXH64_mergeRound(h64, v4); } else { h64 = seed + XXH_PRIME64_5; } h64 += (xxh_u64) len; return XXH64_finalize(h64, input, len, align); } /*! @ingroup xxh64_family */ XXH_PUBLIC_API XXH64_hash_t XXH64 (const void* input, size_t len, XXH64_hash_t seed) { #if 0 /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ XXH64_state_t state; XXH64_reset(&state, seed); XXH64_update(&state, (const xxh_u8*)input, len); return XXH64_digest(&state); #else if (XXH_FORCE_ALIGN_CHECK) { if ((((size_t)input) & 7)==0) { /* Input is aligned, let's leverage the speed advantage */ return XXH64_endian_align((const xxh_u8*)input, len, seed, XXH_aligned); } } return XXH64_endian_align((const xxh_u8*)input, len, seed, XXH_unaligned); #endif } /******* Hash Streaming *******/ /*! @ingroup xxh64_family*/ XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void) { return (XXH64_state_t*)XXH_malloc(sizeof(XXH64_state_t)); } /*! @ingroup xxh64_family */ XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr) { XXH_free(statePtr); return XXH_OK; } /*! @ingroup xxh64_family */ XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dstState, const XXH64_state_t* srcState) { memcpy(dstState, srcState, sizeof(*dstState)); } /*! @ingroup xxh64_family */ XXH_PUBLIC_API XXH_errorcode XXH64_reset(XXH64_state_t* statePtr, XXH64_hash_t seed) { XXH64_state_t state; /* use a local state to memcpy() in order to avoid strict-aliasing warnings */ memset(&state, 0, sizeof(state)); state.v1 = seed + XXH_PRIME64_1 + XXH_PRIME64_2; state.v2 = seed + XXH_PRIME64_2; state.v3 = seed + 0; state.v4 = seed - XXH_PRIME64_1; /* do not write into reserved64, might be removed in a future version */ memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved64)); return XXH_OK; } /*! @ingroup xxh64_family */ XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* state, const void* input, size_t len) { if (input==NULL) #if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1) return XXH_OK; #else return XXH_ERROR; #endif { const xxh_u8* p = (const xxh_u8*)input; const xxh_u8* const bEnd = p + len; state->total_len += len; if (state->memsize + len < 32) { /* fill in tmp buffer */ XXH_memcpy(((xxh_u8*)state->mem64) + state->memsize, input, len); state->memsize += (xxh_u32)len; return XXH_OK; } if (state->memsize) { /* tmp buffer is full */ XXH_memcpy(((xxh_u8*)state->mem64) + state->memsize, input, 32-state->memsize); state->v1 = XXH64_round(state->v1, XXH_readLE64(state->mem64+0)); state->v2 = XXH64_round(state->v2, XXH_readLE64(state->mem64+1)); state->v3 = XXH64_round(state->v3, XXH_readLE64(state->mem64+2)); state->v4 = XXH64_round(state->v4, XXH_readLE64(state->mem64+3)); p += 32 - state->memsize; state->memsize = 0; } if (p+32 <= bEnd) { const xxh_u8* const limit = bEnd - 32; xxh_u64 v1 = state->v1; xxh_u64 v2 = state->v2; xxh_u64 v3 = state->v3; xxh_u64 v4 = state->v4; do { v1 = XXH64_round(v1, XXH_readLE64(p)); p+=8; v2 = XXH64_round(v2, XXH_readLE64(p)); p+=8; v3 = XXH64_round(v3, XXH_readLE64(p)); p+=8; v4 = XXH64_round(v4, XXH_readLE64(p)); p+=8; } while (p<=limit); state->v1 = v1; state->v2 = v2; state->v3 = v3; state->v4 = v4; } if (p < bEnd) { XXH_memcpy(state->mem64, p, (size_t)(bEnd-p)); state->memsize = (unsigned)(bEnd-p); } } return XXH_OK; } /*! @ingroup xxh64_family */ XXH_PUBLIC_API XXH64_hash_t XXH64_digest(const XXH64_state_t* state) { xxh_u64 h64; if (state->total_len >= 32) { xxh_u64 const v1 = state->v1; xxh_u64 const v2 = state->v2; xxh_u64 const v3 = state->v3; xxh_u64 const v4 = state->v4; h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18); h64 = XXH64_mergeRound(h64, v1); h64 = XXH64_mergeRound(h64, v2); h64 = XXH64_mergeRound(h64, v3); h64 = XXH64_mergeRound(h64, v4); } else { h64 = state->v3 /*seed*/ + XXH_PRIME64_5; } h64 += (xxh_u64) state->total_len; return XXH64_finalize(h64, (const xxh_u8*)state->mem64, (size_t)state->total_len, XXH_aligned); } /******* Canonical representation *******/ /*! @ingroup xxh64_family */ XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash) { XXH_STATIC_ASSERT(sizeof(XXH64_canonical_t) == sizeof(XXH64_hash_t)); if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap64(hash); memcpy(dst, &hash, sizeof(*dst)); } /*! @ingroup xxh64_family */ XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src) { return XXH_readBE64(src); } #ifndef XXH_NO_XXH3 /* ********************************************************************* * XXH3 * New generation hash designed for speed on small keys and vectorization ************************************************************************ */ /*! * @} * @defgroup xxh3_impl XXH3 implementation * @ingroup impl * @{ */ /* === Compiler specifics === */ #if ((defined(sun) || defined(__sun)) && __cplusplus) /* Solaris includes __STDC_VERSION__ with C++. Tested with GCC 5.5 */ # define XXH_RESTRICT /* disable */ #elif defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* >= C99 */ # define XXH_RESTRICT restrict #else /* Note: it might be useful to define __restrict or __restrict__ for some C++ compilers */ # define XXH_RESTRICT /* disable */ #endif #if (defined(__GNUC__) && (__GNUC__ >= 3)) \ || (defined(__INTEL_COMPILER) && (__INTEL_COMPILER >= 800)) \ || defined(__clang__) # define XXH_likely(x) __builtin_expect(x, 1) # define XXH_unlikely(x) __builtin_expect(x, 0) #else # define XXH_likely(x) (x) # define XXH_unlikely(x) (x) #endif #if defined(__GNUC__) # if defined(__AVX2__) # include # elif defined(__SSE2__) # include # elif defined(__ARM_NEON__) || defined(__ARM_NEON) # define inline __inline__ /* circumvent a clang bug */ # include # undef inline # endif #elif defined(_MSC_VER) # include #endif /* * One goal of XXH3 is to make it fast on both 32-bit and 64-bit, while * remaining a true 64-bit/128-bit hash function. * * This is done by prioritizing a subset of 64-bit operations that can be * emulated without too many steps on the average 32-bit machine. * * For example, these two lines seem similar, and run equally fast on 64-bit: * * xxh_u64 x; * x ^= (x >> 47); // good * x ^= (x >> 13); // bad * * However, to a 32-bit machine, there is a major difference. * * x ^= (x >> 47) looks like this: * * x.lo ^= (x.hi >> (47 - 32)); * * while x ^= (x >> 13) looks like this: * * // note: funnel shifts are not usually cheap. * x.lo ^= (x.lo >> 13) | (x.hi << (32 - 13)); * x.hi ^= (x.hi >> 13); * * The first one is significantly faster than the second, simply because the * shift is larger than 32. This means: * - All the bits we need are in the upper 32 bits, so we can ignore the lower * 32 bits in the shift. * - The shift result will always fit in the lower 32 bits, and therefore, * we can ignore the upper 32 bits in the xor. * * Thanks to this optimization, XXH3 only requires these features to be efficient: * * - Usable unaligned access * - A 32-bit or 64-bit ALU * - If 32-bit, a decent ADC instruction * - A 32 or 64-bit multiply with a 64-bit result * - For the 128-bit variant, a decent byteswap helps short inputs. * * The first two are already required by XXH32, and almost all 32-bit and 64-bit * platforms which can run XXH32 can run XXH3 efficiently. * * Thumb-1, the classic 16-bit only subset of ARM's instruction set, is one * notable exception. * * First of all, Thumb-1 lacks support for the UMULL instruction which * performs the important long multiply. This means numerous __aeabi_lmul * calls. * * Second of all, the 8 functional registers are just not enough. * Setup for __aeabi_lmul, byteshift loads, pointers, and all arithmetic need * Lo registers, and this shuffling results in thousands more MOVs than A32. * * A32 and T32 don't have this limitation. They can access all 14 registers, * do a 32->64 multiply with UMULL, and the flexible operand allowing free * shifts is helpful, too. * * Therefore, we do a quick sanity check. * * If compiling Thumb-1 for a target which supports ARM instructions, we will * emit a warning, as it is not a "sane" platform to compile for. * * Usually, if this happens, it is because of an accident and you probably need * to specify -march, as you likely meant to compile for a newer architecture. * * Credit: large sections of the vectorial and asm source code paths * have been contributed by @easyaspi314 */ #if defined(__thumb__) && !defined(__thumb2__) && defined(__ARM_ARCH_ISA_ARM) # warning "XXH3 is highly inefficient without ARM or Thumb-2." #endif /* ========================================== * Vectorization detection * ========================================== */ #ifdef XXH_DOXYGEN /*! * @ingroup tuning * @brief Overrides the vectorization implementation chosen for XXH3. * * Can be defined to 0 to disable SIMD or any of the values mentioned in * @ref XXH_VECTOR_TYPE. * * If this is not defined, it uses predefined macros to determine the best * implementation. */ # define XXH_VECTOR XXH_SCALAR /*! * @ingroup tuning * @brief Possible values for @ref XXH_VECTOR. * * Note that these are actually implemented as macros. * * If this is not defined, it is detected automatically. * @ref XXH_X86DISPATCH overrides this. */ enum XXH_VECTOR_TYPE /* fake enum */ { XXH_SCALAR = 0, /*!< Portable scalar version */ XXH_SSE2 = 1, /*!< * SSE2 for Pentium 4, Opteron, all x86_64. * * @note SSE2 is also guaranteed on Windows 10, macOS, and * Android x86. */ XXH_AVX2 = 2, /*!< AVX2 for Haswell and Bulldozer */ XXH_AVX512 = 3, /*!< AVX512 for Skylake and Icelake */ XXH_NEON = 4, /*!< NEON for most ARMv7-A and all AArch64 */ XXH_VSX = 5, /*!< VSX and ZVector for POWER8/z13 (64-bit) */ }; /*! * @ingroup tuning * @brief Selects the minimum alignment for XXH3's accumulators. * * When using SIMD, this should match the alignment reqired for said vector * type, so, for example, 32 for AVX2. * * Default: Auto detected. */ # define XXH_ACC_ALIGN 8 #endif /* Actual definition */ #ifndef XXH_DOXYGEN # define XXH_SCALAR 0 # define XXH_SSE2 1 # define XXH_AVX2 2 # define XXH_AVX512 3 # define XXH_NEON 4 # define XXH_VSX 5 #endif #ifndef XXH_VECTOR /* can be defined on command line */ # if defined(__AVX512F__) # define XXH_VECTOR XXH_AVX512 # elif defined(__AVX2__) # define XXH_VECTOR XXH_AVX2 # elif defined(__SSE2__) || defined(_M_AMD64) || defined(_M_X64) || (defined(_M_IX86_FP) && (_M_IX86_FP == 2)) # define XXH_VECTOR XXH_SSE2 # elif defined(__GNUC__) /* msvc support maybe later */ \ && (defined(__ARM_NEON__) || defined(__ARM_NEON)) \ && (defined(__LITTLE_ENDIAN__) /* We only support little endian NEON */ \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__)) # define XXH_VECTOR XXH_NEON # elif (defined(__PPC64__) && defined(__POWER8_VECTOR__)) \ || (defined(__s390x__) && defined(__VEC__)) \ && defined(__GNUC__) /* TODO: IBM XL */ # define XXH_VECTOR XXH_VSX # else # define XXH_VECTOR XXH_SCALAR # endif #endif /* * Controls the alignment of the accumulator, * for compatibility with aligned vector loads, which are usually faster. */ #ifndef XXH_ACC_ALIGN # if defined(XXH_X86DISPATCH) # define XXH_ACC_ALIGN 64 /* for compatibility with avx512 */ # elif XXH_VECTOR == XXH_SCALAR /* scalar */ # define XXH_ACC_ALIGN 8 # elif XXH_VECTOR == XXH_SSE2 /* sse2 */ # define XXH_ACC_ALIGN 16 # elif XXH_VECTOR == XXH_AVX2 /* avx2 */ # define XXH_ACC_ALIGN 32 # elif XXH_VECTOR == XXH_NEON /* neon */ # define XXH_ACC_ALIGN 16 # elif XXH_VECTOR == XXH_VSX /* vsx */ # define XXH_ACC_ALIGN 16 # elif XXH_VECTOR == XXH_AVX512 /* avx512 */ # define XXH_ACC_ALIGN 64 # endif #endif #if defined(XXH_X86DISPATCH) || XXH_VECTOR == XXH_SSE2 \ || XXH_VECTOR == XXH_AVX2 || XXH_VECTOR == XXH_AVX512 # define XXH_SEC_ALIGN XXH_ACC_ALIGN #else # define XXH_SEC_ALIGN 8 #endif /* * UGLY HACK: * GCC usually generates the best code with -O3 for xxHash. * * However, when targeting AVX2, it is overzealous in its unrolling resulting * in code roughly 3/4 the speed of Clang. * * There are other issues, such as GCC splitting _mm256_loadu_si256 into * _mm_loadu_si128 + _mm256_inserti128_si256. This is an optimization which * only applies to Sandy and Ivy Bridge... which don't even support AVX2. * * That is why when compiling the AVX2 version, it is recommended to use either * -O2 -mavx2 -march=haswell * or * -O2 -mavx2 -mno-avx256-split-unaligned-load * for decent performance, or to use Clang instead. * * Fortunately, we can control the first one with a pragma that forces GCC into * -O2, but the other one we can't control without "failed to inline always * inline function due to target mismatch" warnings. */ #if XXH_VECTOR == XXH_AVX2 /* AVX2 */ \ && defined(__GNUC__) && !defined(__clang__) /* GCC, not Clang */ \ && defined(__OPTIMIZE__) && !defined(__OPTIMIZE_SIZE__) /* respect -O0 and -Os */ # pragma GCC push_options # pragma GCC optimize("-O2") #endif #if XXH_VECTOR == XXH_NEON /* * NEON's setup for vmlal_u32 is a little more complicated than it is on * SSE2, AVX2, and VSX. * * While PMULUDQ and VMULEUW both perform a mask, VMLAL.U32 performs an upcast. * * To do the same operation, the 128-bit 'Q' register needs to be split into * two 64-bit 'D' registers, performing this operation:: * * [ a | b ] * | '---------. .--------' | * | x | * | .---------' '--------. | * [ a & 0xFFFFFFFF | b & 0xFFFFFFFF ],[ a >> 32 | b >> 32 ] * * Due to significant changes in aarch64, the fastest method for aarch64 is * completely different than the fastest method for ARMv7-A. * * ARMv7-A treats D registers as unions overlaying Q registers, so modifying * D11 will modify the high half of Q5. This is similar to how modifying AH * will only affect bits 8-15 of AX on x86. * * VZIP takes two registers, and puts even lanes in one register and odd lanes * in the other. * * On ARMv7-A, this strangely modifies both parameters in place instead of * taking the usual 3-operand form. * * Therefore, if we want to do this, we can simply use a D-form VZIP.32 on the * lower and upper halves of the Q register to end up with the high and low * halves where we want - all in one instruction. * * vzip.32 d10, d11 @ d10 = { d10[0], d11[0] }; d11 = { d10[1], d11[1] } * * Unfortunately we need inline assembly for this: Instructions modifying two * registers at once is not possible in GCC or Clang's IR, and they have to * create a copy. * * aarch64 requires a different approach. * * In order to make it easier to write a decent compiler for aarch64, many * quirks were removed, such as conditional execution. * * NEON was also affected by this. * * aarch64 cannot access the high bits of a Q-form register, and writes to a * D-form register zero the high bits, similar to how writes to W-form scalar * registers (or DWORD registers on x86_64) work. * * The formerly free vget_high intrinsics now require a vext (with a few * exceptions) * * Additionally, VZIP was replaced by ZIP1 and ZIP2, which are the equivalent * of PUNPCKL* and PUNPCKH* in SSE, respectively, in order to only modify one * operand. * * The equivalent of the VZIP.32 on the lower and upper halves would be this * mess: * * ext v2.4s, v0.4s, v0.4s, #2 // v2 = { v0[2], v0[3], v0[0], v0[1] } * zip1 v1.2s, v0.2s, v2.2s // v1 = { v0[0], v2[0] } * zip2 v0.2s, v0.2s, v1.2s // v0 = { v0[1], v2[1] } * * Instead, we use a literal downcast, vmovn_u64 (XTN), and vshrn_n_u64 (SHRN): * * shrn v1.2s, v0.2d, #32 // v1 = (uint32x2_t)(v0 >> 32); * xtn v0.2s, v0.2d // v0 = (uint32x2_t)(v0 & 0xFFFFFFFF); * * This is available on ARMv7-A, but is less efficient than a single VZIP.32. */ /*! * Function-like macro: * void XXH_SPLIT_IN_PLACE(uint64x2_t &in, uint32x2_t &outLo, uint32x2_t &outHi) * { * outLo = (uint32x2_t)(in & 0xFFFFFFFF); * outHi = (uint32x2_t)(in >> 32); * in = UNDEFINED; * } */ # if !defined(XXH_NO_VZIP_HACK) /* define to disable */ \ && defined(__GNUC__) \ && !defined(__aarch64__) && !defined(__arm64__) # define XXH_SPLIT_IN_PLACE(in, outLo, outHi) \ do { \ /* Undocumented GCC/Clang operand modifier: %e0 = lower D half, %f0 = upper D half */ \ /* https://github.com/gcc-mirror/gcc/blob/38cf91e5/gcc/config/arm/arm.c#L22486 */ \ /* https://github.com/llvm-mirror/llvm/blob/2c4ca683/lib/Target/ARM/ARMAsmPrinter.cpp#L399 */ \ __asm__("vzip.32 %e0, %f0" : "+w" (in)); \ (outLo) = vget_low_u32 (vreinterpretq_u32_u64(in)); \ (outHi) = vget_high_u32(vreinterpretq_u32_u64(in)); \ } while (0) # else # define XXH_SPLIT_IN_PLACE(in, outLo, outHi) \ do { \ (outLo) = vmovn_u64 (in); \ (outHi) = vshrn_n_u64 ((in), 32); \ } while (0) # endif #endif /* XXH_VECTOR == XXH_NEON */ /* * VSX and Z Vector helpers. * * This is very messy, and any pull requests to clean this up are welcome. * * There are a lot of problems with supporting VSX and s390x, due to * inconsistent intrinsics, spotty coverage, and multiple endiannesses. */ #if XXH_VECTOR == XXH_VSX # if defined(__s390x__) # include # else /* gcc's altivec.h can have the unwanted consequence to unconditionally * #define bool, vector, and pixel keywords, * with bad consequences for programs already using these keywords for other purposes. * The paragraph defining these macros is skipped when __APPLE_ALTIVEC__ is defined. * __APPLE_ALTIVEC__ is _generally_ defined automatically by the compiler, * but it seems that, in some cases, it isn't. * Force the build macro to be defined, so that keywords are not altered. */ # if defined(__GNUC__) && !defined(__APPLE_ALTIVEC__) # define __APPLE_ALTIVEC__ # endif # include # endif typedef __vector unsigned long long xxh_u64x2; typedef __vector unsigned char xxh_u8x16; typedef __vector unsigned xxh_u32x4; # ifndef XXH_VSX_BE # if defined(__BIG_ENDIAN__) \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) # define XXH_VSX_BE 1 # elif defined(__VEC_ELEMENT_REG_ORDER__) && __VEC_ELEMENT_REG_ORDER__ == __ORDER_BIG_ENDIAN__ # warning "-maltivec=be is not recommended. Please use native endianness." # define XXH_VSX_BE 1 # else # define XXH_VSX_BE 0 # endif # endif /* !defined(XXH_VSX_BE) */ # if XXH_VSX_BE # if defined(__POWER9_VECTOR__) || (defined(__clang__) && defined(__s390x__)) # define XXH_vec_revb vec_revb # else /*! * A polyfill for POWER9's vec_revb(). */ XXH_FORCE_INLINE xxh_u64x2 XXH_vec_revb(xxh_u64x2 val) { xxh_u8x16 const vByteSwap = { 0x07, 0x06, 0x05, 0x04, 0x03, 0x02, 0x01, 0x00, 0x0F, 0x0E, 0x0D, 0x0C, 0x0B, 0x0A, 0x09, 0x08 }; return vec_perm(val, val, vByteSwap); } # endif # endif /* XXH_VSX_BE */ /*! * Performs an unaligned vector load and byte swaps it on big endian. */ XXH_FORCE_INLINE xxh_u64x2 XXH_vec_loadu(const void *ptr) { xxh_u64x2 ret; memcpy(&ret, ptr, sizeof(xxh_u64x2)); # if XXH_VSX_BE ret = XXH_vec_revb(ret); # endif return ret; } /* * vec_mulo and vec_mule are very problematic intrinsics on PowerPC * * These intrinsics weren't added until GCC 8, despite existing for a while, * and they are endian dependent. Also, their meaning swap depending on version. * */ # if defined(__s390x__) /* s390x is always big endian, no issue on this platform */ # define XXH_vec_mulo vec_mulo # define XXH_vec_mule vec_mule # elif defined(__clang__) && XXH_HAS_BUILTIN(__builtin_altivec_vmuleuw) /* Clang has a better way to control this, we can just use the builtin which doesn't swap. */ # define XXH_vec_mulo __builtin_altivec_vmulouw # define XXH_vec_mule __builtin_altivec_vmuleuw # else /* gcc needs inline assembly */ /* Adapted from https://github.com/google/highwayhash/blob/master/highwayhash/hh_vsx.h. */ XXH_FORCE_INLINE xxh_u64x2 XXH_vec_mulo(xxh_u32x4 a, xxh_u32x4 b) { xxh_u64x2 result; __asm__("vmulouw %0, %1, %2" : "=v" (result) : "v" (a), "v" (b)); return result; } XXH_FORCE_INLINE xxh_u64x2 XXH_vec_mule(xxh_u32x4 a, xxh_u32x4 b) { xxh_u64x2 result; __asm__("vmuleuw %0, %1, %2" : "=v" (result) : "v" (a), "v" (b)); return result; } # endif /* XXH_vec_mulo, XXH_vec_mule */ #endif /* XXH_VECTOR == XXH_VSX */ /* prefetch * can be disabled, by declaring XXH_NO_PREFETCH build macro */ #if defined(XXH_NO_PREFETCH) # define XXH_PREFETCH(ptr) (void)(ptr) /* disabled */ #else # if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_IX86)) /* _mm_prefetch() not defined outside of x86/x64 */ # include /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */ # define XXH_PREFETCH(ptr) _mm_prefetch((const char*)(ptr), _MM_HINT_T0) # elif defined(__GNUC__) && ( (__GNUC__ >= 4) || ( (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) ) ) # define XXH_PREFETCH(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 3 /* locality */) # else # define XXH_PREFETCH(ptr) (void)(ptr) /* disabled */ # endif #endif /* XXH_NO_PREFETCH */ /* ========================================== * XXH3 default settings * ========================================== */ #define XXH_SECRET_DEFAULT_SIZE 192 /* minimum XXH3_SECRET_SIZE_MIN */ #if (XXH_SECRET_DEFAULT_SIZE < XXH3_SECRET_SIZE_MIN) # error "default keyset is not large enough" #endif /*! Pseudorandom secret taken directly from FARSH. */ XXH_ALIGN(64) static const xxh_u8 XXH3_kSecret[XXH_SECRET_DEFAULT_SIZE] = { 0xb8, 0xfe, 0x6c, 0x39, 0x23, 0xa4, 0x4b, 0xbe, 0x7c, 0x01, 0x81, 0x2c, 0xf7, 0x21, 0xad, 0x1c, 0xde, 0xd4, 0x6d, 0xe9, 0x83, 0x90, 0x97, 0xdb, 0x72, 0x40, 0xa4, 0xa4, 0xb7, 0xb3, 0x67, 0x1f, 0xcb, 0x79, 0xe6, 0x4e, 0xcc, 0xc0, 0xe5, 0x78, 0x82, 0x5a, 0xd0, 0x7d, 0xcc, 0xff, 0x72, 0x21, 0xb8, 0x08, 0x46, 0x74, 0xf7, 0x43, 0x24, 0x8e, 0xe0, 0x35, 0x90, 0xe6, 0x81, 0x3a, 0x26, 0x4c, 0x3c, 0x28, 0x52, 0xbb, 0x91, 0xc3, 0x00, 0xcb, 0x88, 0xd0, 0x65, 0x8b, 0x1b, 0x53, 0x2e, 0xa3, 0x71, 0x64, 0x48, 0x97, 0xa2, 0x0d, 0xf9, 0x4e, 0x38, 0x19, 0xef, 0x46, 0xa9, 0xde, 0xac, 0xd8, 0xa8, 0xfa, 0x76, 0x3f, 0xe3, 0x9c, 0x34, 0x3f, 0xf9, 0xdc, 0xbb, 0xc7, 0xc7, 0x0b, 0x4f, 0x1d, 0x8a, 0x51, 0xe0, 0x4b, 0xcd, 0xb4, 0x59, 0x31, 0xc8, 0x9f, 0x7e, 0xc9, 0xd9, 0x78, 0x73, 0x64, 0xea, 0xc5, 0xac, 0x83, 0x34, 0xd3, 0xeb, 0xc3, 0xc5, 0x81, 0xa0, 0xff, 0xfa, 0x13, 0x63, 0xeb, 0x17, 0x0d, 0xdd, 0x51, 0xb7, 0xf0, 0xda, 0x49, 0xd3, 0x16, 0x55, 0x26, 0x29, 0xd4, 0x68, 0x9e, 0x2b, 0x16, 0xbe, 0x58, 0x7d, 0x47, 0xa1, 0xfc, 0x8f, 0xf8, 0xb8, 0xd1, 0x7a, 0xd0, 0x31, 0xce, 0x45, 0xcb, 0x3a, 0x8f, 0x95, 0x16, 0x04, 0x28, 0xaf, 0xd7, 0xfb, 0xca, 0xbb, 0x4b, 0x40, 0x7e, }; #ifdef XXH_OLD_NAMES # define kSecret XXH3_kSecret #endif #ifdef XXH_DOXYGEN /*! * @brief Calculates a 32-bit to 64-bit long multiply. * * Implemented as a macro. * * Wraps `__emulu` on MSVC x86 because it tends to call `__allmul` when it doesn't * need to (but it shouldn't need to anyways, it is about 7 instructions to do * a 64x64 multiply...). Since we know that this will _always_ emit `MULL`, we * use that instead of the normal method. * * If you are compiling for platforms like Thumb-1 and don't have a better option, * you may also want to write your own long multiply routine here. * * @param x, y Numbers to be multiplied * @return 64-bit product of the low 32 bits of @p x and @p y. */ XXH_FORCE_INLINE xxh_u64 XXH_mult32to64(xxh_u64 x, xxh_u64 y) { return (x & 0xFFFFFFFF) * (y & 0xFFFFFFFF); } #elif defined(_MSC_VER) && defined(_M_IX86) # include # define XXH_mult32to64(x, y) __emulu((unsigned)(x), (unsigned)(y)) #else /* * Downcast + upcast is usually better than masking on older compilers like * GCC 4.2 (especially 32-bit ones), all without affecting newer compilers. * * The other method, (x & 0xFFFFFFFF) * (y & 0xFFFFFFFF), will AND both operands * and perform a full 64x64 multiply -- entirely redundant on 32-bit. */ # define XXH_mult32to64(x, y) ((xxh_u64)(xxh_u32)(x) * (xxh_u64)(xxh_u32)(y)) #endif /*! * @brief Calculates a 64->128-bit long multiply. * * Uses `__uint128_t` and `_umul128` if available, otherwise uses a scalar * version. * * @param lhs, rhs The 64-bit integers to be multiplied * @return The 128-bit result represented in an @ref XXH128_hash_t. */ static XXH128_hash_t XXH_mult64to128(xxh_u64 lhs, xxh_u64 rhs) { /* * GCC/Clang __uint128_t method. * * On most 64-bit targets, GCC and Clang define a __uint128_t type. * This is usually the best way as it usually uses a native long 64-bit * multiply, such as MULQ on x86_64 or MUL + UMULH on aarch64. * * Usually. * * Despite being a 32-bit platform, Clang (and emscripten) define this type * despite not having the arithmetic for it. This results in a laggy * compiler builtin call which calculates a full 128-bit multiply. * In that case it is best to use the portable one. * https://github.com/Cyan4973/xxHash/issues/211#issuecomment-515575677 */ #if defined(__GNUC__) && !defined(__wasm__) \ && defined(__SIZEOF_INT128__) \ || (defined(_INTEGRAL_MAX_BITS) && _INTEGRAL_MAX_BITS >= 128) __uint128_t const product = (__uint128_t)lhs * (__uint128_t)rhs; XXH128_hash_t r128; r128.low64 = (xxh_u64)(product); r128.high64 = (xxh_u64)(product >> 64); return r128; /* * MSVC for x64's _umul128 method. * * xxh_u64 _umul128(xxh_u64 Multiplier, xxh_u64 Multiplicand, xxh_u64 *HighProduct); * * This compiles to single operand MUL on x64. */ #elif defined(_M_X64) || defined(_M_IA64) #ifndef _MSC_VER # pragma intrinsic(_umul128) #endif xxh_u64 product_high; xxh_u64 const product_low = _umul128(lhs, rhs, &product_high); XXH128_hash_t r128; r128.low64 = product_low; r128.high64 = product_high; return r128; #else /* * Portable scalar method. Optimized for 32-bit and 64-bit ALUs. * * This is a fast and simple grade school multiply, which is shown below * with base 10 arithmetic instead of base 0x100000000. * * 9 3 // D2 lhs = 93 * x 7 5 // D2 rhs = 75 * ---------- * 1 5 // D2 lo_lo = (93 % 10) * (75 % 10) = 15 * 4 5 | // D2 hi_lo = (93 / 10) * (75 % 10) = 45 * 2 1 | // D2 lo_hi = (93 % 10) * (75 / 10) = 21 * + 6 3 | | // D2 hi_hi = (93 / 10) * (75 / 10) = 63 * --------- * 2 7 | // D2 cross = (15 / 10) + (45 % 10) + 21 = 27 * + 6 7 | | // D2 upper = (27 / 10) + (45 / 10) + 63 = 67 * --------- * 6 9 7 5 // D4 res = (27 * 10) + (15 % 10) + (67 * 100) = 6975 * * The reasons for adding the products like this are: * 1. It avoids manual carry tracking. Just like how * (9 * 9) + 9 + 9 = 99, the same applies with this for UINT64_MAX. * This avoids a lot of complexity. * * 2. It hints for, and on Clang, compiles to, the powerful UMAAL * instruction available in ARM's Digital Signal Processing extension * in 32-bit ARMv6 and later, which is shown below: * * void UMAAL(xxh_u32 *RdLo, xxh_u32 *RdHi, xxh_u32 Rn, xxh_u32 Rm) * { * xxh_u64 product = (xxh_u64)*RdLo * (xxh_u64)*RdHi + Rn + Rm; * *RdLo = (xxh_u32)(product & 0xFFFFFFFF); * *RdHi = (xxh_u32)(product >> 32); * } * * This instruction was designed for efficient long multiplication, and * allows this to be calculated in only 4 instructions at speeds * comparable to some 64-bit ALUs. * * 3. It isn't terrible on other platforms. Usually this will be a couple * of 32-bit ADD/ADCs. */ /* First calculate all of the cross products. */ xxh_u64 const lo_lo = XXH_mult32to64(lhs & 0xFFFFFFFF, rhs & 0xFFFFFFFF); xxh_u64 const hi_lo = XXH_mult32to64(lhs >> 32, rhs & 0xFFFFFFFF); xxh_u64 const lo_hi = XXH_mult32to64(lhs & 0xFFFFFFFF, rhs >> 32); xxh_u64 const hi_hi = XXH_mult32to64(lhs >> 32, rhs >> 32); /* Now add the products together. These will never overflow. */ xxh_u64 const cross = (lo_lo >> 32) + (hi_lo & 0xFFFFFFFF) + lo_hi; xxh_u64 const upper = (hi_lo >> 32) + (cross >> 32) + hi_hi; xxh_u64 const lower = (cross << 32) | (lo_lo & 0xFFFFFFFF); XXH128_hash_t r128; r128.low64 = lower; r128.high64 = upper; return r128; #endif } /*! * @brief Calculates a 64-bit to 128-bit multiply, then XOR folds it. * * The reason for the separate function is to prevent passing too many structs * around by value. This will hopefully inline the multiply, but we don't force it. * * @param lhs, rhs The 64-bit integers to multiply * @return The low 64 bits of the product XOR'd by the high 64 bits. * @see XXH_mult64to128() */ static xxh_u64 XXH3_mul128_fold64(xxh_u64 lhs, xxh_u64 rhs) { XXH128_hash_t product = XXH_mult64to128(lhs, rhs); return product.low64 ^ product.high64; } /*! Seems to produce slightly better code on GCC for some reason. */ XXH_FORCE_INLINE xxh_u64 XXH_xorshift64(xxh_u64 v64, int shift) { XXH_ASSERT(0 <= shift && shift < 64); return v64 ^ (v64 >> shift); } /* * This is a fast avalanche stage, * suitable when input bits are already partially mixed */ static XXH64_hash_t XXH3_avalanche(xxh_u64 h64) { h64 = XXH_xorshift64(h64, 37); h64 *= 0x165667919E3779F9ULL; h64 = XXH_xorshift64(h64, 32); return h64; } /* * This is a stronger avalanche, * inspired by Pelle Evensen's rrmxmx * preferable when input has not been previously mixed */ static XXH64_hash_t XXH3_rrmxmx(xxh_u64 h64, xxh_u64 len) { /* this mix is inspired by Pelle Evensen's rrmxmx */ h64 ^= XXH_rotl64(h64, 49) ^ XXH_rotl64(h64, 24); h64 *= 0x9FB21C651E98DF25ULL; h64 ^= (h64 >> 35) + len ; h64 *= 0x9FB21C651E98DF25ULL; return XXH_xorshift64(h64, 28); } /* ========================================== * Short keys * ========================================== * One of the shortcomings of XXH32 and XXH64 was that their performance was * sub-optimal on short lengths. It used an iterative algorithm which strongly * favored lengths that were a multiple of 4 or 8. * * Instead of iterating over individual inputs, we use a set of single shot * functions which piece together a range of lengths and operate in constant time. * * Additionally, the number of multiplies has been significantly reduced. This * reduces latency, especially when emulating 64-bit multiplies on 32-bit. * * Depending on the platform, this may or may not be faster than XXH32, but it * is almost guaranteed to be faster than XXH64. */ /* * At very short lengths, there isn't enough input to fully hide secrets, or use * the entire secret. * * There is also only a limited amount of mixing we can do before significantly * impacting performance. * * Therefore, we use different sections of the secret and always mix two secret * samples with an XOR. This should have no effect on performance on the * seedless or withSeed variants because everything _should_ be constant folded * by modern compilers. * * The XOR mixing hides individual parts of the secret and increases entropy. * * This adds an extra layer of strength for custom secrets. */ XXH_FORCE_INLINE XXH64_hash_t XXH3_len_1to3_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(1 <= len && len <= 3); XXH_ASSERT(secret != NULL); /* * len = 1: combined = { input[0], 0x01, input[0], input[0] } * len = 2: combined = { input[1], 0x02, input[0], input[1] } * len = 3: combined = { input[2], 0x03, input[0], input[1] } */ { xxh_u8 const c1 = input[0]; xxh_u8 const c2 = input[len >> 1]; xxh_u8 const c3 = input[len - 1]; xxh_u32 const combined = ((xxh_u32)c1 << 16) | ((xxh_u32)c2 << 24) | ((xxh_u32)c3 << 0) | ((xxh_u32)len << 8); xxh_u64 const bitflip = (XXH_readLE32(secret) ^ XXH_readLE32(secret+4)) + seed; xxh_u64 const keyed = (xxh_u64)combined ^ bitflip; return XXH64_avalanche(keyed); } } XXH_FORCE_INLINE XXH64_hash_t XXH3_len_4to8_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(4 <= len && len <= 8); seed ^= (xxh_u64)XXH_swap32((xxh_u32)seed) << 32; { xxh_u32 const input1 = XXH_readLE32(input); xxh_u32 const input2 = XXH_readLE32(input + len - 4); xxh_u64 const bitflip = (XXH_readLE64(secret+8) ^ XXH_readLE64(secret+16)) - seed; xxh_u64 const input64 = input2 + (((xxh_u64)input1) << 32); xxh_u64 const keyed = input64 ^ bitflip; return XXH3_rrmxmx(keyed, len); } } XXH_FORCE_INLINE XXH64_hash_t XXH3_len_9to16_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(8 <= len && len <= 16); { xxh_u64 const bitflip1 = (XXH_readLE64(secret+24) ^ XXH_readLE64(secret+32)) + seed; xxh_u64 const bitflip2 = (XXH_readLE64(secret+40) ^ XXH_readLE64(secret+48)) - seed; xxh_u64 const input_lo = XXH_readLE64(input) ^ bitflip1; xxh_u64 const input_hi = XXH_readLE64(input + len - 8) ^ bitflip2; xxh_u64 const acc = len + XXH_swap64(input_lo) + input_hi + XXH3_mul128_fold64(input_lo, input_hi); return XXH3_avalanche(acc); } } XXH_FORCE_INLINE XXH64_hash_t XXH3_len_0to16_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(len <= 16); { if (XXH_likely(len > 8)) return XXH3_len_9to16_64b(input, len, secret, seed); if (XXH_likely(len >= 4)) return XXH3_len_4to8_64b(input, len, secret, seed); if (len) return XXH3_len_1to3_64b(input, len, secret, seed); return XXH64_avalanche(seed ^ (XXH_readLE64(secret+56) ^ XXH_readLE64(secret+64))); } } /* * DISCLAIMER: There are known *seed-dependent* multicollisions here due to * multiplication by zero, affecting hashes of lengths 17 to 240. * * However, they are very unlikely. * * Keep this in mind when using the unseeded XXH3_64bits() variant: As with all * unseeded non-cryptographic hashes, it does not attempt to defend itself * against specially crafted inputs, only random inputs. * * Compared to classic UMAC where a 1 in 2^31 chance of 4 consecutive bytes * cancelling out the secret is taken an arbitrary number of times (addressed * in XXH3_accumulate_512), this collision is very unlikely with random inputs * and/or proper seeding: * * This only has a 1 in 2^63 chance of 8 consecutive bytes cancelling out, in a * function that is only called up to 16 times per hash with up to 240 bytes of * input. * * This is not too bad for a non-cryptographic hash function, especially with * only 64 bit outputs. * * The 128-bit variant (which trades some speed for strength) is NOT affected * by this, although it is always a good idea to use a proper seed if you care * about strength. */ XXH_FORCE_INLINE xxh_u64 XXH3_mix16B(const xxh_u8* XXH_RESTRICT input, const xxh_u8* XXH_RESTRICT secret, xxh_u64 seed64) { #if defined(__GNUC__) && !defined(__clang__) /* GCC, not Clang */ \ && defined(__i386__) && defined(__SSE2__) /* x86 + SSE2 */ \ && !defined(XXH_ENABLE_AUTOVECTORIZE) /* Define to disable like XXH32 hack */ /* * UGLY HACK: * GCC for x86 tends to autovectorize the 128-bit multiply, resulting in * slower code. * * By forcing seed64 into a register, we disrupt the cost model and * cause it to scalarize. See `XXH32_round()` * * FIXME: Clang's output is still _much_ faster -- On an AMD Ryzen 3600, * XXH3_64bits @ len=240 runs at 4.6 GB/s with Clang 9, but 3.3 GB/s on * GCC 9.2, despite both emitting scalar code. * * GCC generates much better scalar code than Clang for the rest of XXH3, * which is why finding a more optimal codepath is an interest. */ XXH_COMPILER_GUARD(seed64); #endif { xxh_u64 const input_lo = XXH_readLE64(input); xxh_u64 const input_hi = XXH_readLE64(input+8); return XXH3_mul128_fold64( input_lo ^ (XXH_readLE64(secret) + seed64), input_hi ^ (XXH_readLE64(secret+8) - seed64) ); } } /* For mid range keys, XXH3 uses a Mum-hash variant. */ XXH_FORCE_INLINE XXH64_hash_t XXH3_len_17to128_64b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(16 < len && len <= 128); { xxh_u64 acc = len * XXH_PRIME64_1; if (len > 32) { if (len > 64) { if (len > 96) { acc += XXH3_mix16B(input+48, secret+96, seed); acc += XXH3_mix16B(input+len-64, secret+112, seed); } acc += XXH3_mix16B(input+32, secret+64, seed); acc += XXH3_mix16B(input+len-48, secret+80, seed); } acc += XXH3_mix16B(input+16, secret+32, seed); acc += XXH3_mix16B(input+len-32, secret+48, seed); } acc += XXH3_mix16B(input+0, secret+0, seed); acc += XXH3_mix16B(input+len-16, secret+16, seed); return XXH3_avalanche(acc); } } #define XXH3_MIDSIZE_MAX 240 XXH_NO_INLINE XXH64_hash_t XXH3_len_129to240_64b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(128 < len && len <= XXH3_MIDSIZE_MAX); #define XXH3_MIDSIZE_STARTOFFSET 3 #define XXH3_MIDSIZE_LASTOFFSET 17 { xxh_u64 acc = len * XXH_PRIME64_1; int const nbRounds = (int)len / 16; int i; for (i=0; i<8; i++) { acc += XXH3_mix16B(input+(16*i), secret+(16*i), seed); } acc = XXH3_avalanche(acc); XXH_ASSERT(nbRounds >= 8); #if defined(__clang__) /* Clang */ \ && (defined(__ARM_NEON) || defined(__ARM_NEON__)) /* NEON */ \ && !defined(XXH_ENABLE_AUTOVECTORIZE) /* Define to disable */ /* * UGLY HACK: * Clang for ARMv7-A tries to vectorize this loop, similar to GCC x86. * In everywhere else, it uses scalar code. * * For 64->128-bit multiplies, even if the NEON was 100% optimal, it * would still be slower than UMAAL (see XXH_mult64to128). * * Unfortunately, Clang doesn't handle the long multiplies properly and * converts them to the nonexistent "vmulq_u64" intrinsic, which is then * scalarized into an ugly mess of VMOV.32 instructions. * * This mess is difficult to avoid without turning autovectorization * off completely, but they are usually relatively minor and/or not * worth it to fix. * * This loop is the easiest to fix, as unlike XXH32, this pragma * _actually works_ because it is a loop vectorization instead of an * SLP vectorization. */ #pragma clang loop vectorize(disable) #endif for (i=8 ; i < nbRounds; i++) { acc += XXH3_mix16B(input+(16*i), secret+(16*(i-8)) + XXH3_MIDSIZE_STARTOFFSET, seed); } /* last bytes */ acc += XXH3_mix16B(input + len - 16, secret + XXH3_SECRET_SIZE_MIN - XXH3_MIDSIZE_LASTOFFSET, seed); return XXH3_avalanche(acc); } } /* ======= Long Keys ======= */ #define XXH_STRIPE_LEN 64 #define XXH_SECRET_CONSUME_RATE 8 /* nb of secret bytes consumed at each accumulation */ #define XXH_ACC_NB (XXH_STRIPE_LEN / sizeof(xxh_u64)) #ifdef XXH_OLD_NAMES # define STRIPE_LEN XXH_STRIPE_LEN # define ACC_NB XXH_ACC_NB #endif XXH_FORCE_INLINE void XXH_writeLE64(void* dst, xxh_u64 v64) { if (!XXH_CPU_LITTLE_ENDIAN) v64 = XXH_swap64(v64); memcpy(dst, &v64, sizeof(v64)); } /* Several intrinsic functions below are supposed to accept __int64 as argument, * as documented in https://software.intel.com/sites/landingpage/IntrinsicsGuide/ . * However, several environments do not define __int64 type, * requiring a workaround. */ #if !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) typedef int64_t xxh_i64; #else /* the following type must have a width of 64-bit */ typedef long long xxh_i64; #endif /* * XXH3_accumulate_512 is the tightest loop for long inputs, and it is the most optimized. * * It is a hardened version of UMAC, based off of FARSH's implementation. * * This was chosen because it adapts quite well to 32-bit, 64-bit, and SIMD * implementations, and it is ridiculously fast. * * We harden it by mixing the original input to the accumulators as well as the product. * * This means that in the (relatively likely) case of a multiply by zero, the * original input is preserved. * * On 128-bit inputs, we swap 64-bit pairs when we add the input to improve * cross-pollination, as otherwise the upper and lower halves would be * essentially independent. * * This doesn't matter on 64-bit hashes since they all get merged together in * the end, so we skip the extra step. * * Both XXH3_64bits and XXH3_128bits use this subroutine. */ #if (XXH_VECTOR == XXH_AVX512) \ || (defined(XXH_DISPATCH_AVX512) && XXH_DISPATCH_AVX512 != 0) #ifndef XXH_TARGET_AVX512 # define XXH_TARGET_AVX512 /* disable attribute target */ #endif XXH_FORCE_INLINE XXH_TARGET_AVX512 void XXH3_accumulate_512_avx512(void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { XXH_ALIGN(64) __m512i* const xacc = (__m512i *) acc; XXH_ASSERT((((size_t)acc) & 63) == 0); XXH_STATIC_ASSERT(XXH_STRIPE_LEN == sizeof(__m512i)); { /* data_vec = input[0]; */ __m512i const data_vec = _mm512_loadu_si512 (input); /* key_vec = secret[0]; */ __m512i const key_vec = _mm512_loadu_si512 (secret); /* data_key = data_vec ^ key_vec; */ __m512i const data_key = _mm512_xor_si512 (data_vec, key_vec); /* data_key_lo = data_key >> 32; */ __m512i const data_key_lo = _mm512_shuffle_epi32 (data_key, (_MM_PERM_ENUM)_MM_SHUFFLE(0, 3, 0, 1)); /* product = (data_key & 0xffffffff) * (data_key_lo & 0xffffffff); */ __m512i const product = _mm512_mul_epu32 (data_key, data_key_lo); /* xacc[0] += swap(data_vec); */ __m512i const data_swap = _mm512_shuffle_epi32(data_vec, (_MM_PERM_ENUM)_MM_SHUFFLE(1, 0, 3, 2)); __m512i const sum = _mm512_add_epi64(*xacc, data_swap); /* xacc[0] += product; */ *xacc = _mm512_add_epi64(product, sum); } } /* * XXH3_scrambleAcc: Scrambles the accumulators to improve mixing. * * Multiplication isn't perfect, as explained by Google in HighwayHash: * * // Multiplication mixes/scrambles bytes 0-7 of the 64-bit result to * // varying degrees. In descending order of goodness, bytes * // 3 4 2 5 1 6 0 7 have quality 228 224 164 160 100 96 36 32. * // As expected, the upper and lower bytes are much worse. * * Source: https://github.com/google/highwayhash/blob/0aaf66b/highwayhash/hh_avx2.h#L291 * * Since our algorithm uses a pseudorandom secret to add some variance into the * mix, we don't need to (or want to) mix as often or as much as HighwayHash does. * * This isn't as tight as XXH3_accumulate, but still written in SIMD to avoid * extraction. * * Both XXH3_64bits and XXH3_128bits use this subroutine. */ XXH_FORCE_INLINE XXH_TARGET_AVX512 void XXH3_scrambleAcc_avx512(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 63) == 0); XXH_STATIC_ASSERT(XXH_STRIPE_LEN == sizeof(__m512i)); { XXH_ALIGN(64) __m512i* const xacc = (__m512i*) acc; const __m512i prime32 = _mm512_set1_epi32((int)XXH_PRIME32_1); /* xacc[0] ^= (xacc[0] >> 47) */ __m512i const acc_vec = *xacc; __m512i const shifted = _mm512_srli_epi64 (acc_vec, 47); __m512i const data_vec = _mm512_xor_si512 (acc_vec, shifted); /* xacc[0] ^= secret; */ __m512i const key_vec = _mm512_loadu_si512 (secret); __m512i const data_key = _mm512_xor_si512 (data_vec, key_vec); /* xacc[0] *= XXH_PRIME32_1; */ __m512i const data_key_hi = _mm512_shuffle_epi32 (data_key, (_MM_PERM_ENUM)_MM_SHUFFLE(0, 3, 0, 1)); __m512i const prod_lo = _mm512_mul_epu32 (data_key, prime32); __m512i const prod_hi = _mm512_mul_epu32 (data_key_hi, prime32); *xacc = _mm512_add_epi64(prod_lo, _mm512_slli_epi64(prod_hi, 32)); } } XXH_FORCE_INLINE XXH_TARGET_AVX512 void XXH3_initCustomSecret_avx512(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 63) == 0); XXH_STATIC_ASSERT(XXH_SEC_ALIGN == 64); XXH_ASSERT(((size_t)customSecret & 63) == 0); (void)(&XXH_writeLE64); { int const nbRounds = XXH_SECRET_DEFAULT_SIZE / sizeof(__m512i); __m512i const seed = _mm512_mask_set1_epi64(_mm512_set1_epi64((xxh_i64)seed64), 0xAA, -(xxh_i64)seed64); XXH_ALIGN(64) const __m512i* const src = (const __m512i*) XXH3_kSecret; XXH_ALIGN(64) __m512i* const dest = ( __m512i*) customSecret; int i; for (i=0; i < nbRounds; ++i) { /* GCC has a bug, _mm512_stream_load_si512 accepts 'void*', not 'void const*', * this will warn "discards ‘const’ qualifier". */ union { XXH_ALIGN(64) const __m512i* cp; XXH_ALIGN(64) void* p; } remote_const_void; remote_const_void.cp = src + i; dest[i] = _mm512_add_epi64(_mm512_stream_load_si512(remote_const_void.p), seed); } } } #endif #if (XXH_VECTOR == XXH_AVX2) \ || (defined(XXH_DISPATCH_AVX2) && XXH_DISPATCH_AVX2 != 0) #ifndef XXH_TARGET_AVX2 # define XXH_TARGET_AVX2 /* disable attribute target */ #endif XXH_FORCE_INLINE XXH_TARGET_AVX2 void XXH3_accumulate_512_avx2( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 31) == 0); { XXH_ALIGN(32) __m256i* const xacc = (__m256i *) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm256_loadu_si256 requires a const __m256i * pointer for some reason. */ const __m256i* const xinput = (const __m256i *) input; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm256_loadu_si256 requires a const __m256i * pointer for some reason. */ const __m256i* const xsecret = (const __m256i *) secret; size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m256i); i++) { /* data_vec = xinput[i]; */ __m256i const data_vec = _mm256_loadu_si256 (xinput+i); /* key_vec = xsecret[i]; */ __m256i const key_vec = _mm256_loadu_si256 (xsecret+i); /* data_key = data_vec ^ key_vec; */ __m256i const data_key = _mm256_xor_si256 (data_vec, key_vec); /* data_key_lo = data_key >> 32; */ __m256i const data_key_lo = _mm256_shuffle_epi32 (data_key, _MM_SHUFFLE(0, 3, 0, 1)); /* product = (data_key & 0xffffffff) * (data_key_lo & 0xffffffff); */ __m256i const product = _mm256_mul_epu32 (data_key, data_key_lo); /* xacc[i] += swap(data_vec); */ __m256i const data_swap = _mm256_shuffle_epi32(data_vec, _MM_SHUFFLE(1, 0, 3, 2)); __m256i const sum = _mm256_add_epi64(xacc[i], data_swap); /* xacc[i] += product; */ xacc[i] = _mm256_add_epi64(product, sum); } } } XXH_FORCE_INLINE XXH_TARGET_AVX2 void XXH3_scrambleAcc_avx2(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 31) == 0); { XXH_ALIGN(32) __m256i* const xacc = (__m256i*) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm256_loadu_si256 requires a const __m256i * pointer for some reason. */ const __m256i* const xsecret = (const __m256i *) secret; const __m256i prime32 = _mm256_set1_epi32((int)XXH_PRIME32_1); size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m256i); i++) { /* xacc[i] ^= (xacc[i] >> 47) */ __m256i const acc_vec = xacc[i]; __m256i const shifted = _mm256_srli_epi64 (acc_vec, 47); __m256i const data_vec = _mm256_xor_si256 (acc_vec, shifted); /* xacc[i] ^= xsecret; */ __m256i const key_vec = _mm256_loadu_si256 (xsecret+i); __m256i const data_key = _mm256_xor_si256 (data_vec, key_vec); /* xacc[i] *= XXH_PRIME32_1; */ __m256i const data_key_hi = _mm256_shuffle_epi32 (data_key, _MM_SHUFFLE(0, 3, 0, 1)); __m256i const prod_lo = _mm256_mul_epu32 (data_key, prime32); __m256i const prod_hi = _mm256_mul_epu32 (data_key_hi, prime32); xacc[i] = _mm256_add_epi64(prod_lo, _mm256_slli_epi64(prod_hi, 32)); } } } XXH_FORCE_INLINE XXH_TARGET_AVX2 void XXH3_initCustomSecret_avx2(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 31) == 0); XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE / sizeof(__m256i)) == 6); XXH_STATIC_ASSERT(XXH_SEC_ALIGN <= 64); (void)(&XXH_writeLE64); XXH_PREFETCH(customSecret); { __m256i const seed = _mm256_set_epi64x(-(xxh_i64)seed64, (xxh_i64)seed64, -(xxh_i64)seed64, (xxh_i64)seed64); XXH_ALIGN(64) const __m256i* const src = (const __m256i*) XXH3_kSecret; XXH_ALIGN(64) __m256i* dest = ( __m256i*) customSecret; # if defined(__GNUC__) || defined(__clang__) /* * On GCC & Clang, marking 'dest' as modified will cause the compiler: * - do not extract the secret from sse registers in the internal loop * - use less common registers, and avoid pushing these reg into stack */ XXH_COMPILER_GUARD(dest); # endif /* GCC -O2 need unroll loop manually */ dest[0] = _mm256_add_epi64(_mm256_stream_load_si256(src+0), seed); dest[1] = _mm256_add_epi64(_mm256_stream_load_si256(src+1), seed); dest[2] = _mm256_add_epi64(_mm256_stream_load_si256(src+2), seed); dest[3] = _mm256_add_epi64(_mm256_stream_load_si256(src+3), seed); dest[4] = _mm256_add_epi64(_mm256_stream_load_si256(src+4), seed); dest[5] = _mm256_add_epi64(_mm256_stream_load_si256(src+5), seed); } } #endif /* x86dispatch always generates SSE2 */ #if (XXH_VECTOR == XXH_SSE2) || defined(XXH_X86DISPATCH) #ifndef XXH_TARGET_SSE2 # define XXH_TARGET_SSE2 /* disable attribute target */ #endif XXH_FORCE_INLINE XXH_TARGET_SSE2 void XXH3_accumulate_512_sse2( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { /* SSE2 is just a half-scale version of the AVX2 version. */ XXH_ASSERT((((size_t)acc) & 15) == 0); { XXH_ALIGN(16) __m128i* const xacc = (__m128i *) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm_loadu_si128 requires a const __m128i * pointer for some reason. */ const __m128i* const xinput = (const __m128i *) input; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm_loadu_si128 requires a const __m128i * pointer for some reason. */ const __m128i* const xsecret = (const __m128i *) secret; size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m128i); i++) { /* data_vec = xinput[i]; */ __m128i const data_vec = _mm_loadu_si128 (xinput+i); /* key_vec = xsecret[i]; */ __m128i const key_vec = _mm_loadu_si128 (xsecret+i); /* data_key = data_vec ^ key_vec; */ __m128i const data_key = _mm_xor_si128 (data_vec, key_vec); /* data_key_lo = data_key >> 32; */ __m128i const data_key_lo = _mm_shuffle_epi32 (data_key, _MM_SHUFFLE(0, 3, 0, 1)); /* product = (data_key & 0xffffffff) * (data_key_lo & 0xffffffff); */ __m128i const product = _mm_mul_epu32 (data_key, data_key_lo); /* xacc[i] += swap(data_vec); */ __m128i const data_swap = _mm_shuffle_epi32(data_vec, _MM_SHUFFLE(1,0,3,2)); __m128i const sum = _mm_add_epi64(xacc[i], data_swap); /* xacc[i] += product; */ xacc[i] = _mm_add_epi64(product, sum); } } } XXH_FORCE_INLINE XXH_TARGET_SSE2 void XXH3_scrambleAcc_sse2(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); { XXH_ALIGN(16) __m128i* const xacc = (__m128i*) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm_loadu_si128 requires a const __m128i * pointer for some reason. */ const __m128i* const xsecret = (const __m128i *) secret; const __m128i prime32 = _mm_set1_epi32((int)XXH_PRIME32_1); size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m128i); i++) { /* xacc[i] ^= (xacc[i] >> 47) */ __m128i const acc_vec = xacc[i]; __m128i const shifted = _mm_srli_epi64 (acc_vec, 47); __m128i const data_vec = _mm_xor_si128 (acc_vec, shifted); /* xacc[i] ^= xsecret[i]; */ __m128i const key_vec = _mm_loadu_si128 (xsecret+i); __m128i const data_key = _mm_xor_si128 (data_vec, key_vec); /* xacc[i] *= XXH_PRIME32_1; */ __m128i const data_key_hi = _mm_shuffle_epi32 (data_key, _MM_SHUFFLE(0, 3, 0, 1)); __m128i const prod_lo = _mm_mul_epu32 (data_key, prime32); __m128i const prod_hi = _mm_mul_epu32 (data_key_hi, prime32); xacc[i] = _mm_add_epi64(prod_lo, _mm_slli_epi64(prod_hi, 32)); } } } XXH_FORCE_INLINE XXH_TARGET_SSE2 void XXH3_initCustomSecret_sse2(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 15) == 0); (void)(&XXH_writeLE64); { int const nbRounds = XXH_SECRET_DEFAULT_SIZE / sizeof(__m128i); # if defined(_MSC_VER) && defined(_M_IX86) && _MSC_VER < 1900 // MSVC 32bit mode does not support _mm_set_epi64x before 2015 XXH_ALIGN(16) const xxh_i64 seed64x2[2] = { (xxh_i64)seed64, -(xxh_i64)seed64 }; __m128i const seed = _mm_load_si128((__m128i const*)seed64x2); # else __m128i const seed = _mm_set_epi64x(-(xxh_i64)seed64, (xxh_i64)seed64); # endif int i; XXH_ALIGN(64) const float* const src = (float const*) XXH3_kSecret; XXH_ALIGN(XXH_SEC_ALIGN) __m128i* dest = (__m128i*) customSecret; # if defined(__GNUC__) || defined(__clang__) /* * On GCC & Clang, marking 'dest' as modified will cause the compiler: * - do not extract the secret from sse registers in the internal loop * - use less common registers, and avoid pushing these reg into stack */ XXH_COMPILER_GUARD(dest); # endif for (i=0; i < nbRounds; ++i) { dest[i] = _mm_add_epi64(_mm_castps_si128(_mm_load_ps(src+i*4)), seed); } } } #endif #if (XXH_VECTOR == XXH_NEON) XXH_FORCE_INLINE void XXH3_accumulate_512_neon( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); { XXH_ALIGN(16) uint64x2_t* const xacc = (uint64x2_t *) acc; /* We don't use a uint32x4_t pointer because it causes bus errors on ARMv7. */ uint8_t const* const xinput = (const uint8_t *) input; uint8_t const* const xsecret = (const uint8_t *) secret; size_t i; for (i=0; i < XXH_STRIPE_LEN / sizeof(uint64x2_t); i++) { /* data_vec = xinput[i]; */ uint8x16_t data_vec = vld1q_u8(xinput + (i * 16)); /* key_vec = xsecret[i]; */ uint8x16_t key_vec = vld1q_u8(xsecret + (i * 16)); uint64x2_t data_key; uint32x2_t data_key_lo, data_key_hi; /* xacc[i] += swap(data_vec); */ uint64x2_t const data64 = vreinterpretq_u64_u8(data_vec); uint64x2_t const swapped = vextq_u64(data64, data64, 1); xacc[i] = vaddq_u64 (xacc[i], swapped); /* data_key = data_vec ^ key_vec; */ data_key = vreinterpretq_u64_u8(veorq_u8(data_vec, key_vec)); /* data_key_lo = (uint32x2_t) (data_key & 0xFFFFFFFF); * data_key_hi = (uint32x2_t) (data_key >> 32); * data_key = UNDEFINED; */ XXH_SPLIT_IN_PLACE(data_key, data_key_lo, data_key_hi); /* xacc[i] += (uint64x2_t) data_key_lo * (uint64x2_t) data_key_hi; */ xacc[i] = vmlal_u32 (xacc[i], data_key_lo, data_key_hi); } } } XXH_FORCE_INLINE void XXH3_scrambleAcc_neon(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); { uint64x2_t* xacc = (uint64x2_t*) acc; uint8_t const* xsecret = (uint8_t const*) secret; uint32x2_t prime = vdup_n_u32 (XXH_PRIME32_1); size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(uint64x2_t); i++) { /* xacc[i] ^= (xacc[i] >> 47); */ uint64x2_t acc_vec = xacc[i]; uint64x2_t shifted = vshrq_n_u64 (acc_vec, 47); uint64x2_t data_vec = veorq_u64 (acc_vec, shifted); /* xacc[i] ^= xsecret[i]; */ uint8x16_t key_vec = vld1q_u8(xsecret + (i * 16)); uint64x2_t data_key = veorq_u64(data_vec, vreinterpretq_u64_u8(key_vec)); /* xacc[i] *= XXH_PRIME32_1 */ uint32x2_t data_key_lo, data_key_hi; /* data_key_lo = (uint32x2_t) (xacc[i] & 0xFFFFFFFF); * data_key_hi = (uint32x2_t) (xacc[i] >> 32); * xacc[i] = UNDEFINED; */ XXH_SPLIT_IN_PLACE(data_key, data_key_lo, data_key_hi); { /* * prod_hi = (data_key >> 32) * XXH_PRIME32_1; * * Avoid vmul_u32 + vshll_n_u32 since Clang 6 and 7 will * incorrectly "optimize" this: * tmp = vmul_u32(vmovn_u64(a), vmovn_u64(b)); * shifted = vshll_n_u32(tmp, 32); * to this: * tmp = "vmulq_u64"(a, b); // no such thing! * shifted = vshlq_n_u64(tmp, 32); * * However, unlike SSE, Clang lacks a 64-bit multiply routine * for NEON, and it scalarizes two 64-bit multiplies instead. * * vmull_u32 has the same timing as vmul_u32, and it avoids * this bug completely. * See https://bugs.llvm.org/show_bug.cgi?id=39967 */ uint64x2_t prod_hi = vmull_u32 (data_key_hi, prime); /* xacc[i] = prod_hi << 32; */ xacc[i] = vshlq_n_u64(prod_hi, 32); /* xacc[i] += (prod_hi & 0xFFFFFFFF) * XXH_PRIME32_1; */ xacc[i] = vmlal_u32(xacc[i], data_key_lo, prime); } } } } #endif #if (XXH_VECTOR == XXH_VSX) XXH_FORCE_INLINE void XXH3_accumulate_512_vsx( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { xxh_u64x2* const xacc = (xxh_u64x2*) acc; /* presumed aligned */ xxh_u64x2 const* const xinput = (xxh_u64x2 const*) input; /* no alignment restriction */ xxh_u64x2 const* const xsecret = (xxh_u64x2 const*) secret; /* no alignment restriction */ xxh_u64x2 const v32 = { 32, 32 }; size_t i; for (i = 0; i < XXH_STRIPE_LEN / sizeof(xxh_u64x2); i++) { /* data_vec = xinput[i]; */ xxh_u64x2 const data_vec = XXH_vec_loadu(xinput + i); /* key_vec = xsecret[i]; */ xxh_u64x2 const key_vec = XXH_vec_loadu(xsecret + i); xxh_u64x2 const data_key = data_vec ^ key_vec; /* shuffled = (data_key << 32) | (data_key >> 32); */ xxh_u32x4 const shuffled = (xxh_u32x4)vec_rl(data_key, v32); /* product = ((xxh_u64x2)data_key & 0xFFFFFFFF) * ((xxh_u64x2)shuffled & 0xFFFFFFFF); */ xxh_u64x2 const product = XXH_vec_mulo((xxh_u32x4)data_key, shuffled); xacc[i] += product; /* swap high and low halves */ #ifdef __s390x__ xacc[i] += vec_permi(data_vec, data_vec, 2); #else xacc[i] += vec_xxpermdi(data_vec, data_vec, 2); #endif } } XXH_FORCE_INLINE void XXH3_scrambleAcc_vsx(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); { xxh_u64x2* const xacc = (xxh_u64x2*) acc; const xxh_u64x2* const xsecret = (const xxh_u64x2*) secret; /* constants */ xxh_u64x2 const v32 = { 32, 32 }; xxh_u64x2 const v47 = { 47, 47 }; xxh_u32x4 const prime = { XXH_PRIME32_1, XXH_PRIME32_1, XXH_PRIME32_1, XXH_PRIME32_1 }; size_t i; for (i = 0; i < XXH_STRIPE_LEN / sizeof(xxh_u64x2); i++) { /* xacc[i] ^= (xacc[i] >> 47); */ xxh_u64x2 const acc_vec = xacc[i]; xxh_u64x2 const data_vec = acc_vec ^ (acc_vec >> v47); /* xacc[i] ^= xsecret[i]; */ xxh_u64x2 const key_vec = XXH_vec_loadu(xsecret + i); xxh_u64x2 const data_key = data_vec ^ key_vec; /* xacc[i] *= XXH_PRIME32_1 */ /* prod_lo = ((xxh_u64x2)data_key & 0xFFFFFFFF) * ((xxh_u64x2)prime & 0xFFFFFFFF); */ xxh_u64x2 const prod_even = XXH_vec_mule((xxh_u32x4)data_key, prime); /* prod_hi = ((xxh_u64x2)data_key >> 32) * ((xxh_u64x2)prime >> 32); */ xxh_u64x2 const prod_odd = XXH_vec_mulo((xxh_u32x4)data_key, prime); xacc[i] = prod_odd + (prod_even << v32); } } } #endif /* scalar variants - universal */ XXH_FORCE_INLINE void XXH3_accumulate_512_scalar(void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { XXH_ALIGN(XXH_ACC_ALIGN) xxh_u64* const xacc = (xxh_u64*) acc; /* presumed aligned */ const xxh_u8* const xinput = (const xxh_u8*) input; /* no alignment restriction */ const xxh_u8* const xsecret = (const xxh_u8*) secret; /* no alignment restriction */ size_t i; XXH_ASSERT(((size_t)acc & (XXH_ACC_ALIGN-1)) == 0); for (i=0; i < XXH_ACC_NB; i++) { xxh_u64 const data_val = XXH_readLE64(xinput + 8*i); xxh_u64 const data_key = data_val ^ XXH_readLE64(xsecret + i*8); xacc[i ^ 1] += data_val; /* swap adjacent lanes */ xacc[i] += XXH_mult32to64(data_key & 0xFFFFFFFF, data_key >> 32); } } XXH_FORCE_INLINE void XXH3_scrambleAcc_scalar(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ALIGN(XXH_ACC_ALIGN) xxh_u64* const xacc = (xxh_u64*) acc; /* presumed aligned */ const xxh_u8* const xsecret = (const xxh_u8*) secret; /* no alignment restriction */ size_t i; XXH_ASSERT((((size_t)acc) & (XXH_ACC_ALIGN-1)) == 0); for (i=0; i < XXH_ACC_NB; i++) { xxh_u64 const key64 = XXH_readLE64(xsecret + 8*i); xxh_u64 acc64 = xacc[i]; acc64 = XXH_xorshift64(acc64, 47); acc64 ^= key64; acc64 *= XXH_PRIME32_1; xacc[i] = acc64; } } XXH_FORCE_INLINE void XXH3_initCustomSecret_scalar(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { /* * We need a separate pointer for the hack below, * which requires a non-const pointer. * Any decent compiler will optimize this out otherwise. */ const xxh_u8* kSecretPtr = XXH3_kSecret; XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 15) == 0); #if defined(__clang__) && defined(__aarch64__) /* * UGLY HACK: * Clang generates a bunch of MOV/MOVK pairs for aarch64, and they are * placed sequentially, in order, at the top of the unrolled loop. * * While MOVK is great for generating constants (2 cycles for a 64-bit * constant compared to 4 cycles for LDR), long MOVK chains stall the * integer pipelines: * I L S * MOVK * MOVK * MOVK * MOVK * ADD * SUB STR * STR * By forcing loads from memory (as the asm line causes Clang to assume * that XXH3_kSecretPtr has been changed), the pipelines are used more * efficiently: * I L S * LDR * ADD LDR * SUB STR * STR * XXH3_64bits_withSeed, len == 256, Snapdragon 835 * without hack: 2654.4 MB/s * with hack: 3202.9 MB/s */ XXH_COMPILER_GUARD(kSecretPtr); #endif /* * Note: in debug mode, this overrides the asm optimization * and Clang will emit MOVK chains again. */ XXH_ASSERT(kSecretPtr == XXH3_kSecret); { int const nbRounds = XXH_SECRET_DEFAULT_SIZE / 16; int i; for (i=0; i < nbRounds; i++) { /* * The asm hack causes Clang to assume that kSecretPtr aliases with * customSecret, and on aarch64, this prevented LDP from merging two * loads together for free. Putting the loads together before the stores * properly generates LDP. */ xxh_u64 lo = XXH_readLE64(kSecretPtr + 16*i) + seed64; xxh_u64 hi = XXH_readLE64(kSecretPtr + 16*i + 8) - seed64; XXH_writeLE64((xxh_u8*)customSecret + 16*i, lo); XXH_writeLE64((xxh_u8*)customSecret + 16*i + 8, hi); } } } typedef void (*XXH3_f_accumulate_512)(void* XXH_RESTRICT, const void*, const void*); typedef void (*XXH3_f_scrambleAcc)(void* XXH_RESTRICT, const void*); typedef void (*XXH3_f_initCustomSecret)(void* XXH_RESTRICT, xxh_u64); #if (XXH_VECTOR == XXH_AVX512) #define XXH3_accumulate_512 XXH3_accumulate_512_avx512 #define XXH3_scrambleAcc XXH3_scrambleAcc_avx512 #define XXH3_initCustomSecret XXH3_initCustomSecret_avx512 #elif (XXH_VECTOR == XXH_AVX2) #define XXH3_accumulate_512 XXH3_accumulate_512_avx2 #define XXH3_scrambleAcc XXH3_scrambleAcc_avx2 #define XXH3_initCustomSecret XXH3_initCustomSecret_avx2 #elif (XXH_VECTOR == XXH_SSE2) #define XXH3_accumulate_512 XXH3_accumulate_512_sse2 #define XXH3_scrambleAcc XXH3_scrambleAcc_sse2 #define XXH3_initCustomSecret XXH3_initCustomSecret_sse2 #elif (XXH_VECTOR == XXH_NEON) #define XXH3_accumulate_512 XXH3_accumulate_512_neon #define XXH3_scrambleAcc XXH3_scrambleAcc_neon #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #elif (XXH_VECTOR == XXH_VSX) #define XXH3_accumulate_512 XXH3_accumulate_512_vsx #define XXH3_scrambleAcc XXH3_scrambleAcc_vsx #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #else /* scalar */ #define XXH3_accumulate_512 XXH3_accumulate_512_scalar #define XXH3_scrambleAcc XXH3_scrambleAcc_scalar #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #endif #ifndef XXH_PREFETCH_DIST # ifdef __clang__ # define XXH_PREFETCH_DIST 320 # else # if (XXH_VECTOR == XXH_AVX512) # define XXH_PREFETCH_DIST 512 # else # define XXH_PREFETCH_DIST 384 # endif # endif /* __clang__ */ #endif /* XXH_PREFETCH_DIST */ /* * XXH3_accumulate() * Loops over XXH3_accumulate_512(). * Assumption: nbStripes will not overflow the secret size */ XXH_FORCE_INLINE void XXH3_accumulate( xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT input, const xxh_u8* XXH_RESTRICT secret, size_t nbStripes, XXH3_f_accumulate_512 f_acc512) { size_t n; for (n = 0; n < nbStripes; n++ ) { const xxh_u8* const in = input + n*XXH_STRIPE_LEN; XXH_PREFETCH(in + XXH_PREFETCH_DIST); f_acc512(acc, in, secret + n*XXH_SECRET_CONSUME_RATE); } } XXH_FORCE_INLINE void XXH3_hashLong_internal_loop(xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble) { size_t const nbStripesPerBlock = (secretSize - XXH_STRIPE_LEN) / XXH_SECRET_CONSUME_RATE; size_t const block_len = XXH_STRIPE_LEN * nbStripesPerBlock; size_t const nb_blocks = (len - 1) / block_len; size_t n; XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); for (n = 0; n < nb_blocks; n++) { XXH3_accumulate(acc, input + n*block_len, secret, nbStripesPerBlock, f_acc512); f_scramble(acc, secret + secretSize - XXH_STRIPE_LEN); } /* last partial block */ XXH_ASSERT(len > XXH_STRIPE_LEN); { size_t const nbStripes = ((len - 1) - (block_len * nb_blocks)) / XXH_STRIPE_LEN; XXH_ASSERT(nbStripes <= (secretSize / XXH_SECRET_CONSUME_RATE)); XXH3_accumulate(acc, input + nb_blocks*block_len, secret, nbStripes, f_acc512); /* last stripe */ { const xxh_u8* const p = input + len - XXH_STRIPE_LEN; #define XXH_SECRET_LASTACC_START 7 /* not aligned on 8, last secret is different from acc & scrambler */ f_acc512(acc, p, secret + secretSize - XXH_STRIPE_LEN - XXH_SECRET_LASTACC_START); } } } XXH_FORCE_INLINE xxh_u64 XXH3_mix2Accs(const xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT secret) { return XXH3_mul128_fold64( acc[0] ^ XXH_readLE64(secret), acc[1] ^ XXH_readLE64(secret+8) ); } static XXH64_hash_t XXH3_mergeAccs(const xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT secret, xxh_u64 start) { xxh_u64 result64 = start; size_t i = 0; for (i = 0; i < 4; i++) { result64 += XXH3_mix2Accs(acc+2*i, secret + 16*i); #if defined(__clang__) /* Clang */ \ && (defined(__arm__) || defined(__thumb__)) /* ARMv7 */ \ && (defined(__ARM_NEON) || defined(__ARM_NEON__)) /* NEON */ \ && !defined(XXH_ENABLE_AUTOVECTORIZE) /* Define to disable */ /* * UGLY HACK: * Prevent autovectorization on Clang ARMv7-a. Exact same problem as * the one in XXH3_len_129to240_64b. Speeds up shorter keys > 240b. * XXH3_64bits, len == 256, Snapdragon 835: * without hack: 2063.7 MB/s * with hack: 2560.7 MB/s */ XXH_COMPILER_GUARD(result64); #endif } return XXH3_avalanche(result64); } #define XXH3_INIT_ACC { XXH_PRIME32_3, XXH_PRIME64_1, XXH_PRIME64_2, XXH_PRIME64_3, \ XXH_PRIME64_4, XXH_PRIME32_2, XXH_PRIME64_5, XXH_PRIME32_1 } XXH_FORCE_INLINE XXH64_hash_t XXH3_hashLong_64b_internal(const void* XXH_RESTRICT input, size_t len, const void* XXH_RESTRICT secret, size_t secretSize, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble) { XXH_ALIGN(XXH_ACC_ALIGN) xxh_u64 acc[XXH_ACC_NB] = XXH3_INIT_ACC; XXH3_hashLong_internal_loop(acc, (const xxh_u8*)input, len, (const xxh_u8*)secret, secretSize, f_acc512, f_scramble); /* converge into final hash */ XXH_STATIC_ASSERT(sizeof(acc) == 64); /* do not align on 8, so that the secret is different from the accumulator */ #define XXH_SECRET_MERGEACCS_START 11 XXH_ASSERT(secretSize >= sizeof(acc) + XXH_SECRET_MERGEACCS_START); return XXH3_mergeAccs(acc, (const xxh_u8*)secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)len * XXH_PRIME64_1); } /* * It's important for performance that XXH3_hashLong is not inlined. */ XXH_NO_INLINE XXH64_hash_t XXH3_hashLong_64b_withSecret(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const xxh_u8* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; return XXH3_hashLong_64b_internal(input, len, secret, secretLen, XXH3_accumulate_512, XXH3_scrambleAcc); } /* * It's important for performance that XXH3_hashLong is not inlined. * Since the function is not inlined, the compiler may not be able to understand that, * in some scenarios, its `secret` argument is actually a compile time constant. * This variant enforces that the compiler can detect that, * and uses this opportunity to streamline the generated code for better performance. */ XXH_NO_INLINE XXH64_hash_t XXH3_hashLong_64b_default(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const xxh_u8* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; (void)secret; (void)secretLen; return XXH3_hashLong_64b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_accumulate_512, XXH3_scrambleAcc); } /* * XXH3_hashLong_64b_withSeed(): * Generate a custom key based on alteration of default XXH3_kSecret with the seed, * and then use this key for long mode hashing. * * This operation is decently fast but nonetheless costs a little bit of time. * Try to avoid it whenever possible (typically when seed==0). * * It's important for performance that XXH3_hashLong is not inlined. Not sure * why (uop cache maybe?), but the difference is large and easily measurable. */ XXH_FORCE_INLINE XXH64_hash_t XXH3_hashLong_64b_withSeed_internal(const void* input, size_t len, XXH64_hash_t seed, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble, XXH3_f_initCustomSecret f_initSec) { if (seed == 0) return XXH3_hashLong_64b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), f_acc512, f_scramble); { XXH_ALIGN(XXH_SEC_ALIGN) xxh_u8 secret[XXH_SECRET_DEFAULT_SIZE]; f_initSec(secret, seed); return XXH3_hashLong_64b_internal(input, len, secret, sizeof(secret), f_acc512, f_scramble); } } /* * It's important for performance that XXH3_hashLong is not inlined. */ XXH_NO_INLINE XXH64_hash_t XXH3_hashLong_64b_withSeed(const void* input, size_t len, XXH64_hash_t seed, const xxh_u8* secret, size_t secretLen) { (void)secret; (void)secretLen; return XXH3_hashLong_64b_withSeed_internal(input, len, seed, XXH3_accumulate_512, XXH3_scrambleAcc, XXH3_initCustomSecret); } typedef XXH64_hash_t (*XXH3_hashLong64_f)(const void* XXH_RESTRICT, size_t, XXH64_hash_t, const xxh_u8* XXH_RESTRICT, size_t); XXH_FORCE_INLINE XXH64_hash_t XXH3_64bits_internal(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen, XXH3_hashLong64_f f_hashLong) { XXH_ASSERT(secretLen >= XXH3_SECRET_SIZE_MIN); /* * If an action is to be taken if `secretLen` condition is not respected, * it should be done here. * For now, it's a contract pre-condition. * Adding a check and a branch here would cost performance at every hash. * Also, note that function signature doesn't offer room to return an error. */ if (len <= 16) return XXH3_len_0to16_64b((const xxh_u8*)input, len, (const xxh_u8*)secret, seed64); if (len <= 128) return XXH3_len_17to128_64b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); if (len <= XXH3_MIDSIZE_MAX) return XXH3_len_129to240_64b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); return f_hashLong(input, len, seed64, (const xxh_u8*)secret, secretLen); } /* === Public entry point === */ /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits(const void* input, size_t len) { return XXH3_64bits_internal(input, len, 0, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_64b_default); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSecret(const void* input, size_t len, const void* secret, size_t secretSize) { return XXH3_64bits_internal(input, len, 0, secret, secretSize, XXH3_hashLong_64b_withSecret); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSeed(const void* input, size_t len, XXH64_hash_t seed) { return XXH3_64bits_internal(input, len, seed, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_64b_withSeed); } /* === XXH3 streaming === */ /* * Malloc's a pointer that is always aligned to align. * * This must be freed with `XXH_alignedFree()`. * * malloc typically guarantees 16 byte alignment on 64-bit systems and 8 byte * alignment on 32-bit. This isn't enough for the 32 byte aligned loads in AVX2 * or on 32-bit, the 16 byte aligned loads in SSE2 and NEON. * * This underalignment previously caused a rather obvious crash which went * completely unnoticed due to XXH3_createState() not actually being tested. * Credit to RedSpah for noticing this bug. * * The alignment is done manually: Functions like posix_memalign or _mm_malloc * are avoided: To maintain portability, we would have to write a fallback * like this anyways, and besides, testing for the existence of library * functions without relying on external build tools is impossible. * * The method is simple: Overallocate, manually align, and store the offset * to the original behind the returned pointer. * * Align must be a power of 2 and 8 <= align <= 128. */ static void* XXH_alignedMalloc(size_t s, size_t align) { XXH_ASSERT(align <= 128 && align >= 8); /* range check */ XXH_ASSERT((align & (align-1)) == 0); /* power of 2 */ XXH_ASSERT(s != 0 && s < (s + align)); /* empty/overflow */ { /* Overallocate to make room for manual realignment and an offset byte */ xxh_u8* base = (xxh_u8*)XXH_malloc(s + align); if (base != NULL) { /* * Get the offset needed to align this pointer. * * Even if the returned pointer is aligned, there will always be * at least one byte to store the offset to the original pointer. */ size_t offset = align - ((size_t)base & (align - 1)); /* base % align */ /* Add the offset for the now-aligned pointer */ xxh_u8* ptr = base + offset; XXH_ASSERT((size_t)ptr % align == 0); /* Store the offset immediately before the returned pointer. */ ptr[-1] = (xxh_u8)offset; return ptr; } return NULL; } } /* * Frees an aligned pointer allocated by XXH_alignedMalloc(). Don't pass * normal malloc'd pointers, XXH_alignedMalloc has a specific data layout. */ static void XXH_alignedFree(void* p) { if (p != NULL) { xxh_u8* ptr = (xxh_u8*)p; /* Get the offset byte we added in XXH_malloc. */ xxh_u8 offset = ptr[-1]; /* Free the original malloc'd pointer */ xxh_u8* base = ptr - offset; XXH_free(base); } } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH3_state_t* XXH3_createState(void) { XXH3_state_t* const state = (XXH3_state_t*)XXH_alignedMalloc(sizeof(XXH3_state_t), 64); if (state==NULL) return NULL; XXH3_INITSTATE(state); return state; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_freeState(XXH3_state_t* statePtr) { XXH_alignedFree(statePtr); return XXH_OK; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API void XXH3_copyState(XXH3_state_t* dst_state, const XXH3_state_t* src_state) { memcpy(dst_state, src_state, sizeof(*dst_state)); } static void XXH3_reset_internal(XXH3_state_t* statePtr, XXH64_hash_t seed, const void* secret, size_t secretSize) { size_t const initStart = offsetof(XXH3_state_t, bufferedSize); size_t const initLength = offsetof(XXH3_state_t, nbStripesPerBlock) - initStart; XXH_ASSERT(offsetof(XXH3_state_t, nbStripesPerBlock) > initStart); XXH_ASSERT(statePtr != NULL); /* set members from bufferedSize to nbStripesPerBlock (excluded) to 0 */ memset((char*)statePtr + initStart, 0, initLength); statePtr->acc[0] = XXH_PRIME32_3; statePtr->acc[1] = XXH_PRIME64_1; statePtr->acc[2] = XXH_PRIME64_2; statePtr->acc[3] = XXH_PRIME64_3; statePtr->acc[4] = XXH_PRIME64_4; statePtr->acc[5] = XXH_PRIME32_2; statePtr->acc[6] = XXH_PRIME64_5; statePtr->acc[7] = XXH_PRIME32_1; statePtr->seed = seed; statePtr->extSecret = (const unsigned char*)secret; XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); statePtr->secretLimit = secretSize - XXH_STRIPE_LEN; statePtr->nbStripesPerBlock = statePtr->secretLimit / XXH_SECRET_CONSUME_RATE; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset(XXH3_state_t* statePtr) { if (statePtr == NULL) return XXH_ERROR; XXH3_reset_internal(statePtr, 0, XXH3_kSecret, XXH_SECRET_DEFAULT_SIZE); return XXH_OK; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecret(XXH3_state_t* statePtr, const void* secret, size_t secretSize) { if (statePtr == NULL) return XXH_ERROR; XXH3_reset_internal(statePtr, 0, secret, secretSize); if (secret == NULL) return XXH_ERROR; if (secretSize < XXH3_SECRET_SIZE_MIN) return XXH_ERROR; return XXH_OK; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSeed(XXH3_state_t* statePtr, XXH64_hash_t seed) { if (statePtr == NULL) return XXH_ERROR; if (seed==0) return XXH3_64bits_reset(statePtr); if (seed != statePtr->seed) XXH3_initCustomSecret(statePtr->customSecret, seed); XXH3_reset_internal(statePtr, seed, NULL, XXH_SECRET_DEFAULT_SIZE); return XXH_OK; } /* Note : when XXH3_consumeStripes() is invoked, * there must be a guarantee that at least one more byte must be consumed from input * so that the function can blindly consume all stripes using the "normal" secret segment */ XXH_FORCE_INLINE void XXH3_consumeStripes(xxh_u64* XXH_RESTRICT acc, size_t* XXH_RESTRICT nbStripesSoFarPtr, size_t nbStripesPerBlock, const xxh_u8* XXH_RESTRICT input, size_t nbStripes, const xxh_u8* XXH_RESTRICT secret, size_t secretLimit, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble) { XXH_ASSERT(nbStripes <= nbStripesPerBlock); /* can handle max 1 scramble per invocation */ XXH_ASSERT(*nbStripesSoFarPtr < nbStripesPerBlock); if (nbStripesPerBlock - *nbStripesSoFarPtr <= nbStripes) { /* need a scrambling operation */ size_t const nbStripesToEndofBlock = nbStripesPerBlock - *nbStripesSoFarPtr; size_t const nbStripesAfterBlock = nbStripes - nbStripesToEndofBlock; XXH3_accumulate(acc, input, secret + nbStripesSoFarPtr[0] * XXH_SECRET_CONSUME_RATE, nbStripesToEndofBlock, f_acc512); f_scramble(acc, secret + secretLimit); XXH3_accumulate(acc, input + nbStripesToEndofBlock * XXH_STRIPE_LEN, secret, nbStripesAfterBlock, f_acc512); *nbStripesSoFarPtr = nbStripesAfterBlock; } else { XXH3_accumulate(acc, input, secret + nbStripesSoFarPtr[0] * XXH_SECRET_CONSUME_RATE, nbStripes, f_acc512); *nbStripesSoFarPtr += nbStripes; } } /* * Both XXH3_64bits_update and XXH3_128bits_update use this routine. */ XXH_FORCE_INLINE XXH_errorcode XXH3_update(XXH3_state_t* state, const xxh_u8* input, size_t len, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble) { if (input==NULL) #if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1) return XXH_OK; #else return XXH_ERROR; #endif { const xxh_u8* const bEnd = input + len; const unsigned char* const secret = (state->extSecret == NULL) ? state->customSecret : state->extSecret; state->totalLen += len; XXH_ASSERT(state->bufferedSize <= XXH3_INTERNALBUFFER_SIZE); if (state->bufferedSize + len <= XXH3_INTERNALBUFFER_SIZE) { /* fill in tmp buffer */ XXH_memcpy(state->buffer + state->bufferedSize, input, len); state->bufferedSize += (XXH32_hash_t)len; return XXH_OK; } /* total input is now > XXH3_INTERNALBUFFER_SIZE */ #define XXH3_INTERNALBUFFER_STRIPES (XXH3_INTERNALBUFFER_SIZE / XXH_STRIPE_LEN) XXH_STATIC_ASSERT(XXH3_INTERNALBUFFER_SIZE % XXH_STRIPE_LEN == 0); /* clean multiple */ /* * Internal buffer is partially filled (always, except at beginning) * Complete it, then consume it. */ if (state->bufferedSize) { size_t const loadSize = XXH3_INTERNALBUFFER_SIZE - state->bufferedSize; XXH_memcpy(state->buffer + state->bufferedSize, input, loadSize); input += loadSize; XXH3_consumeStripes(state->acc, &state->nbStripesSoFar, state->nbStripesPerBlock, state->buffer, XXH3_INTERNALBUFFER_STRIPES, secret, state->secretLimit, f_acc512, f_scramble); state->bufferedSize = 0; } XXH_ASSERT(input < bEnd); /* Consume input by a multiple of internal buffer size */ if (input+XXH3_INTERNALBUFFER_SIZE < bEnd) { const xxh_u8* const limit = bEnd - XXH3_INTERNALBUFFER_SIZE; do { XXH3_consumeStripes(state->acc, &state->nbStripesSoFar, state->nbStripesPerBlock, input, XXH3_INTERNALBUFFER_STRIPES, secret, state->secretLimit, f_acc512, f_scramble); input += XXH3_INTERNALBUFFER_SIZE; } while (inputbuffer + sizeof(state->buffer) - XXH_STRIPE_LEN, input - XXH_STRIPE_LEN, XXH_STRIPE_LEN); } XXH_ASSERT(input < bEnd); /* Some remaining input (always) : buffer it */ XXH_memcpy(state->buffer, input, (size_t)(bEnd-input)); state->bufferedSize = (XXH32_hash_t)(bEnd-input); } return XXH_OK; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_update(XXH3_state_t* state, const void* input, size_t len) { return XXH3_update(state, (const xxh_u8*)input, len, XXH3_accumulate_512, XXH3_scrambleAcc); } XXH_FORCE_INLINE void XXH3_digest_long (XXH64_hash_t* acc, const XXH3_state_t* state, const unsigned char* secret) { /* * Digest on a local copy. This way, the state remains unaltered, and it can * continue ingesting more input afterwards. */ memcpy(acc, state->acc, sizeof(state->acc)); if (state->bufferedSize >= XXH_STRIPE_LEN) { size_t const nbStripes = (state->bufferedSize - 1) / XXH_STRIPE_LEN; size_t nbStripesSoFar = state->nbStripesSoFar; XXH3_consumeStripes(acc, &nbStripesSoFar, state->nbStripesPerBlock, state->buffer, nbStripes, secret, state->secretLimit, XXH3_accumulate_512, XXH3_scrambleAcc); /* last stripe */ XXH3_accumulate_512(acc, state->buffer + state->bufferedSize - XXH_STRIPE_LEN, secret + state->secretLimit - XXH_SECRET_LASTACC_START); } else { /* bufferedSize < XXH_STRIPE_LEN */ xxh_u8 lastStripe[XXH_STRIPE_LEN]; size_t const catchupSize = XXH_STRIPE_LEN - state->bufferedSize; XXH_ASSERT(state->bufferedSize > 0); /* there is always some input buffered */ memcpy(lastStripe, state->buffer + sizeof(state->buffer) - catchupSize, catchupSize); memcpy(lastStripe + catchupSize, state->buffer, state->bufferedSize); XXH3_accumulate_512(acc, lastStripe, secret + state->secretLimit - XXH_SECRET_LASTACC_START); } } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_digest (const XXH3_state_t* state) { const unsigned char* const secret = (state->extSecret == NULL) ? state->customSecret : state->extSecret; if (state->totalLen > XXH3_MIDSIZE_MAX) { XXH_ALIGN(XXH_ACC_ALIGN) XXH64_hash_t acc[XXH_ACC_NB]; XXH3_digest_long(acc, state, secret); return XXH3_mergeAccs(acc, secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)state->totalLen * XXH_PRIME64_1); } /* totalLen <= XXH3_MIDSIZE_MAX: digesting a short input */ if (state->seed) return XXH3_64bits_withSeed(state->buffer, (size_t)state->totalLen, state->seed); return XXH3_64bits_withSecret(state->buffer, (size_t)(state->totalLen), secret, state->secretLimit + XXH_STRIPE_LEN); } #define XXH_MIN(x, y) (((x) > (y)) ? (y) : (x)) /*! @ingroup xxh3_family */ XXH_PUBLIC_API void XXH3_generateSecret(void* secretBuffer, const void* customSeed, size_t customSeedSize) { XXH_ASSERT(secretBuffer != NULL); if (customSeedSize == 0) { memcpy(secretBuffer, XXH3_kSecret, XXH_SECRET_DEFAULT_SIZE); return; } XXH_ASSERT(customSeed != NULL); { size_t const segmentSize = sizeof(XXH128_hash_t); size_t const nbSegments = XXH_SECRET_DEFAULT_SIZE / segmentSize; XXH128_canonical_t scrambler; XXH64_hash_t seeds[12]; size_t segnb; XXH_ASSERT(nbSegments == 12); XXH_ASSERT(segmentSize * nbSegments == XXH_SECRET_DEFAULT_SIZE); /* exact multiple */ XXH128_canonicalFromHash(&scrambler, XXH128(customSeed, customSeedSize, 0)); /* * Copy customSeed to seeds[], truncating or repeating as necessary. */ { size_t toFill = XXH_MIN(customSeedSize, sizeof(seeds)); size_t filled = toFill; memcpy(seeds, customSeed, toFill); while (filled < sizeof(seeds)) { toFill = XXH_MIN(filled, sizeof(seeds) - filled); memcpy((char*)seeds + filled, seeds, toFill); filled += toFill; } } /* generate secret */ memcpy(secretBuffer, &scrambler, sizeof(scrambler)); for (segnb=1; segnb < nbSegments; segnb++) { size_t const segmentStart = segnb * segmentSize; XXH128_canonical_t segment; XXH128_canonicalFromHash(&segment, XXH128(&scrambler, sizeof(scrambler), XXH_readLE64(seeds + segnb) + segnb) ); memcpy((char*)secretBuffer + segmentStart, &segment, sizeof(segment)); } } } /* ========================================== * XXH3 128 bits (a.k.a XXH128) * ========================================== * XXH3's 128-bit variant has better mixing and strength than the 64-bit variant, * even without counting the significantly larger output size. * * For example, extra steps are taken to avoid the seed-dependent collisions * in 17-240 byte inputs (See XXH3_mix16B and XXH128_mix32B). * * This strength naturally comes at the cost of some speed, especially on short * lengths. Note that longer hashes are about as fast as the 64-bit version * due to it using only a slight modification of the 64-bit loop. * * XXH128 is also more oriented towards 64-bit machines. It is still extremely * fast for a _128-bit_ hash on 32-bit (it usually clears XXH64). */ XXH_FORCE_INLINE XXH128_hash_t XXH3_len_1to3_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { /* A doubled version of 1to3_64b with different constants. */ XXH_ASSERT(input != NULL); XXH_ASSERT(1 <= len && len <= 3); XXH_ASSERT(secret != NULL); /* * len = 1: combinedl = { input[0], 0x01, input[0], input[0] } * len = 2: combinedl = { input[1], 0x02, input[0], input[1] } * len = 3: combinedl = { input[2], 0x03, input[0], input[1] } */ { xxh_u8 const c1 = input[0]; xxh_u8 const c2 = input[len >> 1]; xxh_u8 const c3 = input[len - 1]; xxh_u32 const combinedl = ((xxh_u32)c1 <<16) | ((xxh_u32)c2 << 24) | ((xxh_u32)c3 << 0) | ((xxh_u32)len << 8); xxh_u32 const combinedh = XXH_rotl32(XXH_swap32(combinedl), 13); xxh_u64 const bitflipl = (XXH_readLE32(secret) ^ XXH_readLE32(secret+4)) + seed; xxh_u64 const bitfliph = (XXH_readLE32(secret+8) ^ XXH_readLE32(secret+12)) - seed; xxh_u64 const keyed_lo = (xxh_u64)combinedl ^ bitflipl; xxh_u64 const keyed_hi = (xxh_u64)combinedh ^ bitfliph; XXH128_hash_t h128; h128.low64 = XXH64_avalanche(keyed_lo); h128.high64 = XXH64_avalanche(keyed_hi); return h128; } } XXH_FORCE_INLINE XXH128_hash_t XXH3_len_4to8_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(4 <= len && len <= 8); seed ^= (xxh_u64)XXH_swap32((xxh_u32)seed) << 32; { xxh_u32 const input_lo = XXH_readLE32(input); xxh_u32 const input_hi = XXH_readLE32(input + len - 4); xxh_u64 const input_64 = input_lo + ((xxh_u64)input_hi << 32); xxh_u64 const bitflip = (XXH_readLE64(secret+16) ^ XXH_readLE64(secret+24)) + seed; xxh_u64 const keyed = input_64 ^ bitflip; /* Shift len to the left to ensure it is even, this avoids even multiplies. */ XXH128_hash_t m128 = XXH_mult64to128(keyed, XXH_PRIME64_1 + (len << 2)); m128.high64 += (m128.low64 << 1); m128.low64 ^= (m128.high64 >> 3); m128.low64 = XXH_xorshift64(m128.low64, 35); m128.low64 *= 0x9FB21C651E98DF25ULL; m128.low64 = XXH_xorshift64(m128.low64, 28); m128.high64 = XXH3_avalanche(m128.high64); return m128; } } XXH_FORCE_INLINE XXH128_hash_t XXH3_len_9to16_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(9 <= len && len <= 16); { xxh_u64 const bitflipl = (XXH_readLE64(secret+32) ^ XXH_readLE64(secret+40)) - seed; xxh_u64 const bitfliph = (XXH_readLE64(secret+48) ^ XXH_readLE64(secret+56)) + seed; xxh_u64 const input_lo = XXH_readLE64(input); xxh_u64 input_hi = XXH_readLE64(input + len - 8); XXH128_hash_t m128 = XXH_mult64to128(input_lo ^ input_hi ^ bitflipl, XXH_PRIME64_1); /* * Put len in the middle of m128 to ensure that the length gets mixed to * both the low and high bits in the 128x64 multiply below. */ m128.low64 += (xxh_u64)(len - 1) << 54; input_hi ^= bitfliph; /* * Add the high 32 bits of input_hi to the high 32 bits of m128, then * add the long product of the low 32 bits of input_hi and XXH_PRIME32_2 to * the high 64 bits of m128. * * The best approach to this operation is different on 32-bit and 64-bit. */ if (sizeof(void *) < sizeof(xxh_u64)) { /* 32-bit */ /* * 32-bit optimized version, which is more readable. * * On 32-bit, it removes an ADC and delays a dependency between the two * halves of m128.high64, but it generates an extra mask on 64-bit. */ m128.high64 += (input_hi & 0xFFFFFFFF00000000ULL) + XXH_mult32to64((xxh_u32)input_hi, XXH_PRIME32_2); } else { /* * 64-bit optimized (albeit more confusing) version. * * Uses some properties of addition and multiplication to remove the mask: * * Let: * a = input_hi.lo = (input_hi & 0x00000000FFFFFFFF) * b = input_hi.hi = (input_hi & 0xFFFFFFFF00000000) * c = XXH_PRIME32_2 * * a + (b * c) * Inverse Property: x + y - x == y * a + (b * (1 + c - 1)) * Distributive Property: x * (y + z) == (x * y) + (x * z) * a + (b * 1) + (b * (c - 1)) * Identity Property: x * 1 == x * a + b + (b * (c - 1)) * * Substitute a, b, and c: * input_hi.hi + input_hi.lo + ((xxh_u64)input_hi.lo * (XXH_PRIME32_2 - 1)) * * Since input_hi.hi + input_hi.lo == input_hi, we get this: * input_hi + ((xxh_u64)input_hi.lo * (XXH_PRIME32_2 - 1)) */ m128.high64 += input_hi + XXH_mult32to64((xxh_u32)input_hi, XXH_PRIME32_2 - 1); } /* m128 ^= XXH_swap64(m128 >> 64); */ m128.low64 ^= XXH_swap64(m128.high64); { /* 128x64 multiply: h128 = m128 * XXH_PRIME64_2; */ XXH128_hash_t h128 = XXH_mult64to128(m128.low64, XXH_PRIME64_2); h128.high64 += m128.high64 * XXH_PRIME64_2; h128.low64 = XXH3_avalanche(h128.low64); h128.high64 = XXH3_avalanche(h128.high64); return h128; } } } /* * Assumption: `secret` size is >= XXH3_SECRET_SIZE_MIN */ XXH_FORCE_INLINE XXH128_hash_t XXH3_len_0to16_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(len <= 16); { if (len > 8) return XXH3_len_9to16_128b(input, len, secret, seed); if (len >= 4) return XXH3_len_4to8_128b(input, len, secret, seed); if (len) return XXH3_len_1to3_128b(input, len, secret, seed); { XXH128_hash_t h128; xxh_u64 const bitflipl = XXH_readLE64(secret+64) ^ XXH_readLE64(secret+72); xxh_u64 const bitfliph = XXH_readLE64(secret+80) ^ XXH_readLE64(secret+88); h128.low64 = XXH64_avalanche(seed ^ bitflipl); h128.high64 = XXH64_avalanche( seed ^ bitfliph); return h128; } } } /* * A bit slower than XXH3_mix16B, but handles multiply by zero better. */ XXH_FORCE_INLINE XXH128_hash_t XXH128_mix32B(XXH128_hash_t acc, const xxh_u8* input_1, const xxh_u8* input_2, const xxh_u8* secret, XXH64_hash_t seed) { acc.low64 += XXH3_mix16B (input_1, secret+0, seed); acc.low64 ^= XXH_readLE64(input_2) + XXH_readLE64(input_2 + 8); acc.high64 += XXH3_mix16B (input_2, secret+16, seed); acc.high64 ^= XXH_readLE64(input_1) + XXH_readLE64(input_1 + 8); return acc; } XXH_FORCE_INLINE XXH128_hash_t XXH3_len_17to128_128b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(16 < len && len <= 128); { XXH128_hash_t acc; acc.low64 = len * XXH_PRIME64_1; acc.high64 = 0; if (len > 32) { if (len > 64) { if (len > 96) { acc = XXH128_mix32B(acc, input+48, input+len-64, secret+96, seed); } acc = XXH128_mix32B(acc, input+32, input+len-48, secret+64, seed); } acc = XXH128_mix32B(acc, input+16, input+len-32, secret+32, seed); } acc = XXH128_mix32B(acc, input, input+len-16, secret, seed); { XXH128_hash_t h128; h128.low64 = acc.low64 + acc.high64; h128.high64 = (acc.low64 * XXH_PRIME64_1) + (acc.high64 * XXH_PRIME64_4) + ((len - seed) * XXH_PRIME64_2); h128.low64 = XXH3_avalanche(h128.low64); h128.high64 = (XXH64_hash_t)0 - XXH3_avalanche(h128.high64); return h128; } } } XXH_NO_INLINE XXH128_hash_t XXH3_len_129to240_128b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(128 < len && len <= XXH3_MIDSIZE_MAX); { XXH128_hash_t acc; int const nbRounds = (int)len / 32; int i; acc.low64 = len * XXH_PRIME64_1; acc.high64 = 0; for (i=0; i<4; i++) { acc = XXH128_mix32B(acc, input + (32 * i), input + (32 * i) + 16, secret + (32 * i), seed); } acc.low64 = XXH3_avalanche(acc.low64); acc.high64 = XXH3_avalanche(acc.high64); XXH_ASSERT(nbRounds >= 4); for (i=4 ; i < nbRounds; i++) { acc = XXH128_mix32B(acc, input + (32 * i), input + (32 * i) + 16, secret + XXH3_MIDSIZE_STARTOFFSET + (32 * (i - 4)), seed); } /* last bytes */ acc = XXH128_mix32B(acc, input + len - 16, input + len - 32, secret + XXH3_SECRET_SIZE_MIN - XXH3_MIDSIZE_LASTOFFSET - 16, 0ULL - seed); { XXH128_hash_t h128; h128.low64 = acc.low64 + acc.high64; h128.high64 = (acc.low64 * XXH_PRIME64_1) + (acc.high64 * XXH_PRIME64_4) + ((len - seed) * XXH_PRIME64_2); h128.low64 = XXH3_avalanche(h128.low64); h128.high64 = (XXH64_hash_t)0 - XXH3_avalanche(h128.high64); return h128; } } } XXH_FORCE_INLINE XXH128_hash_t XXH3_hashLong_128b_internal(const void* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble) { XXH_ALIGN(XXH_ACC_ALIGN) xxh_u64 acc[XXH_ACC_NB] = XXH3_INIT_ACC; XXH3_hashLong_internal_loop(acc, (const xxh_u8*)input, len, secret, secretSize, f_acc512, f_scramble); /* converge into final hash */ XXH_STATIC_ASSERT(sizeof(acc) == 64); XXH_ASSERT(secretSize >= sizeof(acc) + XXH_SECRET_MERGEACCS_START); { XXH128_hash_t h128; h128.low64 = XXH3_mergeAccs(acc, secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)len * XXH_PRIME64_1); h128.high64 = XXH3_mergeAccs(acc, secret + secretSize - sizeof(acc) - XXH_SECRET_MERGEACCS_START, ~((xxh_u64)len * XXH_PRIME64_2)); return h128; } } /* * It's important for performance that XXH3_hashLong is not inlined. */ XXH_NO_INLINE XXH128_hash_t XXH3_hashLong_128b_default(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; (void)secret; (void)secretLen; return XXH3_hashLong_128b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_accumulate_512, XXH3_scrambleAcc); } /* * It's important for performance that XXH3_hashLong is not inlined. */ XXH_NO_INLINE XXH128_hash_t XXH3_hashLong_128b_withSecret(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; return XXH3_hashLong_128b_internal(input, len, (const xxh_u8*)secret, secretLen, XXH3_accumulate_512, XXH3_scrambleAcc); } XXH_FORCE_INLINE XXH128_hash_t XXH3_hashLong_128b_withSeed_internal(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, XXH3_f_accumulate_512 f_acc512, XXH3_f_scrambleAcc f_scramble, XXH3_f_initCustomSecret f_initSec) { if (seed64 == 0) return XXH3_hashLong_128b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), f_acc512, f_scramble); { XXH_ALIGN(XXH_SEC_ALIGN) xxh_u8 secret[XXH_SECRET_DEFAULT_SIZE]; f_initSec(secret, seed64); return XXH3_hashLong_128b_internal(input, len, (const xxh_u8*)secret, sizeof(secret), f_acc512, f_scramble); } } /* * It's important for performance that XXH3_hashLong is not inlined. */ XXH_NO_INLINE XXH128_hash_t XXH3_hashLong_128b_withSeed(const void* input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen) { (void)secret; (void)secretLen; return XXH3_hashLong_128b_withSeed_internal(input, len, seed64, XXH3_accumulate_512, XXH3_scrambleAcc, XXH3_initCustomSecret); } typedef XXH128_hash_t (*XXH3_hashLong128_f)(const void* XXH_RESTRICT, size_t, XXH64_hash_t, const void* XXH_RESTRICT, size_t); XXH_FORCE_INLINE XXH128_hash_t XXH3_128bits_internal(const void* input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen, XXH3_hashLong128_f f_hl128) { XXH_ASSERT(secretLen >= XXH3_SECRET_SIZE_MIN); /* * If an action is to be taken if `secret` conditions are not respected, * it should be done here. * For now, it's a contract pre-condition. * Adding a check and a branch here would cost performance at every hash. */ if (len <= 16) return XXH3_len_0to16_128b((const xxh_u8*)input, len, (const xxh_u8*)secret, seed64); if (len <= 128) return XXH3_len_17to128_128b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); if (len <= XXH3_MIDSIZE_MAX) return XXH3_len_129to240_128b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); return f_hl128(input, len, seed64, secret, secretLen); } /* === Public XXH128 API === */ /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits(const void* input, size_t len) { return XXH3_128bits_internal(input, len, 0, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_128b_default); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSecret(const void* input, size_t len, const void* secret, size_t secretSize) { return XXH3_128bits_internal(input, len, 0, (const xxh_u8*)secret, secretSize, XXH3_hashLong_128b_withSecret); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSeed(const void* input, size_t len, XXH64_hash_t seed) { return XXH3_128bits_internal(input, len, seed, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_128b_withSeed); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH128_hash_t XXH128(const void* input, size_t len, XXH64_hash_t seed) { return XXH3_128bits_withSeed(input, len, seed); } /* === XXH3 128-bit streaming === */ /* * All the functions are actually the same as for 64-bit streaming variant. * The only difference is the finalization routine. */ /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset(XXH3_state_t* statePtr) { if (statePtr == NULL) return XXH_ERROR; XXH3_reset_internal(statePtr, 0, XXH3_kSecret, XXH_SECRET_DEFAULT_SIZE); return XXH_OK; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecret(XXH3_state_t* statePtr, const void* secret, size_t secretSize) { if (statePtr == NULL) return XXH_ERROR; XXH3_reset_internal(statePtr, 0, secret, secretSize); if (secret == NULL) return XXH_ERROR; if (secretSize < XXH3_SECRET_SIZE_MIN) return XXH_ERROR; return XXH_OK; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSeed(XXH3_state_t* statePtr, XXH64_hash_t seed) { if (statePtr == NULL) return XXH_ERROR; if (seed==0) return XXH3_128bits_reset(statePtr); if (seed != statePtr->seed) XXH3_initCustomSecret(statePtr->customSecret, seed); XXH3_reset_internal(statePtr, seed, NULL, XXH_SECRET_DEFAULT_SIZE); return XXH_OK; } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_update(XXH3_state_t* state, const void* input, size_t len) { return XXH3_update(state, (const xxh_u8*)input, len, XXH3_accumulate_512, XXH3_scrambleAcc); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_digest (const XXH3_state_t* state) { const unsigned char* const secret = (state->extSecret == NULL) ? state->customSecret : state->extSecret; if (state->totalLen > XXH3_MIDSIZE_MAX) { XXH_ALIGN(XXH_ACC_ALIGN) XXH64_hash_t acc[XXH_ACC_NB]; XXH3_digest_long(acc, state, secret); XXH_ASSERT(state->secretLimit + XXH_STRIPE_LEN >= sizeof(acc) + XXH_SECRET_MERGEACCS_START); { XXH128_hash_t h128; h128.low64 = XXH3_mergeAccs(acc, secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)state->totalLen * XXH_PRIME64_1); h128.high64 = XXH3_mergeAccs(acc, secret + state->secretLimit + XXH_STRIPE_LEN - sizeof(acc) - XXH_SECRET_MERGEACCS_START, ~((xxh_u64)state->totalLen * XXH_PRIME64_2)); return h128; } } /* len <= XXH3_MIDSIZE_MAX : short code */ if (state->seed) return XXH3_128bits_withSeed(state->buffer, (size_t)state->totalLen, state->seed); return XXH3_128bits_withSecret(state->buffer, (size_t)(state->totalLen), secret, state->secretLimit + XXH_STRIPE_LEN); } /* 128-bit utility functions */ #include /* memcmp, memcpy */ /* return : 1 is equal, 0 if different */ /*! @ingroup xxh3_family */ XXH_PUBLIC_API int XXH128_isEqual(XXH128_hash_t h1, XXH128_hash_t h2) { /* note : XXH128_hash_t is compact, it has no padding byte */ return !(memcmp(&h1, &h2, sizeof(h1))); } /* This prototype is compatible with stdlib's qsort(). * return : >0 if *h128_1 > *h128_2 * <0 if *h128_1 < *h128_2 * =0 if *h128_1 == *h128_2 */ /*! @ingroup xxh3_family */ XXH_PUBLIC_API int XXH128_cmp(const void* h128_1, const void* h128_2) { XXH128_hash_t const h1 = *(const XXH128_hash_t*)h128_1; XXH128_hash_t const h2 = *(const XXH128_hash_t*)h128_2; int const hcmp = (h1.high64 > h2.high64) - (h2.high64 > h1.high64); /* note : bets that, in most cases, hash values are different */ if (hcmp) return hcmp; return (h1.low64 > h2.low64) - (h2.low64 > h1.low64); } /*====== Canonical representation ======*/ /*! @ingroup xxh3_family */ XXH_PUBLIC_API void XXH128_canonicalFromHash(XXH128_canonical_t* dst, XXH128_hash_t hash) { XXH_STATIC_ASSERT(sizeof(XXH128_canonical_t) == sizeof(XXH128_hash_t)); if (XXH_CPU_LITTLE_ENDIAN) { hash.high64 = XXH_swap64(hash.high64); hash.low64 = XXH_swap64(hash.low64); } memcpy(dst, &hash.high64, sizeof(hash.high64)); memcpy((char*)dst + sizeof(hash.high64), &hash.low64, sizeof(hash.low64)); } /*! @ingroup xxh3_family */ XXH_PUBLIC_API XXH128_hash_t XXH128_hashFromCanonical(const XXH128_canonical_t* src) { XXH128_hash_t h; h.high64 = XXH_readBE64(src); h.low64 = XXH_readBE64(src->digest + 8); return h; } /* Pop our optimization override from above */ #if XXH_VECTOR == XXH_AVX2 /* AVX2 */ \ && defined(__GNUC__) && !defined(__clang__) /* GCC, not Clang */ \ && defined(__OPTIMIZE__) && !defined(__OPTIMIZE_SIZE__) /* respect -O0 and -Os */ # pragma GCC pop_options #endif #endif /* XXH_NO_LONG_LONG */ #endif /* XXH_NO_XXH3 */ /*! * @} */ #endif /* XXH_IMPLEMENTATION */ #if defined (__cplusplus) } #endif zmat-0.9.9/src/blosc2/plugins/codecs/ndlz/ndlz4x4.h0000644000175200007730000000415014515254731022262 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef NDLZ4_H #define NDLZ4_H #include "context.h" #if defined (__cplusplus) extern "C" { #endif #include "ndlz.h" #include "ndlz-private.h" /* #include #include "blosc2/blosc2-common.h" #include "fastcopy.h" */ /** Compress a block of data in the input buffer and returns the size of compressed block. The size of input buffer is specified by length. The minimum input buffer size is 16. The output buffer must be at least 5% larger than the input buffer and can not be smaller than 66 bytes. If the input is not compressible, or output does not fit in maxout bytes, the return value will be 0 and you will have to discard the output buffer. The acceleration parameter is related with the frequency for updating the internal hash. An acceleration of 1 means that the internal hash is updated at full rate. A value < 1 is not allowed and will be silently set to 1. The input buffer and the output buffer can not overlap. */ int ndlz4_compress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_cparams *cparams); /** Decompress a block of compressed data and returns the size of the decompressed block. If error occurs, e.g. the compressed data is corrupted or the output buffer is not large enough, then 0 (zero) will be returned instead. The input buffer and the output buffer can not overlap. Decompression is memory safe and guaranteed not to write the output buffer more than what is specified in maxout. */ int ndlz4_decompress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_dparams *dparams); #if defined (__cplusplus) } #endif #endif /* NDLZ4_H */ zmat-0.9.9/src/blosc2/plugins/codecs/zfp/0000755000175200007730000000000014515254731020432 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/plugins/codecs/zfp/test_zfp_acc_float.c0000644000175200007730000002003114515254731024423 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. Test program demonstrating use of the Blosc codec from C code. To compile this program: $ gcc -O test_zfp_acc_float.c -o test_zfp_acc_float -lblosc2 **********************************************************************/ #include #include #include "blosc2.h" #include "blosc2/codecs-registry.h" #include "blosc-private.h" #include static int test_zfp_acc_float(blosc2_schunk* schunk) { if (schunk->typesize != 4) { printf("Error: This test is only for doubles.\n"); return 0; } int64_t nchunks = schunk->nchunks; int32_t chunksize = (int32_t) (schunk->chunksize); float *data_in = malloc(chunksize); int decompressed; int64_t csize; int64_t dsize; int64_t csize_f = 0; uint8_t *data_out = malloc(chunksize + BLOSC2_MAX_OVERHEAD); float *data_dest = malloc(chunksize); /* Create a context for compression */ int8_t zfp_tol = -2; blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = schunk->typesize; cparams.compcode = BLOSC_CODEC_ZFP_FIXED_ACCURACY; cparams.compcode_meta = zfp_tol; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_NOFILTER; cparams.clevel = 5; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; dparams.schunk = schunk; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress with clevel=5 and shuffle active */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, data_out, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } csize_f += csize; /* Decompress */ dsize = blosc2_decompress_ctx(dctx, data_out, chunksize + BLOSC2_MAX_OVERHEAD, data_dest, chunksize); if (dsize <= 0) { printf("Decompression error. Error code: %" PRId64 "\n", dsize); return (int) dsize; } double tolerance = exp(zfp_tol); for (int i = 0; i < (chunksize / cparams.typesize); i++) { if (fabsf(data_in[i] - data_dest[i]) > tolerance) { printf("i: %d, data %f, dest %f", i, data_in[i], data_dest[i]); printf("\n Decompressed data differs from original!\n"); return -1; } } } csize_f = csize_f / nchunks; free(data_in); free(data_out); free(data_dest); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); printf("Successful roundtrip!\n"); printf("Compression: %d -> %" PRId64 " (%.1fx)\n", chunksize, csize_f, (1. * chunksize) / (double) csize_f); return (int) (chunksize - csize_f); } static int test_zfp_acc_double(blosc2_schunk* schunk) { if (schunk->typesize != 8) { printf("Error: This test is only for doubles.\n"); return 0; } int64_t nchunks = schunk->nchunks; int32_t chunksize = (int32_t) (schunk->chunksize); double *data_in = malloc(chunksize); int decompressed; int64_t csize; int64_t dsize; int64_t csize_f = 0; uint8_t *data_out = malloc(chunksize + BLOSC2_MAX_OVERHEAD); double *data_dest = malloc(chunksize); /* Create a context for compression */ int zfp_tol = -2; blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = schunk->typesize; cparams.compcode = BLOSC_CODEC_ZFP_FIXED_ACCURACY; cparams.compcode_meta = zfp_tol; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_NOFILTER; cparams.clevel = 5; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; dparams.schunk = schunk; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress with clevel=5 and shuffle active */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, data_out, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } csize_f += csize; /* Decompress */ dsize = blosc2_decompress_ctx(dctx, data_out, chunksize + BLOSC2_MAX_OVERHEAD, data_dest, chunksize); if (dsize <= 0) { printf("Decompression error. Error code: %" PRId64 "\n", dsize); return (int) dsize; } double tolerance = exp(zfp_tol); for (int i = 0; i < (chunksize / cparams.typesize); i++) { if (fabs(data_in[i] - data_dest[i]) > tolerance) { printf("i: %d, data %f, dest %f", i, data_in[i], data_dest[i]); printf("\n Decompressed data differs from original!\n"); return -1; } } } csize_f = csize_f / nchunks; free(data_in); free(data_out); free(data_dest); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); printf("Successful roundtrip!\n"); printf("Compression: %d -> %" PRId64 " (%.1fx)\n", chunksize, csize_f, (1. * chunksize) / (double) csize_f); return (int) (chunksize - csize_f); } int float_cyclic() { blosc2_schunk *schunk = blosc2_schunk_open("example_float_cyclic.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_acc_float(schunk); blosc2_schunk_free(schunk); return result; } int double_same_cells() { blosc2_schunk *schunk = blosc2_schunk_open("example_double_same_cells.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_acc_double(schunk); blosc2_schunk_free(schunk); return result; } int day_month_temp() { blosc2_schunk *schunk = blosc2_schunk_open("example_day_month_temp.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_acc_float(schunk); blosc2_schunk_free(schunk); return result; } int item_prices() { blosc2_schunk *schunk = blosc2_schunk_open("example_item_prices.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_acc_float(schunk); blosc2_schunk_free(schunk); return result; } int main(void) { int result; blosc2_init(); // this is mandatory for initiallizing the plugin mechanism result = float_cyclic(); printf("float_cyclic: %d obtained \n \n", result); result = double_same_cells(); printf("double_same_cells: %d obtained \n \n", result); result = day_month_temp(); printf("day_month_temp: %d obtained \n \n", result); result = item_prices(); printf("item_prices: %d obtained \n \n", result); blosc2_destroy(); return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/test_zfp_acc_int.c0000644000175200007730000001103314515254731024112 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. Test program demonstrating use of the Blosc codec from C code. To compile this program: $ gcc -O test_zfp_acc_int.c -o test_zfp_acc_int -lblosc2 **********************************************************************/ #include #include "blosc2.h" #include "blosc2/codecs-registry.h" #include "blosc-private.h" #include static int test_zfp(blosc2_schunk* schunk) { int64_t nchunks = schunk->nchunks; int32_t chunksize = (int32_t) (schunk->chunksize); uint8_t *data_in = malloc(chunksize); int decompressed; int64_t csize; int64_t dsize; int64_t csize_f = 0; uint8_t *data_out = malloc(chunksize + BLOSC2_MAX_OVERHEAD); uint8_t *data_dest = malloc(chunksize); /* Create a context for compression */ blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = schunk->typesize; cparams.compcode = BLOSC_CODEC_ZFP_FIXED_ACCURACY; cparams.compcode_meta = schunk->typesize; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_NOFILTER; cparams.clevel = 5; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; dparams.schunk = schunk; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress with clevel=5 and shuffle active */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, data_out, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } csize_f += csize; /* Decompress */ dsize = blosc2_decompress_ctx(dctx, data_out, chunksize + BLOSC2_MAX_OVERHEAD, data_dest, chunksize); if (dsize <= 0) { printf("Decompression error. Error code: %" PRId64 "\n", dsize); return (int) dsize; } for (int i = 0; i < chunksize; i++) { if ((data_in[i] - data_dest[i]) > 1) { printf("i: %d, data %u, dest %u", i, data_in[i], data_dest[i]); printf("\n Decompressed data differs from original!\n"); return -1; } } } csize_f = csize_f / nchunks; free(data_in); free(data_out); free(data_dest); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); printf("Successful roundtrip!\n"); printf("Compression: %d -> %" PRId64 " (%.1fx)\n", chunksize, csize_f, (1. * chunksize) / (double)csize_f); return (int) (chunksize - csize_f); } int rand_() { blosc2_schunk *schunk = blosc2_schunk_open("example_ndmean_repart_rand.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp(schunk); blosc2_schunk_free(schunk); return result; } int same_cells() { blosc2_schunk *schunk = blosc2_schunk_open("example_ndmean_repart_same_cells.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp(schunk); blosc2_schunk_free(schunk); return result; } int some_matches() { blosc2_schunk *schunk = blosc2_schunk_open("example_ndmean_repart_some_matches.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp(schunk); blosc2_schunk_free(schunk); return result; } int main(void) { int result; blosc2_init(); // this is mandatory for initiallizing the plugin mechanism result = rand_(); printf("rand: %d obtained \n \n", result); result = same_cells(); printf("same_cells: %d obtained \n \n", result); result = some_matches(); printf("some_matches: %d obtained \n \n", result); blosc2_destroy(); return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/zfp-private.h0000644000175200007730000000151714515254731023056 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef ZFP_PRIVATE_H #define ZFP_PRIVATE_H #define ZFP_MAX_DIM 4 #define ZFP_CELL_SHAPE 4 #if defined (__cplusplus) extern "C" { #endif #define XXH_INLINE_ALL #define ZFP_ERROR_NULL(pointer) \ do { \ if ((pointer) == NULL) { \ return 0; \ } \ } while (0) #if defined (__cplusplus) } #endif #endif /* ZFP_PRIVATE_H */ zmat-0.9.9/src/blosc2/plugins/codecs/zfp/include/0000755000175200007730000000000014515254731022055 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/plugins/codecs/zfp/include/bitstream.h0000644000175200007730000000513214515254731024221 0ustar rlaboissrlaboiss#ifndef ZFP_BITSTREAM_H #define ZFP_BITSTREAM_H #include #include "zfp/types.h" #include "zfp/system.h" /* forward declaration of opaque type */ typedef struct bitstream bitstream; extern_ const size_t stream_word_bits; /* bit stream granularity */ #ifndef inline_ #ifdef __cplusplus extern "C" { #endif /* allocate and initialize bit stream */ bitstream* stream_open(void* buffer, size_t bytes); /* close and deallocate bit stream */ void stream_close(bitstream* stream); /* make a copy of bit stream to shared memory buffer */ bitstream* stream_clone(const bitstream* stream); /* word size in bits (equal to stream_word_bits) */ size_t stream_alignment(void); /* pointer to beginning of stream */ void* stream_data(const bitstream* stream); /* current byte size of stream (if flushed) */ size_t stream_size(const bitstream* stream); /* byte capacity of stream */ size_t stream_capacity(const bitstream* stream); /* number of words per block */ size_t stream_stride_block(const bitstream* stream); /* number of blocks between consecutive blocks */ ptrdiff_t stream_stride_delta(const bitstream* stream); /* read single bit (0 or 1) */ uint stream_read_bit(bitstream* stream); /* write single bit */ uint stream_write_bit(bitstream* stream, uint bit); /* read 0 <= n <= 64 bits */ uint64 stream_read_bits(bitstream* stream, uint n); /* write 0 <= n <= 64 low bits of value and return remaining bits */ uint64 stream_write_bits(bitstream* stream, uint64 value, uint n); /* return bit offset to next bit to be read */ size_t stream_rtell(const bitstream* stream); /* return bit offset to next bit to be written */ size_t stream_wtell(const bitstream* stream); /* rewind stream to beginning */ void stream_rewind(bitstream* stream); /* position stream for reading at given bit offset */ void stream_rseek(bitstream* stream, size_t offset); /* position stream for writing at given bit offset */ void stream_wseek(bitstream* stream, size_t offset); /* skip over the next n bits */ void stream_skip(bitstream* stream, uint n); /* append n zero-bits to stream */ void stream_pad(bitstream* stream, uint n); /* align stream on next word boundary */ size_t stream_align(bitstream* stream); /* flush out any remaining buffered bits */ size_t stream_flush(bitstream* stream); /* copy n bits from one bit stream to another */ void stream_copy(bitstream* dst, bitstream* src, size_t n); #ifdef BIT_STREAM_STRIDED /* set block size in number of words and spacing in number of blocks */ int stream_set_stride(bitstream* stream, size_t block, ptrdiff_t delta); #endif #ifdef __cplusplus } #endif #endif /* !inline_ */ #endif zmat-0.9.9/src/blosc2/plugins/codecs/zfp/include/zfp.h0000644000175200007730000010423014515254731023025 0ustar rlaboissrlaboiss/* ** Copyright (c) 2014-2019, Lawrence Livermore National Security, LLC and ** other zfp project contributors. See the top-level LICENSE file for details. ** SPDX-License-Identifier: BSD-3-Clause */ #ifndef ZFP_H #define ZFP_H #include "zfp/types.h" #include "zfp/system.h" #include "zfp/version.h" #include "bitstream.h" /* macros ------------------------------------------------------------------ */ /* default compression parameters */ #define ZFP_MIN_BITS 1 /* minimum number of bits per block */ #define ZFP_MAX_BITS 16658 /* maximum number of bits per block */ #define ZFP_MAX_PREC 64 /* maximum precision supported */ #define ZFP_MIN_EXP (-1074) /* minimum floating-point base-2 exponent */ /* header masks (enable via bitwise or; reader must use same mask) */ #define ZFP_HEADER_NONE 0x0u /* no header */ #define ZFP_HEADER_MAGIC 0x1u /* embed 64-bit magic */ #define ZFP_HEADER_META 0x2u /* embed 52-bit field metadata */ #define ZFP_HEADER_MODE 0x4u /* embed 12- or 64-bit compression mode */ #define ZFP_HEADER_FULL 0x7u /* embed all of the above */ /* bit masks for specifying storage class */ #define ZFP_DATA_UNUSED 0x01u /* allocated but unused storage */ #define ZFP_DATA_PADDING 0x02u /* padding for alignment purposes */ #define ZFP_DATA_META 0x04u /* class members and other fixed-size storage */ #define ZFP_DATA_MISC 0x08u /* miscellaneous uncategorized storage */ #define ZFP_DATA_PAYLOAD 0x10u /* compressed data */ #define ZFP_DATA_INDEX 0x20u /* variable-rate block index information */ #define ZFP_DATA_CACHE 0x40u /* uncompressed cached data */ #define ZFP_DATA_HEADER 0x80u /* header information */ #define ZFP_DATA_ALL 0xffu /* all storage */ /* field metadata indeterminate state and error code */ #define ZFP_META_NULL (UINT64C(-1)) /* number of bits per header entry */ #define ZFP_MAGIC_BITS 32 /* number of magic word bits */ #define ZFP_META_BITS 52 /* number of field metadata bits */ #define ZFP_MODE_SHORT_BITS 12 /* number of mode bits in short format */ #define ZFP_MODE_LONG_BITS 64 /* number of mode bits in long format */ #define ZFP_HEADER_MAX_BITS 148 /* max number of header bits */ #define ZFP_MODE_SHORT_MAX ((1u << ZFP_MODE_SHORT_BITS) - 2) /* rounding mode for reducing bias; see build option ZFP_ROUNDING_MODE */ #define ZFP_ROUND_FIRST (-1) /* round during compression */ #define ZFP_ROUND_NEVER 0 /* never round */ #define ZFP_ROUND_LAST 1 /* round during decompression */ /* types ------------------------------------------------------------------- */ /* Boolean constants */ enum { zfp_false = 0, /* false */ zfp_true = !zfp_false /* true */ }; typedef int zfp_bool; /* Boolean type */ /* execution policy */ typedef enum { zfp_exec_serial = 0, /* serial execution (default) */ zfp_exec_omp = 1, /* OpenMP multi-threaded execution */ zfp_exec_cuda = 2 /* CUDA parallel execution */ } zfp_exec_policy; /* OpenMP execution parameters */ typedef struct { uint threads; /* number of requested threads */ uint chunk_size; /* number of blocks per chunk (1D only) */ } zfp_exec_params_omp; /* execution parameters */ typedef union { zfp_exec_params_omp omp; /* OpenMP parameters */ } zfp_exec_params; typedef struct { zfp_exec_policy policy; /* execution policy (serial, omp, ...) */ zfp_exec_params params; /* execution parameters */ } zfp_execution; /* compressed stream; use accessors to get/set members */ typedef struct { uint minbits; /* minimum number of bits to store per block */ uint maxbits; /* maximum number of bits to store per block */ uint maxprec; /* maximum number of bit planes to store */ int minexp; /* minimum floating point bit plane number to store */ bitstream* stream; /* compressed bit stream */ zfp_execution exec; /* execution policy and parameters */ } zfp_stream; /* compression mode */ typedef enum { zfp_mode_null = 0, /* an invalid configuration of the 4 params */ zfp_mode_expert = 1, /* expert mode (4 params set manually) */ zfp_mode_fixed_rate = 2, /* fixed rate mode */ zfp_mode_fixed_precision = 3, /* fixed precision mode */ zfp_mode_fixed_accuracy = 4, /* fixed accuracy mode */ zfp_mode_reversible = 5 /* reversible (lossless) mode */ } zfp_mode; /* compression mode and parameter settings */ typedef struct { zfp_mode mode; /* compression mode */ union { double rate; /* compressed bits/value (negative for word alignment) */ uint precision; /* uncompressed bits/value */ double tolerance; /* absolute error tolerance */ struct { uint minbits; /* min number of compressed bits/block */ uint maxbits; /* max number of compressed bits/block */ uint maxprec; /* max number of uncompressed bits/value */ int minexp; /* min floating point bit plane number to store */ } expert; /* expert mode arguments */ } arg; /* arguments corresponding to compression mode */ } zfp_config; /* scalar type */ typedef enum { zfp_type_none = 0, /* unspecified type */ zfp_type_int32 = 1, /* 32-bit signed integer */ zfp_type_int64 = 2, /* 64-bit signed integer */ zfp_type_float = 3, /* single precision floating point */ zfp_type_double = 4 /* double precision floating point */ } zfp_type; /* uncompressed array; use accessors to get/set members */ typedef struct { zfp_type type; /* scalar type (e.g. int32, double) */ size_t nx, ny, nz, nw; /* sizes (zero for unused dimensions) */ ptrdiff_t sx, sy, sz, sw; /* strides (zero for contiguous array a[nw][nz][ny][nx]) */ void* data; /* pointer to array data */ } zfp_field; #ifdef __cplusplus extern "C" { #endif /* public data ------------------------------------------------------------- */ extern_ const uint zfp_codec_version; /* codec version ZFP_CODEC */ extern_ const uint zfp_library_version; /* library version ZFP_VERSION */ extern_ const char* const zfp_version_string; /* verbose version string */ /* high-level API: utility functions --------------------------------------- */ size_t /* byte size of scalar type */ zfp_type_size( zfp_type type /* scalar type */ ); /* high-level API: compressed stream construction/destruction -------------- */ /* open compressed stream and associate with bit stream */ zfp_stream* /* allocated compressed stream */ zfp_stream_open( bitstream* stream /* bit stream to read from and write to (may be NULL) */ ); /* close and deallocate compressed stream (does not affect bit stream) */ void zfp_stream_close( zfp_stream* stream /* compressed stream */ ); /* high-level API: compressed stream inspectors ---------------------------- */ /* bit stream associated with compressed stream */ bitstream* /* bit stream associated with compressed stream */ zfp_stream_bit_stream( const zfp_stream* stream /* compressed stream */ ); /* enumerated compression mode */ zfp_mode /* compression mode or zfp_mode_null if not set */ zfp_stream_compression_mode( const zfp_stream* stream /* compressed stream */ ); /* rate in compressed bits/scalar (when in fixed-rate mode) */ double /* rate or zero upon failure */ zfp_stream_rate( const zfp_stream* stream, /* compressed stream */ uint dims /* array dimensionality (1, 2, 3, or 4) */ ); /* precision in uncompressed bits/scalar (when in fixed-precision mode) */ uint /* precision or zero upon failure */ zfp_stream_precision( const zfp_stream* stream /* compressed stream */ ); /* accuracy as absolute error tolerance (when in fixed-accuracy mode) */ double /* tolerance or zero upon failure */ zfp_stream_accuracy( const zfp_stream* stream /* compressed stream */ ); /* get all compression parameters in a compact representation */ uint64 /* 12- or 64-bit encoding of parameters */ zfp_stream_mode( const zfp_stream* stream /* compressed stream */ ); /* get all compression parameters (pointers may be NULL) */ void zfp_stream_params( const zfp_stream* stream, /* compressed stream */ uint* minbits, /* minimum number of bits per 4^d block */ uint* maxbits, /* maximum number of bits per 4^d block */ uint* maxprec, /* maximum precision (# bit planes coded) */ int* minexp /* minimum base-2 exponent; error <= 2^minexp */ ); /* byte size of sequentially compressed stream (call after compression) */ size_t /* actual number of bytes of compressed storage */ zfp_stream_compressed_size( const zfp_stream* stream /* compressed stream */ ); /* conservative estimate of compressed size in bytes */ size_t /* maximum number of bytes of compressed storage */ zfp_stream_maximum_size( const zfp_stream* stream, /* compressed stream */ const zfp_field* field /* array to compress */ ); /* high-level API: initialization of compressed stream parameters ---------- */ /* rewind bit stream to beginning for compression or decompression */ void zfp_stream_rewind( zfp_stream* stream /* compressed bit stream */ ); /* associate bit stream with compressed stream */ void zfp_stream_set_bit_stream( zfp_stream* stream, /* compressed stream */ bitstream* bs /* bit stream to read from and write to */ ); /* enable reversible (lossless) compression */ void zfp_stream_set_reversible( zfp_stream* stream /* compressed stream */ ); /* set size in compressed bits/scalar (fixed-rate mode) */ double /* actual rate in compressed bits/scalar */ zfp_stream_set_rate( zfp_stream* stream, /* compressed stream */ double rate, /* desired rate in compressed bits/scalar */ zfp_type type, /* scalar type to compress */ uint dims, /* array dimensionality (1, 2, 3, or 4) */ zfp_bool align /* word-aligned blocks, e.g., for write random access */ ); /* set precision in uncompressed bits/scalar (fixed-precision mode) */ uint /* actual precision */ zfp_stream_set_precision( zfp_stream* stream, /* compressed stream */ uint precision /* desired precision in uncompressed bits/scalar */ ); /* set accuracy as absolute error tolerance (fixed-accuracy mode) */ double /* actual error tolerance */ zfp_stream_set_accuracy( zfp_stream* stream, /* compressed stream */ double tolerance /* desired error tolerance */ ); /* set parameters from compact encoding; leaves stream intact on failure */ zfp_mode /* compression mode or zfp_mode_null upon failure */ zfp_stream_set_mode( zfp_stream* stream, /* compressed stream */ uint64 mode /* 12- or 64-bit encoding of parameters */ ); /* set all parameters (expert mode); leaves stream intact on failure */ zfp_bool /* true upon success */ zfp_stream_set_params( zfp_stream* stream, /* compressed stream */ uint minbits, /* minimum number of bits per 4^d block */ uint maxbits, /* maximum number of bits per 4^d block */ uint maxprec, /* maximum precision (# bit planes coded) */ int minexp /* minimum base-2 exponent; error <= 2^minexp */ ); /* high-level API: execution policy ---------------------------------------- */ /* current execution policy */ zfp_exec_policy zfp_stream_execution( const zfp_stream* stream /* compressed stream */ ); /* number of OpenMP threads to use */ uint /* number of threads (0 for default) */ zfp_stream_omp_threads( const zfp_stream* stream /* compressed stream */ ); /* number of blocks per OpenMP chunk (1D only) */ uint /* number of blocks per chunk (0 for default) */ zfp_stream_omp_chunk_size( const zfp_stream* stream /* compressed stream */ ); /* set execution policy */ zfp_bool /* true upon success */ zfp_stream_set_execution( zfp_stream* stream, /* compressed stream */ zfp_exec_policy policy /* execution policy */ ); /* set OpenMP execution policy and number of threads */ zfp_bool /* true upon success */ zfp_stream_set_omp_threads( zfp_stream* stream, /* compressed stream */ uint threads /* number of OpenMP threads to use (0 for default) */ ); /* set OpenMP execution policy and number of blocks per chunk (1D only) */ zfp_bool /* true upon success */ zfp_stream_set_omp_chunk_size( zfp_stream* stream, /* compressed stream */ uint chunk_size /* number of blocks per chunk (0 for default) */ ); /* high-level API: compression mode and parameter settings ----------------- */ /* unspecified configuration */ zfp_config /* compression mode and parameter settings */ zfp_config_none(void); /* fixed-rate configuration */ zfp_config /* compression mode and parameter settings */ zfp_config_rate( double rate, /* desired rate in compressed bits/scalar */ zfp_bool align /* word-aligned blocks, e.g., for write random access */ ); /* fixed-precision configuration */ zfp_config /* compression mode and parameter settings */ zfp_config_precision( uint precision /* desired precision in uncompressed bits/scalar */ ); /* fixed-accuracy configuration */ zfp_config /* compression mode and parameter settings */ zfp_config_accuracy( double tolerance /* desired error tolerance */ ); /* reversible (lossless) configuration */ zfp_config /* compression mode and parameter settings */ zfp_config_reversible(void); /* expert configuration */ zfp_config /* compression mode and parameter settings */ zfp_config_expert( uint minbits, /* minimum number of bits per 4^d block */ uint maxbits, /* maximum number of bits per 4^d block */ uint maxprec, /* maximum precision (# bit planes coded) */ int minexp /* minimum base-2 exponent; error <= 2^minexp */ ); /* high-level API: uncompressed array construction/destruction ------------- */ /* allocate field struct */ zfp_field* /* pointer to default initialized field */ zfp_field_alloc(void); /* allocate metadata for 1D field f[nx] */ zfp_field* /* allocated field metadata */ zfp_field_1d( void* pointer, /* pointer to uncompressed scalars (may be NULL) */ zfp_type type, /* scalar type */ size_t nx /* number of scalars */ ); /* allocate metadata for 2D field f[ny][nx] */ zfp_field* /* allocated field metadata */ zfp_field_2d( void* pointer, /* pointer to uncompressed scalars (may be NULL) */ zfp_type type, /* scalar type */ size_t nx, /* number of scalars in x dimension */ size_t ny /* number of scalars in y dimension */ ); /* allocate metadata for 3D field f[nz][ny][nx] */ zfp_field* /* allocated field metadata */ zfp_field_3d( void* pointer, /* pointer to uncompressed scalars (may be NULL) */ zfp_type type, /* scalar type */ size_t nx, /* number of scalars in x dimension */ size_t ny, /* number of scalars in y dimension */ size_t nz /* number of scalars in z dimension */ ); /* allocate metadata for 4D field f[nw][nz][ny][nx] */ zfp_field* /* allocated field metadata */ zfp_field_4d( void* pointer, /* pointer to uncompressed scalars (may be NULL) */ zfp_type type, /* scalar type */ size_t nx, /* number of scalars in x dimension */ size_t ny, /* number of scalars in y dimension */ size_t nz, /* number of scalars in z dimension */ size_t nw /* number of scalars in w dimension */ ); /* deallocate field metadata */ void zfp_field_free( zfp_field* field /* field metadata */ ); /* high-level API: uncompressed array inspectors --------------------------- */ /* pointer to first scalar in field */ void* /* array pointer */ zfp_field_pointer( const zfp_field* field /* field metadata */ ); /* pointer to lowest memory address spanned by field */ void* zfp_field_begin( const zfp_field* field /* field metadata */ ); /* field scalar type */ zfp_type /* scalar type */ zfp_field_type( const zfp_field* field /* field metadata */ ); /* precision of field scalar type */ uint /* scalar type precision in number of bits */ zfp_field_precision( const zfp_field* field /* field metadata */ ); /* field dimensionality (1, 2, 3, or 4) */ uint /* number of dimensions */ zfp_field_dimensionality( const zfp_field* field /* field metadata */ ); /* field size in number of scalars */ size_t /* total number of scalars */ zfp_field_size( const zfp_field* field, /* field metadata */ size_t* size /* number of scalars per dimension (may be NULL) */ ); /* number of bytes spanned by field data including gaps (if any) */ size_t zfp_field_size_bytes( const zfp_field* field /* field metadata */ ); /* field strides per dimension */ zfp_bool /* true if array is not contiguous */ zfp_field_stride( const zfp_field* field, /* field metadata */ ptrdiff_t* stride /* stride in scalars per dimension (may be NULL) */ ); /* field contiguity test */ zfp_bool /* true if field layout is contiguous */ zfp_field_is_contiguous( const zfp_field* field /* field metadata */ ); /* field scalar type and dimensions */ uint64 /* compact 52-bit encoding of metadata */ zfp_field_metadata( const zfp_field* field /* field metadata */ ); /* high-level API: uncompressed array specification ------------------------ */ /* set pointer to first scalar in field */ void zfp_field_set_pointer( zfp_field* field, /* field metadata */ void* pointer /* pointer to first scalar */ ); /* set field scalar type */ zfp_type /* actual scalar type */ zfp_field_set_type( zfp_field* field, /* field metadata */ zfp_type type /* desired scalar type */ ); /* set 1D field size */ void zfp_field_set_size_1d( zfp_field* field, /* field metadata */ size_t nx /* number of scalars */ ); /* set 2D field size */ void zfp_field_set_size_2d( zfp_field* field, /* field metadata */ size_t nx, /* number of scalars in x dimension */ size_t ny /* number of scalars in y dimension */ ); /* set 3D field size */ void zfp_field_set_size_3d( zfp_field* field, /* field metadata */ size_t nx, /* number of scalars in x dimension */ size_t ny, /* number of scalars in y dimension */ size_t nz /* number of scalars in z dimension */ ); /* set 4D field size */ void zfp_field_set_size_4d( zfp_field* field, /* field metadata */ size_t nx, /* number of scalars in x dimension */ size_t ny, /* number of scalars in y dimension */ size_t nz, /* number of scalars in z dimension */ size_t nw /* number of scalars in w dimension */ ); /* set 1D field stride in number of scalars */ void zfp_field_set_stride_1d( zfp_field* field, /* field metadata */ ptrdiff_t sx /* stride in number of scalars: &f[1] - &f[0] */ ); /* set 2D field strides in number of scalars */ void zfp_field_set_stride_2d( zfp_field* field, /* field metadata */ ptrdiff_t sx, /* stride in x dimension: &f[0][1] - &f[0][0] */ ptrdiff_t sy /* stride in y dimension: &f[1][0] - &f[0][0] */ ); /* set 3D field strides in number of scalars */ void zfp_field_set_stride_3d( zfp_field* field, /* field metadata */ ptrdiff_t sx, /* stride in x dimension: &f[0][0][1] - &f[0][0][0] */ ptrdiff_t sy, /* stride in y dimension: &f[0][1][0] - &f[0][0][0] */ ptrdiff_t sz /* stride in z dimension: &f[1][0][0] - &f[0][0][0] */ ); /* set 4D field strides in number of scalars */ void zfp_field_set_stride_4d( zfp_field* field, /* field metadata */ ptrdiff_t sx, /* stride in x dimension: &f[0][0][0][1] - &f[0][0][0][0] */ ptrdiff_t sy, /* stride in y dimension: &f[0][0][1][0] - &f[0][0][0][0] */ ptrdiff_t sz, /* stride in z dimension: &f[0][1][0][0] - &f[0][0][0][0] */ ptrdiff_t sw /* stride in w dimension: &f[1][0][0][0] - &f[0][0][0][0] */ ); /* set field scalar type and dimensions */ zfp_bool /* true upon success */ zfp_field_set_metadata( zfp_field* field, /* field metadata */ uint64 meta /* compact 52-bit encoding of metadata */ ); /* high-level API: compression and decompression --------------------------- */ /* compress entire field (nonzero return value upon success) */ size_t /* cumulative number of bytes of compressed storage */ zfp_compress( zfp_stream* stream, /* compressed stream */ const zfp_field* field /* field metadata */ ); /* decompress entire field (nonzero return value upon success) */ size_t /* cumulative number of bytes of compressed storage */ zfp_decompress( zfp_stream* stream, /* compressed stream */ zfp_field* field /* field metadata */ ); /* write compression parameters and field metadata (optional) */ size_t /* number of bits written or zero upon failure */ zfp_write_header( zfp_stream* stream, /* compressed stream */ const zfp_field* field, /* field metadata */ uint mask /* information to write */ ); /* read compression parameters and field metadata when previously written */ size_t /* number of bits read or zero upon failure */ zfp_read_header( zfp_stream* stream, /* compressed stream */ zfp_field* field, /* field metadata */ uint mask /* information to read */ ); /* low-level API: stream manipulation -------------------------------------- */ /* flush bit stream--must be called after last encode call or between seeks */ size_t zfp_stream_flush( zfp_stream* stream /* compressed bit stream */ ); /* align bit stream on next word boundary (decoding analogy to flush) */ size_t zfp_stream_align( zfp_stream* stream /* compressed bit stream */ ); /* low-level API: encoder -------------------------------------------------- */ /* The functions below all compress either a complete contiguous d-dimensional block of 4^d scalars or a complete or partial block assembled from a strided array. In the latter case, p points to the first scalar; (nx, ny, nz) specify the size of the block, with 1 <= nx, ny, nz <= 4; and (sx, sy, sz) specify the strides, i.e. the number of scalars to advance to get to the next scalar along each dimension. The functions return the number of bits of compressed storage needed for the compressed block. */ /* encode 1D contiguous block of 4 values */ size_t zfp_encode_block_int32_1(zfp_stream* stream, const int32* block); size_t zfp_encode_block_int64_1(zfp_stream* stream, const int64* block); size_t zfp_encode_block_float_1(zfp_stream* stream, const float* block); size_t zfp_encode_block_double_1(zfp_stream* stream, const double* block); /* encode 1D complete or partial block from strided array */ size_t zfp_encode_block_strided_int32_1(zfp_stream* stream, const int32* p, ptrdiff_t sx); size_t zfp_encode_block_strided_int64_1(zfp_stream* stream, const int64* p, ptrdiff_t sx); size_t zfp_encode_block_strided_float_1(zfp_stream* stream, const float* p, ptrdiff_t sx); size_t zfp_encode_block_strided_double_1(zfp_stream* stream, const double* p, ptrdiff_t sx); size_t zfp_encode_partial_block_strided_int32_1(zfp_stream* stream, const int32* p, size_t nx, ptrdiff_t sx); size_t zfp_encode_partial_block_strided_int64_1(zfp_stream* stream, const int64* p, size_t nx, ptrdiff_t sx); size_t zfp_encode_partial_block_strided_float_1(zfp_stream* stream, const float* p, size_t nx, ptrdiff_t sx); size_t zfp_encode_partial_block_strided_double_1(zfp_stream* stream, const double* p, size_t nx, ptrdiff_t sx); /* encode 2D contiguous block of 4x4 values */ size_t zfp_encode_block_int32_2(zfp_stream* stream, const int32* block); size_t zfp_encode_block_int64_2(zfp_stream* stream, const int64* block); size_t zfp_encode_block_float_2(zfp_stream* stream, const float* block); size_t zfp_encode_block_double_2(zfp_stream* stream, const double* block); /* encode 2D complete or partial block from strided array */ size_t zfp_encode_partial_block_strided_int32_2(zfp_stream* stream, const int32* p, size_t nx, size_t ny, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_encode_partial_block_strided_int64_2(zfp_stream* stream, const int64* p, size_t nx, size_t ny, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_encode_partial_block_strided_float_2(zfp_stream* stream, const float* p, size_t nx, size_t ny, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_encode_partial_block_strided_double_2(zfp_stream* stream, const double* p, size_t nx, size_t ny, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_encode_block_strided_int32_2(zfp_stream* stream, const int32* p, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_encode_block_strided_int64_2(zfp_stream* stream, const int64* p, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_encode_block_strided_float_2(zfp_stream* stream, const float* p, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_encode_block_strided_double_2(zfp_stream* stream, const double* p, ptrdiff_t sx, ptrdiff_t sy); /* encode 3D contiguous block of 4x4x4 values */ size_t zfp_encode_block_int32_3(zfp_stream* stream, const int32* block); size_t zfp_encode_block_int64_3(zfp_stream* stream, const int64* block); size_t zfp_encode_block_float_3(zfp_stream* stream, const float* block); size_t zfp_encode_block_double_3(zfp_stream* stream, const double* block); /* encode 3D complete or partial block from strided array */ size_t zfp_encode_block_strided_int32_3(zfp_stream* stream, const int32* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_encode_block_strided_int64_3(zfp_stream* stream, const int64* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_encode_block_strided_float_3(zfp_stream* stream, const float* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_encode_block_strided_double_3(zfp_stream* stream, const double* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_encode_partial_block_strided_int32_3(zfp_stream* stream, const int32* p, size_t nx, size_t ny, size_t nz, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_encode_partial_block_strided_int64_3(zfp_stream* stream, const int64* p, size_t nx, size_t ny, size_t nz, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_encode_partial_block_strided_float_3(zfp_stream* stream, const float* p, size_t nx, size_t ny, size_t nz, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_encode_partial_block_strided_double_3(zfp_stream* stream, const double* p, size_t nx, size_t ny, size_t nz, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); /* encode 4D contiguous block of 4x4x4x4 values */ size_t zfp_encode_block_int32_4(zfp_stream* stream, const int32* block); size_t zfp_encode_block_int64_4(zfp_stream* stream, const int64* block); size_t zfp_encode_block_float_4(zfp_stream* stream, const float* block); size_t zfp_encode_block_double_4(zfp_stream* stream, const double* block); /* encode 4D complete or partial block from strided array */ size_t zfp_encode_block_strided_int32_4(zfp_stream* stream, const int32* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_encode_block_strided_int64_4(zfp_stream* stream, const int64* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_encode_block_strided_float_4(zfp_stream* stream, const float* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_encode_block_strided_double_4(zfp_stream* stream, const double* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_encode_partial_block_strided_int32_4(zfp_stream* stream, const int32* p, size_t nx, size_t ny, size_t nz, size_t nw, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_encode_partial_block_strided_int64_4(zfp_stream* stream, const int64* p, size_t nx, size_t ny, size_t nz, size_t nw, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_encode_partial_block_strided_float_4(zfp_stream* stream, const float* p, size_t nx, size_t ny, size_t nz, size_t nw, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_encode_partial_block_strided_double_4(zfp_stream* stream, const double* p, size_t nx, size_t ny, size_t nz, size_t nw, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); /* low-level API: decoder -------------------------------------------------- */ /* Each function below decompresses a single block and returns the number of bits of compressed storage consumed. See corresponding encoder functions above for further details. */ /* decode 1D contiguous block of 4 values */ size_t zfp_decode_block_int32_1(zfp_stream* stream, int32* block); size_t zfp_decode_block_int64_1(zfp_stream* stream, int64* block); size_t zfp_decode_block_float_1(zfp_stream* stream, float* block); size_t zfp_decode_block_double_1(zfp_stream* stream, double* block); /* decode 1D complete or partial block from strided array */ size_t zfp_decode_block_strided_int32_1(zfp_stream* stream, int32* p, ptrdiff_t sx); size_t zfp_decode_block_strided_int64_1(zfp_stream* stream, int64* p, ptrdiff_t sx); size_t zfp_decode_block_strided_float_1(zfp_stream* stream, float* p, ptrdiff_t sx); size_t zfp_decode_block_strided_double_1(zfp_stream* stream, double* p, ptrdiff_t sx); size_t zfp_decode_partial_block_strided_int32_1(zfp_stream* stream, int32* p, size_t nx, ptrdiff_t sx); size_t zfp_decode_partial_block_strided_int64_1(zfp_stream* stream, int64* p, size_t nx, ptrdiff_t sx); size_t zfp_decode_partial_block_strided_float_1(zfp_stream* stream, float* p, size_t nx, ptrdiff_t sx); size_t zfp_decode_partial_block_strided_double_1(zfp_stream* stream, double* p, size_t nx, ptrdiff_t sx); /* decode 2D contiguous block of 4x4 values */ size_t zfp_decode_block_int32_2(zfp_stream* stream, int32* block); size_t zfp_decode_block_int64_2(zfp_stream* stream, int64* block); size_t zfp_decode_block_float_2(zfp_stream* stream, float* block); size_t zfp_decode_block_double_2(zfp_stream* stream, double* block); /* decode 2D complete or partial block from strided array */ size_t zfp_decode_block_strided_int32_2(zfp_stream* stream, int32* p, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_decode_block_strided_int64_2(zfp_stream* stream, int64* p, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_decode_block_strided_float_2(zfp_stream* stream, float* p, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_decode_block_strided_double_2(zfp_stream* stream, double* p, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_decode_partial_block_strided_int32_2(zfp_stream* stream, int32* p, size_t nx, size_t ny, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_decode_partial_block_strided_int64_2(zfp_stream* stream, int64* p, size_t nx, size_t ny, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_decode_partial_block_strided_float_2(zfp_stream* stream, float* p, size_t nx, size_t ny, ptrdiff_t sx, ptrdiff_t sy); size_t zfp_decode_partial_block_strided_double_2(zfp_stream* stream, double* p, size_t nx, size_t ny, ptrdiff_t sx, ptrdiff_t sy); /* decode 3D contiguous block of 4x4x4 values */ size_t zfp_decode_block_int32_3(zfp_stream* stream, int32* block); size_t zfp_decode_block_int64_3(zfp_stream* stream, int64* block); size_t zfp_decode_block_float_3(zfp_stream* stream, float* block); size_t zfp_decode_block_double_3(zfp_stream* stream, double* block); /* decode 3D complete or partial block from strided array */ size_t zfp_decode_block_strided_int32_3(zfp_stream* stream, int32* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_decode_block_strided_int64_3(zfp_stream* stream, int64* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_decode_block_strided_float_3(zfp_stream* stream, float* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_decode_block_strided_double_3(zfp_stream* stream, double* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_decode_partial_block_strided_int32_3(zfp_stream* stream, int32* p, size_t nx, size_t ny, size_t nz, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_decode_partial_block_strided_int64_3(zfp_stream* stream, int64* p, size_t nx, size_t ny, size_t nz, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_decode_partial_block_strided_float_3(zfp_stream* stream, float* p, size_t nx, size_t ny, size_t nz, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); size_t zfp_decode_partial_block_strided_double_3(zfp_stream* stream, double* p, size_t nx, size_t ny, size_t nz, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz); /* decode 4D contiguous block of 4x4x4x4 values */ size_t zfp_decode_block_int32_4(zfp_stream* stream, int32* block); size_t zfp_decode_block_int64_4(zfp_stream* stream, int64* block); size_t zfp_decode_block_float_4(zfp_stream* stream, float* block); size_t zfp_decode_block_double_4(zfp_stream* stream, double* block); /* decode 4D complete or partial block from strided array */ size_t zfp_decode_block_strided_int32_4(zfp_stream* stream, int32* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_decode_block_strided_int64_4(zfp_stream* stream, int64* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_decode_block_strided_float_4(zfp_stream* stream, float* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_decode_block_strided_double_4(zfp_stream* stream, double* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_decode_partial_block_strided_int32_4(zfp_stream* stream, int32* p, size_t nx, size_t ny, size_t nz, size_t nw, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_decode_partial_block_strided_int64_4(zfp_stream* stream, int64* p, size_t nx, size_t ny, size_t nz, size_t nw, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_decode_partial_block_strided_float_4(zfp_stream* stream, float* p, size_t nx, size_t ny, size_t nz, size_t nw, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); size_t zfp_decode_partial_block_strided_double_4(zfp_stream* stream, double* p, size_t nx, size_t ny, size_t nz, size_t nw, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw); /* low-level API: utility functions ---------------------------------------- */ /* convert dims-dimensional contiguous block to 32-bit integer type */ void zfp_promote_int8_to_int32(int32* oblock, const int8* iblock, uint dims); void zfp_promote_uint8_to_int32(int32* oblock, const uint8* iblock, uint dims); void zfp_promote_int16_to_int32(int32* oblock, const int16* iblock, uint dims); void zfp_promote_uint16_to_int32(int32* oblock, const uint16* iblock, uint dims); /* convert dims-dimensional contiguous block from 32-bit integer type */ void zfp_demote_int32_to_int8(int8* oblock, const int32* iblock, uint dims); void zfp_demote_int32_to_uint8(uint8* oblock, const int32* iblock, uint dims); void zfp_demote_int32_to_int16(int16* oblock, const int32* iblock, uint dims); void zfp_demote_int32_to_uint16(uint16* oblock, const int32* iblock, uint dims); #ifdef __cplusplus } #endif #endif zmat-0.9.9/src/blosc2/plugins/codecs/zfp/include/zfp/0000755000175200007730000000000014515254731022654 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/plugins/codecs/zfp/include/zfp/macros.h0000644000175200007730000000020714515254731024310 0ustar rlaboissrlaboiss#ifndef ZFP_MACROS_H #define ZFP_MACROS_H #define MIN(x, y) ((x) < (y) ? (x) : (y)) #define MAX(x, y) ((x) > (y) ? (x) : (y)) #endif zmat-0.9.9/src/blosc2/plugins/codecs/zfp/include/zfp/system.h0000644000175200007730000000230314515254731024347 0ustar rlaboissrlaboiss#ifndef ZFP_SYSTEM_H #define ZFP_SYSTEM_H #if defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99: use restrict */ #define restrict_ restrict #else /* C89: no restrict keyword */ #define restrict_ #endif /* macros for exporting and importing symbols */ #if defined(_MSC_VER) && defined(ZFP_SHARED_LIBS) /* export (import) symbols when ZFP_SOURCE is (is not) defined */ #ifdef ZFP_SOURCE #ifdef __cplusplus #define extern_ extern "C" __declspec(dllexport) #else #define extern_ extern __declspec(dllexport) #endif #else #ifdef __cplusplus #define extern_ extern "C" __declspec(dllimport) #else #define extern_ extern __declspec(dllimport) #endif #endif #else /* !(_MSC_VER && ZFP_SHARED_LIBS) */ #ifdef __cplusplus #define extern_ extern "C" #else #define extern_ extern #endif #endif /* L1 cache line size for alignment purposes */ #ifndef ZFP_CACHE_LINE_SIZE #define ZFP_CACHE_LINE_SIZE 0x100 #endif /* ZFP_CACHE_LINE_SIZE=0 disables alignment */ #if defined(__GNUC__) && ZFP_CACHE_LINE_SIZE #define cache_align_(x) x __attribute__((aligned(ZFP_CACHE_LINE_SIZE))) #else #define cache_align_(x) x #endif #endif zmat-0.9.9/src/blosc2/plugins/codecs/zfp/include/zfp/version.h0000644000175200007730000000151714515254731024516 0ustar rlaboissrlaboiss#ifndef ZFP_VERSION_H #define ZFP_VERSION_H /* stringification */ #define _zfp_str_(x) # x #define _zfp_str(x) _zfp_str_(x) /* library version information */ #define ZFP_VERSION_MAJOR 0 /* library major version number */ #define ZFP_VERSION_MINOR 5 /* library minor version number */ #define ZFP_VERSION_PATCH 5 /* library patch version number */ #define ZFP_VERSION_RELEASE ZFP_VERSION_PATCH /* codec version number (see also zfp_codec_version) */ #define ZFP_CODEC 5 /* library version number (see also zfp_library_version) */ #define ZFP_VERSION \ ((ZFP_VERSION_MAJOR << 8) + \ (ZFP_VERSION_MINOR << 4) + \ (ZFP_VERSION_PATCH << 0)) /* library version string (see also zfp_version_string) */ #define ZFP_VERSION_STRING \ _zfp_str(ZFP_VERSION_MAJOR) "." \ _zfp_str(ZFP_VERSION_MINOR) "." \ _zfp_str(ZFP_VERSION_PATCH) #endif zmat-0.9.9/src/blosc2/plugins/codecs/zfp/include/zfp/types.h0000644000175200007730000000721014515254731024171 0ustar rlaboissrlaboiss#ifndef ZFP_TYPES_H #define ZFP_TYPES_H typedef unsigned char uchar; typedef unsigned short ushort; typedef unsigned int uint; typedef unsigned long ulong; #if defined(__cplusplus) && __cplusplus >= 201103L /* C++11: use standard integer types */ #include #include #define INT64C(x) INT64_C(x) #define UINT64C(x) UINT64_C(x) #define INT64PRId PRId64 #define INT64PRIi PRIi64 #define UINT64PRIo PRIo64 #define UINT64PRIu PRIu64 #define UINT64PRIx PRIx64 #define INT64SCNd SCNd64 #define INT64SCNi SCNi64 #define UINT64SCNo SCNo64 #define UINT64SCNu SCNu64 #define UINT64SCNx SCNx64 typedef std::int8_t int8; typedef std::uint8_t uint8; typedef std::int16_t int16; typedef std::uint16_t uint16; typedef std::int32_t int32; typedef std::uint32_t uint32; typedef std::int64_t int64; typedef std::uint64_t uint64; #elif defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99: use standard integer types */ #include #include #define INT64C(x) INT64_C(x) #define UINT64C(x) UINT64_C(x) #define INT64PRId PRId64 #define INT64PRIi PRIi64 #define UINT64PRIo PRIo64 #define UINT64PRIu PRIu64 #define UINT64PRIx PRIx64 #define INT64SCNd SCNd64 #define INT64SCNi SCNi64 #define UINT64SCNo SCNo64 #define UINT64SCNu SCNu64 #define UINT64SCNx SCNx64 typedef int8_t int8; typedef uint8_t uint8; typedef int16_t int16; typedef uint16_t uint16; typedef int32_t int32; typedef uint32_t uint32; typedef int64_t int64; typedef uint64_t uint64; #else /* C89: assume common integer types */ typedef signed char int8; typedef unsigned char uint8; typedef signed short int16; typedef unsigned short uint16; /* assume 32-bit integers (LP64, LLP64) */ typedef signed int int32; typedef unsigned int uint32; /* determine 64-bit data model */ #if defined(_WIN32) || defined(_WIN64) /* assume ILP32 or LLP64 (MSVC, MinGW) */ #define ZFP_LLP64 1 #else /* assume LP64 (Linux, macOS, ...) */ #define ZFP_LP64 1 #endif /* concatenation for literal suffixes */ #define _zfp_cat_(x, y) x ## y #define _zfp_cat(x, y) _zfp_cat_(x, y) /* signed 64-bit integers */ #if defined(ZFP_INT64) && defined(ZFP_INT64_SUFFIX) #define INT64C(x) _zfp_cat(x, ZFP_INT64_SUFFIX) #define INT64PRId #ZFP_INT64_SUFFIX "d" #define INT64PRIi #ZFP_INT64_SUFFIX "i" typedef ZFP_INT64 int64; #elif ZFP_LP64 #define INT64C(x) x ## l #define INT64PRId "ld" #define INT64PRIi "li" typedef signed long int64; #elif ZFP_LLP64 #define INT64C(x) x ## ll #define INT64PRId "lld" #define INT64PRIi "lli" typedef signed long long int64; #else #error "unknown 64-bit signed integer type" #endif #define INT64SCNd INT64PRId #define INT64SCNi INT64PRIi /* unsigned 64-bit integers */ #if defined(ZFP_UINT64) && defined(ZFP_UINT64_SUFFIX) #define UINT64C(x) _zfp_cat(x, ZFP_UINT64_SUFFIX) #ifdef ZFP_INT64_SUFFIX #define UINT64PRIo #ZFP_INT64_SUFFIX "o" #define UINT64PRIu #ZFP_INT64_SUFFIX "u" #define UINT64PRIx #ZFP_INT64_SUFFIX "x" #endif typedef ZFP_UINT64 uint64; #elif ZFP_LP64 #define UINT64C(x) x ## ul #define UINT64PRIo "lo" #define UINT64PRIu "lu" #define UINT64PRIx "lx" typedef unsigned long uint64; #elif ZFP_LLP64 #define UINT64C(x) x ## ull #define UINT64PRIo "llo" #define UINT64PRIu "llu" #define UINT64PRIx "llx" typedef unsigned long long uint64; #else #error "unknown 64-bit unsigned integer type" #endif #define UINT64SCNo UINT64PRIo #define UINT64SCNu UINT64PRIu #define UINT64SCNx UINT64PRIx #endif #endif zmat-0.9.9/src/blosc2/plugins/codecs/zfp/test_zfp_rate_float.c0000644000175200007730000001643314515254731024643 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. Test program demonstrating use of the Blosc codec from C code. To compile this program: $ gcc -O test_zfp_rate_float.c -o test_zfp_rate_float -lblosc2 **********************************************************************/ #include #include "blosc2.h" #include "blosc2/codecs-registry.h" #include "blosc-private.h" #include static int test_zfp_rate_float(blosc2_schunk* schunk) { if (schunk->typesize != 4) { printf("Error: This test is only for doubles.\n"); return 0; } int64_t nchunks = schunk->nchunks; int32_t chunksize = (int32_t) (schunk->chunksize); float *data_in = malloc(chunksize); int decompressed; int64_t csize; int64_t dsize; int64_t csize_f = 0; uint8_t *data_out = malloc(chunksize + BLOSC2_MAX_OVERHEAD); float *data_dest = malloc(chunksize); /* Create a context for compression */ int8_t zfp_rate = 37; blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = schunk->typesize; cparams.compcode = BLOSC_CODEC_ZFP_FIXED_RATE; cparams.compcode_meta = zfp_rate; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_NOFILTER; cparams.clevel = 5; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; dparams.schunk = schunk; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress with clevel=5 and shuffle active */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, data_out, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } csize_f += csize; /* Decompress */ dsize = blosc2_decompress_ctx(dctx, data_out, chunksize + BLOSC2_MAX_OVERHEAD, data_dest, chunksize); if (dsize <= 0) { printf("Decompression error. Error code: %" PRId64 "\n", dsize); return (int) dsize; } } csize_f = csize_f / nchunks; free(data_in); free(data_out); free(data_dest); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); printf("Successful roundtrip!\n"); printf("Compression: %d -> %" PRId64 " (%.1fx)\n", chunksize, csize_f, (1. * chunksize) / (double) csize_f); return (int) (chunksize - csize_f); } static int test_zfp_rate_double(blosc2_schunk* schunk) { if (schunk->typesize != 8) { printf("Error: This test is only for doubles.\n"); return 0; } int64_t nchunks = schunk->nchunks; int32_t chunksize = (int32_t) (schunk->chunksize); double *data_in = malloc(chunksize); int decompressed; int64_t csize; int64_t dsize; int64_t csize_f = 0; uint8_t *data_out = malloc(chunksize + BLOSC2_MAX_OVERHEAD); double *data_dest = malloc(chunksize); /* Create a context for compression */ int zfp_rate = 37; blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = schunk->typesize; cparams.compcode = BLOSC_CODEC_ZFP_FIXED_RATE; cparams.compcode_meta = zfp_rate; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_NOFILTER; cparams.clevel = 5; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; dparams.schunk = schunk; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress with clevel=5 and shuffle active */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, data_out, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } csize_f += csize; /* Decompress */ dsize = blosc2_decompress_ctx(dctx, data_out, chunksize + BLOSC2_MAX_OVERHEAD, data_dest, chunksize); if (dsize <= 0) { printf("Decompression error. Error code: %" PRId64 "\n", dsize); return (int) dsize; } } csize_f = csize_f / nchunks; free(data_in); free(data_out); free(data_dest); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); printf("Successful roundtrip!\n"); printf("Compression: %d -> %" PRId64 " (%.1fx)\n", chunksize, csize_f, (1. * chunksize) / (double) csize_f); return (int) (chunksize - csize_f); } int float_cyclic() { blosc2_schunk *schunk = blosc2_schunk_open("example_float_cyclic.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_rate_float(schunk); blosc2_schunk_free(schunk); return result; } int double_same_cells() { blosc2_schunk *schunk = blosc2_schunk_open("example_double_same_cells.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_rate_double(schunk); blosc2_schunk_free(schunk); return result; } int day_month_temp() { blosc2_schunk *schunk = blosc2_schunk_open("example_day_month_temp.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_rate_float(schunk); blosc2_schunk_free(schunk); return result; } int item_prices() { blosc2_schunk *schunk = blosc2_schunk_open("example_item_prices.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_rate_float(schunk); blosc2_schunk_free(schunk); return result; } int main(void) { int result; blosc2_init(); // this is mandatory for initiallizing the plugin mechanism result = float_cyclic(); printf("float_cyclic: %d obtained \n \n", result); result = double_same_cells(); printf("double_same_cells: %d obtained \n \n", result); result = day_month_temp(); printf("day_month_temp: %d obtained \n \n", result); result = item_prices(); printf("item_prices: %d obtained \n \n", result); blosc2_destroy(); return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/README.md0000644000175200007730000001055514515254731021717 0ustar rlaboissrlaboissZFP: a multidimensional lossy codec ============================================================================= *ZFP* is a codec for lossy data compression, designed for being mainly used in multidimensional (up to 4-dim) floating point datasets. Plugin motivation -------------------- A lossy codec like ZFP allows for much better compression ratios at the expense of losing some precision in floating point data. For a discussion on how it works and specially, how it performs, see our blog at: https://www.blosc.org/posts/support-lossy-zfp/. Plugin usage ------------------- The codec consists of different encoders to codify data and decoders to recover the original data, located at `blosc2-zfp.c`. This plugin only works with 1 to 4-dim datasets of floats or doubles, so if one tries to work with another type of dataset, it will return an error value. Also, the blocksize requires to be a minimum of 4 items for 1 dim, 4x4 for 2 dim, 4x4x4 for 3 dim and 4x4x4x4 for 4 dim. Finally, the ZFP codecs do interpret the *values*, so they are meant to be used *without any shuffle filter* (that would break byte/bit ordering, and hence, changing the values before they would reach the ZFP codec). The parameters used by *ZFP* are the ones specified in the `blosc2_codec` structure in the `blosc2.h` header. Furthermore, *ZFP* allows to work in three modes, BLOSC_CODEC_ZFP_FIXED_ACCURACY, BLOSC_CODEC_ZFP_FIXED_PRECISION and BLOSC_CODEC_ZFP_FIXED_RATE, each of one can be fine-tuned via the `meta` parameter as follows: - BLOSC_CODEC_ZFP_FIXED_ACCURACY: `meta` is used as absolute error in truncation. For example, if meta = -2, each value loss must be less than or equal to 10^(-2) = 0,01. Then, if 23,0567 is a value of the original input, after compressing and decompressing this input with meta = -2, this value must be between 23,0467 and 23,0667. As `meta` is unsigned in the codec plugin interface, if user does not want the compiler to show a warning, a cast to a `uint8_t` must be done before calling the plugin. - BLOSC_CODEC_ZFP_FIXED_PRECISION: `meta` must be a number between 1 and ZFP_MAX_PREC. It is called *precision* and represents the maximum number of bit planes encoded. This is, for each input value, the number of most significant bits that will be encoded. For more info, see: https://zfp.readthedocs.io/en/latest/faq.html#q-relerr - BLOSC_CODEC_ZFP_FIXED_RATE: `meta` must be a number between 1 and 100. It is called *ratio* and represents the size that the compressed cells must have based on the input cell size. For example, if the cell size is 2000 bytes and meta = 50, the output cell size will be 50% of 2000 = 1000 bytes. To choose one of the compression modes, user must use its corresponding identifier as `blosc2_cparams.compcode`. For example: cparams.compcode = BLOSC_CODEC_ZFP_FIXED_ACCURACY; or: cparams.compcode = BLOSC_CODEC_ZFP_FIXED_PRECISION; or: cparams.compcode = BLOSC_CODEC_ZFP_FIXED_RATE; Remember that all available codec IDs (including ZFP modes) are defined at: https://github.com/Blosc/c-blosc2/blob/main/include/blosc2/codecs-registry.h How ZFP works ------------------- In order to compress n-dimensional arrays of floating-point data, ZFP partitions them into cells of size 4^n so; for example, in a 3-dim dataset, the cellshape will be 4x4x4. Then, depending on the compression mode, each block is compressed or decompressed individually into a fixed- or variable-length bit string, and these bit strings are concatenated into a single stream of bits. For more info you can see ZFP official documentation: https://zfp.readthedocs.io/en/latest/ And the official repo: https://github.com/LLNL/zfp Advantages and disadvantages ------------------------------ The main advantage of *ZFP* when compared with others is that *ZFP* makes use of the multidimensionality of datasets and takes advantage of this instead of processing all data as serial. For example, the difference between *ZFP* and *NDLZ* is that *ZFP* codec uses lossy data compression (which implies better ratios) and lets user to work with floating-point datasets. Furthermore, *ZFP* implements different compression modes that brings interesting new possibilities. The main disadvantage of *ZFP* is that it is mainly useful for 2 to 4-dim datasets only. It can also be used for 1-dim datasets, but then it loses most of its advantages (i.e. using spacial locality to better reduce data entropy). zmat-0.9.9/src/blosc2/plugins/codecs/zfp/test_zfp_prec_float.c0000644000175200007730000002146514515254731024642 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. Test program demonstrating use of the Blosc codec from C code. To compile this program: $ gcc -O test_zfp_prec_float.c -o test_zfp_prec_float -lblosc2 **********************************************************************/ #include #include #include "blosc2.h" #include "blosc2/codecs-registry.h" #include "blosc-private.h" #include static int test_zfp_prec_float(blosc2_schunk* schunk) { if (schunk->typesize != 4) { printf("Error: This test is only for doubles.\n"); return 0; } int64_t nchunks = schunk->nchunks; int32_t chunksize = (int32_t) (schunk->chunksize); float *data_in = malloc(chunksize); int decompressed; int64_t csize; int64_t dsize; int64_t csize_f = 0; uint8_t *data_out = malloc(chunksize + BLOSC2_MAX_OVERHEAD); float *data_dest = malloc(chunksize); /* Create a context for compression */ int8_t zfp_prec = 25; blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = schunk->typesize; cparams.compcode = BLOSC_CODEC_ZFP_FIXED_PRECISION; cparams.compcode_meta = zfp_prec; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_NOFILTER; cparams.clevel = 5; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; dparams.schunk = schunk; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress with clevel=5 and shuffle active */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, data_out, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } csize_f += csize; /* Decompress */ dsize = blosc2_decompress_ctx(dctx, data_out, chunksize + BLOSC2_MAX_OVERHEAD, data_dest, chunksize); if (dsize <= 0) { printf("Decompression error. Error code: %" PRId64 "\n", dsize); return (int) dsize; } double tolerance = 0.01; for (int i = 0; i < (chunksize / cparams.typesize); i++) { if ((data_in[i] == 0) || (data_dest[i] == 0)) { if (fabsf(data_in[i] - data_dest[i]) > tolerance) { printf("i: %d, data %.8f, dest %.8f", i, data_in[i], data_dest[i]); printf("\n Decompressed data differs from original!\n"); return -1; } } else if (fabsf(data_in[i] - data_dest[i]) > tolerance * fmaxf(fabsf(data_in[i]), fabsf(data_dest[i]))) { printf("i: %d, data %.8f, dest %.8f", i, data_in[i], data_dest[i]); printf("\n Decompressed data differs from original!\n"); return -1; } } } csize_f = csize_f / nchunks; free(data_in); free(data_out); free(data_dest); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); printf("Successful roundtrip!\n"); printf("Compression: %d -> %" PRId64 " (%.1fx)\n", chunksize, csize_f, (1. * chunksize) / (double) csize_f); return (int) (chunksize - csize_f); } static int test_zfp_prec_double(blosc2_schunk* schunk) { if (schunk->typesize != 8) { printf("Error: This test is only for doubles.\n"); return 0; } int64_t nchunks = schunk->nchunks; int32_t chunksize = (int32_t) (schunk->chunksize); double *data_in = malloc(chunksize); int decompressed; int64_t csize; int64_t dsize; int64_t csize_f = 0; uint8_t *data_out = malloc(chunksize + BLOSC2_MAX_OVERHEAD); double *data_dest = malloc(chunksize); /* Create a context for compression */ int zfp_prec = 25; blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = schunk->typesize; cparams.compcode = BLOSC_CODEC_ZFP_FIXED_PRECISION; cparams.compcode_meta = zfp_prec; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_NOFILTER; cparams.clevel = 5; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; dparams.schunk = schunk; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress with clevel=5 and shuffle active */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, data_out, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } csize_f += csize; /* Decompress */ dsize = blosc2_decompress_ctx(dctx, data_out, chunksize + BLOSC2_MAX_OVERHEAD, data_dest, chunksize); if (dsize <= 0) { printf("Decompression error. Error code: %" PRId64 "\n", dsize); return (int) dsize; } double tolerance = 0.01; for (int i = 0; i < (chunksize / cparams.typesize); i++) { if ((data_in[i] == 0) || (data_dest[i] == 0)) { if (fabs(data_in[i] - data_dest[i]) > tolerance) { printf("i: %d, data %.16f, dest %.16f", i, data_in[i], data_dest[i]); printf("\n Decompressed data differs from original!\n"); return -1; } } else if (fabs(data_in[i] - data_dest[i]) > tolerance * fmax(fabs(data_in[i]), fabs(data_dest[i]))) { printf("i: %d, data %.16f, dest %.16f", i, data_in[i], data_dest[i]); printf("\n Decompressed data differs from original!\n"); return -1; } } } csize_f = csize_f / nchunks; free(data_in); free(data_out); free(data_dest); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); printf("Successful roundtrip!\n"); printf("Compression: %d -> %" PRId64 " (%.1fx)\n", chunksize, csize_f, (1. * chunksize) / (double) csize_f); return (int) (chunksize - csize_f); } int float_cyclic() { blosc2_schunk *schunk = blosc2_schunk_open("example_float_cyclic.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_prec_float(schunk); blosc2_schunk_free(schunk); return result; } int double_same_cells() { blosc2_schunk *schunk = blosc2_schunk_open("example_double_same_cells.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_prec_double(schunk); blosc2_schunk_free(schunk); return result; } int day_month_temp() { blosc2_schunk *schunk = blosc2_schunk_open("example_day_month_temp.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_prec_float(schunk); blosc2_schunk_free(schunk); return result; } int item_prices() { blosc2_schunk *schunk = blosc2_schunk_open("example_item_prices.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_prec_float(schunk); blosc2_schunk_free(schunk); return result; } int main(void) { int result; blosc2_init(); // this is mandatory for initiallizing the plugin mechanism result = float_cyclic(); printf("float_cyclic: %d obtained \n \n", result); result = double_same_cells(); printf("double_same_cells: %d obtained \n \n", result); result = day_month_temp(); printf("day_month_temp: %d obtained \n \n", result); result = item_prices(); printf("item_prices: %d obtained \n \n", result); blosc2_destroy(); return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/blosc2-zfp.h0000644000175200007730000000347714515254731022577 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ #ifndef BLOSC2_ZFP_H #define BLOSC2_ZFP_H #include "zfp-private.h" #include "../plugins/plugin_utils.h" #if defined (__cplusplus) extern "C" { #endif int zfp_acc_compress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_cparams *cparams, const void* chunk); int zfp_acc_decompress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_dparams *dparams, const void* chunk); int zfp_prec_compress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_cparams *cparams, const void* chunk); int zfp_prec_decompress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_dparams *dparams, const void* chunk); int zfp_rate_compress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_cparams *cparams, const void* chunk); int zfp_rate_decompress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_dparams *dparams, const void* chunk); int zfp_getcell(void *thread_context, const uint8_t *block, int32_t cbytes, uint8_t *dest, int32_t destsize); #if defined (__cplusplus) } #endif #endif /* BLOSC2_ZFP_H */ zmat-0.9.9/src/blosc2/plugins/codecs/zfp/CMakeLists.txt0000644000175200007730000000546014515254731023177 0ustar rlaboissrlaboissget_filename_component(ZFP_LOCAL_DIR ./ ABSOLUTE) file(GLOB ZFP_SRC_FILES ${ZFP_LOCAL_DIR}/blosc2-zfp.* ${ZFP_LOCAL_DIR}/zfp-private.h ${ZFP_LOCAL_DIR}/src/zfp.c ${ZFP_LOCAL_DIR}/src/bitstream.c ${ZFP_LOCAL_DIR}/src/decode*.c ${ZFP_LOCAL_DIR}/src/encode*.c) #message(status "--------------!!!!zfp:" ${ZFP_SRC_FILES}) set(SOURCES ${SOURCES} ${ZFP_SRC_FILES} PARENT_SCOPE) # targets if(BUILD_TESTS) add_executable(test_zfp_acc_int test_zfp_acc_int.c) add_executable(test_zfp_acc_float test_zfp_acc_float.c) add_executable(test_zfp_prec_float test_zfp_prec_float.c) add_executable(test_zfp_rate_float test_zfp_rate_float.c) add_executable(test_zfp_rate_getitem test_zfp_rate_getitem.c) # Define the BLOSC_TESTING symbol so normally-hidden functions # aren't hidden from the view of the test programs. set_property( TARGET test_zfp_acc_int APPEND PROPERTY COMPILE_DEFINITIONS BLOSC_TESTING) set_property( TARGET test_zfp_acc_float APPEND PROPERTY COMPILE_DEFINITIONS BLOSC_TESTING) set_property( TARGET test_zfp_prec_float APPEND PROPERTY COMPILE_DEFINITIONS BLOSC_TESTING) set_property( TARGET test_zfp_rate_float APPEND PROPERTY COMPILE_DEFINITIONS BLOSC_TESTING) set_property( TARGET test_zfp_rate_getitem APPEND PROPERTY COMPILE_DEFINITIONS BLOSC_TESTING) target_link_libraries(test_zfp_acc_int blosc_testing) target_link_libraries(test_zfp_acc_float blosc_testing) target_link_libraries(test_zfp_prec_float blosc_testing) target_link_libraries(test_zfp_rate_float blosc_testing) target_link_libraries(test_zfp_rate_getitem blosc_testing) # tests add_test(NAME test_plugin_test_zfp_acc_int COMMAND ${CMAKE_CROSSCOMPILING_EMULATOR} $) add_test(NAME test_plugin_test_zfp_acc_float COMMAND ${CMAKE_CROSSCOMPILING_EMULATOR} $) add_test(NAME test_plugin_test_zfp_prec_float COMMAND ${CMAKE_CROSSCOMPILING_EMULATOR} $) add_test(NAME test_plugin_test_zfp_rate_float COMMAND ${CMAKE_CROSSCOMPILING_EMULATOR} $) add_test(NAME test_plugin_test_zfp_rate_getitem COMMAND ${CMAKE_CROSSCOMPILING_EMULATOR} $) # Copy test files file(GLOB TESTS_DATA ../../test_data/example_float_cyclic.caterva ../../test_data/example_double_same_cells.caterva ../../test_data/example_day_month_temp.caterva ../../test_data/example_item_prices.caterva ../../filters/ndmean/example_ndmean_r*.caterva) foreach (data ${TESTS_DATA}) file(COPY ${data} DESTINATION ${CMAKE_CURRENT_BINARY_DIR}/) endforeach(data) endif() zmat-0.9.9/src/blosc2/plugins/codecs/zfp/test_zfp_rate_getitem.c0000644000175200007730000002452014515254731025170 0ustar rlaboissrlaboiss/********************************************************************* Blosc - Blocked Shuffling and Compression Library Copyright (C) 2021 The Blosc Developers https://blosc.org License: BSD 3-Clause (see LICENSE.txt) See LICENSE.txt for details about copyright and rights to use. Test program demonstrating use of the Blosc codec from C code. To compile this program: $ gcc -O test_zfp_rate_getitem.c -o test_zfp_rate_getitem -lblosc2 **********************************************************************/ #include #include #include #include "blosc2.h" #include "blosc2/codecs-registry.h" #include "blosc-private.h" static int test_zfp_rate_getitem_float(blosc2_schunk* schunk) { if (schunk->typesize != 4) { printf("Error: This test is only for doubles.\n"); return 0; } int64_t nchunks = schunk->nchunks; int32_t chunksize = (int32_t) (schunk->chunksize); float *data_in = malloc(chunksize); int decompressed; int64_t csize; uint8_t *chunk_zfp = malloc(chunksize + BLOSC2_MAX_OVERHEAD); uint8_t *chunk_blosc = malloc(chunksize + BLOSC2_MAX_OVERHEAD); float *data_dest = malloc(chunksize); /* Create a context for compression */ int8_t zfp_rate = 37; blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = schunk->typesize; cparams.compcode = BLOSC_CODEC_ZFP_FIXED_RATE; cparams.compcode_meta = zfp_rate; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_NOFILTER; cparams.clevel = 5; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; dparams.schunk = schunk; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); int32_t zfp_chunk_nbytes, zfp_chunk_cbytes, blosc_chunk_cbytes; uint8_t *lossy_chunk = malloc(chunksize + BLOSC2_MAX_OVERHEAD); for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress using ZFP fixed-rate */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, chunk_zfp, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } blosc2_cbuffer_sizes(chunk_zfp, &zfp_chunk_nbytes, &zfp_chunk_cbytes, NULL); decompressed = blosc2_decompress_ctx(dctx, chunk_zfp, zfp_chunk_cbytes, lossy_chunk, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress not using ZFP fixed-rate */ csize = blosc2_compress_ctx(schunk->cctx, lossy_chunk, chunksize, chunk_blosc, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } blosc2_cbuffer_sizes(chunk_blosc, NULL, &blosc_chunk_cbytes, NULL); /* Get item */ int index, dsize_zfp, dsize_blosc; float item_zfp, item_blosc; int nelems = schunk->chunksize / schunk->typesize; for (int i = 0; i < 100; ++i) { srand(i); index = rand() % nelems; // Usual getitem dsize_blosc = blosc2_getitem_ctx(schunk->dctx, chunk_blosc, blosc_chunk_cbytes, index, 1, &item_blosc, sizeof(item_blosc)); // Optimized getitem using ZFP cell machinery dsize_zfp = blosc2_getitem_ctx(dctx, chunk_zfp, zfp_chunk_cbytes, index, 1, &item_zfp, sizeof(item_zfp)); if (dsize_blosc != dsize_zfp) { printf("Different amount of items gotten\n"); return -1; } if (item_blosc != item_zfp) { printf("\nIn index %d different items extracted zfp %f blosc %f\n", index, item_zfp, item_blosc); return -1; } } } free(data_in); free(data_dest); free(chunk_zfp); free(chunk_blosc); free(lossy_chunk); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); printf("Successful roundtrip!\n"); return (int) (BLOSC2_ERROR_SUCCESS); } static int test_zfp_rate_getitem_double(blosc2_schunk* schunk) { if (schunk->typesize != 8) { printf("Error: This test is only for doubles.\n"); return 0; } int64_t nchunks = schunk->nchunks; int32_t chunksize = (int32_t) (schunk->chunksize); double *data_in = malloc(chunksize); int decompressed; int64_t csize; uint8_t *chunk_zfp = malloc(chunksize + BLOSC2_MAX_OVERHEAD); uint8_t *chunk_blosc = malloc(chunksize + BLOSC2_MAX_OVERHEAD); double *data_dest = malloc(chunksize); /* Create a context for compression */ int zfp_rate = 37; blosc2_cparams cparams = BLOSC2_CPARAMS_DEFAULTS; cparams.splitmode = BLOSC_NEVER_SPLIT; cparams.typesize = schunk->typesize; cparams.compcode = BLOSC_CODEC_ZFP_FIXED_RATE; cparams.compcode_meta = zfp_rate; cparams.filters[BLOSC2_MAX_FILTERS - 1] = BLOSC_NOFILTER; cparams.clevel = 5; cparams.nthreads = 1; cparams.blocksize = schunk->blocksize; cparams.schunk = schunk; blosc2_context *cctx; cctx = blosc2_create_cctx(cparams); blosc2_dparams dparams = BLOSC2_DPARAMS_DEFAULTS; dparams.nthreads = 1; dparams.schunk = schunk; blosc2_context *dctx; dctx = blosc2_create_dctx(dparams); int32_t zfp_chunk_nbytes, zfp_chunk_cbytes, blosc_chunk_cbytes; uint8_t *lossy_chunk = malloc(chunksize + BLOSC2_MAX_OVERHEAD); for (int ci = 0; ci < nchunks; ci++) { decompressed = blosc2_schunk_decompress_chunk(schunk, ci, data_in, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress using ZFP fixed-rate */ csize = blosc2_compress_ctx(cctx, data_in, chunksize, chunk_zfp, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } blosc2_cbuffer_sizes(chunk_zfp, &zfp_chunk_nbytes, &zfp_chunk_cbytes, NULL); decompressed = blosc2_decompress_ctx(dctx, chunk_zfp, zfp_chunk_cbytes, lossy_chunk, chunksize); if (decompressed < 0) { printf("Error decompressing chunk \n"); return -1; } /* Compress not using ZFP fixed-rate */ csize = blosc2_compress_ctx(schunk->cctx, lossy_chunk, chunksize, chunk_blosc, chunksize + BLOSC2_MAX_OVERHEAD); if (csize == 0) { printf("Buffer is uncompressible. Giving up.\n"); return 0; } else if (csize < 0) { printf("Compression error. Error code: %" PRId64 "\n", csize); return (int) csize; } blosc2_cbuffer_sizes(chunk_blosc, NULL, &blosc_chunk_cbytes, NULL); /* Get item */ int index, dsize_zfp, dsize_blosc; double item_zfp, item_blosc; int nelems = schunk->chunksize / schunk->typesize; for (int i = 0; i < 100; ++i) { srand(i); index = rand() % nelems; // Usual getitem dsize_blosc = blosc2_getitem_ctx(schunk->dctx, chunk_blosc, blosc_chunk_cbytes, index, 1, &item_blosc, sizeof(item_blosc)); // Optimized getitem using ZFP cell machinery dsize_zfp = blosc2_getitem_ctx(dctx, chunk_zfp, zfp_chunk_cbytes, index, 1, &item_zfp, sizeof(item_zfp)); if (dsize_blosc != dsize_zfp) { printf("Different amount of items gotten\n"); return -1; } if (item_blosc != item_zfp) { printf("\nIn index %d different items extracted zfp %f blosc %f\n", index, item_zfp, item_blosc); return -1; } } } free(data_in); free(data_dest); free(chunk_zfp); free(chunk_blosc); free(lossy_chunk); blosc2_free_ctx(cctx); blosc2_free_ctx(dctx); printf("Successful roundtrip!\n"); return (int) (BLOSC2_ERROR_SUCCESS); } int float_cyclic() { blosc2_schunk *schunk = blosc2_schunk_open("example_float_cyclic.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_rate_getitem_float(schunk); blosc2_schunk_free(schunk); return result; } int double_same_cells() { blosc2_schunk *schunk = blosc2_schunk_open("example_double_same_cells.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_rate_getitem_double(schunk); blosc2_schunk_free(schunk); return result; } int day_month_temp() { blosc2_schunk *schunk = blosc2_schunk_open("example_day_month_temp.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_rate_getitem_float(schunk); blosc2_schunk_free(schunk); return result; } int item_prices() { blosc2_schunk *schunk = blosc2_schunk_open("example_item_prices.caterva"); BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_FILE_OPEN); /* Run the test. */ int result = test_zfp_rate_getitem_float(schunk); blosc2_schunk_free(schunk); return result; } int main(void) { blosc2_init(); // this is mandatory for initiallizing the plugin mechanism printf("float_cyclic: "); float_cyclic(); printf("double_same_cells: "); double_same_cells(); printf("day_month_temp: "); day_month_temp(); printf("item_prices: "); item_prices(); blosc2_destroy(); return BLOSC2_ERROR_SUCCESS; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/0000755000175200007730000000000014515254731021221 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode1l.c0000644000175200007730000000063114515254731023057 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block1.h" #include "traitsl.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec1.c" #include "template/encode.c" #include "template/encodei.c" #include "template/encode1.c" #include "template/revencode.c" #include "template/revencode1.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/inline/0000755000175200007730000000000014515254731022477 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/inline/bitstream.c0000644000175200007730000003123514515254731024641 0ustar rlaboissrlaboiss/* High-speed in-memory bit stream I/O that supports reading and writing between 0 and 64 bits at a time. The implementation, which relies heavily on bit shifts, has been carefully written to ensure that all shifts are between zero and one less the width of the type being shifted to avoid undefined behavior. This occasionally causes somewhat convoluted code. The following assumptions and restrictions apply: 1. The user must allocate a memory buffer large enough to hold the bit stream, whether for reading, writing, or both. This buffer is associated with the bit stream via stream_open(buffer, bytes), which allocates and returns a pointer to an opaque bit stream struct. Call stream_close(stream) to deallocate this struct. 2. The stream is either in a read or write state (or, initially, in both states). When done writing, call stream_flush(stream) before entering read mode to ensure any buffered bits are output. To enter read mode, call stream_rewind(stream) or stream_rseek(stream, offset) to position the stream at the beginning or at a particular bit offset. Conversely, stream_rewind(stream) or stream_wseek(stream, offset) positions the stream for writing. In read mode, the following functions may be called: size_t stream_size(stream); size_t stream_rtell(stream); void stream_rewind(stream); void stream_rseek(stream, offset); void stream_skip(stream, uint n); size_t stream_align(stream); uint stream_read_bit(stream); uint64 stream_read_bits(stream, n); Each of the above read calls has a corresponding write call: size_t stream_size(stream); size_t stream_wtell(stream); void stream_rewind(stream); void stream_wseek(stream, offset); void stream_pad(stream, n); size_t stream_flush(stream); uint stream_write_bit(stream, bit); uint64 stream_write_bits(stream, value, n); 3. The stream buffer is an unsigned integer of a user-specified type given by the BIT_STREAM_WORD_TYPE macro. Bits are read and written in units of this integer word type. Supported types are 8, 16, 32, or 64 bits wide. The bit width of the buffer is denoted by 'wsize' and can be accessed via the global constant stream_word_bits. A small wsize allows for fine granularity reads and writes, and may be preferable when working with many small blocks of data that require non-sequential access. The default maximum size of 64 bits ensures maximum speed. Note that even when wsize < 64, it is still possible to read and write up to 64 bits at a time using stream_read_bits() and stream_write_bits(). 4. If BIT_STREAM_STRIDED is defined, words read from or written to the stream may be accessed noncontiguously by setting a power-of-two block size (which by default is one word) and a block stride (defaults to zero blocks). The word pointer is always incremented by one word each time a word is accessed. Once advanced past a block boundary, the word pointer is also advanced by the stride to the next block. This feature may be used to store blocks of data interleaved, e.g. for progressive coding or for noncontiguous parallel access to the bit stream Note that the block size is measured in words, while the stride is measured in multiples of the block size. Strided access can have a significant performance penalty. 5. Multiple bits are read and written in order of least to most significant bit. Thus, the statement value = stream_write_bits(stream, value, n); is essentially equivalent to (but faster than) for (i = 0; i < n; i++, value >>= 1) stream_write_bit(value & 1); when 0 <= n <= 64. The same holds for read calls, and thus value = stream_read_bits(stream, n); is essentially equivalent to for (i = 0, value = 0; i < n; i++) value += (uint64)stream_read_bit() << i; Note that it is possible to write fewer bits than the argument 'value' holds (possibly even no bits), in which case any unwritten bits are returned. 6. Although the stream_wseek(stream, offset) call allows positioning the stream for writing at any bit offset without any data loss (i.e. all previously written bits preceding the offset remain valid), for efficiency the stream_flush(stream) operation will zero all bits up to the next multiple of wsize bits, thus overwriting bits that were previously stored at that location. Consequently, random write access is effectively supported only at wsize granularity. For sequential access, the largest possible wsize is preferred due to higher speed. 7. It is up to the user to adhere to these rules. For performance reasons, no error checking is done, and in particular buffer overruns are not caught. */ #include #include #ifndef inline_ #define inline_ #endif /* satisfy compiler when args unused */ #define unused_(x) ((void)(x)) /* bit stream word/buffer type; granularity of stream I/O operations */ #ifdef BIT_STREAM_WORD_TYPE /* may be 8-, 16-, 32-, or 64-bit unsigned integer type */ typedef BIT_STREAM_WORD_TYPE word; #else /* use maximum word size by default for highest speed */ typedef uint64 word; #endif /* number of bits in a buffered word */ #define wsize ((uint)(CHAR_BIT * sizeof(word))) /* bit stream structure (opaque to caller) */ struct bitstream { uint bits; /* number of buffered bits (0 <= bits < wsize) */ word buffer; /* buffer for incoming/outgoing bits (buffer < 2^bits) */ word* ptr; /* pointer to next word to be read/written */ word* begin; /* beginning of stream */ word* end; /* end of stream (currently unused) */ #ifdef BIT_STREAM_STRIDED size_t mask; /* one less the block size in number of words */ ptrdiff_t delta; /* number of words between consecutive blocks */ #endif }; /* private functions ------------------------------------------------------- */ /* read a single word from memory */ static word stream_read_word(bitstream* s) { word w = *s->ptr++; #ifdef BIT_STREAM_STRIDED if (!((s->ptr - s->begin) & s->mask)) s->ptr += s->delta; #endif return w; } /* write a single word to memory */ static void stream_write_word(bitstream* s, word value) { *s->ptr++ = value; #ifdef BIT_STREAM_STRIDED if (!((s->ptr - s->begin) & s->mask)) s->ptr += s->delta; #endif } /* public functions -------------------------------------------------------- */ /* word size in bits (equals stream_word_bits) */ inline_ size_t stream_alignment(void) { return wsize; } /* pointer to beginning of stream */ inline_ void* stream_data(const bitstream* s) { return s->begin; } /* current byte size of stream (if flushed) */ inline_ size_t stream_size(const bitstream* s) { return sizeof(word) * (size_t)(s->ptr - s->begin); } /* byte capacity of stream */ inline_ size_t stream_capacity(const bitstream* s) { return sizeof(word) * (size_t)(s->end - s->begin); } /* number of words per block */ inline_ size_t stream_stride_block(const bitstream* s) { #ifdef BIT_STREAM_STRIDED return s->mask + 1; #else unused_(s); return 1; #endif } /* number of blocks between consecutive stream blocks */ inline_ ptrdiff_t stream_stride_delta(const bitstream* s) { #ifdef BIT_STREAM_STRIDED return s->delta / (s->mask + 1); #else unused_(s); return 0; #endif } /* read single bit (0 or 1) */ inline_ uint stream_read_bit(bitstream* s) { uint bit; if (!s->bits) { s->buffer = stream_read_word(s); s->bits = wsize; } s->bits--; bit = (uint)s->buffer & 1u; s->buffer >>= 1; return bit; } /* write single bit (must be 0 or 1) */ inline_ uint stream_write_bit(bitstream* s, uint bit) { s->buffer += (word)bit << s->bits; if (++s->bits == wsize) { stream_write_word(s, s->buffer); s->buffer = 0; s->bits = 0; } return bit; } /* read 0 <= n <= 64 bits */ inline_ uint64 stream_read_bits(bitstream* s, uint n) { uint64 value = s->buffer; if (s->bits < n) { /* keep fetching wsize bits until enough bits are buffered */ do { /* assert: 0 <= s->bits < n <= 64 */ s->buffer = stream_read_word(s); value += (uint64)s->buffer << s->bits; s->bits += wsize; } while (sizeof(s->buffer) < sizeof(value) && s->bits < n); /* assert: 1 <= n <= s->bits < n + wsize */ s->bits -= n; if (!s->bits) { /* value holds exactly n bits; no need for masking */ s->buffer = 0; } else { /* assert: 1 <= s->bits < wsize */ s->buffer >>= wsize - s->bits; /* assert: 1 <= n <= 64 */ value &= ((uint64)2 << (n - 1)) - 1; } } else { /* assert: 0 <= n <= s->bits < wsize <= 64 */ s->bits -= n; s->buffer >>= n; value &= ((uint64)1 << n) - 1; } return value; } /* write 0 <= n <= 64 low bits of value and return remaining bits */ inline_ uint64 stream_write_bits(bitstream* s, uint64 value, uint n) { /* append bit string to buffer */ s->buffer += (word)(value << s->bits); s->bits += n; /* is buffer full? */ if (s->bits >= wsize) { /* 1 <= n <= 64; decrement n to ensure valid right shifts below */ value >>= 1; n--; /* assert: 0 <= n < 64; wsize <= s->bits <= wsize + n */ do { /* output wsize bits while buffer is full */ s->bits -= wsize; /* assert: 0 <= s->bits <= n */ stream_write_word(s, s->buffer); /* assert: 0 <= n - s->bits < 64 */ s->buffer = (word)(value >> (n - s->bits)); } while (sizeof(s->buffer) < sizeof(value) && s->bits >= wsize); } /* assert: 0 <= s->bits < wsize */ s->buffer &= ((word)1 << s->bits) - 1; /* assert: 0 <= n < 64 */ return value >> n; } /* return bit offset to next bit to be read */ inline_ size_t stream_rtell(const bitstream* s) { return wsize * (size_t)(s->ptr - s->begin) - s->bits; } /* return bit offset to next bit to be written */ inline_ size_t stream_wtell(const bitstream* s) { return wsize * (size_t)(s->ptr - s->begin) + s->bits; } /* position stream for reading or writing at beginning */ inline_ void stream_rewind(bitstream* s) { s->ptr = s->begin; s->buffer = 0; s->bits = 0; } /* position stream for reading at given bit offset */ inline_ void stream_rseek(bitstream* s, size_t offset) { uint n = offset % wsize; s->ptr = s->begin + offset / wsize; if (n) { s->buffer = stream_read_word(s) >> n; s->bits = wsize - n; } else { s->buffer = 0; s->bits = 0; } } /* position stream for writing at given bit offset */ inline_ void stream_wseek(bitstream* s, size_t offset) { uint n = offset % wsize; s->ptr = s->begin + offset / wsize; if (n) { word buffer = *s->ptr; buffer &= ((word)1 << n) - 1; s->buffer = buffer; s->bits = n; } else { s->buffer = 0; s->bits = 0; } } /* skip over the next n bits (n >= 0) */ inline_ void stream_skip(bitstream* s, uint n) { stream_rseek(s, stream_rtell(s) + n); } /* append n zero-bits to stream (n >= 0) */ inline_ void stream_pad(bitstream* s, uint n) { for (s->bits += n; s->bits >= wsize; s->bits -= wsize) { stream_write_word(s, s->buffer); s->buffer = 0; } } /* align stream on next word boundary */ inline_ size_t stream_align(bitstream* s) { uint bits = s->bits; if (bits) stream_skip(s, bits); return bits; } /* write any remaining buffered bits and align stream on next word boundary */ inline_ size_t stream_flush(bitstream* s) { uint bits = (wsize - s->bits) % wsize; if (bits) stream_pad(s, bits); return bits; } /* copy n bits from one bit stream to another */ inline_ void stream_copy(bitstream* dst, bitstream* src, size_t n) { while (n > wsize) { word w = (word)stream_read_bits(src, wsize); stream_write_bits(dst, w, wsize); n -= wsize; } if (n) { word w = (word)stream_read_bits(src, (uint)n); stream_write_bits(dst, w, (uint)n); } } #ifdef BIT_STREAM_STRIDED /* set block size in number of words and spacing in number of blocks */ inline_ int stream_set_stride(bitstream* s, size_t block, ptrdiff_t delta) { /* ensure block size is a power of two */ if (block & (block - 1)) return 0; s->mask = block - 1; s->delta = delta * block; return 1; } #endif /* allocate and initialize bit stream to user-allocated buffer */ inline_ bitstream* stream_open(void* buffer, size_t bytes) { bitstream* s = (bitstream*)malloc(sizeof(bitstream)); if (s) { s->begin = (word*)buffer; s->end = s->begin + bytes / sizeof(word); #ifdef BIT_STREAM_STRIDED stream_set_stride(s, 0, 0); #endif stream_rewind(s); } return s; } /* close and deallocate bit stream */ inline_ void stream_close(bitstream* s) { free(s); } /* make a copy of bit stream to shared memory buffer */ inline_ bitstream* stream_clone(const bitstream* s) { bitstream* c = (bitstream*)malloc(sizeof(bitstream)); if (c) *c = *s; return c; } #undef unused_ zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/inline/inline.h0000644000175200007730000000026214515254731024126 0ustar rlaboissrlaboiss#ifndef INLINE_H #define INLINE_H #ifndef inline_ #if __STDC_VERSION__ >= 199901L #define inline_ static inline #else #define inline_ static #endif #endif #endif zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode3l.c0000644000175200007730000000063114515254731023047 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block3.h" #include "traitsl.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec3.c" #include "template/decode.c" #include "template/decodei.c" #include "template/decode3.c" #include "template/revdecode.c" #include "template/revdecode3.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode2d.c0000644000175200007730000000076714515254731023062 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block2.h" #include "traitsd.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec2.c" #include "template/encode.c" #include "template/encodef.c" #include "template/encode2.c" #include "template/revcodecf.c" #include "template/revencode.c" #include "template/revencodef.c" #include "template/revencode2.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode3i.c0000644000175200007730000000063114515254731023056 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block3.h" #include "traitsi.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec3.c" #include "template/encode.c" #include "template/encodei.c" #include "template/encode3.c" #include "template/revencode.c" #include "template/revencode3.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/block4.h0000644000175200007730000000001714515254731022546 0ustar rlaboissrlaboiss#define DIMS 4 zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode1d.c0000644000175200007730000000076714515254731023061 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block1.h" #include "traitsd.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec1.c" #include "template/encode.c" #include "template/encodef.c" #include "template/encode1.c" #include "template/revcodecf.c" #include "template/revencode.c" #include "template/revencodef.c" #include "template/revencode1.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode3f.c0000644000175200007730000000076714515254731023053 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block3.h" #include "traitsf.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec3.c" #include "template/decode.c" #include "template/decodef.c" #include "template/decode3.c" #include "template/revcodecf.c" #include "template/revdecode.c" #include "template/revdecodef.c" #include "template/revdecode3.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode4i.c0000644000175200007730000000063114515254731023045 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block4.h" #include "traitsi.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec4.c" #include "template/decode.c" #include "template/decodei.c" #include "template/decode4.c" #include "template/revdecode.c" #include "template/revdecode4.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/traitsf.h0000644000175200007730000000144114515254731023046 0ustar rlaboissrlaboiss/* single-precision floating-point traits */ #define Scalar float /* floating-point type */ #define Int int32 /* corresponding signed integer type */ #define UInt uint32 /* corresponding unsigned integer type */ #define EBITS 8 /* number of exponent bits */ #define PBITS 5 /* number of bits needed to encode precision */ #define NBMASK 0xaaaaaaaau /* negabinary mask */ #define TCMASK 0x7fffffffu /* two's complement mask */ #define SCALAR_MIN FLT_MIN /* smallest positive normal number */ #if __STDC_VERSION__ >= 199901L #define FABS(x) fabsf(x) #define FREXP(x, e) frexpf(x, e) #define LDEXP(x, e) ldexpf(x, e) #else #define FABS(x) (float)fabs(x) #define FREXP(x, e) (void)frexp(x, e) #define LDEXP(x, e) (float)ldexp(x, e) #endif zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode4f.c0000644000175200007730000000076714515254731023066 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block4.h" #include "traitsf.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec4.c" #include "template/encode.c" #include "template/encodef.c" #include "template/encode4.c" #include "template/revcodecf.c" #include "template/revencode.c" #include "template/revencodef.c" #include "template/revencode4.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode3i.c0000644000175200007730000000063114515254731023044 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block3.h" #include "traitsi.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec3.c" #include "template/decode.c" #include "template/decodei.c" #include "template/decode3.c" #include "template/revdecode.c" #include "template/revdecode3.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/bitstream.c0000644000175200007730000000013514515254731023356 0ustar rlaboissrlaboiss#include "bitstream.h" #include "inline/bitstream.c" const size_t stream_word_bits = wsize; zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode3l.c0000644000175200007730000000063114515254731023061 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block3.h" #include "traitsl.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec3.c" #include "template/encode.c" #include "template/encodei.c" #include "template/encode3.c" #include "template/revencode.c" #include "template/revencode3.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/traitsl.h0000644000175200007730000000063714515254731023062 0ustar rlaboissrlaboiss/* 64-bit integer traits */ #define Scalar int64 /* integer type */ #define Int int64 /* corresponding signed integer type */ #define UInt uint64 /* corresponding unsigned integer type */ #define PBITS 6 /* number of bits needed to encode precision */ #define NBMASK UINT64C(0xaaaaaaaaaaaaaaaa) /* negabinary mask */ zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode2f.c0000644000175200007730000000076714515254731023052 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block2.h" #include "traitsf.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec2.c" #include "template/decode.c" #include "template/decodef.c" #include "template/decode2.c" #include "template/revcodecf.c" #include "template/revdecode.c" #include "template/revdecodef.c" #include "template/revdecode2.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode4i.c0000644000175200007730000000063114515254731023057 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block4.h" #include "traitsi.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec4.c" #include "template/encode.c" #include "template/encodei.c" #include "template/encode4.c" #include "template/revencode.c" #include "template/revencode4.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode2f.c0000644000175200007730000000076714515254731023064 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block2.h" #include "traitsf.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec2.c" #include "template/encode.c" #include "template/encodef.c" #include "template/encode2.c" #include "template/revcodecf.c" #include "template/revencode.c" #include "template/revencodef.c" #include "template/revencode2.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode3d.c0000644000175200007730000000076714515254731023051 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block3.h" #include "traitsd.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec3.c" #include "template/decode.c" #include "template/decodef.c" #include "template/decode3.c" #include "template/revcodecf.c" #include "template/revdecode.c" #include "template/revdecodef.c" #include "template/revdecode3.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode1f.c0000644000175200007730000000076714515254731023063 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block1.h" #include "traitsf.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec1.c" #include "template/encode.c" #include "template/encodef.c" #include "template/encode1.c" #include "template/revcodecf.c" #include "template/revencode.c" #include "template/revencodef.c" #include "template/revencode1.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode4l.c0000644000175200007730000000063114515254731023062 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block4.h" #include "traitsl.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec4.c" #include "template/encode.c" #include "template/encodei.c" #include "template/encode4.c" #include "template/revencode.c" #include "template/revencode4.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode1i.c0000644000175200007730000000063114515254731023054 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block1.h" #include "traitsi.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec1.c" #include "template/encode.c" #include "template/encodei.c" #include "template/encode1.c" #include "template/revencode.c" #include "template/revencode1.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode2l.c0000644000175200007730000000063114515254731023060 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block2.h" #include "traitsl.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec2.c" #include "template/encode.c" #include "template/encodei.c" #include "template/encode2.c" #include "template/revencode.c" #include "template/revencode2.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode4d.c0000644000175200007730000000076714515254731023052 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block4.h" #include "traitsd.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec4.c" #include "template/decode.c" #include "template/decodef.c" #include "template/decode4.c" #include "template/revcodecf.c" #include "template/revdecode.c" #include "template/revdecodef.c" #include "template/revdecode4.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode2i.c0000644000175200007730000000063114515254731023043 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block2.h" #include "traitsi.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec2.c" #include "template/decode.c" #include "template/decodei.c" #include "template/decode2.c" #include "template/revdecode.c" #include "template/revdecode2.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/block1.h0000644000175200007730000000001714515254731022543 0ustar rlaboissrlaboiss#define DIMS 1 zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode2i.c0000644000175200007730000000063114515254731023055 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block2.h" #include "traitsi.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec2.c" #include "template/encode.c" #include "template/encodei.c" #include "template/encode2.c" #include "template/revencode.c" #include "template/revencode2.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode2d.c0000644000175200007730000000076714515254731023050 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block2.h" #include "traitsd.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec2.c" #include "template/decode.c" #include "template/decodef.c" #include "template/decode2.c" #include "template/revcodecf.c" #include "template/revdecode.c" #include "template/revdecodef.c" #include "template/revdecode2.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/block2.h0000644000175200007730000000001714515254731022544 0ustar rlaboissrlaboiss#define DIMS 2 zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/0000755000175200007730000000000014515254731023034 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/revencodef.c0000644000175200007730000000513114515254731025320 0ustar rlaboissrlaboiss#include /* private functions ------------------------------------------------------- */ /* test if block-floating-point encoding is reversible */ static int _t1(rev_fwd_reversible, Scalar)(const Int* iblock, const Scalar* fblock, uint n, int emax) { /* reconstruct block */ cache_align_(Scalar gblock[BLOCK_SIZE]); _t1(rev_inv_cast, Scalar)(iblock, gblock, n, emax); /* perform bit-wise comparison */ return !memcmp(fblock, gblock, n * sizeof(*fblock)); } /* forward block-floating-point transform to signed integers */ static void _t1(rev_fwd_cast, Scalar)(Int* iblock, const Scalar* fblock, uint n, int emax) { /* test for all-zero block, which needs special treatment */ if (emax != -EBIAS) _t1(fwd_cast, Scalar)(iblock, fblock, BLOCK_SIZE, emax); else while (n--) *iblock++ = 0; } /* reinterpret floating values as two's complement integers */ static void _t1(rev_fwd_reinterpret, Scalar)(Int* iblock, const Scalar* fblock, uint n) { /* reinterpret floating values as sign-magnitude integers */ memcpy(iblock, fblock, n * sizeof(*iblock)); /* convert sign-magnitude integers to two's complement integers */ while (n--) { Int x = *iblock; if (x < 0) *iblock = (Int)((UInt)x ^ TCMASK); iblock++; } } /* encode contiguous floating-point block using reversible algorithm */ static uint _t2(rev_encode_block, Scalar, DIMS)(zfp_stream* zfp, const Scalar* fblock) { uint bits = 0; cache_align_(Int iblock[BLOCK_SIZE]); /* compute maximum exponent */ int emax = _t1(exponent_block, Scalar)(fblock, BLOCK_SIZE); /* perform forward block-floating-point transform */ _t1(rev_fwd_cast, Scalar)(iblock, fblock, BLOCK_SIZE, emax); /* test if block-floating-point transform is reversible */ if (_t1(rev_fwd_reversible, Scalar)(iblock, fblock, BLOCK_SIZE, emax)) { /* transform is reversible; test if block has any non-zeros */ uint e = emax + EBIAS; if (e) { /* encode common exponent */ bits += 2; stream_write_bits(zfp->stream, 1, 2); bits += EBITS; stream_write_bits(zfp->stream, e, EBITS); } else { /* emit single bit for all-zero block */ bits++; stream_write_bit(zfp->stream, 0); return bits; } } else { /* transform is irreversible; reinterpret floating values as integers */ _t1(rev_fwd_reinterpret, Scalar)(iblock, fblock, BLOCK_SIZE); bits += 2; stream_write_bits(zfp->stream, 3, 2); } /* losslessly encode integers */ bits += _t2(rev_encode_block, Int, DIMS)(zfp->stream, zfp->minbits - bits, zfp->maxbits - bits, zfp->maxprec, iblock); return bits; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/encode3.c0000644000175200007730000000470114515254731024522 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* gather 4*4*4 block from strided array */ static void _t2(gather, Scalar, 3)(Scalar* q, const Scalar* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz) { uint x, y, z; for (z = 0; z < 4; z++, p += sz - 4 * sy) for (y = 0; y < 4; y++, p += sy - 4 * sx) for (x = 0; x < 4; x++, p += sx) *q++ = *p; } /* gather nx*ny*nz block from strided array */ static void _t2(gather_partial, Scalar, 3)(Scalar* q, const Scalar* p, size_t nx, size_t ny, size_t nz, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz) { size_t x, y, z; for (z = 0; z < nz; z++, p += sz - (ptrdiff_t)ny * sy) { for (y = 0; y < ny; y++, p += sy - (ptrdiff_t)nx * sx) { for (x = 0; x < nx; x++, p += sx) q[16 * z + 4 * y + x] = *p; _t1(pad_block, Scalar)(q + 16 * z + 4 * y, nx, 1); } for (x = 0; x < 4; x++) _t1(pad_block, Scalar)(q + 16 * z + x, ny, 4); } for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) _t1(pad_block, Scalar)(q + 4 * y + x, nz, 16); } /* forward decorrelating 3D transform */ static void _t2(fwd_xform, Int, 3)(Int* p) { uint x, y, z; /* transform along x */ for (z = 0; z < 4; z++) for (y = 0; y < 4; y++) _t1(fwd_lift, Int)(p + 4 * y + 16 * z, 1); /* transform along y */ for (x = 0; x < 4; x++) for (z = 0; z < 4; z++) _t1(fwd_lift, Int)(p + 16 * z + 1 * x, 4); /* transform along z */ for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) _t1(fwd_lift, Int)(p + 1 * x + 4 * y, 16); } /* public functions -------------------------------------------------------- */ /* encode 4*4*4 block stored at p using strides (sx, sy, sz) */ size_t _t2(zfp_encode_block_strided, Scalar, 3)(zfp_stream* stream, const Scalar* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz) { /* gather block from strided array */ cache_align_(Scalar block[64]); _t2(gather, Scalar, 3)(block, p, sx, sy, sz); /* encode block */ return _t2(zfp_encode_block, Scalar, 3)(stream, block); } /* encode nx*ny*nz block stored at p using strides (sx, sy, sz) */ size_t _t2(zfp_encode_partial_block_strided, Scalar, 3)(zfp_stream* stream, const Scalar* p, size_t nx, size_t ny, size_t nz, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz) { /* gather block from strided array */ cache_align_(Scalar block[64]); _t2(gather_partial, Scalar, 3)(block, p, nx, ny, nz, sx, sy, sz); /* encode block */ return _t2(zfp_encode_block, Scalar, 3)(stream, block); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/encodef.c0000644000175200007730000000570514515254731024612 0ustar rlaboissrlaboiss#include #include #include static uint _t2(rev_encode_block, Scalar, DIMS)(zfp_stream* zfp, const Scalar* fblock); /* private functions ------------------------------------------------------- */ /* return normalized floating-point exponent for x >= 0 */ static int _t1(exponent, Scalar)(Scalar x) { /* use e = -EBIAS when x = 0 */ int e = -EBIAS; #ifdef ZFP_WITH_DAZ /* treat subnormals as zero; resolves issue #119 by avoiding overflow */ if (x >= SCALAR_MIN) FREXP(x, &e); #else if (x > 0) { FREXP(x, &e); /* clamp exponent in case x is subnormal; may still result in overflow */ e = MAX(e, 1 - EBIAS); } #endif return e; } /* compute maximum floating-point exponent in block of n values */ static int _t1(exponent_block, Scalar)(const Scalar* p, uint n) { Scalar max = 0; do { Scalar f = FABS(*p++); if (max < f) max = f; } while (--n); return _t1(exponent, Scalar)(max); } /* map floating-point number x to integer relative to exponent e */ static Scalar _t1(quantize, Scalar)(Scalar x, int e) { return LDEXP(x, ((int)(CHAR_BIT * sizeof(Scalar)) - 2) - e); } /* forward block-floating-point transform to signed integers */ static void _t1(fwd_cast, Scalar)(Int* iblock, const Scalar* fblock, uint n, int emax) { /* compute power-of-two scale factor s */ Scalar s = _t1(quantize, Scalar)(1, emax); /* compute p-bit int y = s*x where x is floating and |y| <= 2^(p-2) - 1 */ do *iblock++ = (Int)(s * *fblock++); while (--n); } /* encode contiguous floating-point block using lossy algorithm */ static uint _t2(encode_block, Scalar, DIMS)(zfp_stream* zfp, const Scalar* fblock) { uint bits = 1; /* compute maximum exponent */ int emax = _t1(exponent_block, Scalar)(fblock, BLOCK_SIZE); int maxprec = precision(emax, zfp->maxprec, zfp->minexp, DIMS); uint e = maxprec ? emax + EBIAS : 0; /* encode block only if biased exponent is nonzero */ if (e) { cache_align_(Int iblock[BLOCK_SIZE]); /* encode common exponent; LSB indicates that exponent is nonzero */ bits += EBITS; stream_write_bits(zfp->stream, 2 * e + 1, bits); /* perform forward block-floating-point transform */ _t1(fwd_cast, Scalar)(iblock, fblock, BLOCK_SIZE, emax); /* encode integer block */ bits += _t2(encode_block, Int, DIMS)(zfp->stream, zfp->minbits - bits, zfp->maxbits - bits, maxprec, iblock); } else { /* write single zero-bit to indicate that all values are zero */ stream_write_bit(zfp->stream, 0); if (zfp->minbits > bits) { stream_pad(zfp->stream, zfp->minbits - bits); bits = zfp->minbits; } } return bits; } /* public functions -------------------------------------------------------- */ /* encode contiguous floating-point block */ size_t _t2(zfp_encode_block, Scalar, DIMS)(zfp_stream* zfp, const Scalar* fblock) { return REVERSIBLE(zfp) ? _t2(rev_encode_block, Scalar, DIMS)(zfp, fblock) : _t2(encode_block, Scalar, DIMS)(zfp, fblock); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/codec.c0000644000175200007730000000024214515254731024253 0ustar rlaboissrlaboiss/* true if max compressed size exceeds maxbits */ static int with_maxbits(uint maxbits, uint maxprec, uint size) { return (maxprec + 1) * size - 1 > maxbits; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/decode1.c0000644000175200007730000000273114515254731024507 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* scatter 4-value block to strided array */ static void _t2(scatter, Scalar, 1)(const Scalar* q, Scalar* p, ptrdiff_t sx) { uint x; for (x = 0; x < 4; x++, p += sx) *p = *q++; } /* scatter nx-value block to strided array */ static void _t2(scatter_partial, Scalar, 1)(const Scalar* q, Scalar* p, size_t nx, ptrdiff_t sx) { size_t x; for (x = 0; x < nx; x++, p += sx) *p = *q++; } /* inverse decorrelating 1D transform */ static void _t2(inv_xform, Int, 1)(Int* p) { /* transform along x */ _t1(inv_lift, Int)(p, 1); } /* public functions -------------------------------------------------------- */ /* decode 4-value block and store at p using stride sx */ size_t _t2(zfp_decode_block_strided, Scalar, 1)(zfp_stream* stream, Scalar* p, ptrdiff_t sx) { /* decode contiguous block */ cache_align_(Scalar block[4]); size_t bits = _t2(zfp_decode_block, Scalar, 1)(stream, block); /* scatter block to strided array */ _t2(scatter, Scalar, 1)(block, p, sx); return bits; } /* decode nx-value block and store at p using stride sx */ size_t _t2(zfp_decode_partial_block_strided, Scalar, 1)(zfp_stream* stream, Scalar* p, size_t nx, ptrdiff_t sx) { /* decode contiguous block */ cache_align_(Scalar block[4]); size_t bits = _t2(zfp_decode_block, Scalar, 1)(stream, block); /* scatter block to strided array */ _t2(scatter_partial, Scalar, 1)(block, p, nx, sx); return bits; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/encode2.c0000644000175200007730000000356714515254731024532 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* gather 4*4 block from strided array */ static void _t2(gather, Scalar, 2)(Scalar* q, const Scalar* p, ptrdiff_t sx, ptrdiff_t sy) { uint x, y; for (y = 0; y < 4; y++, p += sy - 4 * sx) for (x = 0; x < 4; x++, p += sx) *q++ = *p; } /* gather nx*ny block from strided array */ static void _t2(gather_partial, Scalar, 2)(Scalar* q, const Scalar* p, size_t nx, size_t ny, ptrdiff_t sx, ptrdiff_t sy) { size_t x, y; for (y = 0; y < ny; y++, p += sy - (ptrdiff_t)nx * sx) { for (x = 0; x < nx; x++, p += sx) q[4 * y + x] = *p; _t1(pad_block, Scalar)(q + 4 * y, nx, 1); } for (x = 0; x < 4; x++) _t1(pad_block, Scalar)(q + x, ny, 4); } /* forward decorrelating 2D transform */ static void _t2(fwd_xform, Int, 2)(Int* p) { uint x, y; /* transform along x */ for (y = 0; y < 4; y++) _t1(fwd_lift, Int)(p + 4 * y, 1); /* transform along y */ for (x = 0; x < 4; x++) _t1(fwd_lift, Int)(p + 1 * x, 4); } /* public functions -------------------------------------------------------- */ /* encode 4*4 block stored at p using strides (sx, sy) */ size_t _t2(zfp_encode_block_strided, Scalar, 2)(zfp_stream* stream, const Scalar* p, ptrdiff_t sx, ptrdiff_t sy) { /* gather block from strided array */ cache_align_(Scalar block[16]); _t2(gather, Scalar, 2)(block, p, sx, sy); /* encode block */ return _t2(zfp_encode_block, Scalar, 2)(stream, block); } /* encode nx*ny block stored at p using strides (sx, sy) */ size_t _t2(zfp_encode_partial_block_strided, Scalar, 2)(zfp_stream* stream, const Scalar* p, size_t nx, size_t ny, ptrdiff_t sx, ptrdiff_t sy) { /* gather block from strided array */ cache_align_(Scalar block[16]); _t2(gather_partial, Scalar, 2)(block, p, nx, ny, sx, sy); /* encode block */ return _t2(zfp_encode_block, Scalar, 2)(stream, block); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/revdecode.c0000644000175200007730000000257514515254731025151 0ustar rlaboissrlaboissstatic void _t2(rev_inv_xform, Int, DIMS)(Int* p); /* private functions ------------------------------------------------------- */ /* reversible inverse lifting transform of 4-vector */ static void _t1(rev_inv_lift, Int)(Int* p, uint s) { Int x, y, z, w; x = *p; p += s; y = *p; p += s; z = *p; p += s; w = *p; p += s; /* ** high-order Lorenzo transform (P4 Pascal matrix) ** ( 1 0 0 0) (x) ** ( 1 1 0 0) (y) ** ( 1 2 1 0) (z) ** ( 1 3 3 1) (w) */ w += z; z += y; w += z; y += x; z += y; w += z; p -= s; *p = w; p -= s; *p = z; p -= s; *p = y; p -= s; *p = x; } /* decode block of integers using reversible algorithm */ static uint _t2(rev_decode_block, Int, DIMS)(bitstream* stream, int minbits, int maxbits, Int* iblock) { /* decode number of significant bits */ int bits = PBITS; int prec = (int)stream_read_bits(stream, PBITS) + 1; cache_align_(UInt ublock[BLOCK_SIZE]); /* decode integer coefficients */ bits += _t1(decode_ints, UInt)(stream, maxbits - bits, prec, ublock, BLOCK_SIZE); /* read at least minbits bits */ if (bits < minbits) { stream_skip(stream, minbits - bits); bits = minbits; } /* reorder unsigned coefficients and convert to signed integer */ _t1(inv_order, Int)(ublock, iblock, PERM, BLOCK_SIZE); /* perform decorrelating transform */ _t2(rev_inv_xform, Int, DIMS)(iblock); return bits; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/encode.c0000644000175200007730000002042414515254731024437 0ustar rlaboissrlaboiss#include static void _t2(fwd_xform, Int, DIMS)(Int* p); /* private functions ------------------------------------------------------- */ /* pad partial block of width n <= 4 and stride s */ static void _t1(pad_block, Scalar)(Scalar* p, size_t n, ptrdiff_t s) { switch (n) { case 0: p[0 * s] = 0; /* FALLTHROUGH */ case 1: p[1 * s] = p[0 * s]; /* FALLTHROUGH */ case 2: p[2 * s] = p[1 * s]; /* FALLTHROUGH */ case 3: p[3 * s] = p[0 * s]; /* FALLTHROUGH */ default: break; } } /* forward lifting transform of 4-vector */ static void _t1(fwd_lift, Int)(Int* p, ptrdiff_t s) { Int x, y, z, w; x = *p; p += s; y = *p; p += s; z = *p; p += s; w = *p; p += s; /* ** non-orthogonal transform ** ( 4 4 4 4) (x) ** 1/16 * ( 5 1 -1 -5) (y) ** (-4 4 4 -4) (z) ** (-2 6 -6 2) (w) */ x += w; x >>= 1; w -= x; z += y; z >>= 1; y -= z; x += z; x >>= 1; z -= x; w += y; w >>= 1; y -= w; w += y >> 1; y -= w >> 1; p -= s; *p = w; p -= s; *p = z; p -= s; *p = y; p -= s; *p = x; } #if ZFP_ROUNDING_MODE == ZFP_ROUND_FIRST /* bias values such that truncation is equivalent to round to nearest */ static void _t1(fwd_round, Int)(Int* iblock, uint n, uint maxprec) { /* add or subtract 1/6 ulp to unbias errors */ if (maxprec < (uint)(CHAR_BIT * sizeof(Int))) { Int bias = (NBMASK >> 2) >> maxprec; if (maxprec & 1u) do *iblock++ += bias; while (--n); else do *iblock++ -= bias; while (--n); } } #endif /* map two's complement signed integer to negabinary unsigned integer */ static UInt _t1(int2uint, Int)(Int x) { return ((UInt)x + NBMASK) ^ NBMASK; } /* reorder signed coefficients and convert to unsigned integer */ static void _t1(fwd_order, Int)(UInt* ublock, const Int* iblock, const uchar* perm, uint n) { do *ublock++ = _t1(int2uint, Int)(iblock[*perm++]); while (--n); } /* compress sequence of size <= 64 unsigned integers */ static uint _t1(encode_few_ints, UInt)(bitstream* restrict_ stream, uint maxbits, uint maxprec, const UInt* restrict_ data, uint size) { /* make a copy of bit stream to avoid aliasing */ bitstream s = *stream; uint intprec = (uint)(CHAR_BIT * sizeof(UInt)); uint kmin = intprec > maxprec ? intprec - maxprec : 0; uint bits = maxbits; uint i, k, m, n; uint64 x; /* encode one bit plane at a time from MSB to LSB */ for (k = intprec, n = 0; bits && k-- > kmin;) { /* step 1: extract bit plane #k to x */ x = 0; for (i = 0; i < size; i++) x += (uint64)((data[i] >> k) & 1u) << i; /* step 2: encode first n bits of bit plane */ m = MIN(n, bits); bits -= m; x = stream_write_bits(&s, x, m); /* step 3: unary run-length encode remainder of bit plane */ for (; bits && n < size; x >>= 1, n++) { bits--; if (stream_write_bit(&s, !!x)) { /* positive group test (x != 0); scan for one-bit */ for (; bits && n < size - 1; x >>= 1, n++) { bits--; if (stream_write_bit(&s, x & 1u)) break; } } else { /* negative group test (x == 0); done with bit plane */ break; } } } *stream = s; return maxbits - bits; } /* compress sequence of size > 64 unsigned integers */ static uint _t1(encode_many_ints, UInt)(bitstream* restrict_ stream, uint maxbits, uint maxprec, const UInt* restrict_ data, uint size) { /* make a copy of bit stream to avoid aliasing */ bitstream s = *stream; uint intprec = (uint)(CHAR_BIT * sizeof(UInt)); uint kmin = intprec > maxprec ? intprec - maxprec : 0; uint bits = maxbits; uint i, k, m, n, c; /* encode one bit plane at a time from MSB to LSB */ for (k = intprec, n = 0; bits && k-- > kmin;) { /* step 1: encode first n bits of bit plane #k */ m = MIN(n, bits); bits -= m; for (i = 0; i < m; i++) stream_write_bit(&s, (data[i] >> k) & 1u); /* step 2: count remaining one-bits in bit plane */ c = 0; for (i = m; i < size; i++) c += (data[i] >> k) & 1u; /* step 3: unary run-length encode remainder of bit plane */ for (; bits && n < size; n++) { bits--; if (stream_write_bit(&s, !!c)) { /* positive group test (c > 0); scan for one-bit */ for (c--; bits && n < size - 1; n++) { bits--; if (stream_write_bit(&s, (data[n] >> k) & 1u)) break; } } else { /* negative group test (c == 0); done with bit plane */ break; } } } *stream = s; return maxbits - bits; } /* compress sequence of size <= 64 unsigned integers with no rate constraint */ static uint _t1(encode_few_ints_prec, UInt)(bitstream* restrict_ stream, uint maxprec, const UInt* restrict_ data, uint size) { /* make a copy of bit stream to avoid aliasing */ bitstream s = *stream; size_t offset = stream_wtell(&s); uint intprec = (uint)(CHAR_BIT * sizeof(UInt)); uint kmin = intprec > maxprec ? intprec - maxprec : 0; uint i, k, n; /* encode one bit plane at a time from MSB to LSB */ for (k = intprec, n = 0; k-- > kmin;) { /* step 1: extract bit plane #k to x */ uint64 x = 0; for (i = 0; i < size; i++) x += (uint64)((data[i] >> k) & 1u) << i; /* step 2: encode first n bits of bit plane */ x = stream_write_bits(&s, x, n); /* step 3: unary run-length encode remainder of bit plane */ for (; n < size && stream_write_bit(&s, !!x); x >>= 1, n++) for (; n < size - 1 && !stream_write_bit(&s, x & 1u); x >>= 1, n++) ; } *stream = s; return (uint)(stream_wtell(&s) - offset); } /* compress sequence of size > 64 unsigned integers with no rate constraint */ static uint _t1(encode_many_ints_prec, UInt)(bitstream* restrict_ stream, uint maxprec, const UInt* restrict_ data, uint size) { /* make a copy of bit stream to avoid aliasing */ bitstream s = *stream; size_t offset = stream_wtell(&s); uint intprec = (uint)(CHAR_BIT * sizeof(UInt)); uint kmin = intprec > maxprec ? intprec - maxprec : 0; uint i, k, n, c; /* encode one bit plane at a time from MSB to LSB */ for (k = intprec, n = 0; k-- > kmin;) { /* step 1: encode first n bits of bit plane #k */ for (i = 0; i < n; i++) stream_write_bit(&s, (data[i] >> k) & 1u); /* step 2: count remaining one-bits in bit plane */ c = 0; for (i = n; i < size; i++) c += (data[i] >> k) & 1u; /* step 3: unary run-length encode remainder of bit plane */ for (; n < size && stream_write_bit(&s, !!c); n++) for (c--; n < size - 1 && !stream_write_bit(&s, (data[n] >> k) & 1u); n++) ; } *stream = s; return (uint)(stream_wtell(&s) - offset); } /* compress sequence of size unsigned integers */ static uint _t1(encode_ints, UInt)(bitstream* restrict_ stream, uint maxbits, uint maxprec, const UInt* restrict_ data, uint size) { /* use fastest available encoder implementation */ if (with_maxbits(maxbits, maxprec, size)) { /* rate constrained path: encode partial bit planes */ if (size <= 64) return _t1(encode_few_ints, UInt)(stream, maxbits, maxprec, data, size); /* 1D, 2D, 3D blocks */ else return _t1(encode_many_ints, UInt)(stream, maxbits, maxprec, data, size); /* 4D blocks */ } else { /* variable-rate path: encode whole bit planes */ if (size <= 64) return _t1(encode_few_ints_prec, UInt)(stream, maxprec, data, size); /* 1D, 2D, 3D blocks */ else return _t1(encode_many_ints_prec, UInt)(stream, maxprec, data, size); /* 4D blocks */ } } /* encode block of integers */ static uint _t2(encode_block, Int, DIMS)(bitstream* stream, int minbits, int maxbits, int maxprec, Int* iblock) { int bits; cache_align_(UInt ublock[BLOCK_SIZE]); /* perform decorrelating transform */ _t2(fwd_xform, Int, DIMS)(iblock); #if ZFP_ROUNDING_MODE == ZFP_ROUND_FIRST /* bias values to achieve proper rounding */ _t1(fwd_round, Int)(iblock, BLOCK_SIZE, maxprec); #endif /* reorder signed coefficients and convert to unsigned integer */ _t1(fwd_order, Int)(ublock, iblock, PERM, BLOCK_SIZE); /* encode integer coefficients */ bits = _t1(encode_ints, UInt)(stream, maxbits, maxprec, ublock, BLOCK_SIZE); /* write at least minbits bits by padding with zeros */ if (bits < minbits) { stream_pad(stream, minbits - bits); bits = minbits; } return bits; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/encode1.c0000644000175200007730000000270114515254731024516 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* gather 4-value block from strided array */ static void _t2(gather, Scalar, 1)(Scalar* q, const Scalar* p, ptrdiff_t sx) { uint x; for (x = 0; x < 4; x++, p += sx) *q++ = *p; } /* gather nx-value block from strided array */ static void _t2(gather_partial, Scalar, 1)(Scalar* q, const Scalar* p, size_t nx, ptrdiff_t sx) { size_t x; for (x = 0; x < nx; x++, p += sx) q[x] = *p; _t1(pad_block, Scalar)(q, nx, 1); } /* forward decorrelating 1D transform */ static void _t2(fwd_xform, Int, 1)(Int* p) { /* transform along x */ _t1(fwd_lift, Int)(p, 1); } /* public functions -------------------------------------------------------- */ /* encode 4-value block stored at p using stride sx */ size_t _t2(zfp_encode_block_strided, Scalar, 1)(zfp_stream* stream, const Scalar* p, ptrdiff_t sx) { /* gather block from strided array */ cache_align_(Scalar block[4]); _t2(gather, Scalar, 1)(block, p, sx); /* encode block */ return _t2(zfp_encode_block, Scalar, 1)(stream, block); } /* encode nx-value block stored at p using stride sx */ size_t _t2(zfp_encode_partial_block_strided, Scalar, 1)(zfp_stream* stream, const Scalar* p, size_t nx, ptrdiff_t sx) { /* gather block from strided array */ cache_align_(Scalar block[4]); _t2(gather_partial, Scalar, 1)(block, p, nx, sx); /* encode block */ return _t2(zfp_encode_block, Scalar, 1)(stream, block); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/revdecodef.c0000644000175200007730000000363114515254731025311 0ustar rlaboissrlaboiss#include /* private functions ------------------------------------------------------- */ /* reinterpret two's complement integers as floating values */ static void _t1(rev_inv_reinterpret, Scalar)(Int* iblock, Scalar* fblock, uint n) { /* convert two's complement integers to sign-magnitude integers */ uint i; for (i = 0; i < n; i++) { Int x = iblock[i]; if (x < 0) iblock[i] = (Int)((UInt)x ^ TCMASK); } /* reinterpret sign-magnitude integers as floating values */ memcpy(fblock, iblock, n * sizeof(*fblock)); } /* decode contiguous floating-point block using reversible algorithm */ static uint _t2(rev_decode_block, Scalar, DIMS)(zfp_stream* zfp, Scalar* fblock) { uint bits = 0; cache_align_(Int iblock[BLOCK_SIZE]); /* test whether block is all-zero */ bits++; if (stream_read_bit(zfp->stream)) { /* non-zero block; test whether to use block-floating-point transform */ bits++; if (stream_read_bit(zfp->stream)) { /* decode integer block */ bits += _t2(rev_decode_block, Int, DIMS)(zfp->stream, zfp->minbits - bits, zfp->maxbits - bits, iblock); /* reinterpret integers as floating values */ _t1(rev_inv_reinterpret, Scalar)(iblock, fblock, BLOCK_SIZE); } else { /* decode common exponent */ int emax; bits += EBITS; emax = (int)stream_read_bits(zfp->stream, EBITS) - EBIAS; /* decode integer block */ bits += _t2(rev_decode_block, Int, DIMS)(zfp->stream, zfp->minbits - bits, zfp->maxbits - bits, iblock); /* perform inverse block-floating-point transform */ _t1(rev_inv_cast, Scalar)(iblock, fblock, BLOCK_SIZE, emax); } } else { /* all-zero block; set all values to zero */ uint i; for (i = 0; i < BLOCK_SIZE; i++) *fblock++ = 0; if (zfp->minbits > bits) { stream_skip(zfp->stream, zfp->minbits - bits); bits = zfp->minbits; } } return bits; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/codec4.c0000644000175200007730000002237214515254731024347 0ustar rlaboissrlaboiss#define index(i, j, k, l) ((i) + 4 * ((j) + 4 * ((k) + 4 * (l)))) /* order coefficients (i, j, k, l) by i + j + k + l, then i^2 + j^2 + k^2 + l^2 */ cache_align_(static const uchar perm_4[256]) = { index(0, 0, 0, 0), /* 0 : 0 */ index(1, 0, 0, 0), /* 1 : 1 */ index(0, 1, 0, 0), /* 2 : 1 */ index(0, 0, 1, 0), /* 3 : 1 */ index(0, 0, 0, 1), /* 4 : 1 */ index(1, 1, 0, 0), /* 5 : 2 */ index(0, 0, 1, 1), /* 6 : 2 */ index(1, 0, 1, 0), /* 7 : 2 */ index(0, 1, 0, 1), /* 8 : 2 */ index(1, 0, 0, 1), /* 9 : 2 */ index(0, 1, 1, 0), /* 10 : 2 */ index(2, 0, 0, 0), /* 11 : 2 */ index(0, 2, 0, 0), /* 12 : 2 */ index(0, 0, 2, 0), /* 13 : 2 */ index(0, 0, 0, 2), /* 14 : 2 */ index(0, 1, 1, 1), /* 15 : 3 */ index(1, 0, 1, 1), /* 16 : 3 */ index(1, 1, 0, 1), /* 17 : 3 */ index(1, 1, 1, 0), /* 18 : 3 */ index(2, 1, 0, 0), /* 19 : 3 */ index(2, 0, 1, 0), /* 20 : 3 */ index(2, 0, 0, 1), /* 21 : 3 */ index(0, 2, 1, 0), /* 22 : 3 */ index(0, 2, 0, 1), /* 23 : 3 */ index(1, 2, 0, 0), /* 24 : 3 */ index(0, 0, 2, 1), /* 25 : 3 */ index(1, 0, 2, 0), /* 26 : 3 */ index(0, 1, 2, 0), /* 27 : 3 */ index(1, 0, 0, 2), /* 28 : 3 */ index(0, 1, 0, 2), /* 29 : 3 */ index(0, 0, 1, 2), /* 30 : 3 */ index(3, 0, 0, 0), /* 31 : 3 */ index(0, 3, 0, 0), /* 32 : 3 */ index(0, 0, 3, 0), /* 33 : 3 */ index(0, 0, 0, 3), /* 34 : 3 */ index(1, 1, 1, 1), /* 35 : 4 */ index(2, 0, 1, 1), /* 36 : 4 */ index(2, 1, 0, 1), /* 37 : 4 */ index(2, 1, 1, 0), /* 38 : 4 */ index(1, 2, 0, 1), /* 39 : 4 */ index(1, 2, 1, 0), /* 40 : 4 */ index(0, 2, 1, 1), /* 41 : 4 */ index(1, 1, 2, 0), /* 42 : 4 */ index(0, 1, 2, 1), /* 43 : 4 */ index(1, 0, 2, 1), /* 44 : 4 */ index(0, 1, 1, 2), /* 45 : 4 */ index(1, 0, 1, 2), /* 46 : 4 */ index(1, 1, 0, 2), /* 47 : 4 */ index(2, 2, 0, 0), /* 48 : 4 */ index(0, 0, 2, 2), /* 49 : 4 */ index(2, 0, 2, 0), /* 50 : 4 */ index(0, 2, 0, 2), /* 51 : 4 */ index(2, 0, 0, 2), /* 52 : 4 */ index(0, 2, 2, 0), /* 53 : 4 */ index(3, 1, 0, 0), /* 54 : 4 */ index(3, 0, 1, 0), /* 55 : 4 */ index(3, 0, 0, 1), /* 56 : 4 */ index(0, 3, 1, 0), /* 57 : 4 */ index(0, 3, 0, 1), /* 58 : 4 */ index(1, 3, 0, 0), /* 59 : 4 */ index(0, 0, 3, 1), /* 60 : 4 */ index(1, 0, 3, 0), /* 61 : 4 */ index(0, 1, 3, 0), /* 62 : 4 */ index(1, 0, 0, 3), /* 63 : 4 */ index(0, 1, 0, 3), /* 64 : 4 */ index(0, 0, 1, 3), /* 65 : 4 */ index(2, 1, 1, 1), /* 66 : 5 */ index(1, 2, 1, 1), /* 67 : 5 */ index(1, 1, 2, 1), /* 68 : 5 */ index(1, 1, 1, 2), /* 69 : 5 */ index(1, 0, 2, 2), /* 70 : 5 */ index(1, 2, 0, 2), /* 71 : 5 */ index(1, 2, 2, 0), /* 72 : 5 */ index(2, 1, 0, 2), /* 73 : 5 */ index(2, 1, 2, 0), /* 74 : 5 */ index(0, 1, 2, 2), /* 75 : 5 */ index(2, 2, 1, 0), /* 76 : 5 */ index(0, 2, 1, 2), /* 77 : 5 */ index(2, 0, 1, 2), /* 78 : 5 */ index(0, 2, 2, 1), /* 79 : 5 */ index(2, 0, 2, 1), /* 80 : 5 */ index(2, 2, 0, 1), /* 81 : 5 */ index(3, 0, 1, 1), /* 82 : 5 */ index(3, 1, 0, 1), /* 83 : 5 */ index(3, 1, 1, 0), /* 84 : 5 */ index(1, 3, 0, 1), /* 85 : 5 */ index(1, 3, 1, 0), /* 86 : 5 */ index(0, 3, 1, 1), /* 87 : 5 */ index(1, 1, 3, 0), /* 88 : 5 */ index(0, 1, 3, 1), /* 89 : 5 */ index(1, 0, 3, 1), /* 90 : 5 */ index(0, 1, 1, 3), /* 91 : 5 */ index(1, 0, 1, 3), /* 92 : 5 */ index(1, 1, 0, 3), /* 93 : 5 */ index(3, 2, 0, 0), /* 94 : 5 */ index(3, 0, 2, 0), /* 95 : 5 */ index(3, 0, 0, 2), /* 96 : 5 */ index(0, 3, 2, 0), /* 97 : 5 */ index(0, 3, 0, 2), /* 98 : 5 */ index(2, 3, 0, 0), /* 99 : 5 */ index(0, 0, 3, 2), /* 100 : 5 */ index(2, 0, 3, 0), /* 101 : 5 */ index(0, 2, 3, 0), /* 102 : 5 */ index(2, 0, 0, 3), /* 103 : 5 */ index(0, 2, 0, 3), /* 104 : 5 */ index(0, 0, 2, 3), /* 105 : 5 */ index(2, 2, 1, 1), /* 106 : 6 */ index(1, 1, 2, 2), /* 107 : 6 */ index(2, 1, 2, 1), /* 108 : 6 */ index(1, 2, 1, 2), /* 109 : 6 */ index(2, 1, 1, 2), /* 110 : 6 */ index(1, 2, 2, 1), /* 111 : 6 */ index(0, 2, 2, 2), /* 112 : 6 */ index(2, 0, 2, 2), /* 113 : 6 */ index(2, 2, 0, 2), /* 114 : 6 */ index(2, 2, 2, 0), /* 115 : 6 */ index(3, 1, 1, 1), /* 116 : 6 */ index(1, 3, 1, 1), /* 117 : 6 */ index(1, 1, 3, 1), /* 118 : 6 */ index(1, 1, 1, 3), /* 119 : 6 */ index(3, 2, 1, 0), /* 120 : 6 */ index(3, 2, 0, 1), /* 121 : 6 */ index(3, 0, 2, 1), /* 122 : 6 */ index(3, 1, 2, 0), /* 123 : 6 */ index(3, 1, 0, 2), /* 124 : 6 */ index(3, 0, 1, 2), /* 125 : 6 */ index(0, 3, 2, 1), /* 126 : 6 */ index(1, 3, 2, 0), /* 127 : 6 */ index(1, 3, 0, 2), /* 128 : 6 */ index(0, 3, 1, 2), /* 129 : 6 */ index(2, 3, 1, 0), /* 130 : 6 */ index(2, 3, 0, 1), /* 131 : 6 */ index(1, 0, 3, 2), /* 132 : 6 */ index(0, 1, 3, 2), /* 133 : 6 */ index(2, 1, 3, 0), /* 134 : 6 */ index(2, 0, 3, 1), /* 135 : 6 */ index(0, 2, 3, 1), /* 136 : 6 */ index(1, 2, 3, 0), /* 137 : 6 */ index(2, 1, 0, 3), /* 138 : 6 */ index(2, 0, 1, 3), /* 139 : 6 */ index(0, 2, 1, 3), /* 140 : 6 */ index(1, 2, 0, 3), /* 141 : 6 */ index(1, 0, 2, 3), /* 142 : 6 */ index(0, 1, 2, 3), /* 143 : 6 */ index(3, 3, 0, 0), /* 144 : 6 */ index(0, 0, 3, 3), /* 145 : 6 */ index(3, 0, 3, 0), /* 146 : 6 */ index(0, 3, 0, 3), /* 147 : 6 */ index(3, 0, 0, 3), /* 148 : 6 */ index(0, 3, 3, 0), /* 149 : 6 */ index(1, 2, 2, 2), /* 150 : 7 */ index(2, 1, 2, 2), /* 151 : 7 */ index(2, 2, 1, 2), /* 152 : 7 */ index(2, 2, 2, 1), /* 153 : 7 */ index(3, 2, 1, 1), /* 154 : 7 */ index(3, 1, 2, 1), /* 155 : 7 */ index(3, 1, 1, 2), /* 156 : 7 */ index(1, 3, 2, 1), /* 157 : 7 */ index(1, 3, 1, 2), /* 158 : 7 */ index(2, 3, 1, 1), /* 159 : 7 */ index(1, 1, 3, 2), /* 160 : 7 */ index(2, 1, 3, 1), /* 161 : 7 */ index(1, 2, 3, 1), /* 162 : 7 */ index(2, 1, 1, 3), /* 163 : 7 */ index(1, 2, 1, 3), /* 164 : 7 */ index(1, 1, 2, 3), /* 165 : 7 */ index(3, 0, 2, 2), /* 166 : 7 */ index(3, 2, 0, 2), /* 167 : 7 */ index(3, 2, 2, 0), /* 168 : 7 */ index(2, 3, 0, 2), /* 169 : 7 */ index(2, 3, 2, 0), /* 170 : 7 */ index(0, 3, 2, 2), /* 171 : 7 */ index(2, 2, 3, 0), /* 172 : 7 */ index(0, 2, 3, 2), /* 173 : 7 */ index(2, 0, 3, 2), /* 174 : 7 */ index(0, 2, 2, 3), /* 175 : 7 */ index(2, 0, 2, 3), /* 176 : 7 */ index(2, 2, 0, 3), /* 177 : 7 */ index(1, 0, 3, 3), /* 178 : 7 */ index(1, 3, 0, 3), /* 179 : 7 */ index(1, 3, 3, 0), /* 180 : 7 */ index(3, 1, 0, 3), /* 181 : 7 */ index(3, 1, 3, 0), /* 182 : 7 */ index(0, 1, 3, 3), /* 183 : 7 */ index(3, 3, 1, 0), /* 184 : 7 */ index(0, 3, 1, 3), /* 185 : 7 */ index(3, 0, 1, 3), /* 186 : 7 */ index(0, 3, 3, 1), /* 187 : 7 */ index(3, 0, 3, 1), /* 188 : 7 */ index(3, 3, 0, 1), /* 189 : 7 */ index(2, 2, 2, 2), /* 190 : 8 */ index(3, 1, 2, 2), /* 191 : 8 */ index(3, 2, 1, 2), /* 192 : 8 */ index(3, 2, 2, 1), /* 193 : 8 */ index(2, 3, 1, 2), /* 194 : 8 */ index(2, 3, 2, 1), /* 195 : 8 */ index(1, 3, 2, 2), /* 196 : 8 */ index(2, 2, 3, 1), /* 197 : 8 */ index(1, 2, 3, 2), /* 198 : 8 */ index(2, 1, 3, 2), /* 199 : 8 */ index(1, 2, 2, 3), /* 200 : 8 */ index(2, 1, 2, 3), /* 201 : 8 */ index(2, 2, 1, 3), /* 202 : 8 */ index(3, 3, 1, 1), /* 203 : 8 */ index(1, 1, 3, 3), /* 204 : 8 */ index(3, 1, 3, 1), /* 205 : 8 */ index(1, 3, 1, 3), /* 206 : 8 */ index(3, 1, 1, 3), /* 207 : 8 */ index(1, 3, 3, 1), /* 208 : 8 */ index(2, 0, 3, 3), /* 209 : 8 */ index(2, 3, 0, 3), /* 210 : 8 */ index(2, 3, 3, 0), /* 211 : 8 */ index(3, 2, 0, 3), /* 212 : 8 */ index(3, 2, 3, 0), /* 213 : 8 */ index(0, 2, 3, 3), /* 214 : 8 */ index(3, 3, 2, 0), /* 215 : 8 */ index(0, 3, 2, 3), /* 216 : 8 */ index(3, 0, 2, 3), /* 217 : 8 */ index(0, 3, 3, 2), /* 218 : 8 */ index(3, 0, 3, 2), /* 219 : 8 */ index(3, 3, 0, 2), /* 220 : 8 */ index(3, 2, 2, 2), /* 221 : 9 */ index(2, 3, 2, 2), /* 222 : 9 */ index(2, 2, 3, 2), /* 223 : 9 */ index(2, 2, 2, 3), /* 224 : 9 */ index(2, 1, 3, 3), /* 225 : 9 */ index(2, 3, 1, 3), /* 226 : 9 */ index(2, 3, 3, 1), /* 227 : 9 */ index(3, 2, 1, 3), /* 228 : 9 */ index(3, 2, 3, 1), /* 229 : 9 */ index(1, 2, 3, 3), /* 230 : 9 */ index(3, 3, 2, 1), /* 231 : 9 */ index(1, 3, 2, 3), /* 232 : 9 */ index(3, 1, 2, 3), /* 233 : 9 */ index(1, 3, 3, 2), /* 234 : 9 */ index(3, 1, 3, 2), /* 235 : 9 */ index(3, 3, 1, 2), /* 236 : 9 */ index(0, 3, 3, 3), /* 237 : 9 */ index(3, 0, 3, 3), /* 238 : 9 */ index(3, 3, 0, 3), /* 239 : 9 */ index(3, 3, 3, 0), /* 240 : 9 */ index(3, 3, 2, 2), /* 241 : 10 */ index(2, 2, 3, 3), /* 242 : 10 */ index(3, 2, 3, 2), /* 243 : 10 */ index(2, 3, 2, 3), /* 244 : 10 */ index(3, 2, 2, 3), /* 245 : 10 */ index(2, 3, 3, 2), /* 246 : 10 */ index(1, 3, 3, 3), /* 247 : 10 */ index(3, 1, 3, 3), /* 248 : 10 */ index(3, 3, 1, 3), /* 249 : 10 */ index(3, 3, 3, 1), /* 250 : 10 */ index(2, 3, 3, 3), /* 251 : 11 */ index(3, 2, 3, 3), /* 252 : 11 */ index(3, 3, 2, 3), /* 253 : 11 */ index(3, 3, 3, 2), /* 254 : 11 */ index(3, 3, 3, 3), /* 255 : 12 */ }; #undef index zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/decodei.c0000644000175200007730000000076614515254731024605 0ustar rlaboissrlaboissstatic uint _t2(rev_decode_block, Int, DIMS)(bitstream* stream, int minbits, int maxbits, Int* iblock); /* public functions -------------------------------------------------------- */ /* decode contiguous integer block */ size_t _t2(zfp_decode_block, Int, DIMS)(zfp_stream* zfp, Int* iblock) { return REVERSIBLE(zfp) ? _t2(rev_decode_block, Int, DIMS)(zfp->stream, zfp->minbits, zfp->maxbits, iblock) : _t2(decode_block, Int, DIMS)(zfp->stream, zfp->minbits, zfp->maxbits, zfp->maxprec, iblock); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/codec1.c0000644000175200007730000000016714515254731024342 0ustar rlaboissrlaboiss/* order coefficients by polynomial degree/frequency */ cache_align_(static const uchar perm_1[4]) = { 0, 1, 2, 3 }; zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/codec.h0000644000175200007730000000040514515254731024261 0ustar rlaboissrlaboiss#define PERM _t1(perm, DIMS) /* coefficient order */ #define BLOCK_SIZE (1 << (2 * DIMS)) /* values per block */ #define EBIAS ((1 << (EBITS - 1)) - 1) /* exponent bias */ #define REVERSIBLE(zfp) ((zfp)->minexp < ZFP_MIN_EXP) /* reversible mode? */ zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/ompcompress.c0000644000175200007730000002267714515254731025565 0ustar rlaboissrlaboiss#ifdef _OPENMP /* compress 1d contiguous array in parallel */ static void _t2(compress_omp, Scalar, 1)(zfp_stream* stream, const zfp_field* field) { /* array metadata */ const Scalar* data = (const Scalar*)field->data; size_t nx = field->nx; /* number of omp threads, blocks, and chunks */ uint threads = thread_count_omp(stream); size_t blocks = (nx + 3) / 4; size_t chunks = chunk_count_omp(stream, blocks, threads); int chunk; /* OpenMP 2.0 requires int loop counter */ /* allocate per-thread streams */ bitstream** bs = compress_init_par(stream, field, chunks, blocks); if (!bs) return; /* compress chunks of blocks in parallel */ #pragma omp parallel for num_threads(threads) for (chunk = 0; chunk < (int)chunks; chunk++) { /* determine range of block indices assigned to this thread */ size_t bmin = chunk_offset(blocks, chunks, chunk + 0); size_t bmax = chunk_offset(blocks, chunks, chunk + 1); size_t block; /* set up thread-local bit stream */ zfp_stream s = *stream; zfp_stream_set_bit_stream(&s, bs[chunk]); /* compress sequence of blocks */ for (block = bmin; block < bmax; block++) { /* determine block origin x within array */ const Scalar* p = data; size_t x = 4 * block; p += x; /* compress partial or full block */ if (nx - x < 4u) _t2(zfp_encode_partial_block_strided, Scalar, 1)(&s, p, nx - x, 1); else _t2(zfp_encode_block, Scalar, 1)(&s, p); } } /* concatenate per-thread streams */ compress_finish_par(stream, bs, chunks); } /* compress 1d strided array in parallel */ static void _t2(compress_strided_omp, Scalar, 1)(zfp_stream* stream, const zfp_field* field) { /* array metadata */ const Scalar* data = (const Scalar*)field->data; size_t nx = field->nx; ptrdiff_t sx = field->sx ? field->sx : 1; /* number of omp threads, blocks, and chunks */ uint threads = thread_count_omp(stream); size_t blocks = (nx + 3) / 4; size_t chunks = chunk_count_omp(stream, blocks, threads); int chunk; /* OpenMP 2.0 requires int loop counter */ /* allocate per-thread streams */ bitstream** bs = compress_init_par(stream, field, chunks, blocks); if (!bs) return; /* compress chunks of blocks in parallel */ #pragma omp parallel for num_threads(threads) for (chunk = 0; chunk < (int)chunks; chunk++) { /* determine range of block indices assigned to this thread */ size_t bmin = chunk_offset(blocks, chunks, chunk + 0); size_t bmax = chunk_offset(blocks, chunks, chunk + 1); size_t block; /* set up thread-local bit stream */ zfp_stream s = *stream; zfp_stream_set_bit_stream(&s, bs[chunk]); /* compress sequence of blocks */ for (block = bmin; block < bmax; block++) { /* determine block origin x within array */ const Scalar* p = data; size_t x = 4 * block; p += sx * (ptrdiff_t)x; /* compress partial or full block */ if (nx - x < 4u) _t2(zfp_encode_partial_block_strided, Scalar, 1)(&s, p, nx - x, sx); else _t2(zfp_encode_block_strided, Scalar, 1)(&s, p, sx); } } /* concatenate per-thread streams */ compress_finish_par(stream, bs, chunks); } /* compress 2d strided array in parallel */ static void _t2(compress_strided_omp, Scalar, 2)(zfp_stream* stream, const zfp_field* field) { /* array metadata */ const Scalar* data = (const Scalar*)field->data; size_t nx = field->nx; size_t ny = field->ny; ptrdiff_t sx = field->sx ? field->sx : 1; ptrdiff_t sy = field->sy ? field->sy : (ptrdiff_t)nx; /* number of omp threads, blocks, and chunks */ uint threads = thread_count_omp(stream); size_t bx = (nx + 3) / 4; size_t by = (ny + 3) / 4; size_t blocks = bx * by; size_t chunks = chunk_count_omp(stream, blocks, threads); int chunk; /* OpenMP 2.0 requires int loop counter */ /* allocate per-thread streams */ bitstream** bs = compress_init_par(stream, field, chunks, blocks); if (!bs) return; /* compress chunks of blocks in parallel */ #pragma omp parallel for num_threads(threads) for (chunk = 0; chunk < (int)chunks; chunk++) { /* determine range of block indices assigned to this thread */ size_t bmin = chunk_offset(blocks, chunks, chunk + 0); size_t bmax = chunk_offset(blocks, chunks, chunk + 1); size_t block; /* set up thread-local bit stream */ zfp_stream s = *stream; zfp_stream_set_bit_stream(&s, bs[chunk]); /* compress sequence of blocks */ for (block = bmin; block < bmax; block++) { /* determine block origin (x, y) within array */ const Scalar* p = data; size_t b = block; size_t x, y; x = 4 * (b % bx); b /= bx; y = 4 * b; p += sx * (ptrdiff_t)x + sy * (ptrdiff_t)y; /* compress partial or full block */ if (nx - x < 4u || ny - y < 4u) _t2(zfp_encode_partial_block_strided, Scalar, 2)(&s, p, MIN(nx - x, 4u), MIN(ny - y, 4u), sx, sy); else _t2(zfp_encode_block_strided, Scalar, 2)(&s, p, sx, sy); } } /* concatenate per-thread streams */ compress_finish_par(stream, bs, chunks); } /* compress 3d strided array in parallel */ static void _t2(compress_strided_omp, Scalar, 3)(zfp_stream* stream, const zfp_field* field) { /* array metadata */ const Scalar* data = (const Scalar*)field->data; size_t nx = field->nx; size_t ny = field->ny; size_t nz = field->nz; ptrdiff_t sx = field->sx ? field->sx : 1; ptrdiff_t sy = field->sy ? field->sy : (ptrdiff_t)nx; ptrdiff_t sz = field->sz ? field->sz : (ptrdiff_t)(nx * ny); /* number of omp threads, blocks, and chunks */ uint threads = thread_count_omp(stream); size_t bx = (nx + 3) / 4; size_t by = (ny + 3) / 4; size_t bz = (nz + 3) / 4; size_t blocks = bx * by * bz; size_t chunks = chunk_count_omp(stream, blocks, threads); int chunk; /* OpenMP 2.0 requires int loop counter */ /* allocate per-thread streams */ bitstream** bs = compress_init_par(stream, field, chunks, blocks); if (!bs) return; /* compress chunks of blocks in parallel */ #pragma omp parallel for num_threads(threads) for (chunk = 0; chunk < (int)chunks; chunk++) { /* determine range of block indices assigned to this thread */ size_t bmin = chunk_offset(blocks, chunks, chunk + 0); size_t bmax = chunk_offset(blocks, chunks, chunk + 1); size_t block; /* set up thread-local bit stream */ zfp_stream s = *stream; zfp_stream_set_bit_stream(&s, bs[chunk]); /* compress sequence of blocks */ for (block = bmin; block < bmax; block++) { /* determine block origin (x, y, z) within array */ const Scalar* p = data; size_t b = block; size_t x, y, z; x = 4 * (b % bx); b /= bx; y = 4 * (b % by); b /= by; z = 4 * b; p += sx * (ptrdiff_t)x + sy * (ptrdiff_t)y + sz * (ptrdiff_t)z; /* compress partial or full block */ if (nx - x < 4u || ny - y < 4u || nz - z < 4u) _t2(zfp_encode_partial_block_strided, Scalar, 3)(&s, p, MIN(nx - x, 4u), MIN(ny - y, 4u), MIN(nz - z, 4u), sx, sy, sz); else _t2(zfp_encode_block_strided, Scalar, 3)(&s, p, sx, sy, sz); } } /* concatenate per-thread streams */ compress_finish_par(stream, bs, chunks); } /* compress 4d strided array in parallel */ static void _t2(compress_strided_omp, Scalar, 4)(zfp_stream* stream, const zfp_field* field) { /* array metadata */ const Scalar* data = field->data; size_t nx = field->nx; size_t ny = field->ny; size_t nz = field->nz; size_t nw = field->nw; ptrdiff_t sx = field->sx ? field->sx : 1; ptrdiff_t sy = field->sy ? field->sy : (ptrdiff_t)nx; ptrdiff_t sz = field->sz ? field->sz : (ptrdiff_t)(nx * ny); ptrdiff_t sw = field->sw ? field->sw : (ptrdiff_t)(nx * ny * nz); /* number of omp threads, blocks, and chunks */ uint threads = thread_count_omp(stream); size_t bx = (nx + 3) / 4; size_t by = (ny + 3) / 4; size_t bz = (nz + 3) / 4; size_t bw = (nw + 3) / 4; size_t blocks = bx * by * bz * bw; size_t chunks = chunk_count_omp(stream, blocks, threads); int chunk; /* OpenMP 2.0 requires int loop counter */ /* allocate per-thread streams */ bitstream** bs = compress_init_par(stream, field, chunks, blocks); if (!bs) return; /* compress chunks of blocks in parallel */ #pragma omp parallel for num_threads(threads) for (chunk = 0; chunk < (int)chunks; chunk++) { /* determine range of block indices assigned to this thread */ size_t bmin = chunk_offset(blocks, chunks, chunk + 0); size_t bmax = chunk_offset(blocks, chunks, chunk + 1); size_t block; /* set up thread-local bit stream */ zfp_stream s = *stream; zfp_stream_set_bit_stream(&s, bs[chunk]); /* compress sequence of blocks */ for (block = bmin; block < bmax; block++) { /* determine block origin (x, y, z, w) within array */ const Scalar* p = data; size_t b = block; size_t x, y, z, w; x = 4 * (b % bx); b /= bx; y = 4 * (b % by); b /= by; z = 4 * (b % bz); b /= bz; w = 4 * b; p += sx * (ptrdiff_t)x + sy * (ptrdiff_t)y + sz * (ptrdiff_t)z + sw * (ptrdiff_t)w; /* compress partial or full block */ if (nx - x < 4u || ny - y < 4u || nz - z < 4u || nw - w < 4u) _t2(zfp_encode_partial_block_strided, Scalar, 4)(&s, p, MIN(nx - x, 4u), MIN(ny - y, 4u), MIN(nz - z, 4u), MIN(nw - w, 4u), sx, sy, sz, sw); else _t2(zfp_encode_block_strided, Scalar, 4)(&s, p, sx, sy, sz, sw); } } /* concatenate per-thread streams */ compress_finish_par(stream, bs, chunks); } #endif zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/revdecode4.c0000644000175200007730000000160314515254731025224 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* reversible inverse decorrelating 4D transform */ static void _t2(rev_inv_xform, Int, 4)(Int* p) { uint x, y, z, w; /* transform along w */ for (z = 0; z < 4; z++) for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) _t1(rev_inv_lift, Int)(p + 1 * x + 4 * y + 16 * z, 64); /* transform along z */ for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) for (w = 0; w < 4; w++) _t1(rev_inv_lift, Int)(p + 64 * w + 1 * x + 4 * y, 16); /* transform along y */ for (x = 0; x < 4; x++) for (w = 0; w < 4; w++) for (z = 0; z < 4; z++) _t1(rev_inv_lift, Int)(p + 16 * z + 64 * w + 1 * x, 4); /* transform along x */ for (w = 0; w < 4; w++) for (z = 0; z < 4; z++) for (y = 0; y < 4; y++) _t1(rev_inv_lift, Int)(p + 4 * y + 16 * z + 64 * w, 1); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/decodef.c0000644000175200007730000000273114515254731024574 0ustar rlaboissrlaboissstatic uint _t2(rev_decode_block, Scalar, DIMS)(zfp_stream* zfp, Scalar* fblock); /* private functions ------------------------------------------------------- */ /* decode contiguous floating-point block using lossy algorithm */ static uint _t2(decode_block, Scalar, DIMS)(zfp_stream* zfp, Scalar* fblock) { uint bits = 1; /* test if block has nonzero values */ if (stream_read_bit(zfp->stream)) { cache_align_(Int iblock[BLOCK_SIZE]); int emax, maxprec; /* decode common exponent */ bits += EBITS; emax = (int)stream_read_bits(zfp->stream, EBITS) - EBIAS; maxprec = precision(emax, zfp->maxprec, zfp->minexp, DIMS); /* decode integer block */ bits += _t2(decode_block, Int, DIMS)(zfp->stream, zfp->minbits - bits, zfp->maxbits - bits, maxprec, iblock); /* perform inverse block-floating-point transform */ _t1(inv_cast, Scalar)(iblock, fblock, BLOCK_SIZE, emax); } else { /* set all values to zero */ uint i; for (i = 0; i < BLOCK_SIZE; i++) *fblock++ = 0; if (zfp->minbits > bits) { stream_skip(zfp->stream, zfp->minbits - bits); bits = zfp->minbits; } } return bits; } /* public functions -------------------------------------------------------- */ /* decode contiguous floating-point block */ size_t _t2(zfp_decode_block, Scalar, DIMS)(zfp_stream* zfp, Scalar* fblock) { return REVERSIBLE(zfp) ? _t2(rev_decode_block, Scalar, DIMS)(zfp, fblock) : _t2(decode_block, Scalar, DIMS)(zfp, fblock); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/codec2.c0000644000175200007730000000115014515254731024334 0ustar rlaboissrlaboiss#define index(i, j) ((i) + 4 * (j)) /* order coefficients (i, j) by i + j, then i^2 + j^2 */ cache_align_(static const uchar perm_2[16]) = { index(0, 0), /* 0 : 0 */ index(1, 0), /* 1 : 1 */ index(0, 1), /* 2 : 1 */ index(1, 1), /* 3 : 2 */ index(2, 0), /* 4 : 2 */ index(0, 2), /* 5 : 2 */ index(2, 1), /* 6 : 3 */ index(1, 2), /* 7 : 3 */ index(3, 0), /* 8 : 3 */ index(0, 3), /* 9 : 3 */ index(2, 2), /* 10 : 4 */ index(3, 1), /* 11 : 4 */ index(1, 3), /* 12 : 4 */ index(3, 2), /* 13 : 5 */ index(2, 3), /* 14 : 5 */ index(3, 3), /* 15 : 6 */ }; #undef index zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/revcodecf.c0000644000175200007730000000052614515254731025143 0ustar rlaboissrlaboiss/* inverse block-floating-point transform from signed integers */ static void _t1(rev_inv_cast, Scalar)(const Int* iblock, Scalar* fblock, uint n, int emax) { /* test for all-zero block, which needs special treatment */ if (emax != -EBIAS) _t1(inv_cast, Scalar)(iblock, fblock, n, emax); else while (n--) *fblock++ = 0; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/compress.c0000644000175200007730000000751314515254731025041 0ustar rlaboissrlaboiss/* compress 1d contiguous array */ static void _t2(compress, Scalar, 1)(zfp_stream* stream, const zfp_field* field) { const Scalar* data = (const Scalar*)field->data; size_t nx = field->nx; size_t mx = nx & ~3u; size_t x; /* compress array one block of 4 values at a time */ for (x = 0; x < mx; x += 4, data += 4) _t2(zfp_encode_block, Scalar, 1)(stream, data); if (x < nx) _t2(zfp_encode_partial_block_strided, Scalar, 1)(stream, data, nx - x, 1); } /* compress 1d strided array */ static void _t2(compress_strided, Scalar, 1)(zfp_stream* stream, const zfp_field* field) { const Scalar* data = field->data; size_t nx = field->nx; ptrdiff_t sx = field->sx ? field->sx : 1; size_t x; /* compress array one block of 4 values at a time */ for (x = 0; x < nx; x += 4) { const Scalar* p = data + sx * (ptrdiff_t)x; if (nx - x < 4) _t2(zfp_encode_partial_block_strided, Scalar, 1)(stream, p, nx - x, sx); else _t2(zfp_encode_block_strided, Scalar, 1)(stream, p, sx); } } /* compress 2d strided array */ static void _t2(compress_strided, Scalar, 2)(zfp_stream* stream, const zfp_field* field) { const Scalar* data = (const Scalar*)field->data; size_t nx = field->nx; size_t ny = field->ny; ptrdiff_t sx = field->sx ? field->sx : 1; ptrdiff_t sy = field->sy ? field->sy : (ptrdiff_t)nx; size_t x, y; /* compress array one block of 4x4 values at a time */ for (y = 0; y < ny; y += 4) for (x = 0; x < nx; x += 4) { const Scalar* p = data + sx * (ptrdiff_t)x + sy * (ptrdiff_t)y; if (nx - x < 4 || ny - y < 4) _t2(zfp_encode_partial_block_strided, Scalar, 2)(stream, p, MIN(nx - x, 4u), MIN(ny - y, 4u), sx, sy); else _t2(zfp_encode_block_strided, Scalar, 2)(stream, p, sx, sy); } } /* compress 3d strided array */ static void _t2(compress_strided, Scalar, 3)(zfp_stream* stream, const zfp_field* field) { const Scalar* data = (const Scalar*)field->data; size_t nx = field->nx; size_t ny = field->ny; size_t nz = field->nz; ptrdiff_t sx = field->sx ? field->sx : 1; ptrdiff_t sy = field->sy ? field->sy : (ptrdiff_t)nx; ptrdiff_t sz = field->sz ? field->sz : (ptrdiff_t)(nx * ny); size_t x, y, z; /* compress array one block of 4x4x4 values at a time */ for (z = 0; z < nz; z += 4) for (y = 0; y < ny; y += 4) for (x = 0; x < nx; x += 4) { const Scalar* p = data + sx * (ptrdiff_t)x + sy * (ptrdiff_t)y + sz * (ptrdiff_t)z; if (nx - x < 4 || ny - y < 4 || nz - z < 4) _t2(zfp_encode_partial_block_strided, Scalar, 3)(stream, p, MIN(nx - x, 4u), MIN(ny - y, 4u), MIN(nz - z, 4u), sx, sy, sz); else _t2(zfp_encode_block_strided, Scalar, 3)(stream, p, sx, sy, sz); } } /* compress 4d strided array */ static void _t2(compress_strided, Scalar, 4)(zfp_stream* stream, const zfp_field* field) { const Scalar* data = field->data; size_t nx = field->nx; size_t ny = field->ny; size_t nz = field->nz; size_t nw = field->nw; ptrdiff_t sx = field->sx ? field->sx : 1; ptrdiff_t sy = field->sy ? field->sy : (ptrdiff_t)nx; ptrdiff_t sz = field->sz ? field->sz : (ptrdiff_t)(nx * ny); ptrdiff_t sw = field->sw ? field->sw : (ptrdiff_t)(nx * ny * nz); size_t x, y, z, w; /* compress array one block of 4x4x4x4 values at a time */ for (w = 0; w < nw; w += 4) for (z = 0; z < nz; z += 4) for (y = 0; y < ny; y += 4) for (x = 0; x < nx; x += 4) { const Scalar* p = data + sx * (ptrdiff_t)x + sy * (ptrdiff_t)y + sz * (ptrdiff_t)z + sw * (ptrdiff_t)w; if (nx - x < 4 || ny - y < 4 || nz - z < 4 || nw - w < 4) _t2(zfp_encode_partial_block_strided, Scalar, 4)(stream, p, MIN(nx - x, 4u), MIN(ny - y, 4u), MIN(nz - z, 4u), MIN(nw - w, 4u), sx, sy, sz, sw); else _t2(zfp_encode_block_strided, Scalar, 4)(stream, p, sx, sy, sz, sw); } } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/decode2.c0000644000175200007730000000350314515254731024506 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* scatter 4*4 block to strided array */ static void _t2(scatter, Scalar, 2)(const Scalar* q, Scalar* p, ptrdiff_t sx, ptrdiff_t sy) { uint x, y; for (y = 0; y < 4; y++, p += sy - 4 * sx) for (x = 0; x < 4; x++, p += sx) *p = *q++; } /* scatter nx*ny block to strided array */ static void _t2(scatter_partial, Scalar, 2)(const Scalar* q, Scalar* p, size_t nx, size_t ny, ptrdiff_t sx, ptrdiff_t sy) { size_t x, y; for (y = 0; y < ny; y++, p += sy - (ptrdiff_t)nx * sx, q += 4 - nx) for (x = 0; x < nx; x++, p += sx, q++) *p = *q; } /* inverse decorrelating 2D transform */ static void _t2(inv_xform, Int, 2)(Int* p) { uint x, y; /* transform along y */ for (x = 0; x < 4; x++) _t1(inv_lift, Int)(p + 1 * x, 4); /* transform along x */ for (y = 0; y < 4; y++) _t1(inv_lift, Int)(p + 4 * y, 1); } /* public functions -------------------------------------------------------- */ /* decode 4*4 block and store at p using strides (sx, sy) */ size_t _t2(zfp_decode_block_strided, Scalar, 2)(zfp_stream* stream, Scalar* p, ptrdiff_t sx, ptrdiff_t sy) { /* decode contiguous block */ cache_align_(Scalar block[16]); size_t bits = _t2(zfp_decode_block, Scalar, 2)(stream, block); /* scatter block to strided array */ _t2(scatter, Scalar, 2)(block, p, sx, sy); return bits; } /* decode nx*ny block and store at p using strides (sx, sy) */ size_t _t2(zfp_decode_partial_block_strided, Scalar, 2)(zfp_stream* stream, Scalar* p, size_t nx, size_t ny, ptrdiff_t sx, ptrdiff_t sy) { /* decode contiguous block */ cache_align_(Scalar block[16]); size_t bits = _t2(zfp_decode_block, Scalar, 2)(stream, block); /* scatter block to strided array */ _t2(scatter_partial, Scalar, 2)(block, p, nx, ny, sx, sy); return bits; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/decode3.c0000644000175200007730000000442114515254731024507 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* scatter 4*4*4 block to strided array */ static void _t2(scatter, Scalar, 3)(const Scalar* q, Scalar* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz) { uint x, y, z; for (z = 0; z < 4; z++, p += sz - 4 * sy) for (y = 0; y < 4; y++, p += sy - 4 * sx) for (x = 0; x < 4; x++, p += sx) *p = *q++; } /* scatter nx*ny*nz block to strided array */ static void _t2(scatter_partial, Scalar, 3)(const Scalar* q, Scalar* p, size_t nx, size_t ny, size_t nz, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz) { size_t x, y, z; for (z = 0; z < nz; z++, p += sz - (ptrdiff_t)ny * sy, q += 4 * (4 - ny)) for (y = 0; y < ny; y++, p += sy - (ptrdiff_t)nx * sx, q += 1 * (4 - nx)) for (x = 0; x < nx; x++, p += sx, q++) *p = *q; } /* inverse decorrelating 3D transform */ static void _t2(inv_xform, Int, 3)(Int* p) { uint x, y, z; /* transform along z */ for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) _t1(inv_lift, Int)(p + 1 * x + 4 * y, 16); /* transform along y */ for (x = 0; x < 4; x++) for (z = 0; z < 4; z++) _t1(inv_lift, Int)(p + 16 * z + 1 * x, 4); /* transform along x */ for (z = 0; z < 4; z++) for (y = 0; y < 4; y++) _t1(inv_lift, Int)(p + 4 * y + 16 * z, 1); } /* public functions -------------------------------------------------------- */ /* decode 4*4*4 block and store at p using strides (sx, sy, sz) */ size_t _t2(zfp_decode_block_strided, Scalar, 3)(zfp_stream* stream, Scalar* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz) { /* decode contiguous block */ cache_align_(Scalar block[64]); size_t bits = _t2(zfp_decode_block, Scalar, 3)(stream, block); /* scatter block to strided array */ _t2(scatter, Scalar, 3)(block, p, sx, sy, sz); return bits; } /* decode nx*ny*nz block and store at p using strides (sx, sy, sz) */ size_t _t2(zfp_decode_partial_block_strided, Scalar, 3)(zfp_stream* stream, Scalar* p, size_t nx, size_t ny, size_t nz, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz) { /* decode contiguous block */ cache_align_(Scalar block[64]); size_t bits = _t2(zfp_decode_block, Scalar, 3)(stream, block); /* scatter block to strided array */ _t2(scatter_partial, Scalar, 3)(block, p, nx, ny, nz, sx, sy, sz); return bits; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/decode4.c0000644000175200007730000000546614515254731024522 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* scatter 4*4*4*4 block to strided array */ static void _t2(scatter, Scalar, 4)(const Scalar* q, Scalar* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw) { uint x, y, z, w; for (w = 0; w < 4; w++, p += sw - 4 * sz) for (z = 0; z < 4; z++, p += sz - 4 * sy) for (y = 0; y < 4; y++, p += sy - 4 * sx) for (x = 0; x < 4; x++, p += sx) *p = *q++; } /* scatter nx*ny*nz*nw block to strided array */ static void _t2(scatter_partial, Scalar, 4)(const Scalar* q, Scalar* p, size_t nx, size_t ny, size_t nz, size_t nw, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw) { size_t x, y, z, w; for (w = 0; w < nw; w++, p += sw - (ptrdiff_t)nz * sz, q += 16 * (4 - nz)) for (z = 0; z < nz; z++, p += sz - (ptrdiff_t)ny * sy, q += 4 * (4 - ny)) for (y = 0; y < ny; y++, p += sy - (ptrdiff_t)nx * sx, q += 1 * (4 - nx)) for (x = 0; x < nx; x++, p += sx, q++) *p = *q; } /* inverse decorrelating 4D transform */ static void _t2(inv_xform, Int, 4)(Int* p) { uint x, y, z, w; /* transform along w */ for (z = 0; z < 4; z++) for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) _t1(inv_lift, Int)(p + 1 * x + 4 * y + 16 * z, 64); /* transform along z */ for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) for (w = 0; w < 4; w++) _t1(inv_lift, Int)(p + 64 * w + 1 * x + 4 * y, 16); /* transform along y */ for (x = 0; x < 4; x++) for (w = 0; w < 4; w++) for (z = 0; z < 4; z++) _t1(inv_lift, Int)(p + 16 * z + 64 * w + 1 * x, 4); /* transform along x */ for (w = 0; w < 4; w++) for (z = 0; z < 4; z++) for (y = 0; y < 4; y++) _t1(inv_lift, Int)(p + 4 * y + 16 * z + 64 * w, 1); } /* public functions -------------------------------------------------------- */ /* decode 4*4*4*4 block and store at p using strides (sx, sy, sz, sw) */ size_t _t2(zfp_decode_block_strided, Scalar, 4)(zfp_stream* stream, Scalar* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw) { /* decode contiguous block */ cache_align_(Scalar block[256]); size_t bits = _t2(zfp_decode_block, Scalar, 4)(stream, block); /* scatter block to strided array */ _t2(scatter, Scalar, 4)(block, p, sx, sy, sz, sw); return bits; } /* decode nx*ny*nz*nw block and store at p using strides (sx, sy, sz, sw) */ size_t _t2(zfp_decode_partial_block_strided, Scalar, 4)(zfp_stream* stream, Scalar* p, size_t nx, size_t ny, size_t nz, size_t nw, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw) { /* decode contiguous block */ cache_align_(Scalar block[256]); size_t bits = _t2(zfp_decode_block, Scalar, 4)(stream, block); /* scatter block to strided array */ _t2(scatter_partial, Scalar, 4)(block, p, nx, ny, nz, nw, sx, sy, sz, sw); return bits; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/encodei.c0000644000175200007730000000122614515254731024607 0ustar rlaboissrlaboissstatic uint _t2(rev_encode_block, Int, DIMS)(bitstream* stream, int minbits, int maxbits, int maxprec, Int* iblock); /* public functions -------------------------------------------------------- */ /* encode contiguous integer block */ size_t _t2(zfp_encode_block, Int, DIMS)(zfp_stream* zfp, const Int* iblock) { cache_align_(Int block[BLOCK_SIZE]); uint i; /* copy block */ for (i = 0; i < BLOCK_SIZE; i++) block[i] = iblock[i]; return REVERSIBLE(zfp) ? _t2(rev_encode_block, Int, DIMS)(zfp->stream, zfp->minbits, zfp->maxbits, zfp->maxprec, block) : _t2(encode_block, Int, DIMS)(zfp->stream, zfp->minbits, zfp->maxbits, zfp->maxprec, block); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/template.h0000644000175200007730000000046414515254731025024 0ustar rlaboissrlaboiss#ifndef TEMPLATE_H #define TEMPLATE_H /* concatenation */ #define _cat2(x, y) x ## _ ## y #define _cat3(x, y, z) x ## _ ## y ## _ ## z /* 1- and 2-argument function templates */ #define _t1(function, arg) _cat2(function, arg) #define _t2(function, type, dims) _cat3(function, type, dims) #endif zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/revencode.c0000644000175200007730000000370114515254731025153 0ustar rlaboissrlaboissstatic void _t2(rev_fwd_xform, Int, DIMS)(Int* p); /* private functions ------------------------------------------------------- */ /* reversible forward lifting transform of 4-vector */ static void _t1(rev_fwd_lift, Int)(Int* p, uint s) { Int x, y, z, w; x = *p; p += s; y = *p; p += s; z = *p; p += s; w = *p; p += s; /* ** high-order Lorenzo transform ** ( 1 0 0 0) (x) ** (-1 1 0 0) (y) ** ( 1 -2 1 0) (z) ** (-1 3 -3 1) (w) */ w -= z; z -= y; y -= x; w -= z; z -= y; w -= z; p -= s; *p = w; p -= s; *p = z; p -= s; *p = y; p -= s; *p = x; } /* return precision required to encode block reversibly */ static uint _t1(rev_precision, UInt)(const UInt* block, uint n) { uint p = 0; uint s; /* compute bitwise OR of all values */ UInt m = 0; while (n--) m |= *block++; /* count trailing zeros via binary search */ for (s = (uint)(CHAR_BIT * sizeof(UInt)); m; s /= 2) if ((UInt)(m << (s - 1))) { m <<= s - 1; m <<= 1; p += s; } return p; } /* encode block of integers using reversible algorithm */ static uint _t2(rev_encode_block, Int, DIMS)(bitstream* stream, int minbits, int maxbits, int maxprec, Int* iblock) { int bits = PBITS; int prec; cache_align_(UInt ublock[BLOCK_SIZE]); /* perform decorrelating transform */ _t2(rev_fwd_xform, Int, DIMS)(iblock); /* reorder signed coefficients and convert to unsigned integer */ _t1(fwd_order, Int)(ublock, iblock, PERM, BLOCK_SIZE); /* determine and encode number of significant bits */ prec = _t1(rev_precision, UInt)(ublock, BLOCK_SIZE); prec = MIN(prec, maxprec); prec = MAX(prec, 1); stream_write_bits(stream, prec - 1, PBITS); /* encode integer coefficients */ bits += _t1(encode_ints, UInt)(stream, maxbits - bits, prec, ublock, BLOCK_SIZE); /* write at least minbits bits by padding with zeros */ if (bits < minbits) { stream_pad(stream, minbits - bits); bits = minbits; } return bits; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/codec3.c0000644000175200007730000000421614515254731024343 0ustar rlaboissrlaboiss#define index(i, j, k) ((i) + 4 * ((j) + 4 * (k))) /* order coefficients (i, j, k) by i + j + k, then i^2 + j^2 + k^2 */ cache_align_(static const uchar perm_3[64]) = { index(0, 0, 0), /* 0 : 0 */ index(1, 0, 0), /* 1 : 1 */ index(0, 1, 0), /* 2 : 1 */ index(0, 0, 1), /* 3 : 1 */ index(0, 1, 1), /* 4 : 2 */ index(1, 0, 1), /* 5 : 2 */ index(1, 1, 0), /* 6 : 2 */ index(2, 0, 0), /* 7 : 2 */ index(0, 2, 0), /* 8 : 2 */ index(0, 0, 2), /* 9 : 2 */ index(1, 1, 1), /* 10 : 3 */ index(2, 1, 0), /* 11 : 3 */ index(2, 0, 1), /* 12 : 3 */ index(0, 2, 1), /* 13 : 3 */ index(1, 2, 0), /* 14 : 3 */ index(1, 0, 2), /* 15 : 3 */ index(0, 1, 2), /* 16 : 3 */ index(3, 0, 0), /* 17 : 3 */ index(0, 3, 0), /* 18 : 3 */ index(0, 0, 3), /* 19 : 3 */ index(2, 1, 1), /* 20 : 4 */ index(1, 2, 1), /* 21 : 4 */ index(1, 1, 2), /* 22 : 4 */ index(0, 2, 2), /* 23 : 4 */ index(2, 0, 2), /* 24 : 4 */ index(2, 2, 0), /* 25 : 4 */ index(3, 1, 0), /* 26 : 4 */ index(3, 0, 1), /* 27 : 4 */ index(0, 3, 1), /* 28 : 4 */ index(1, 3, 0), /* 29 : 4 */ index(1, 0, 3), /* 30 : 4 */ index(0, 1, 3), /* 31 : 4 */ index(1, 2, 2), /* 32 : 5 */ index(2, 1, 2), /* 33 : 5 */ index(2, 2, 1), /* 34 : 5 */ index(3, 1, 1), /* 35 : 5 */ index(1, 3, 1), /* 36 : 5 */ index(1, 1, 3), /* 37 : 5 */ index(3, 2, 0), /* 38 : 5 */ index(3, 0, 2), /* 39 : 5 */ index(0, 3, 2), /* 40 : 5 */ index(2, 3, 0), /* 41 : 5 */ index(2, 0, 3), /* 42 : 5 */ index(0, 2, 3), /* 43 : 5 */ index(2, 2, 2), /* 44 : 6 */ index(3, 2, 1), /* 45 : 6 */ index(3, 1, 2), /* 46 : 6 */ index(1, 3, 2), /* 47 : 6 */ index(2, 3, 1), /* 48 : 6 */ index(2, 1, 3), /* 49 : 6 */ index(1, 2, 3), /* 50 : 6 */ index(0, 3, 3), /* 51 : 6 */ index(3, 0, 3), /* 52 : 6 */ index(3, 3, 0), /* 53 : 6 */ index(3, 2, 2), /* 54 : 7 */ index(2, 3, 2), /* 55 : 7 */ index(2, 2, 3), /* 56 : 7 */ index(1, 3, 3), /* 57 : 7 */ index(3, 1, 3), /* 58 : 7 */ index(3, 3, 1), /* 59 : 7 */ index(2, 3, 3), /* 60 : 8 */ index(3, 2, 3), /* 61 : 8 */ index(3, 3, 2), /* 62 : 8 */ index(3, 3, 3), /* 63 : 9 */ }; #undef index zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/revencode3.c0000644000175200007730000000112714515254731025236 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* reversible forward decorrelating 3D transform */ static void _t2(rev_fwd_xform, Int, 3)(Int* p) { uint x, y, z; /* transform along x */ for (z = 0; z < 4; z++) for (y = 0; y < 4; y++) _t1(rev_fwd_lift, Int)(p + 4 * y + 16 * z, 1); /* transform along y */ for (x = 0; x < 4; x++) for (z = 0; z < 4; z++) _t1(rev_fwd_lift, Int)(p + 16 * z + 1 * x, 4); /* transform along z */ for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) _t1(rev_fwd_lift, Int)(p + 1 * x + 4 * y, 16); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/codecf.c0000644000175200007730000000167014515254731024427 0ustar rlaboissrlaboiss#include #include /* maximum number of bit planes to encode */ static uint precision(int maxexp, uint maxprec, int minexp, int dims) { #if (ZFP_ROUNDING_MODE != ZFP_ROUND_NEVER) && defined(ZFP_WITH_TIGHT_ERROR) return MIN(maxprec, (uint)MAX(0, maxexp - minexp + 2 * dims + 1)); #else return MIN(maxprec, (uint)MAX(0, maxexp - minexp + 2 * dims + 2)); #endif } /* map integer x relative to exponent e to floating-point number */ static Scalar _t1(dequantize, Scalar)(Int x, int e) { return LDEXP((Scalar)x, e - ((int)(CHAR_BIT * sizeof(Scalar)) - 2)); } /* inverse block-floating-point transform from signed integers */ static void _t1(inv_cast, Scalar)(const Int* iblock, Scalar* fblock, uint n, int emax) { /* compute power-of-two scale factor s */ Scalar s = _t1(dequantize, Scalar)(1, emax); /* compute p-bit float x = s*y where |y| <= 2^(p-2) - 1 */ do *fblock++ = (Scalar)(s * *iblock++); while (--n); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/revencode1.c0000644000175200007730000000036214515254731025234 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* reversible forward decorrelating 1D transform */ static void _t2(rev_fwd_xform, Int, 1)(Int* p) { /* transform along x */ _t1(rev_fwd_lift, Int)(p, 1); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/revencode4.c0000644000175200007730000000160314515254731025236 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* reversible forward decorrelating 4D transform */ static void _t2(rev_fwd_xform, Int, 4)(Int* p) { uint x, y, z, w; /* transform along x */ for (w = 0; w < 4; w++) for (z = 0; z < 4; z++) for (y = 0; y < 4; y++) _t1(rev_fwd_lift, Int)(p + 4 * y + 16 * z + 64 * w, 1); /* transform along y */ for (x = 0; x < 4; x++) for (w = 0; w < 4; w++) for (z = 0; z < 4; z++) _t1(rev_fwd_lift, Int)(p + 16 * z + 64 * w + 1 * x, 4); /* transform along z */ for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) for (w = 0; w < 4; w++) _t1(rev_fwd_lift, Int)(p + 64 * w + 1 * x + 4 * y, 16); /* transform along w */ for (z = 0; z < 4; z++) for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) _t1(rev_fwd_lift, Int)(p + 1 * x + 4 * y + 16 * z, 64); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/revdecode1.c0000644000175200007730000000036214515254731025222 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* reversible inverse decorrelating 1D transform */ static void _t2(rev_inv_xform, Int, 1)(Int* p) { /* transform along x */ _t1(rev_inv_lift, Int)(p, 1); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/revdecode2.c0000644000175200007730000000060114515254731025217 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* reversible inverse decorrelating 2D transform */ static void _t2(rev_inv_xform, Int, 2)(Int* p) { uint x, y; /* transform along y */ for (x = 0; x < 4; x++) _t1(rev_inv_lift, Int)(p + 1 * x, 4); /* transform along x */ for (y = 0; y < 4; y++) _t1(rev_inv_lift, Int)(p + 4 * y, 1); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/revencode2.c0000644000175200007730000000060114515254731025231 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* reversible forward decorrelating 2D transform */ static void _t2(rev_fwd_xform, Int, 2)(Int* p) { uint x, y; /* transform along x */ for (y = 0; y < 4; y++) _t1(rev_fwd_lift, Int)(p + 4 * y, 1); /* transform along y */ for (x = 0; x < 4; x++) _t1(rev_fwd_lift, Int)(p + 1 * x, 4); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/decode.c0000644000175200007730000002100414515254731024420 0ustar rlaboissrlaboiss#include static void _t2(inv_xform, Int, DIMS)(Int* p); /* private functions ------------------------------------------------------- */ /* inverse lifting transform of 4-vector */ static void _t1(inv_lift, Int)(Int* p, ptrdiff_t s) { Int x, y, z, w; x = *p; p += s; y = *p; p += s; z = *p; p += s; w = *p; p += s; /* ** non-orthogonal transform ** ( 4 6 -4 -1) (x) ** 1/4 * ( 4 2 4 5) (y) ** ( 4 -2 4 -5) (z) ** ( 4 -6 -4 1) (w) */ y += w >> 1; w -= y >> 1; y += w; w <<= 1; w -= y; z += x; x <<= 1; x -= z; y += z; z <<= 1; z -= y; w += x; x <<= 1; x -= w; p -= s; *p = w; p -= s; *p = z; p -= s; *p = y; p -= s; *p = x; } #if ZFP_ROUNDING_MODE == ZFP_ROUND_LAST /* bias values such that truncation is equivalent to round to nearest */ static void _t1(inv_round, UInt)(UInt* ublock, uint n, uint m, uint prec) { /* add 1/6 ulp to unbias errors */ if (prec < (uint)(CHAR_BIT * sizeof(UInt) - 1)) { /* the first m values (0 <= m <= n) have one more bit of precision */ n -= m; while (m--) *ublock++ += ((NBMASK >> 2) >> prec); while (n--) *ublock++ += ((NBMASK >> 1) >> prec); } } #endif /* map two's complement signed integer to negabinary unsigned integer */ static Int _t1(uint2int, UInt)(UInt x) { return (Int)((x ^ NBMASK) - NBMASK); } /* reorder unsigned coefficients and convert to signed integer */ static void _t1(inv_order, Int)(const UInt* ublock, Int* iblock, const uchar* perm, uint n) { do iblock[*perm++] = _t1(uint2int, UInt)(*ublock++); while (--n); } /* decompress sequence of size <= 64 unsigned integers */ static uint _t1(decode_few_ints, UInt)(bitstream* restrict_ stream, uint maxbits, uint maxprec, UInt* restrict_ data, uint size) { /* make a copy of bit stream to avoid aliasing */ bitstream s = *stream; uint intprec = (uint)(CHAR_BIT * sizeof(UInt)); uint kmin = intprec > maxprec ? intprec - maxprec : 0; uint bits = maxbits; uint i, k, m, n; uint64 x; /* initialize data array to all zeros */ for (i = 0; i < size; i++) data[i] = 0; /* decode one bit plane at a time from MSB to LSB */ for (k = intprec, m = n = 0; bits && (m = 0, k-- > kmin);) { /* step 1: decode first n bits of bit plane #k */ m = MIN(n, bits); bits -= m; x = stream_read_bits(&s, m); /* step 2: unary run-length decode remainder of bit plane */ for (; bits && n < size; n++, m = n) { bits--; if (stream_read_bit(&s)) { /* positive group test; scan for next one-bit */ for (; bits && n < size - 1; n++) { bits--; if (stream_read_bit(&s)) break; } /* set bit and continue decoding bit plane */ x += (uint64)1 << n; } else { /* negative group test; done with bit plane */ m = size; break; } } /* step 3: deposit bit plane from x */ for (i = 0; x; i++, x >>= 1) data[i] += (UInt)(x & 1u) << k; } #if ZFP_ROUNDING_MODE == ZFP_ROUND_LAST /* bias values to achieve proper rounding */ _t1(inv_round, UInt)(data, size, m, intprec - k); #endif *stream = s; return maxbits - bits; } /* decompress sequence of size > 64 unsigned integers */ static uint _t1(decode_many_ints, UInt)(bitstream* restrict_ stream, uint maxbits, uint maxprec, UInt* restrict_ data, uint size) { /* make a copy of bit stream to avoid aliasing */ bitstream s = *stream; uint intprec = (uint)(CHAR_BIT * sizeof(UInt)); uint kmin = intprec > maxprec ? intprec - maxprec : 0; uint bits = maxbits; uint i, k, m, n; /* initialize data array to all zeros */ for (i = 0; i < size; i++) data[i] = 0; /* decode one bit plane at a time from MSB to LSB */ for (k = intprec, m = n = 0; bits && (m = 0, k-- > kmin);) { /* step 1: decode first n bits of bit plane #k */ m = MIN(n, bits); bits -= m; for (i = 0; i < m; i++) if (stream_read_bit(&s)) data[i] += (UInt)1 << k; /* step 2: unary run-length decode remainder of bit plane */ for (; bits && n < size; n++, m = n) { bits--; if (stream_read_bit(&s)) { /* positive group test; scan for next one-bit */ for (; bits && n < size - 1; n++) { bits--; if (stream_read_bit(&s)) break; } /* set bit and continue decoding bit plane */ data[n] += (UInt)1 << k; } else { /* negative group test; done with bit plane */ m = size; break; } } } #if ZFP_ROUNDING_MODE == ZFP_ROUND_LAST /* bias values to achieve proper rounding */ _t1(inv_round, UInt)(data, size, m, intprec - k); #endif *stream = s; return maxbits - bits; } /* decompress sequence of size <= 64 unsigned integers with no rate constraint */ static uint _t1(decode_few_ints_prec, UInt)(bitstream* restrict_ stream, uint maxprec, UInt* restrict_ data, uint size) { /* make a copy of bit stream to avoid aliasing */ bitstream s = *stream; size_t offset = stream_rtell(&s); uint intprec = (uint)(CHAR_BIT * sizeof(UInt)); uint kmin = intprec > maxprec ? intprec - maxprec : 0; uint i, k, n; /* initialize data array to all zeros */ for (i = 0; i < size; i++) data[i] = 0; /* decode one bit plane at a time from MSB to LSB */ for (k = intprec, n = 0; k-- > kmin;) { /* step 1: decode first n bits of bit plane #k */ uint64 x = stream_read_bits(&s, n); /* step 2: unary run-length decode remainder of bit plane */ for (; n < size && stream_read_bit(&s); x += (uint64)1 << n, n++) for (; n < size - 1 && !stream_read_bit(&s); n++) ; /* step 3: deposit bit plane from x */ for (i = 0; x; i++, x >>= 1) data[i] += (UInt)(x & 1u) << k; } #if ZFP_ROUNDING_MODE == ZFP_ROUND_LAST /* bias values to achieve proper rounding */ _t1(inv_round, UInt)(data, size, 0, intprec - k); #endif *stream = s; return (uint)(stream_rtell(&s) - offset); } /* decompress sequence of size > 64 unsigned integers with no rate constraint */ static uint _t1(decode_many_ints_prec, UInt)(bitstream* restrict_ stream, uint maxprec, UInt* restrict_ data, uint size) { /* make a copy of bit stream to avoid aliasing */ bitstream s = *stream; size_t offset = stream_rtell(&s); uint intprec = (uint)(CHAR_BIT * sizeof(UInt)); uint kmin = intprec > maxprec ? intprec - maxprec : 0; uint i, k, n; /* initialize data array to all zeros */ for (i = 0; i < size; i++) data[i] = 0; /* decode one bit plane at a time from MSB to LSB */ for (k = intprec, n = 0; k-- > kmin;) { /* step 1: decode first n bits of bit plane #k */ for (i = 0; i < n; i++) if (stream_read_bit(&s)) data[i] += (UInt)1 << k; /* step 2: unary run-length decode remainder of bit plane */ for (; n < size && stream_read_bit(&s); data[n] += (UInt)1 << k, n++) for (; n < size - 1 && !stream_read_bit(&s); n++) ; } #if ZFP_ROUNDING_MODE == ZFP_ROUND_LAST /* bias values to achieve proper rounding */ _t1(inv_round, UInt)(data, size, 0, intprec - k); #endif *stream = s; return (uint)(stream_rtell(&s) - offset); } /* decompress sequence of size unsigned integers */ static uint _t1(decode_ints, UInt)(bitstream* restrict_ stream, uint maxbits, uint maxprec, UInt* restrict_ data, uint size) { /* use fastest available decoder implementation */ if (with_maxbits(maxbits, maxprec, size)) { /* rate constrained path: decode partial bit planes */ if (size <= 64) return _t1(decode_few_ints, UInt)(stream, maxbits, maxprec, data, size); /* 1D, 2D, 3D blocks */ else return _t1(decode_many_ints, UInt)(stream, maxbits, maxprec, data, size); /* 4D blocks */ } else { /* variable-rate path: decode whole bit planes */ if (size <= 64) return _t1(decode_few_ints_prec, UInt)(stream, maxprec, data, size); /* 1D, 2D, 3D blocks */ else return _t1(decode_many_ints_prec, UInt)(stream, maxprec, data, size); /* 4D blocks */ } } /* decode block of integers */ static uint _t2(decode_block, Int, DIMS)(bitstream* stream, int minbits, int maxbits, int maxprec, Int* iblock) { int bits; cache_align_(UInt ublock[BLOCK_SIZE]); /* decode integer coefficients */ bits = _t1(decode_ints, UInt)(stream, maxbits, maxprec, ublock, BLOCK_SIZE); /* read at least minbits bits */ if (bits < minbits) { stream_skip(stream, minbits - bits); bits = minbits; } /* reorder unsigned coefficients and convert to signed integer */ _t1(inv_order, Int)(ublock, iblock, PERM, BLOCK_SIZE); /* perform decorrelating transform */ _t2(inv_xform, Int, DIMS)(iblock); return bits; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/decompress.c0000644000175200007730000000740314515254731025350 0ustar rlaboissrlaboiss/* decompress 1d contiguous array */ static void _t2(decompress, Scalar, 1)(zfp_stream* stream, zfp_field* field) { Scalar* data = (Scalar*)field->data; size_t nx = field->nx; size_t mx = nx & ~3u; size_t x; /* decompress array one block of 4 values at a time */ for (x = 0; x < mx; x += 4, data += 4) _t2(zfp_decode_block, Scalar, 1)(stream, data); if (x < nx) _t2(zfp_decode_partial_block_strided, Scalar, 1)(stream, data, nx - x, 1); } /* decompress 1d strided array */ static void _t2(decompress_strided, Scalar, 1)(zfp_stream* stream, zfp_field* field) { Scalar* data = field->data; size_t nx = field->nx; ptrdiff_t sx = field->sx ? field->sx : 1; size_t x; /* decompress array one block of 4 values at a time */ for (x = 0; x < nx; x += 4) { Scalar* p = data + sx * (ptrdiff_t)x; if (nx - x < 4) _t2(zfp_decode_partial_block_strided, Scalar, 1)(stream, p, nx - x, sx); else _t2(zfp_decode_block_strided, Scalar, 1)(stream, p, sx); } } /* decompress 2d strided array */ static void _t2(decompress_strided, Scalar, 2)(zfp_stream* stream, zfp_field* field) { Scalar* data = (Scalar*)field->data; size_t nx = field->nx; size_t ny = field->ny; ptrdiff_t sx = field->sx ? field->sx : 1; ptrdiff_t sy = field->sy ? field->sy : (ptrdiff_t)nx; size_t x, y; /* decompress array one block of 4x4 values at a time */ for (y = 0; y < ny; y += 4) for (x = 0; x < nx; x += 4) { Scalar* p = data + sx * (ptrdiff_t)x + sy * (ptrdiff_t)y; if (nx - x < 4 || ny - y < 4) _t2(zfp_decode_partial_block_strided, Scalar, 2)(stream, p, MIN(nx - x, 4u), MIN(ny - y, 4u), sx, sy); else _t2(zfp_decode_block_strided, Scalar, 2)(stream, p, sx, sy); } } /* decompress 3d strided array */ static void _t2(decompress_strided, Scalar, 3)(zfp_stream* stream, zfp_field* field) { Scalar* data = (Scalar*)field->data; size_t nx = field->nx; size_t ny = field->ny; size_t nz = field->nz; ptrdiff_t sx = field->sx ? field->sx : 1; ptrdiff_t sy = field->sy ? field->sy : (ptrdiff_t)nx; ptrdiff_t sz = field->sz ? field->sz : (ptrdiff_t)(nx * ny); size_t x, y, z; /* decompress array one block of 4x4x4 values at a time */ for (z = 0; z < nz; z += 4) for (y = 0; y < ny; y += 4) for (x = 0; x < nx; x += 4) { Scalar* p = data + sx * (ptrdiff_t)x + sy * (ptrdiff_t)y + sz * (ptrdiff_t)z; if (nx - x < 4 || ny - y < 4 || nz - z < 4) _t2(zfp_decode_partial_block_strided, Scalar, 3)(stream, p, MIN(nx - x, 4u), MIN(ny - y, 4u), MIN(nz - z, 4u), sx, sy, sz); else _t2(zfp_decode_block_strided, Scalar, 3)(stream, p, sx, sy, sz); } } /* decompress 4d strided array */ static void _t2(decompress_strided, Scalar, 4)(zfp_stream* stream, zfp_field* field) { Scalar* data = field->data; size_t nx = field->nx; size_t ny = field->ny; size_t nz = field->nz; size_t nw = field->nw; ptrdiff_t sx = field->sx ? field->sx : 1; ptrdiff_t sy = field->sy ? field->sy : (ptrdiff_t)nx; ptrdiff_t sz = field->sz ? field->sz : (ptrdiff_t)(nx * ny); ptrdiff_t sw = field->sw ? field->sw : (ptrdiff_t)(nx * ny * nz); size_t x, y, z, w; /* decompress array one block of 4x4x4x4 values at a time */ for (w = 0; w < nw; w += 4) for (z = 0; z < nz; z += 4) for (y = 0; y < ny; y += 4) for (x = 0; x < nx; x += 4) { Scalar* p = data + sx * (ptrdiff_t)x + sy * (ptrdiff_t)y + sz * (ptrdiff_t)z + sw * (ptrdiff_t)w; if (nx - x < 4 || ny - y < 4 || nz - z < 4 || nw - w < 4) _t2(zfp_decode_partial_block_strided, Scalar, 4)(stream, p, MIN(nx - x, 4u), MIN(ny - y, 4u), MIN(nz - z, 4u), MIN(nw - w, 4u), sx, sy, sz, sw); else _t2(zfp_decode_block_strided, Scalar, 4)(stream, p, sx, sy, sz, sw); } } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/encode4.c0000644000175200007730000000624014515254731024523 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* gather 4*4*4*4 block from strided array */ static void _t2(gather, Scalar, 4)(Scalar* q, const Scalar* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw) { uint x, y, z, w; for (w = 0; w < 4; w++, p += sw - 4 * sz) for (z = 0; z < 4; z++, p += sz - 4 * sy) for (y = 0; y < 4; y++, p += sy - 4 * sx) for (x = 0; x < 4; x++, p += sx) *q++ = *p; } /* gather nx*ny*nz*nw block from strided array */ static void _t2(gather_partial, Scalar, 4)(Scalar* q, const Scalar* p, size_t nx, size_t ny, size_t nz, size_t nw, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw) { size_t x, y, z, w; for (w = 0; w < nw; w++, p += sw - (ptrdiff_t)nz * sz) { for (z = 0; z < nz; z++, p += sz - (ptrdiff_t)ny * sy) { for (y = 0; y < ny; y++, p += sy - (ptrdiff_t)nx * sx) { for (x = 0; x < nx; x++, p += sx) q[64 * w + 16 * z + 4 * y + x] = *p; _t1(pad_block, Scalar)(q + 64 * w + 16 * z + 4 * y, nx, 1); } for (x = 0; x < 4; x++) _t1(pad_block, Scalar)(q + 64 * w + 16 * z + x, ny, 4); } for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) _t1(pad_block, Scalar)(q + 64 * w + 4 * y + x, nz, 16); } for (z = 0; z < 4; z++) for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) _t1(pad_block, Scalar)(q + 16 * z + 4 * y + x, nw, 64); } /* forward decorrelating 4D transform */ static void _t2(fwd_xform, Int, 4)(Int* p) { uint x, y, z, w; /* transform along x */ for (w = 0; w < 4; w++) for (z = 0; z < 4; z++) for (y = 0; y < 4; y++) _t1(fwd_lift, Int)(p + 4 * y + 16 * z + 64 * w, 1); /* transform along y */ for (x = 0; x < 4; x++) for (w = 0; w < 4; w++) for (z = 0; z < 4; z++) _t1(fwd_lift, Int)(p + 16 * z + 64 * w + 1 * x, 4); /* transform along z */ for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) for (w = 0; w < 4; w++) _t1(fwd_lift, Int)(p + 64 * w + 1 * x + 4 * y, 16); /* transform along w */ for (z = 0; z < 4; z++) for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) _t1(fwd_lift, Int)(p + 1 * x + 4 * y + 16 * z, 64); } /* public functions -------------------------------------------------------- */ /* encode 4*4*4*4 block stored at p using strides (sx, sy, sz, sw) */ size_t _t2(zfp_encode_block_strided, Scalar, 4)(zfp_stream* stream, const Scalar* p, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw) { /* gather block from strided array */ cache_align_(Scalar block[256]); _t2(gather, Scalar, 4)(block, p, sx, sy, sz, sw); /* encode block */ return _t2(zfp_encode_block, Scalar, 4)(stream, block); } /* encode nx*ny*nz*nw block stored at p using strides (sx, sy, sz, sw) */ size_t _t2(zfp_encode_partial_block_strided, Scalar, 4)(zfp_stream* stream, const Scalar* p, size_t nx, size_t ny, size_t nz, size_t nw, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw) { /* gather block from strided array */ cache_align_(Scalar block[256]); _t2(gather_partial, Scalar, 4)(block, p, nx, ny, nz, nw, sx, sy, sz, sw); /* encode block */ return _t2(zfp_encode_block, Scalar, 4)(stream, block); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/template/revdecode3.c0000644000175200007730000000112714515254731025224 0ustar rlaboissrlaboiss/* private functions ------------------------------------------------------- */ /* reversible inverse decorrelating 3D transform */ static void _t2(rev_inv_xform, Int, 3)(Int* p) { uint x, y, z; /* transform along z */ for (y = 0; y < 4; y++) for (x = 0; x < 4; x++) _t1(rev_inv_lift, Int)(p + 1 * x + 4 * y, 16); /* transform along y */ for (x = 0; x < 4; x++) for (z = 0; z < 4; z++) _t1(rev_inv_lift, Int)(p + 16 * z + 1 * x, 4); /* transform along x */ for (z = 0; z < 4; z++) for (y = 0; y < 4; y++) _t1(rev_inv_lift, Int)(p + 4 * y + 16 * z, 1); } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode1d.c0000644000175200007730000000076714515254731023047 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block1.h" #include "traitsd.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec1.c" #include "template/decode.c" #include "template/decodef.c" #include "template/decode1.c" #include "template/revcodecf.c" #include "template/revdecode.c" #include "template/revdecodef.c" #include "template/revdecode1.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode1f.c0000644000175200007730000000076714515254731023051 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block1.h" #include "traitsf.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec1.c" #include "template/decode.c" #include "template/decodef.c" #include "template/decode1.c" #include "template/revcodecf.c" #include "template/revdecode.c" #include "template/revdecodef.c" #include "template/revdecode1.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode3d.c0000644000175200007730000000076714515254731023063 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block3.h" #include "traitsd.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec3.c" #include "template/encode.c" #include "template/encodef.c" #include "template/encode3.c" #include "template/revcodecf.c" #include "template/revencode.c" #include "template/revencodef.c" #include "template/revencode3.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode4l.c0000644000175200007730000000063114515254731023050 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block4.h" #include "traitsl.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec4.c" #include "template/decode.c" #include "template/decodei.c" #include "template/decode4.c" #include "template/revdecode.c" #include "template/revdecode4.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode1i.c0000644000175200007730000000063114515254731023042 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block1.h" #include "traitsi.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec1.c" #include "template/decode.c" #include "template/decodei.c" #include "template/decode1.c" #include "template/revdecode.c" #include "template/revdecode1.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/traitsi.h0000644000175200007730000000051714515254731023054 0ustar rlaboissrlaboiss/* 32-bit integer traits */ #define Scalar int32 /* integer type */ #define Int int32 /* corresponding signed integer type */ #define UInt uint32 /* corresponding unsigned integer type */ #define PBITS 5 /* number of bits needed to encode precision */ #define NBMASK 0xaaaaaaaau /* negabinary mask */ zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode3f.c0000644000175200007730000000076714515254731023065 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block3.h" #include "traitsf.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec3.c" #include "template/encode.c" #include "template/encodef.c" #include "template/encode3.c" #include "template/revcodecf.c" #include "template/revencode.c" #include "template/revencodef.c" #include "template/revencode3.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode4f.c0000644000175200007730000000076714515254731023054 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block4.h" #include "traitsf.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec4.c" #include "template/decode.c" #include "template/decodef.c" #include "template/decode4.c" #include "template/revcodecf.c" #include "template/revdecode.c" #include "template/revdecodef.c" #include "template/revdecode4.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/traitsd.h0000644000175200007730000000136114515254731023045 0ustar rlaboissrlaboiss/* double-precision floating-point traits */ #define Scalar double /* floating-point type */ #define Int int64 /* corresponding signed integer type */ #define UInt uint64 /* corresponding unsigned integer type */ #define EBITS 11 /* number of exponent bits */ #define PBITS 6 /* number of bits needed to encode precision */ #define NBMASK UINT64C(0xaaaaaaaaaaaaaaaa) /* negabinary mask */ #define TCMASK UINT64C(0x7fffffffffffffff) /* two's complement mask */ #define SCALAR_MIN DBL_MIN /* smallest positive normal number */ #define FABS(x) fabs(x) #define FREXP(x, e) frexp(x, e) #define LDEXP(x, e) ldexp(x, e) zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/encode4d.c0000644000175200007730000000076714515254731023064 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block4.h" #include "traitsd.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codecf.c" #include "template/codec4.c" #include "template/encode.c" #include "template/encodef.c" #include "template/encode4.c" #include "template/revcodecf.c" #include "template/revencode.c" #include "template/revencodef.c" #include "template/revencode4.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode1l.c0000644000175200007730000000063114515254731023045 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block1.h" #include "traitsl.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec1.c" #include "template/decode.c" #include "template/decodei.c" #include "template/decode1.c" #include "template/revdecode.c" #include "template/revdecode1.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/block3.h0000644000175200007730000000001714515254731022545 0ustar rlaboissrlaboiss#define DIMS 3 zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/decode2l.c0000644000175200007730000000063114515254731023046 0ustar rlaboissrlaboiss#include "inline/inline.h" #include "zfp.h" #include "zfp/macros.h" #include "block2.h" #include "traitsl.h" #include "template/template.h" #include "template/codec.h" #include "inline/bitstream.c" #include "template/codec.c" #include "template/codec2.c" #include "template/decode.c" #include "template/decodei.c" #include "template/decode2.c" #include "template/revdecode.c" #include "template/revdecode2.c" zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/zfp.c0000644000175200007730000007775314515254731022207 0ustar rlaboissrlaboiss#include #include #include #include #include "zfp.h" #include "zfp/macros.h" #include "zfp/version.h" #include "template/template.h" /* public data ------------------------------------------------------------- */ const uint zfp_codec_version = ZFP_CODEC; const uint zfp_library_version = ZFP_VERSION; const char* const zfp_version_string = "zfp version " ZFP_VERSION_STRING " (May 5, 2019)"; /* private functions ------------------------------------------------------- */ static size_t field_index_span(const zfp_field* field, ptrdiff_t* min, ptrdiff_t* max) { /* compute strides */ ptrdiff_t sx = field->sx ? field->sx : 1; ptrdiff_t sy = field->sy ? field->sy : (ptrdiff_t)field->nx; ptrdiff_t sz = field->sz ? field->sz : (ptrdiff_t)(field->nx * field->ny); ptrdiff_t sw = field->sw ? field->sw : (ptrdiff_t)(field->nx * field->ny * field->nz); /* compute largest offsets from base pointer */ ptrdiff_t dx = field->nx ? sx * (ptrdiff_t)(field->nx - 1) : 0; ptrdiff_t dy = field->ny ? sy * (ptrdiff_t)(field->ny - 1) : 0; ptrdiff_t dz = field->nz ? sz * (ptrdiff_t)(field->nz - 1) : 0; ptrdiff_t dw = field->nw ? sw * (ptrdiff_t)(field->nw - 1) : 0; /* compute lowest and highest offset */ ptrdiff_t imin = MIN(dx, 0) + MIN(dy, 0) + MIN(dz, 0) + MIN(dw, 0); ptrdiff_t imax = MAX(dx, 0) + MAX(dy, 0) + MAX(dz, 0) + MAX(dw, 0); if (min) *min = imin; if (max) *max = imax; return imax - imin + 1; } static zfp_bool is_reversible(const zfp_stream* zfp) { return zfp->minexp < ZFP_MIN_EXP; } /* shared code across template instances ------------------------------------*/ #include "share/parallel.c" #include "share/omp.c" /* template instantiation of integer and float compressor -------------------*/ #define Scalar int32 #include "template/compress.c" #include "template/decompress.c" #include "template/ompcompress.c" #undef Scalar #define Scalar int64 #include "template/compress.c" #include "template/decompress.c" #include "template/ompcompress.c" #undef Scalar #define Scalar float #include "template/compress.c" #include "template/decompress.c" #include "template/ompcompress.c" #undef Scalar #define Scalar double #include "template/compress.c" #include "template/decompress.c" #include "template/ompcompress.c" #undef Scalar /* public functions: miscellaneous ----------------------------------------- */ size_t zfp_type_size(zfp_type type) { switch (type) { case zfp_type_int32: return sizeof(int32); case zfp_type_int64: return sizeof(int64); case zfp_type_float: return sizeof(float); case zfp_type_double: return sizeof(double); default: return 0; } } /* public functions: fields ------------------------------------------------ */ zfp_field* zfp_field_alloc() { zfp_field* field = (zfp_field*)malloc(sizeof(zfp_field)); if (field) { field->type = zfp_type_none; field->nx = field->ny = field->nz = field->nw = 0; field->sx = field->sy = field->sz = field->sw = 0; field->data = 0; } return field; } zfp_field* zfp_field_1d(void* data, zfp_type type, size_t nx) { zfp_field* field = zfp_field_alloc(); if (field) { field->type = type; field->nx = nx; field->data = data; } return field; } zfp_field* zfp_field_2d(void* data, zfp_type type, size_t nx, size_t ny) { zfp_field* field = zfp_field_alloc(); if (field) { field->type = type; field->nx = nx; field->ny = ny; field->data = data; } return field; } zfp_field* zfp_field_3d(void* data, zfp_type type, size_t nx, size_t ny, size_t nz) { zfp_field* field = zfp_field_alloc(); if (field) { field->type = type; field->nx = nx; field->ny = ny; field->nz = nz; field->data = data; } return field; } zfp_field* zfp_field_4d(void* data, zfp_type type, size_t nx, size_t ny, size_t nz, size_t nw) { zfp_field* field = zfp_field_alloc(); if (field) { field->type = type; field->nx = nx; field->ny = ny; field->nz = nz; field->nw = nw; field->data = data; } return field; } void zfp_field_free(zfp_field* field) { free(field); } void* zfp_field_pointer(const zfp_field* field) { return field->data; } void* zfp_field_begin(const zfp_field* field) { if (field->data) { ptrdiff_t min; field_index_span(field, &min, NULL); return (void*)((uchar*)field->data + min * (ptrdiff_t)zfp_type_size(field->type)); } else return NULL; } zfp_type zfp_field_type(const zfp_field* field) { return field->type; } uint zfp_field_precision(const zfp_field* field) { return (uint)(CHAR_BIT * zfp_type_size(field->type)); } uint zfp_field_dimensionality(const zfp_field* field) { return field->nx ? field->ny ? field->nz ? field->nw ? 4 : 3 : 2 : 1 : 0; } size_t zfp_field_size(const zfp_field* field, size_t* size) { if (size) switch (zfp_field_dimensionality(field)) { case 4: size[3] = field->nw; /* FALLTHROUGH */ case 3: size[2] = field->nz; /* FALLTHROUGH */ case 2: size[1] = field->ny; /* FALLTHROUGH */ case 1: size[0] = field->nx; break; } return MAX(field->nx, 1u) * MAX(field->ny, 1u) * MAX(field->nz, 1u) * MAX(field->nw, 1u); } size_t zfp_field_size_bytes(const zfp_field* field) { return field_index_span(field, NULL, NULL) * zfp_type_size(field->type); } zfp_bool zfp_field_stride(const zfp_field* field, ptrdiff_t* stride) { if (stride) switch (zfp_field_dimensionality(field)) { case 4: stride[3] = field->sw ? field->sw : (ptrdiff_t)(field->nx * field->ny * field->nz); /* FALLTHROUGH */ case 3: stride[2] = field->sz ? field->sz : (ptrdiff_t)(field->nx * field->ny); /* FALLTHROUGH */ case 2: stride[1] = field->sy ? field->sy : (ptrdiff_t)field->nx; /* FALLTHROUGH */ case 1: stride[0] = field->sx ? field->sx : 1; break; } return field->sx || field->sy || field->sz || field->sw; } zfp_bool zfp_field_is_contiguous(const zfp_field* field) { return field_index_span(field, NULL, NULL) == zfp_field_size(field, NULL); } uint64 zfp_field_metadata(const zfp_field* field) { uint64 meta = 0; /* 48 bits for dimensions */ switch (zfp_field_dimensionality(field)) { case 1: if ((uint64)(field->nx - 1) >> 48) return ZFP_META_NULL; meta <<= 48; meta += field->nx - 1; break; case 2: if (((field->nx - 1) >> 24) || ((field->ny - 1) >> 24)) return ZFP_META_NULL; meta <<= 24; meta += field->ny - 1; meta <<= 24; meta += field->nx - 1; break; case 3: if (((field->nx - 1) >> 16) || ((field->ny - 1) >> 16) || ((field->nz - 1) >> 16)) return ZFP_META_NULL; meta <<= 16; meta += field->nz - 1; meta <<= 16; meta += field->ny - 1; meta <<= 16; meta += field->nx - 1; break; case 4: if (((field->nx - 1) >> 12) || ((field->ny - 1) >> 12) || ((field->nz - 1) >> 12) || ((field->nw - 1) >> 12)) return ZFP_META_NULL; meta <<= 12; meta += field->nw - 1; meta <<= 12; meta += field->nz - 1; meta <<= 12; meta += field->ny - 1; meta <<= 12; meta += field->nx - 1; break; } /* 2 bits for dimensionality (1D, 2D, 3D, 4D) */ meta <<= 2; meta += zfp_field_dimensionality(field) - 1; /* 2 bits for scalar type */ meta <<= 2; meta += field->type - 1; return meta; } void zfp_field_set_pointer(zfp_field* field, void* data) { field->data = data; } zfp_type zfp_field_set_type(zfp_field* field, zfp_type type) { switch (type) { case zfp_type_int32: case zfp_type_int64: case zfp_type_float: case zfp_type_double: field->type = type; return type; default: return zfp_type_none; } } void zfp_field_set_size_1d(zfp_field* field, size_t n) { field->nx = n; field->ny = 0; field->nz = 0; field->nw = 0; } void zfp_field_set_size_2d(zfp_field* field, size_t nx, size_t ny) { field->nx = nx; field->ny = ny; field->nz = 0; field->nw = 0; } void zfp_field_set_size_3d(zfp_field* field, size_t nx, size_t ny, size_t nz) { field->nx = nx; field->ny = ny; field->nz = nz; field->nw = 0; } void zfp_field_set_size_4d(zfp_field* field, size_t nx, size_t ny, size_t nz, size_t nw) { field->nx = nx; field->ny = ny; field->nz = nz; field->nw = nw; } void zfp_field_set_stride_1d(zfp_field* field, ptrdiff_t sx) { field->sx = sx; field->sy = 0; field->sz = 0; field->sw = 0; } void zfp_field_set_stride_2d(zfp_field* field, ptrdiff_t sx, ptrdiff_t sy) { field->sx = sx; field->sy = sy; field->sz = 0; field->sw = 0; } void zfp_field_set_stride_3d(zfp_field* field, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz) { field->sx = sx; field->sy = sy; field->sz = sz; field->sw = 0; } void zfp_field_set_stride_4d(zfp_field* field, ptrdiff_t sx, ptrdiff_t sy, ptrdiff_t sz, ptrdiff_t sw) { field->sx = sx; field->sy = sy; field->sz = sz; field->sw = sw; } zfp_bool zfp_field_set_metadata(zfp_field* field, uint64 meta) { uint64 dims; /* ensure value is in range */ if (meta >> ZFP_META_BITS) return zfp_false; field->type = (zfp_type)((meta & 0x3u) + 1); meta >>= 2; dims = (meta & 0x3u) + 1; meta >>= 2; switch (dims) { case 1: /* currently dimensions are limited to 2^32 - 1 */ field->nx = (size_t)(meta & UINT64C(0x0000ffffffff)) + 1; meta >>= 48; field->ny = 0; field->nz = 0; field->nw = 0; break; case 2: field->nx = (size_t)(meta & UINT64C(0xffffff)) + 1; meta >>= 24; field->ny = (size_t)(meta & UINT64C(0xffffff)) + 1; meta >>= 24; field->nz = 0; field->nw = 0; break; case 3: field->nx = (size_t)(meta & UINT64C(0xffff)) + 1; meta >>= 16; field->ny = (size_t)(meta & UINT64C(0xffff)) + 1; meta >>= 16; field->nz = (size_t)(meta & UINT64C(0xffff)) + 1; meta >>= 16; field->nw = 0; break; case 4: field->nx = (size_t)(meta & UINT64C(0xfff)) + 1; meta >>= 12; field->ny = (size_t)(meta & UINT64C(0xfff)) + 1; meta >>= 12; field->nz = (size_t)(meta & UINT64C(0xfff)) + 1; meta >>= 12; field->nw = (size_t)(meta & UINT64C(0xfff)) + 1; meta >>= 12; break; } field->sx = field->sy = field->sz = field->sw = 0; return zfp_true; } /* public functions: compression mode and parameter settings --------------- */ zfp_config zfp_config_none() { zfp_config config; config.mode = zfp_mode_null; return config; } zfp_config zfp_config_rate( double rate, zfp_bool align ) { zfp_config config; config.mode = zfp_mode_fixed_rate; config.arg.rate = align ? -rate : +rate; return config; } zfp_config zfp_config_precision( uint precision ) { zfp_config config; config.mode = zfp_mode_fixed_precision; config.arg.precision = precision; return config; } zfp_config zfp_config_accuracy( double tolerance ) { zfp_config config; config.mode = zfp_mode_fixed_accuracy; config.arg.tolerance = tolerance; return config; } zfp_config zfp_config_reversible() { zfp_config config; config.mode = zfp_mode_reversible; return config; } zfp_config zfp_config_expert( uint minbits, uint maxbits, uint maxprec, int minexp ) { zfp_config config; config.mode = zfp_mode_expert; config.arg.expert.minbits = minbits; config.arg.expert.maxbits = maxbits; config.arg.expert.maxprec = maxprec; config.arg.expert.minexp = minexp; return config; } /* public functions: zfp compressed stream --------------------------------- */ zfp_stream* zfp_stream_open(bitstream* stream) { zfp_stream* zfp = (zfp_stream*)malloc(sizeof(zfp_stream)); if (zfp) { zfp->stream = stream; zfp->minbits = ZFP_MIN_BITS; zfp->maxbits = ZFP_MAX_BITS; zfp->maxprec = ZFP_MAX_PREC; zfp->minexp = ZFP_MIN_EXP; zfp->exec.policy = zfp_exec_serial; } return zfp; } void zfp_stream_close(zfp_stream* zfp) { free(zfp); } bitstream* zfp_stream_bit_stream(const zfp_stream* zfp) { return zfp->stream; } zfp_mode zfp_stream_compression_mode(const zfp_stream* zfp) { if (zfp->minbits > zfp->maxbits || !(0 < zfp->maxprec && zfp->maxprec <= 64)) return zfp_mode_null; /* default values are considered expert mode */ if (zfp->minbits == ZFP_MIN_BITS && zfp->maxbits == ZFP_MAX_BITS && zfp->maxprec == ZFP_MAX_PREC && zfp->minexp == ZFP_MIN_EXP) return zfp_mode_expert; /* fixed rate? */ if (zfp->minbits == zfp->maxbits && 1 <= zfp->maxbits && zfp->maxbits <= ZFP_MAX_BITS && zfp->maxprec >= ZFP_MAX_PREC && zfp->minexp == ZFP_MIN_EXP) return zfp_mode_fixed_rate; /* fixed precision? */ if (zfp->minbits <= ZFP_MIN_BITS && zfp->maxbits >= ZFP_MAX_BITS && zfp->maxprec >= 1 && zfp->minexp == ZFP_MIN_EXP) return zfp_mode_fixed_precision; /* fixed accuracy? */ if (zfp->minbits <= ZFP_MIN_BITS && zfp->maxbits >= ZFP_MAX_BITS && zfp->maxprec >= ZFP_MAX_PREC && zfp->minexp >= ZFP_MIN_EXP) return zfp_mode_fixed_accuracy; /* reversible? */ if (zfp->minbits <= ZFP_MIN_BITS && zfp->maxbits >= ZFP_MAX_BITS && zfp->maxprec >= ZFP_MAX_PREC && zfp->minexp < ZFP_MIN_EXP) return zfp_mode_reversible; return zfp_mode_expert; } double zfp_stream_rate(const zfp_stream* zfp, uint dims) { return (zfp_stream_compression_mode(zfp) == zfp_mode_fixed_rate) ? (double)zfp->maxbits / (1u << (2 * dims)) : 0.0; } uint zfp_stream_precision(const zfp_stream* zfp) { return (zfp_stream_compression_mode(zfp) == zfp_mode_fixed_precision) ? zfp->maxprec : 0; } double zfp_stream_accuracy(const zfp_stream* zfp) { return (zfp_stream_compression_mode(zfp) == zfp_mode_fixed_accuracy) ? ldexp(1.0, zfp->minexp) : 0.0; } uint64 zfp_stream_mode(const zfp_stream* zfp) { uint64 mode = 0; uint minbits; uint maxbits; uint maxprec; uint minexp; /* common configurations mapped to short representation */ switch (zfp_stream_compression_mode(zfp)) { case zfp_mode_fixed_rate: if (zfp->maxbits <= 2048) /* maxbits is [1, 2048] */ /* returns [0, 2047] */ return (zfp->maxbits - 1); else break; case zfp_mode_fixed_precision: if (zfp->maxprec <= 128) /* maxprec is [1, 128] */ /* returns [2048, 2175] */ return (zfp->maxprec - 1) + (2048); else break; case zfp_mode_fixed_accuracy: if (zfp->minexp <= 843) /* minexp is [ZFP_MIN_EXP=-1074, 843] */ /* returns [2177, ZFP_MODE_SHORT_MAX=4094] */ /* +1 because skipped 2176 */ return (zfp->minexp - ZFP_MIN_EXP) + (2048 + 128 + 1); else break; case zfp_mode_reversible: /* returns 2176 */ return 2048 + 128; default: break; } /* encode each parameter separately */ minbits = MAX(1, MIN(zfp->minbits, 0x8000u)) - 1; maxbits = MAX(1, MIN(zfp->maxbits, 0x8000u)) - 1; maxprec = MAX(1, MIN(zfp->maxprec, 0x0080u)) - 1; minexp = MAX(0, MIN(zfp->minexp + 16495, 0x7fff)); mode <<= 15; mode += minexp; mode <<= 7; mode += maxprec; mode <<= 15; mode += maxbits; mode <<= 15; mode += minbits; mode <<= 12; mode += 0xfffu; return mode; } void zfp_stream_params(const zfp_stream* zfp, uint* minbits, uint* maxbits, uint* maxprec, int* minexp) { if (minbits) *minbits = zfp->minbits; if (maxbits) *maxbits = zfp->maxbits; if (maxprec) *maxprec = zfp->maxprec; if (minexp) *minexp = zfp->minexp; } size_t zfp_stream_compressed_size(const zfp_stream* zfp) { return stream_size(zfp->stream); } size_t zfp_stream_maximum_size(const zfp_stream* zfp, const zfp_field* field) { zfp_bool reversible = is_reversible(zfp); uint dims = zfp_field_dimensionality(field); size_t mx = (MAX(field->nx, 1u) + 3) / 4; size_t my = (MAX(field->ny, 1u) + 3) / 4; size_t mz = (MAX(field->nz, 1u) + 3) / 4; size_t mw = (MAX(field->nw, 1u) + 3) / 4; size_t blocks = mx * my * mz * mw; uint values = 1u << (2 * dims); uint maxbits = 0; if (!dims) return 0; switch (field->type) { case zfp_type_int32: maxbits += reversible ? 5 : 0; break; case zfp_type_int64: maxbits += reversible ? 6 : 0; break; case zfp_type_float: maxbits += reversible ? 1 + 1 + 8 + 5 : 1 + 8; break; case zfp_type_double: maxbits += reversible ? 1 + 1 + 11 + 6 : 1 + 11; break; default: return 0; } maxbits += values - 1 + values * MIN(zfp->maxprec, zfp_field_precision(field)); maxbits = MIN(maxbits, zfp->maxbits); maxbits = MAX(maxbits, zfp->minbits); return ((ZFP_HEADER_MAX_BITS + blocks * maxbits + stream_word_bits - 1) & ~(stream_word_bits - 1)) / CHAR_BIT; } void zfp_stream_set_bit_stream(zfp_stream* zfp, bitstream* stream) { zfp->stream = stream; } void zfp_stream_set_reversible(zfp_stream* zfp) { zfp->minbits = ZFP_MIN_BITS; zfp->maxbits = ZFP_MAX_BITS; zfp->maxprec = ZFP_MAX_PREC; zfp->minexp = ZFP_MIN_EXP - 1; } double zfp_stream_set_rate(zfp_stream* zfp, double rate, zfp_type type, uint dims, zfp_bool align) { uint n = 1u << (2 * dims); uint bits = (uint)floor(n * rate + 0.5); switch (type) { case zfp_type_float: bits = MAX(bits, 1 + 8u); break; case zfp_type_double: bits = MAX(bits, 1 + 11u); break; default: break; } if (align) { /* for write random access, round up to next multiple of stream word size */ bits += (uint)stream_word_bits - 1; bits &= ~(stream_word_bits - 1); } zfp->minbits = bits; zfp->maxbits = bits; zfp->maxprec = ZFP_MAX_PREC; zfp->minexp = ZFP_MIN_EXP; return (double)bits / n; } uint zfp_stream_set_precision(zfp_stream* zfp, uint precision) { zfp->minbits = ZFP_MIN_BITS; zfp->maxbits = ZFP_MAX_BITS; zfp->maxprec = precision ? MIN(precision, ZFP_MAX_PREC) : ZFP_MAX_PREC; zfp->minexp = ZFP_MIN_EXP; return zfp->maxprec; } double zfp_stream_set_accuracy(zfp_stream* zfp, double tolerance) { int emin = ZFP_MIN_EXP; if (tolerance > 0) { /* tolerance = x * 2^emin, with 0.5 <= x < 1 */ frexp(tolerance, &emin); emin--; /* assert: 2^emin <= tolerance < 2^(emin+1) */ } zfp->minbits = ZFP_MIN_BITS; zfp->maxbits = ZFP_MAX_BITS; zfp->maxprec = ZFP_MAX_PREC; zfp->minexp = emin; return tolerance > 0 ? ldexp(1.0, emin) : 0; } zfp_mode zfp_stream_set_mode(zfp_stream* zfp, uint64 mode) { uint minbits, maxbits, maxprec; int minexp; if (mode <= ZFP_MODE_SHORT_MAX) { /* 12-bit (short) encoding of one of four modes */ if (mode < 2048) { /* fixed rate */ minbits = maxbits = (uint)mode + 1; maxprec = ZFP_MAX_PREC; minexp = ZFP_MIN_EXP; } else if (mode < (2048 + 128)) { /* fixed precision */ minbits = ZFP_MIN_BITS; maxbits = ZFP_MAX_BITS; maxprec = (uint)mode + 1 - (2048); minexp = ZFP_MIN_EXP; } else if (mode == (2048 + 128)) { /* reversible */ minbits = ZFP_MIN_BITS; maxbits = ZFP_MAX_BITS; maxprec = ZFP_MAX_PREC; minexp = ZFP_MIN_EXP - 1; } else { /* fixed accuracy */ minbits = ZFP_MIN_BITS; maxbits = ZFP_MAX_BITS; maxprec = ZFP_MAX_PREC; minexp = (int)mode + ZFP_MIN_EXP - (2048 + 128 + 1); } } else { /* 64-bit encoding */ mode >>= 12; minbits = (uint)(mode & 0x7fffu) + 1; mode >>= 15; maxbits = (uint)(mode & 0x7fffu) + 1; mode >>= 15; maxprec = (uint)(mode & 0x007fu) + 1; mode >>= 7; minexp = (int)(mode & 0x7fffu) - 16495; } if (!zfp_stream_set_params(zfp, minbits, maxbits, maxprec, minexp)) return zfp_mode_null; return zfp_stream_compression_mode(zfp); } zfp_bool zfp_stream_set_params(zfp_stream* zfp, uint minbits, uint maxbits, uint maxprec, int minexp) { if (minbits > maxbits || !(0 < maxprec && maxprec <= 64)) return zfp_false; zfp->minbits = minbits; zfp->maxbits = maxbits; zfp->maxprec = maxprec; zfp->minexp = minexp; return zfp_true; } size_t zfp_stream_flush(zfp_stream* zfp) { return stream_flush(zfp->stream); } size_t zfp_stream_align(zfp_stream* zfp) { return stream_align(zfp->stream); } void zfp_stream_rewind(zfp_stream* zfp) { stream_rewind(zfp->stream); } /* public functions: execution policy -------------------------------------- */ zfp_exec_policy zfp_stream_execution(const zfp_stream* zfp) { return zfp->exec.policy; } uint zfp_stream_omp_threads(const zfp_stream* zfp) { return zfp->exec.params.omp.threads; } uint zfp_stream_omp_chunk_size(const zfp_stream* zfp) { return zfp->exec.params.omp.chunk_size; } zfp_bool zfp_stream_set_execution(zfp_stream* zfp, zfp_exec_policy policy) { switch (policy) { case zfp_exec_serial: break; #ifdef ZFP_WITH_CUDA case zfp_exec_cuda: break; #endif case zfp_exec_omp: #ifdef _OPENMP if (zfp->exec.policy != policy) { zfp->exec.params.omp.threads = 0; zfp->exec.params.omp.chunk_size = 0; } break; #else return zfp_false; #endif default: return zfp_false; } zfp->exec.policy = policy; return zfp_true; } zfp_bool zfp_stream_set_omp_threads(zfp_stream* zfp, uint threads) { if (!zfp_stream_set_execution(zfp, zfp_exec_omp)) return zfp_false; zfp->exec.params.omp.threads = threads; return zfp_true; } zfp_bool zfp_stream_set_omp_chunk_size(zfp_stream* zfp, uint chunk_size) { if (!zfp_stream_set_execution(zfp, zfp_exec_omp)) return zfp_false; zfp->exec.params.omp.chunk_size = chunk_size; return zfp_true; } /* public functions: utility functions --------------------------------------*/ void zfp_promote_int8_to_int32(int32* oblock, const int8* iblock, uint dims) { uint count = 1u << (2 * dims); while (count--) *oblock++ = (int32)*iblock++ << 23; } void zfp_promote_uint8_to_int32(int32* oblock, const uint8* iblock, uint dims) { uint count = 1u << (2 * dims); while (count--) *oblock++ = ((int32)*iblock++ - 0x80) << 23; } void zfp_promote_int16_to_int32(int32* oblock, const int16* iblock, uint dims) { uint count = 1u << (2 * dims); while (count--) *oblock++ = (int32)*iblock++ << 15; } void zfp_promote_uint16_to_int32(int32* oblock, const uint16* iblock, uint dims) { uint count = 1u << (2 * dims); while (count--) *oblock++ = ((int32)*iblock++ - 0x8000) << 15; } void zfp_demote_int32_to_int8(int8* oblock, const int32* iblock, uint dims) { uint count = 1u << (2 * dims); while (count--) { int32 i = *iblock++ >> 23; *oblock++ = (int8)MAX(-0x80, MIN(i, 0x7f)); } } void zfp_demote_int32_to_uint8(uint8* oblock, const int32* iblock, uint dims) { uint count = 1u << (2 * dims); while (count--) { int32 i = (*iblock++ >> 23) + 0x80; *oblock++ = (uint8)MAX(0x00, MIN(i, 0xff)); } } void zfp_demote_int32_to_int16(int16* oblock, const int32* iblock, uint dims) { uint count = 1u << (2 * dims); while (count--) { int32 i = *iblock++ >> 15; *oblock++ = (int16)MAX(-0x8000, MIN(i, 0x7fff)); } } void zfp_demote_int32_to_uint16(uint16* oblock, const int32* iblock, uint dims) { uint count = 1u << (2 * dims); while (count--) { int32 i = (*iblock++ >> 15) + 0x8000; *oblock++ = (uint16)MAX(0x0000, MIN(i, 0xffff)); } } /* public functions: compression and decompression --------------------------*/ size_t zfp_compress(zfp_stream* zfp, const zfp_field* field) { /* function table [execution][strided][dimensionality][scalar type] */ void (*ftable[3][2][4][4])(zfp_stream*, const zfp_field*) = { /* serial */ {{{ compress_int32_1, compress_int64_1, compress_float_1, compress_double_1 }, { compress_strided_int32_2, compress_strided_int64_2, compress_strided_float_2, compress_strided_double_2 }, { compress_strided_int32_3, compress_strided_int64_3, compress_strided_float_3, compress_strided_double_3 }, { compress_strided_int32_4, compress_strided_int64_4, compress_strided_float_4, compress_strided_double_4 }}, {{ compress_strided_int32_1, compress_strided_int64_1, compress_strided_float_1, compress_strided_double_1 }, { compress_strided_int32_2, compress_strided_int64_2, compress_strided_float_2, compress_strided_double_2 }, { compress_strided_int32_3, compress_strided_int64_3, compress_strided_float_3, compress_strided_double_3 }, { compress_strided_int32_4, compress_strided_int64_4, compress_strided_float_4, compress_strided_double_4 }}}, /* OpenMP */ #ifdef _OPENMP {{{ compress_omp_int32_1, compress_omp_int64_1, compress_omp_float_1, compress_omp_double_1 }, { compress_strided_omp_int32_2, compress_strided_omp_int64_2, compress_strided_omp_float_2, compress_strided_omp_double_2 }, { compress_strided_omp_int32_3, compress_strided_omp_int64_3, compress_strided_omp_float_3, compress_strided_omp_double_3 }, { compress_strided_omp_int32_4, compress_strided_omp_int64_4, compress_strided_omp_float_4, compress_strided_omp_double_4 }}, {{ compress_strided_omp_int32_1, compress_strided_omp_int64_1, compress_strided_omp_float_1, compress_strided_omp_double_1 }, { compress_strided_omp_int32_2, compress_strided_omp_int64_2, compress_strided_omp_float_2, compress_strided_omp_double_2 }, { compress_strided_omp_int32_3, compress_strided_omp_int64_3, compress_strided_omp_float_3, compress_strided_omp_double_3 }, { compress_strided_omp_int32_4, compress_strided_omp_int64_4, compress_strided_omp_float_4, compress_strided_omp_double_4 }}}, #else {{{ NULL }}}, #endif /* CUDA */ #ifdef ZFP_WITH_CUDA {{{ compress_cuda_int32_1, compress_cuda_int64_1, compress_cuda_float_1, compress_cuda_double_1 }, { compress_strided_cuda_int32_2, compress_strided_cuda_int64_2, compress_strided_cuda_float_2, compress_strided_cuda_double_2 }, { compress_strided_cuda_int32_3, compress_strided_cuda_int64_3, compress_strided_cuda_float_3, compress_strided_cuda_double_3 }, { NULL, NULL, NULL, NULL }}, {{ compress_strided_cuda_int32_1, compress_strided_cuda_int64_1, compress_strided_cuda_float_1, compress_strided_cuda_double_1 }, { compress_strided_cuda_int32_2, compress_strided_cuda_int64_2, compress_strided_cuda_float_2, compress_strided_cuda_double_2 }, { compress_strided_cuda_int32_3, compress_strided_cuda_int64_3, compress_strided_cuda_float_3, compress_strided_cuda_double_3 }, { NULL, NULL, NULL, NULL }}}, #else {{{ NULL }}}, #endif }; uint exec = zfp->exec.policy; uint strided = zfp_field_stride(field, NULL); uint dims = zfp_field_dimensionality(field); uint type = field->type; void (*compress)(zfp_stream*, const zfp_field*); switch (type) { case zfp_type_int32: case zfp_type_int64: case zfp_type_float: case zfp_type_double: break; default: return 0; } /* return 0 if compression mode is not supported */ compress = ftable[exec][strided][dims - 1][type - zfp_type_int32]; if (!compress) return 0; /* compress field and align bit stream on word boundary */ compress(zfp, field); stream_flush(zfp->stream); return stream_size(zfp->stream); } size_t zfp_decompress(zfp_stream* zfp, zfp_field* field) { /* function table [execution][strided][dimensionality][scalar type] */ void (*ftable[3][2][4][4])(zfp_stream*, zfp_field*) = { /* serial */ {{{ decompress_int32_1, decompress_int64_1, decompress_float_1, decompress_double_1 }, { decompress_strided_int32_2, decompress_strided_int64_2, decompress_strided_float_2, decompress_strided_double_2 }, { decompress_strided_int32_3, decompress_strided_int64_3, decompress_strided_float_3, decompress_strided_double_3 }, { decompress_strided_int32_4, decompress_strided_int64_4, decompress_strided_float_4, decompress_strided_double_4 }}, {{ decompress_strided_int32_1, decompress_strided_int64_1, decompress_strided_float_1, decompress_strided_double_1 }, { decompress_strided_int32_2, decompress_strided_int64_2, decompress_strided_float_2, decompress_strided_double_2 }, { decompress_strided_int32_3, decompress_strided_int64_3, decompress_strided_float_3, decompress_strided_double_3 }, { decompress_strided_int32_4, decompress_strided_int64_4, decompress_strided_float_4, decompress_strided_double_4 }}}, /* OpenMP; not yet supported */ {{{ NULL }}}, /* CUDA */ #ifdef ZFP_WITH_CUDA {{{ decompress_cuda_int32_1, decompress_cuda_int64_1, decompress_cuda_float_1, decompress_cuda_double_1 }, { decompress_strided_cuda_int32_2, decompress_strided_cuda_int64_2, decompress_strided_cuda_float_2, decompress_strided_cuda_double_2 }, { decompress_strided_cuda_int32_3, decompress_strided_cuda_int64_3, decompress_strided_cuda_float_3, decompress_strided_cuda_double_3 }, { NULL, NULL, NULL, NULL }}, {{ decompress_strided_cuda_int32_1, decompress_strided_cuda_int64_1, decompress_strided_cuda_float_1, decompress_strided_cuda_double_1 }, { decompress_strided_cuda_int32_2, decompress_strided_cuda_int64_2, decompress_strided_cuda_float_2, decompress_strided_cuda_double_2 }, { decompress_strided_cuda_int32_3, decompress_strided_cuda_int64_3, decompress_strided_cuda_float_3, decompress_strided_cuda_double_3 }, { NULL, NULL, NULL, NULL }}}, #else {{{ NULL }}}, #endif }; uint exec = zfp->exec.policy; uint strided = zfp_field_stride(field, NULL); uint dims = zfp_field_dimensionality(field); uint type = field->type; void (*decompress)(zfp_stream*, zfp_field*); switch (type) { case zfp_type_int32: case zfp_type_int64: case zfp_type_float: case zfp_type_double: break; default: return 0; } /* return 0 if decompression mode is not supported */ decompress = ftable[exec][strided][dims - 1][type - zfp_type_int32]; if (!decompress) return 0; /* decompress field and align bit stream on word boundary */ decompress(zfp, field); stream_align(zfp->stream); return stream_size(zfp->stream); } size_t zfp_write_header(zfp_stream* zfp, const zfp_field* field, uint mask) { size_t bits = 0; uint64 meta = 0; /* first make sure field dimensions fit in header */ if (mask & ZFP_HEADER_META) { meta = zfp_field_metadata(field); if (meta == ZFP_META_NULL) return 0; } /* 32-bit magic */ if (mask & ZFP_HEADER_MAGIC) { stream_write_bits(zfp->stream, 'z', 8); stream_write_bits(zfp->stream, 'f', 8); stream_write_bits(zfp->stream, 'p', 8); stream_write_bits(zfp->stream, zfp_codec_version, 8); bits += ZFP_MAGIC_BITS; } /* 52-bit field metadata */ if (mask & ZFP_HEADER_META) { stream_write_bits(zfp->stream, meta, ZFP_META_BITS); bits += ZFP_META_BITS; } /* 12- or 64-bit compression parameters */ if (mask & ZFP_HEADER_MODE) { uint64 mode = zfp_stream_mode(zfp); uint size = mode > ZFP_MODE_SHORT_MAX ? ZFP_MODE_LONG_BITS : ZFP_MODE_SHORT_BITS; stream_write_bits(zfp->stream, mode, size); bits += size; } return bits; } size_t zfp_read_header(zfp_stream* zfp, zfp_field* field, uint mask) { size_t bits = 0; if (mask & ZFP_HEADER_MAGIC) { if (stream_read_bits(zfp->stream, 8) != 'z' || stream_read_bits(zfp->stream, 8) != 'f' || stream_read_bits(zfp->stream, 8) != 'p' || stream_read_bits(zfp->stream, 8) != zfp_codec_version) return 0; bits += ZFP_MAGIC_BITS; } if (mask & ZFP_HEADER_META) { uint64 meta = stream_read_bits(zfp->stream, ZFP_META_BITS); if (!zfp_field_set_metadata(field, meta)) return 0; bits += ZFP_META_BITS; } if (mask & ZFP_HEADER_MODE) { uint64 mode = stream_read_bits(zfp->stream, ZFP_MODE_SHORT_BITS); bits += ZFP_MODE_SHORT_BITS; if (mode > ZFP_MODE_SHORT_MAX) { uint size = ZFP_MODE_LONG_BITS - ZFP_MODE_SHORT_BITS; mode += stream_read_bits(zfp->stream, size) << ZFP_MODE_SHORT_BITS; bits += size; } if (zfp_stream_set_mode(zfp, mode) == zfp_mode_null) return 0; } return bits; } zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/share/0000755000175200007730000000000014515254731022323 5ustar rlaboissrlaboisszmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/share/parallel.c0000644000175200007730000000516514515254731024272 0ustar rlaboissrlaboiss#ifdef _OPENMP /* block index at which chunk begins */ static size_t chunk_offset(size_t blocks, size_t chunks, size_t chunk) { return (size_t)(((uint64)blocks * (uint64)chunk) / chunks); } /* initialize per-thread bit streams for parallel compression */ static bitstream** compress_init_par(zfp_stream* stream, const zfp_field* field, size_t chunks, size_t blocks) { bitstream** bs; zfp_bool copy; size_t n = 4 * (blocks + chunks - 1) / chunks; size_t size; size_t chunk; /* determine maximum size buffer needed per thread */ zfp_field f = *field; switch (zfp_field_dimensionality(field)) { case 1: f.nx = n; break; case 2: f.nx = 4; f.ny = n; break; case 3: f.nx = 4; f.ny = 4; f.nz = n; break; case 4: f.nx = 4; f.ny = 4; f.nz = 4; f.nw = n; break; default: return NULL; } size = zfp_stream_maximum_size(stream, &f); /* avoid copies in fixed-rate mode when each bitstream is word aligned */ copy = (stream->minbits != stream->maxbits) || (stream->maxbits % stream_word_bits != 0) || (stream_wtell(stream->stream) % stream_word_bits != 0); /* set up buffer for each thread to compress to */ bs = (bitstream**)malloc(chunks * sizeof(bitstream*)); if (!bs) return NULL; for (chunk = 0; chunk < chunks; chunk++) { size_t block = chunk_offset(blocks, chunks, chunk); void* buffer = copy ? malloc(size) : (uchar*)stream_data(stream->stream) + stream_size(stream->stream) + block * (stream->maxbits / CHAR_BIT); if (!buffer) break; bs[chunk] = stream_open(buffer, size); } /* handle memory allocation failure */ if (copy && chunk < chunks) { while (chunk--) { free(stream_data(bs[chunk])); stream_close(bs[chunk]); } free(bs); bs = NULL; } return bs; } /* flush and concatenate bit streams if needed */ static void compress_finish_par(zfp_stream* stream, bitstream** src, size_t chunks) { bitstream* dst = zfp_stream_bit_stream(stream); zfp_bool copy = (stream_data(dst) != stream_data(*src)); size_t offset = stream_wtell(dst); size_t chunk; /* flush each stream and concatenate if necessary */ for (chunk = 0; chunk < chunks; chunk++) { size_t bits = stream_wtell(src[chunk]); offset += bits; stream_flush(src[chunk]); /* concatenate streams if they are not already contiguous */ if (copy) { stream_rewind(src[chunk]); stream_copy(dst, src[chunk], bits); free(stream_data(src[chunk])); } stream_close(src[chunk]); } free(src); if (!copy) stream_wseek(dst, offset); } #endif zmat-0.9.9/src/blosc2/plugins/codecs/zfp/src/share/omp.c0000644000175200007730000000153514515254731023266 0ustar rlaboissrlaboiss#ifdef _OPENMP #include #include /* number of omp threads to use */ static uint thread_count_omp(const zfp_stream* stream) { uint count = stream->exec.params.omp.threads; /* if no thread count is specified, use default number of threads */ if (!count) count = omp_get_max_threads(); return count; } /* number of chunks to partition array into */ static size_t chunk_count_omp(const zfp_stream* stream, size_t blocks, uint threads) { size_t chunk_size = stream->exec.params.omp.chunk_size; /* if no chunk size is specified, assign one chunk per thread */ size_t chunks = chunk_size ? (blocks + chunk_size - 1) / chunk_size : threads; /* each chunk must contain at least one block */ chunks = MIN(chunks, blocks); /* OpenMP 2.0 loop counters must be ints */ chunks = MIN(chunks, INT_MAX); return chunks; } #endif zmat-0.9.9/src/blosc2/plugins/codecs/zfp/blosc2-zfp.c0000644000175200007730000007046214515254731022570 0ustar rlaboissrlaboiss#include "blosc2.h" #include "frame.h" #include "blosc2/codecs-registry.h" #include "zfp.h" #include "blosc2-zfp.h" #include #include "context.h" #include "assert.h" #include "../plugins/plugin_utils.h" int zfp_acc_compress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_cparams *cparams, const void* chunk) { BLOSC_UNUSED_PARAM(chunk); ZFP_ERROR_NULL(input); ZFP_ERROR_NULL(output); ZFP_ERROR_NULL(cparams); double tol = (int8_t) meta; int8_t ndim; int64_t *shape = malloc(8 * sizeof(int64_t)); int32_t *chunkshape = malloc(8 * sizeof(int32_t)); int32_t *blockshape = malloc(8 * sizeof(int32_t)); uint8_t *smeta; int32_t smeta_len; if (blosc2_meta_get(cparams->schunk, "caterva", &smeta, &smeta_len) < 0) { printf("Blosc error"); free(shape); free(chunkshape); free(blockshape); return -1; } deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape); free(smeta); zfp_type type; /* array scalar type */ zfp_field *field; /* array meta data */ zfp_stream *zfp; /* stream containing the real output buffer */ zfp_stream *zfp_aux; /* auxiliary compressed stream */ bitstream *stream; /* bit stream to write to or read from */ bitstream *stream_aux; /* auxiliary bit stream to write to or read from */ size_t zfpsize; /* byte size of compressed stream */ double tolerance = pow(10, tol); int32_t typesize = cparams->typesize; switch (typesize) { case sizeof(float): type = zfp_type_float; break; case sizeof(double): type = zfp_type_double; break; default: printf("\n ZFP is not available for this typesize \n"); free(shape); free(chunkshape); free(blockshape); return 0; } zfp = zfp_stream_open(NULL); zfp_stream_set_accuracy(zfp, tolerance); stream = stream_open(output, output_len); zfp_stream_set_bit_stream(zfp, stream); zfp_stream_rewind(zfp); switch (ndim) { case 1: field = zfp_field_1d((void *) input, type, blockshape[0]); break; case 2: field = zfp_field_2d((void *) input, type, blockshape[1], blockshape[0]); break; case 3: field = zfp_field_3d((void *) input, type, blockshape[2], blockshape[1], blockshape[0]); break; case 4: field = zfp_field_4d((void *) input, type, blockshape[3], blockshape[2], blockshape[1], blockshape[0]); break; default: printf("\n ZFP is not available for this number of dims \n"); free(shape); free(chunkshape); free(blockshape); return 0; } int zfp_maxout = (int) zfp_stream_maximum_size(zfp, field); zfp_stream_close(zfp); stream_close(stream); uint8_t *aux_out = malloc(zfp_maxout); zfp_aux = zfp_stream_open(NULL); zfp_stream_set_accuracy(zfp_aux, tolerance); stream_aux = stream_open(aux_out, zfp_maxout); zfp_stream_set_bit_stream(zfp_aux, stream_aux); zfp_stream_rewind(zfp_aux); zfpsize = zfp_compress(zfp_aux, field); /* clean up */ zfp_field_free(field); zfp_stream_close(zfp_aux); stream_close(stream_aux); free(shape); free(chunkshape); free(blockshape); if (zfpsize == 0) { BLOSC_TRACE_ERROR("\n ZFP: Compression failed\n"); free(aux_out); return (int) zfpsize; } if ((int32_t) zfpsize >= input_len) { BLOSC_TRACE_ERROR("\n ZFP: Compressed data is bigger than input! \n"); free(aux_out); return 0; } memcpy(output, aux_out, zfpsize); free(aux_out); return (int) zfpsize; } int zfp_acc_decompress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_dparams *dparams, const void* chunk) { ZFP_ERROR_NULL(input); ZFP_ERROR_NULL(output); ZFP_ERROR_NULL(dparams); size_t typesize; int flags; blosc1_cbuffer_metainfo(chunk, &typesize, &flags); double tol = (int8_t) meta; int8_t ndim; int64_t *shape = malloc(8 * sizeof(int64_t)); int32_t *chunkshape = malloc(8 * sizeof(int32_t)); int32_t *blockshape = malloc(8 * sizeof(int32_t)); uint8_t *smeta; int32_t smeta_len; if (blosc2_meta_get(dparams->schunk, "caterva", &smeta, &smeta_len) < 0) { printf("Blosc error"); free(shape); free(chunkshape); free(blockshape); return -1; } deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape); free(smeta); zfp_type type; /* array scalar type */ zfp_field *field; /* array meta data */ zfp_stream *zfp; /* compressed stream */ bitstream *stream; /* bit stream to write to or read from */ size_t zfpsize; /* byte size of compressed stream */ double tolerance = pow(10, tol); switch (typesize) { case sizeof(float): type = zfp_type_float; break; case sizeof(double): type = zfp_type_double; break; default: printf("\n ZFP is not available for this typesize \n"); free(shape); free(chunkshape); free(blockshape); return 0; } zfp = zfp_stream_open(NULL); zfp_stream_set_accuracy(zfp, tolerance); stream = stream_open((void*) input, input_len); zfp_stream_set_bit_stream(zfp, stream); zfp_stream_rewind(zfp); switch (ndim) { case 1: field = zfp_field_1d((void *) output, type, blockshape[0]); break; case 2: field = zfp_field_2d((void *) output, type, blockshape[1], blockshape[0]); break; case 3: field = zfp_field_3d((void *) output, type, blockshape[2], blockshape[1], blockshape[0]); break; case 4: field = zfp_field_4d((void *) output, type, blockshape[3], blockshape[2], blockshape[1], blockshape[0]); break; default: printf("\n ZFP is not available for this number of dims \n"); free(shape); free(chunkshape); free(blockshape); return 0; } zfpsize = zfp_decompress(zfp, field); /* clean up */ zfp_field_free(field); zfp_stream_close(zfp); stream_close(stream); free(shape); free(chunkshape); free(blockshape); if (zfpsize == 0) { BLOSC_TRACE_ERROR("\n ZFP: Decompression failed\n"); return (int) zfpsize; } return (int) output_len; } int zfp_prec_compress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_cparams *cparams, const void* chunk) { BLOSC_UNUSED_PARAM(chunk); ZFP_ERROR_NULL(input); ZFP_ERROR_NULL(output); ZFP_ERROR_NULL(cparams); int8_t ndim; int64_t *shape = malloc(8 * sizeof(int64_t)); int32_t *chunkshape = malloc(8 * sizeof(int32_t)); int32_t *blockshape = malloc(8 * sizeof(int32_t)); uint8_t *smeta; int32_t smeta_len; if (blosc2_meta_get(cparams->schunk, "caterva", &smeta, &smeta_len) < 0) { printf("Blosc error"); free(shape); free(chunkshape); free(blockshape); return -1; } deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape); free(smeta); zfp_type type; /* array scalar type */ zfp_field *field; /* array meta data */ zfp_stream *zfp; /* stream containing the real output buffer */ zfp_stream *zfp_aux; /* auxiliary compressed stream */ bitstream *stream; /* bit stream to write to or read from */ bitstream *stream_aux; /* auxiliary bit stream to write to or read from */ size_t zfpsize; /* byte size of compressed stream */ uint prec; switch (ndim) { case 1: prec = meta + 5; break; case 2: prec = meta + 7; break; case 3: prec = meta + 9; break; case 4: prec = meta + 11; break; default: printf("\n ZFP is not available for this ndim \n"); free(shape); free(chunkshape); free(blockshape); return 0; } if(prec > ZFP_MAX_PREC) { BLOSC_TRACE_ERROR("Max precision for this codecs is %d", ZFP_MAX_PREC); prec = ZFP_MAX_PREC; } int32_t typesize = cparams->typesize; switch (typesize) { case sizeof(float): type = zfp_type_float; break; case sizeof(double): type = zfp_type_double; break; default: printf("\n ZFP is not available for this typesize \n"); free(shape); free(chunkshape); free(blockshape); return 0; } zfp = zfp_stream_open(NULL); zfp_stream_set_precision(zfp, prec); stream = stream_open(output, output_len); zfp_stream_set_bit_stream(zfp, stream); zfp_stream_rewind(zfp); switch (ndim) { case 1: field = zfp_field_1d((void *) input, type, blockshape[0]); break; case 2: field = zfp_field_2d((void *) input, type, blockshape[1], blockshape[0]); break; case 3: field = zfp_field_3d((void *) input, type, blockshape[2], blockshape[1], blockshape[0]); break; case 4: field = zfp_field_4d((void *) input, type, blockshape[3], blockshape[2], blockshape[1], blockshape[0]); break; default: printf("\n ZFP is not available for this number of dims \n"); free(shape); free(chunkshape); free(blockshape); return 0; } int zfp_maxout = (int) zfp_stream_maximum_size(zfp, field); zfp_stream_close(zfp); stream_close(stream); uint8_t *aux_out = malloc(zfp_maxout); zfp_aux = zfp_stream_open(NULL); zfp_stream_set_precision(zfp_aux, prec); stream_aux = stream_open(aux_out, zfp_maxout); zfp_stream_set_bit_stream(zfp_aux, stream_aux); zfp_stream_rewind(zfp_aux); zfpsize = zfp_compress(zfp_aux, field); /* clean up */ zfp_field_free(field); zfp_stream_close(zfp_aux); stream_close(stream_aux); free(shape); free(chunkshape); free(blockshape); if (zfpsize == 0) { BLOSC_TRACE_ERROR("\n ZFP: Compression failed\n"); free(aux_out); return (int) zfpsize; } if ((int32_t) zfpsize >= input_len) { BLOSC_TRACE_ERROR("\n ZFP: Compressed data is bigger than input! \n"); free(aux_out); return 0; } memcpy(output, aux_out, zfpsize); free(aux_out); return (int) zfpsize; } int zfp_prec_decompress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_dparams *dparams, const void* chunk) { ZFP_ERROR_NULL(input); ZFP_ERROR_NULL(output); ZFP_ERROR_NULL(dparams); size_t typesize; int flags; blosc1_cbuffer_metainfo(chunk, &typesize, &flags); int8_t ndim; int64_t *shape = malloc(8 * sizeof(int64_t)); int32_t *chunkshape = malloc(8 * sizeof(int32_t)); int32_t *blockshape = malloc(8 * sizeof(int32_t)); uint8_t *smeta; int32_t smeta_len; if (blosc2_meta_get(dparams->schunk, "caterva", &smeta, &smeta_len) < 0) { printf("Blosc error"); free(shape); free(chunkshape); free(blockshape); return -1; } deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape); free(smeta); zfp_type type; /* array scalar type */ zfp_field *field; /* array meta data */ zfp_stream *zfp; /* compressed stream */ bitstream *stream; /* bit stream to write to or read from */ size_t zfpsize; /* byte size of compressed stream */ uint prec; switch (ndim) { case 1: prec = meta + 5; break; case 2: prec = meta + 7; break; case 3: prec = meta + 9; break; case 4: prec = meta + 11; break; default: printf("\n ZFP is not available for this ndim \n"); free(shape); free(chunkshape); free(blockshape); return 0; } if(prec > ZFP_MAX_PREC) { BLOSC_TRACE_ERROR("Max precision for this codecs is %d", ZFP_MAX_PREC); prec = ZFP_MAX_PREC; } switch (typesize) { case sizeof(float): type = zfp_type_float; break; case sizeof(double): type = zfp_type_double; break; default: printf("\n ZFP is not available for this typesize \n"); free(shape); free(chunkshape); free(blockshape); return 0; } zfp = zfp_stream_open(NULL); zfp_stream_set_precision(zfp, prec); stream = stream_open((void*) input, input_len); zfp_stream_set_bit_stream(zfp, stream); zfp_stream_rewind(zfp); switch (ndim) { case 1: field = zfp_field_1d((void *) output, type, blockshape[0]); break; case 2: field = zfp_field_2d((void *) output, type, blockshape[1], blockshape[0]); break; case 3: field = zfp_field_3d((void *) output, type, blockshape[2], blockshape[1], blockshape[0]); break; case 4: field = zfp_field_4d((void *) output, type, blockshape[3], blockshape[2], blockshape[1], blockshape[0]); break; default: printf("\n ZFP is not available for this number of dims \n"); free(shape); free(chunkshape); free(blockshape); return 0; } zfpsize = zfp_decompress(zfp, field); /* clean up */ zfp_field_free(field); zfp_stream_close(zfp); stream_close(stream); free(shape); free(chunkshape); free(blockshape); if (zfpsize == 0) { BLOSC_TRACE_ERROR("\n ZFP: Decompression failed\n"); return (int) zfpsize; } return (int) output_len; } int zfp_rate_compress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_cparams *cparams, const void* chunk) { BLOSC_UNUSED_PARAM(chunk); ZFP_ERROR_NULL(input); ZFP_ERROR_NULL(output); ZFP_ERROR_NULL(cparams); double ratio = (double) meta / 100.0; int8_t ndim; int64_t *shape = malloc(8 * sizeof(int64_t)); int32_t *chunkshape = malloc(8 * sizeof(int32_t)); int32_t *blockshape = malloc(8 * sizeof(int32_t)); uint8_t *smeta; int32_t smeta_len; if (blosc2_meta_get(cparams->schunk, "caterva", &smeta, &smeta_len) < 0) { printf("Blosc error"); free(shape); free(chunkshape); free(blockshape); return -1; } deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape); free(smeta); zfp_type type; /* array scalar type */ zfp_field *field; /* array meta data */ zfp_stream *zfp; /* stream containing the real output buffer */ zfp_stream *zfp_aux; /* auxiliary compressed stream */ bitstream *stream; /* bit stream to write to or read from */ bitstream *stream_aux; /* auxiliary bit stream to write to or read from */ size_t zfpsize; /* byte size of compressed stream */ int32_t typesize = cparams->typesize; switch (typesize) { case sizeof(float): type = zfp_type_float; break; case sizeof(double): type = zfp_type_double; break; default: printf("\n ZFP is not available for this typesize \n"); return 0; } double rate = ratio * typesize * 8; // convert from output size / input size to output bits per input value uint cellsize = 1u << (2 * ndim); double min_rate; switch (type) { case zfp_type_float: min_rate = (double) (1 + 8u) / cellsize; if (rate < min_rate) { BLOSC_TRACE_ERROR("\n ZFP minimum rate for this item type is %f. Compression will be done using this rate \n", min_rate); } break; case zfp_type_double: min_rate = (double) (1 + 11u) / cellsize; if (rate < min_rate) { BLOSC_TRACE_ERROR("\n ZFP minimum rate for this item type is %f. Compression will be done using this rate \n", min_rate); } break; default: free(shape); free(chunkshape); free(blockshape); return 0; } zfp = zfp_stream_open(NULL); stream = stream_open(output, output_len); zfp_stream_set_bit_stream(zfp, stream); zfp_stream_rewind(zfp); switch (ndim) { case 1: field = zfp_field_1d((void *) input, type, blockshape[0]); break; case 2: field = zfp_field_2d((void *) input, type, blockshape[1], blockshape[0]); break; case 3: field = zfp_field_3d((void *) input, type, blockshape[2], blockshape[1], blockshape[0]); break; case 4: field = zfp_field_4d((void *) input, type, blockshape[3], blockshape[2], blockshape[1], blockshape[0]); break; default: printf("\n ZFP is not available for this number of dims \n"); free(shape); free(chunkshape); free(blockshape); return 0; } int zfp_maxout = (int) zfp_stream_maximum_size(zfp, field); zfp_stream_close(zfp); stream_close(stream); uint8_t *aux_out = malloc(zfp_maxout); zfp_aux = zfp_stream_open(NULL); stream_aux = stream_open(aux_out, zfp_maxout); zfp_stream_set_bit_stream(zfp_aux, stream_aux); zfp_stream_rewind(zfp_aux); zfp_stream_set_rate(zfp_aux, rate, type, ndim, zfp_false); zfpsize = zfp_compress(zfp_aux, field); /* clean up */ zfp_field_free(field); zfp_stream_close(zfp_aux); stream_close(stream_aux); free(shape); free(chunkshape); free(blockshape); if (zfpsize == 0) { BLOSC_TRACE_ERROR("\n ZFP: Compression failed\n"); free(aux_out); return (int) zfpsize; } if ((int32_t) zfpsize >= input_len) { BLOSC_TRACE_ERROR("\n ZFP: Compressed data is bigger than input! \n"); free(aux_out); return 0; } memcpy(output, aux_out, zfpsize); free(aux_out); return (int) zfpsize; } int zfp_rate_decompress(const uint8_t *input, int32_t input_len, uint8_t *output, int32_t output_len, uint8_t meta, blosc2_dparams *dparams, const void* chunk) { ZFP_ERROR_NULL(input); ZFP_ERROR_NULL(output); ZFP_ERROR_NULL(dparams); size_t typesize; int flags; blosc1_cbuffer_metainfo(chunk, &typesize, &flags); double ratio = (double) meta / 100.0; int8_t ndim; int64_t *shape = malloc(8 * sizeof(int64_t)); int32_t *chunkshape = malloc(8 * sizeof(int32_t)); int32_t *blockshape = malloc(8 * sizeof(int32_t)); uint8_t *smeta; int32_t smeta_len; if (blosc2_meta_get(dparams->schunk, "caterva", &smeta, &smeta_len) < 0) { printf("Blosc error"); free(shape); free(chunkshape); free(blockshape); return -1; } deserialize_meta(smeta, smeta_len, &ndim, shape, chunkshape, blockshape); free(smeta); zfp_type type; /* array scalar type */ zfp_field *field; /* array meta data */ zfp_stream *zfp; /* compressed stream */ bitstream *stream; /* bit stream to write to or read from */ size_t zfpsize; /* byte size of compressed stream */ switch (typesize) { case sizeof(float): type = zfp_type_float; break; case sizeof(double): type = zfp_type_double; break; default: printf("\n ZFP is not available for this typesize \n"); free(shape); free(chunkshape); free(blockshape); return 0; } double rate = ratio * (double) typesize * 8; // convert from output size / input size to output bits per input value zfp = zfp_stream_open(NULL); zfp_stream_set_rate(zfp, rate, type, ndim, zfp_false); stream = stream_open((void*) input, input_len); zfp_stream_set_bit_stream(zfp, stream); zfp_stream_rewind(zfp); switch (ndim) { case 1: field = zfp_field_1d((void *) output, type, blockshape[0]); break; case 2: field = zfp_field_2d((void *) output, type, blockshape[1], blockshape[0]); break; case 3: field = zfp_field_3d((void *) output, type, blockshape[2], blockshape[1], blockshape[0]); break; case 4: field = zfp_field_4d((void *) output, type, blockshape[3], blockshape[2], blockshape[1], blockshape[0]); break; default: printf("\n ZFP is not available for this number of dims \n"); free(shape); free(chunkshape); free(blockshape); return 0; } zfpsize = zfp_decompress(zfp, field); /* clean up */ zfp_field_free(field); zfp_stream_close(zfp); stream_close(stream); free(shape); free(chunkshape); free(blockshape); if (zfpsize == 0) { BLOSC_TRACE_ERROR("\n ZFP: Decompression failed\n"); return (int) zfpsize; } return (int) output_len; } int zfp_getcell(void *thread_context, const uint8_t *block, int32_t cbytes, uint8_t *dest, int32_t destsize) { struct thread_context *thread_ctx = thread_context; blosc2_context *context = thread_ctx->parent_context; bool meta = false; int8_t ndim = ZFP_MAX_DIM + 1; int32_t blockmeta[ZFP_MAX_DIM]; if (context->schunk->blockshape == NULL) { // blockshape is not filled yet. Use the Caterva layer to populate it. for (int nmetalayer = 0; nmetalayer < context->schunk->nmetalayers; nmetalayer++) { if (strcmp("caterva", context->schunk->metalayers[nmetalayer]->name) == 0) { meta = true; uint8_t *pmeta = context->schunk->metalayers[nmetalayer]->content; ndim = (int8_t) pmeta[2]; assert(ndim <= ZFP_MAX_DIM); pmeta += (6 + ndim * 9 + ndim * 5); for (int8_t i = 0; (uint8_t) i < ndim; i++) { pmeta += 1; swap_store(blockmeta + i, pmeta, sizeof(int32_t)); pmeta += sizeof(int32_t); } } } if (!meta) { return -1; } context->schunk->ndim = ndim; context->schunk->blockshape = malloc(sizeof(int64_t) * ndim); for (int i = 0; i < ndim; ++i) { context->schunk->blockshape[i] = (int64_t) blockmeta[i]; } } ndim = context->schunk->ndim; int64_t *blockshape = context->schunk->blockshape; // Compute the coordinates of the cell int64_t cell_start_ndim[ZFP_MAX_DIM]; int64_t cell_ind_ndim[ZFP_MAX_DIM]; int64_t ncell_ndim[ZFP_MAX_DIM]; int64_t ind_strides[ZFP_MAX_DIM]; int64_t cell_strides[ZFP_MAX_DIM]; int64_t cell_ind, ncell; blosc2_unidim_to_multidim(ndim, blockshape, thread_ctx->zfp_cell_start, cell_start_ndim); for (int i = 0; i < ndim; ++i) { cell_ind_ndim[i] = cell_start_ndim[i] % ZFP_MAX_DIM; ncell_ndim[i] = cell_start_ndim[i] / ZFP_MAX_DIM; } ind_strides[ndim - 1] = cell_strides[ndim - 1] = 1; for (int i = ndim - 2; i >= 0; --i) { ind_strides[i] = ZFP_MAX_DIM * ind_strides[i + 1]; cell_strides[i] = ((blockshape[i + 1] - 1) / ZFP_MAX_DIM + 1) * cell_strides[i + 1]; } blosc2_multidim_to_unidim(cell_ind_ndim, (int8_t) ndim, ind_strides, &cell_ind); blosc2_multidim_to_unidim(ncell_ndim, (int8_t) ndim, cell_strides, &ncell); int cell_nitems = (int) (1u << (2 * ndim)); if ((thread_ctx->zfp_cell_nitems > cell_nitems) || ((cell_ind + thread_ctx->zfp_cell_nitems) > cell_nitems)) { return 0; } // Get the ZFP stream zfp_type type; /* array scalar type */ zfp_stream *zfp; /* compressed stream */ bitstream *stream; /* bit stream to write to or read from */ int32_t typesize = context->typesize; zfp = zfp_stream_open(NULL); switch (typesize) { case sizeof(float): type = zfp_type_float; break; case sizeof(double): type = zfp_type_double; break; default: printf("\n ZFP is not available for this typesize \n"); return 0; } uint8_t compmeta = context->compcode_meta; // access to compressed chunk header double rate = (double) (compmeta * typesize * 8) / 100.0; // convert from output size / input size to output bits per input value zfp_stream_set_rate(zfp, rate, type, ndim, zfp_false); stream = stream_open((void *) block, cbytes); zfp_stream_set_bit_stream(zfp, stream); zfp_stream_rewind(zfp); // Check that ncell is a valid index int ncells = (int) ((cbytes * 8) / zfp->maxbits); if (ncell >= ncells) { BLOSC_TRACE_ERROR("Invalid cell index"); return -1; } // Position the stream at the ncell bit offset for reading stream_rseek(zfp->stream, (size_t) (ncell * zfp->maxbits)); // Get the cell size_t zfpsize; uint8_t *cell = malloc(cell_nitems * typesize); switch (ndim) { case 1: switch (type) { case zfp_type_float: zfpsize = zfp_decode_block_float_1(zfp, (float *) cell); break; case zfp_type_double: zfpsize = zfp_decode_block_double_1(zfp, (double *) cell); break; default: printf("\n ZFP is not available for this typesize \n"); zfp_stream_close(zfp); stream_close(stream); return 0; } break; case 2: switch (type) { case zfp_type_float: zfpsize = zfp_decode_block_float_2(zfp, (float *) cell); break; case zfp_type_double: zfpsize = zfp_decode_block_double_2(zfp, (double *) cell); break; default: printf("\n ZFP is not available for this typesize \n"); zfp_stream_close(zfp); stream_close(stream); return 0; } break; case 3: switch (type) { case zfp_type_float: zfpsize = zfp_decode_block_float_3(zfp, (float *) cell); break; case zfp_type_double: zfpsize = zfp_decode_block_double_3(zfp, (double *) cell); break; default: printf("\n ZFP is not available for this typesize \n"); zfp_stream_close(zfp); stream_close(stream); return 0; } break; case 4: switch (type) { case zfp_type_float: zfpsize = zfp_decode_block_float_4(zfp, (float *) cell); break; case zfp_type_double: zfpsize = zfp_decode_block_double_4(zfp, (double *) cell); break; default: printf("\n ZFP is not available for this typesize \n"); zfp_stream_close(zfp); stream_close(stream); return 0; } break; default: printf("\n ZFP is not available for this number of dims \n"); return 0; } memcpy(dest, &cell[cell_ind * typesize], thread_ctx->zfp_cell_nitems * typesize); zfp_stream_close(zfp); stream_close(stream); free(cell); if ((zfpsize == 0) || ((int32_t)zfpsize > (destsize * 8)) || ((int32_t)zfpsize > (cell_nitems * typesize * 8)) || ((thread_ctx->zfp_cell_nitems * typesize * 8) > (int32_t)zfpsize)) { BLOSC_TRACE_ERROR("ZFP error or small destsize"); return -1; } return (int) (thread_ctx->zfp_cell_nitems * typesize); } zmat-0.9.9/src/blosc2/README.rst0000644000175200007730000003761414515254731016414 0ustar rlaboissrlaboiss======== C-Blosc2 ======== A fast, compressed and persistent data store library for C ========================================================== :Author: The Blosc Development Team :Contact: blosc@blosc.org :URL: https://www.blosc.org :Gitter: |gitter| :Actions: |actions| :NumFOCUS: |numfocus| :Code of Conduct: |Contributor Covenant| .. |gitter| image:: https://badges.gitter.im/Blosc/c-blosc.svg :alt: Join the chat at https://gitter.im/Blosc/c-blosc :target: https://gitter.im/Blosc/c-blosc?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge .. |actions| image:: https://github.com/Blosc/c-blosc2/workflows/CI%20CMake/badge.svg :target: https://github.com/Blosc/c-blosc2/actions?query=workflow%3A%22CI+CMake%22 .. |numfocus| image:: https://img.shields.io/badge/powered%20by-NumFOCUS-orange.svg?style=flat&colorA=E1523D&colorB=007D8A :target: https://numfocus.org .. |Contributor Covenant| image:: https://img.shields.io/badge/Contributor%20Covenant-v2.0%20adopted-ff69b4.svg :target: https://github.com/Blosc/community/blob/master/code_of_conduct.md What is it? =========== `Blosc `_ is a high performance compressor optimized for binary data (i.e. floating point numbers, integers and booleans, although it can handle string data too). It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Blosc main goal is not just to reduce the size of large datasets on-disk or in-memory, but also to accelerate memory-bound computations. C-Blosc2 is the new major version of `C-Blosc `_, and is backward compatible with both the C-Blosc1 API and its in-memory format. However, the reverse thing is generally not true for the format; buffers generated with C-Blosc2 are not format-compatible with C-Blosc1 (i.e. forward compatibility is not supported). In case you want to ensure full API compatibility with C-Blosc1 API, define the `BLOSC1_COMPAT` symbol. See a 3 minutes `introductory video to Blosc2 `_. Blosc2 NDim: an N-Dimensional store =================================== One of the latest and more exciting additions in C-Blosc2 is the `Blosc2 NDim layer `_ (or b2nd for short), allowing to create *and* read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, that allows to slice and dice arbitrary large and compressed data in a more fine-grained way: .. image:: https://github.com/Blosc/c-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true :width: 75% To wet you appetite, here it is how the `NDArray` object in the `Python wrapper`_ performs on getting slices orthogonal to the different axis of a 4-dim dataset: .. image:: https://github.com/Blosc/c-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true :width: 75% We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro We also have a ~2 min explanatory video on `why slicing in a pineapple-style (aka double partition) is useful `_: .. image:: https://github.com/Blosc/blogsite/blob/master/files/images/slicing-pineapple-style.png?raw=true :width: 50% :alt: Slicing a dataset in pineapple-style :target: https://www.youtube.com/watch?v=LvP9zxMGBng New features in C-Blosc2 ======================== * **64-bit containers:** the first-class container in C-Blosc2 is the `super-chunk` or, for brevity, `schunk`, that is made by smaller chunks which are essentially C-Blosc1 32-bit containers. The super-chunk can be backed or not by another container which is called a `frame` (see later). * **NDim containers (b2nd):** allow to store n-dimensional data that can efficiently read datasets in slices that can be n-dimensional too. To achieve this, a n-dimensional 2-level partitioning has been implemented. This capabilities were formerly part of `Caterva `_, and now it is included in C-Blosc2 for convenience. Caterva is now deprecated. * **More filters:** besides `shuffle` and `bitshuffle` already present in C-Blosc1, C-Blosc2 already implements: - `delta`: the stored blocks inside a chunk are diff'ed with respect to first block in the chunk. The idea is that, in some situations, the diff will have more zeros than the original data, leading to better compression. - `trunc_prec`: it zeroes the least significant bits of the mantissa of float32 and float64 types. When combined with the `shuffle` or `bitshuffle` filter, this leads to more contiguous zeros, which are compressed better. * **A filter pipeline:** the different filters can be pipelined so that the output of one can the input for the other. A possible example is a `delta` followed by `shuffle`, or as described above, `trunc_prec` followed by `bitshuffle`. * **Prefilters:** allow to apply user-defined C callbacks **prior** the filter pipeline during compression. See `test_prefilter.c `_ for an example of use. * **Postfilters:** allow to apply user-defined C callbacks **after** the filter pipeline during decompression. The combination of prefilters and postfilters could be interesting for supporting e.g. encryption (via prefilters) and decryption (via postfilters). Also, a postfilter alone can be used to produce on-the-flight computation based on existing data (or other metadata, like e.g. coordinates). See `test_postfilter.c `_ for an example of use. * **SIMD support for ARM (NEON):** this allows for faster operation on ARM architectures. Only `shuffle` is supported right now, but the idea is to implement `bitshuffle` for NEON too. Thanks to Lucian Marc. * **SIMD support for PowerPC (ALTIVEC):** this allows for faster operation on PowerPC architectures. Both `shuffle` and `bitshuffle` are supported; however, this has been done via a transparent mapping from SSE2 into ALTIVEC emulation in GCC 8, so performance could be better (but still, it is already a nice improvement over native C code; see PR https://github.com/Blosc/c-blosc2/pull/59 for details). Thanks to Jerome Kieffer and `ESRF `_ for sponsoring the Blosc team in helping him in this task. * **Dictionaries:** when a block is going to be compressed, C-Blosc2 can use a previously made dictionary (stored in the header of the super-chunk) for compressing all the blocks that are part of the chunks. This usually improves the compression ratio, as well as the decompression speed, at the expense of a (small) overhead in compression speed. Currently, it is only supported in the `zstd` codec, but would be nice to extend it to `lz4` and `blosclz` at least. * **Contiguous frames:** allow to store super-chunks contiguously, either on-disk or in-memory. When a super-chunk is backed by a frame, instead of storing all the chunks sparsely in-memory, they are serialized inside the frame container. The frame can be stored on-disk too, meaning that persistence of super-chunks is supported. * **Sparse frames:** each chunk in a super-chunk is stored in a separate file or different memory area, as well as the metadata. This is allows for more efficient updates/deletes than in contiguous frames (i.e. avoiding 'holes' in monolithic files). The drawback is that it consumes more inodes when on-disk. Thanks to Marta Iborra for this contribution. * **Partial chunk reads:** there is support for reading just part of chunks, so avoiding to read the whole thing and then discard the unnecessary data. * **Parallel chunk reads:** when several blocks of a chunk are to be read, this is done in parallel by the decompressing machinery. That means that every thread is responsible to read, post-filter and decompress a block by itself, leading to an efficient overlap of I/O and CPU usage that optimizes reads to a maximum. * **Meta-layers:** optionally, the user can add meta-data for different uses and in different layers. For example, one may think on providing a meta-layer for `NumPy `_ so that most of the meta-data for it is stored in a meta-layer; then, one can place another meta-layer on top of the latter for adding more high-level info if desired (e.g. geo-spatial, meteorological...). * **Variable length meta-layers:** the user may want to add variable-length meta information that can be potentially very large (up to 2 GB). The regular meta-layer described above is very quick to read, but meant to store fixed-length and relatively small meta information. Variable length metalayers are stored in the trailer of a frame, whereas regular meta-layers are in the header. * **Efficient support for special values:** large sequences of repeated values can be represented with an efficient, simple and fast run-length representation, without the need to use regular codecs. With that, chunks or super-chunks with values that are the same (zeros, NaNs or any value in general) can be built in constant time, regardless of the size. This can be useful in situations where a lot of zeros (or NaNs) need to be stored (e.g. sparse matrices). * **Nice markup for documentation:** we are currently using a combination of Sphinx + Doxygen + Breathe for documenting the C-API. See https://www.blosc.org/c-blosc2/c-blosc2.html. Thanks to Alberto Sabater and Aleix Alcacer for contributing the support for this. * **Plugin capabilities for filters and codecs:** we have a plugin register capability inplace so that the info about the new filters and codecs can be persisted and transmitted to different machines. See https://github.com/Blosc/c-blosc2/blob/main/examples/urfilters.c for a self-contained example. Thanks to the NumFOCUS foundation for providing a grant for doing this, and Oscar Griñón and Aleix Alcacer for the implementation. * **Pluggable tuning capabilities:** this will allow users with different needs to define an interface so as to better tune different parameters like the codec, the compression level, the filters to use, the blocksize or the shuffle size. Thanks to ironArray for sponsoring us in doing this. * **Support for I/O plugins:** so that users can extend the I/O capabilities beyond the current filesystem support. Things like the use of databases or S3 interfaces should be possible by implementing these interfaces. Thanks to ironArray for sponsoring us in doing this. * **Python wrapper:** we have a preliminary wrapper in the works. You can have a look at our ongoing efforts in the `python-blosc2 repo `_. Thanks to the Python Software Foundation for providing a grant for doing this. * **Security:** we are actively using using the `OSS-Fuzz `_ and `ClusterFuzz `_ for uncovering programming errors in C-Blosc2. Thanks to Google for sponsoring us in doing this, and to Nathan Moinvaziri for most of the work here. More info about the `improved capabilities of C-Blosc2 can be found in this talk `_. C-Blosc2 API and format have been frozen, and that means that there is guarantee that your programs will continue to work with future versions of the library, and that next releases will be able to read from persistent storage generated from previous releases (as of 2.0.0). Python wrapper ============== We are officially supporting (thanks to the Python Software Foundation) a `Python wrapper for Blosc2 `_. It supports all the features of the predecessor `python-blosc `_ package plus most of the bells and whistles of C-Blosc2, like 64-bit and multidimensional containers. As a bonus, the `python-blosc2` package comes with wheels and binary versions of the C-Blosc2 libraries, so anyone, even non-Python users can install C-Blosc2 binaries easily with: .. code-block:: console pip install blosc2 Compiling the C-Blosc2 library with CMake ========================================= Blosc can be built, tested and installed using `CMake `_. The following procedure describes a typical CMake build. Create the build directory inside the sources and move into it: .. code-block:: console git clone https://github.com/Blosc/c-blosc2 cd c-blosc2 mkdir build cd build Now run CMake configuration and optionally specify the installation directory (e.g. '/usr' or '/usr/local'): .. code-block:: console cmake -DCMAKE_INSTALL_PREFIX=your_install_prefix_directory .. CMake allows to configure Blosc in many different ways, like preferring internal or external sources for compressors or enabling/disabling them. Please note that configuration can also be performed using UI tools provided by CMake (`ccmake` or `cmake-gui`): .. code-block:: console ccmake .. # run a curses-based interface cmake-gui .. # run a graphical interface Build, test and install Blosc: .. code-block:: console cmake --build . ctest cmake --build . --target install The static and dynamic version of the Blosc library, together with header files, will be installed into the specified CMAKE_INSTALL_PREFIX. Once you have compiled your Blosc library, you can easily link your apps with it as shown in the `examples/ directory `_. Handling support for codecs (LZ4, LZ4HC, Zstd, Zlib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C-Blosc2 comes with full sources for LZ4, LZ4HC, Zstd, and Zlib and in general, you should not worry about not having (or CMake not finding) the libraries in your system because by default the included sources will be automatically compiled and included in the C-Blosc2 library. This means that you can be confident in having a complete support for all these codecs in all the official Blosc deployments. Of course, if you consider this is too bloated, you can exclude support for some of them. For example, let's suppose that you want to disable support for Zstd: .. code-block:: console cmake -DDEACTIVATE_ZSTD=ON .. Or, you may want to use a codec in an external library already in the system: .. code-block:: console cmake -DPREFER_EXTERNAL_LZ4=ON .. Supported platforms ~~~~~~~~~~~~~~~~~~~ C-Blosc2 is meant to support all platforms where a C99 compliant C compiler can be found. The ones that are mostly tested are Intel (Linux, Mac OSX and Windows), ARM (Linux, Mac), and PowerPC (Linux). More on ARM support in `README_ARM.rst`. For Windows, you will need at least VS2015 or higher on x86 and x64 targets (i.e. ARM is not supported on Windows). For Mac OSX, make sure that you have the command line developer tools available. You can always install them with: .. code-block:: console xcode-select --install For Mac OSX on arm64 architecture, you may want to compile it like this: .. code-block:: console CC="clang -arch arm64" cmake .. Display error messages ~~~~~~~~~~~~~~~~~~~~~~ By default error messages are disabled. To display them, you just need to activate the Blosc tracing machinery by setting the ``BLOSC_TRACE`` environment variable. Contributing ============ If you want to collaborate in this development you are welcome. We need help in the different areas listed at the `ROADMAP `_; also, be sure to read our `DEVELOPING-GUIDE `_ and our `Code of Conduct `_. Blosc is distributed using the `BSD license `_. Tweeter feed ============ Follow `@Blosc2 `_ so as to get informed about the latest developments. Acknowledgments =============== See `THANKS document `_. ---- **Enjoy data!** zmat-0.9.9/src/blosc2/LICENSE.txt0000644000175200007730000000315414515254731016540 0ustar rlaboissrlaboissBSD License For Blosc - A blocking, shuffling and lossless compression library Copyright (C) 2009-2018 Francesc Alted Copyright (C) 2019-present Blosc Development team Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name Francesc Alted nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. zmat-0.9.9/src/zmat.cpp0000644000175200007730000002566314515254731015221 0ustar rlaboissrlaboiss/***************************************************************************//** ** \mainpage ZMat - A portable C-library and MATLAB/Octave toolbox for inline data compression ** ** \author Qianqian Fang ** \copyright Qianqian Fang, 2019,2020,2022 ** ** ZMat provides an easy-to-use interface for stream compression and decompression. ** ** It can be compiled as a MATLAB/Octave mex function (zipmat.mex/zmat.m) and compresses ** arrays and strings in MATLAB/Octave. It can also be compiled as a lightweight ** C-library (libzmat.a/libzmat.so) that can be called in C/C++/FORTRAN etc to ** provide stream-level compression and decompression. ** ** Currently, zmat/libzmat supports 6 different compression algorthms, including ** - zlib and gzip : the most widely used algorithm algorithms for .zip and .gz files ** - lzma and lzip : high compression ratio LZMA based algorithms for .lzma and .lzip files ** - lz4 and lz4hc : real-time compression based on LZ4 and LZ4HC algorithms ** - base64 : base64 encoding and decoding ** ** ZMat is part of the NeuroJSON project (https://neurojson.org) ** More information can be found at https://github.com/NeuroJSON/zmat ** ** Depencency: ZLib library: https://www.zlib.net/ ** author: (C) 1995-2017 Jean-loup Gailly and Mark Adler ** ** Depencency: LZ4 library: https://lz4.github.io/lz4/ ** author: (C) 2011-2019, Yann Collet, ** ** Depencency: Original LZMA library ** author: Igor Pavlov ** ** Depencency: Eazylzma: https://github.com/lloyd/easylzma ** author: Lloyd Hilaiel (lloyd) ** ** Depencency: base64_encode()/base64_decode() ** \copyright 2005-2011, Jouni Malinen ** ** \section slicense License ** GPL v3, see LICENSE.txt for details *******************************************************************************/ /***************************************************************************//** \file zmat.cpp @brief mex function for ZMAT *******************************************************************************/ #include #include #include #include #include #include "mex.h" #include "zmatlib.h" #include "zlib.h" void zmat_usage(); const char* metadata[] = {"type", "size", "byte", "method", "status", "level"}; /** @brief Mex function for the zmat - an interface to compress/decompress binary data * This is the master function to interface for zipping and unzipping a char/int8 buffer */ void mexFunction(int nlhs, mxArray* plhs[], int nrhs, const mxArray* prhs[]) { TZipMethod zipid = zmZlib; const char* zipmethods[] = { "zlib", "gzip", "base64", #if !defined(NO_LZMA) "lzip", "lzma", #endif #if !defined(NO_LZ4) "lz4", "lz4hc", #endif #if !defined(NO_ZSTD) "zstd", #endif #if !defined(NO_BLOSC2) "blosc2blosclz", "blosc2lz4", "blosc2lz4hc", "blosc2zlib", "blosc2zstd", #endif "" }; const TZipMethod zipmethodid[] = { zmZlib, zmGzip, zmBase64, #if !defined(NO_LZMA) zmLzip, zmLzma, #endif #if !defined(NO_LZ4) zmLz4, zmLz4hc, #endif #if !defined(NO_ZSTD) zmZstd, #endif #if !defined(NO_BLOSC2) zmBlosc2Blosclz, zmBlosc2Lz4, zmBlosc2Lz4hc, zmBlosc2Zlib, zmBlosc2Zstd, #endif zmUnknown }; int use4bytedim = 0; /** * The TZMatFlags union data structure is defined in zmatlib.h * it allows users to set advanced parameters to be used for * blosc2 meta-compressor, including number of threads, * byte-shuffle and element byte size; the iscompress element is * used to pass on to zmat_run * * union TZMatFlags { * int iscompress; // combined flag used to pass on to zmat_run * struct settings { // unpacked flags * char clevel; // compression level, 0: decompression, 1: use default level; negative: set compression level (-1 to -19) * char nthread; // number of compression/decompression threads * char shuffle; // byte shuffle length * char typesize; // for ND-array, the byte-size for each array element * } param; * }; */ union TZMatFlags flags = {0}; /** * If no input is given for this function, it prints help information and return. */ if (nrhs == 0) { zmat_usage(); return; } /** * Octave and MATLAB had changed mwSize from 4 byte to size_t (8 byte on 64bit machines) * in Octave 5/R2016. When running a mex/oct file compiled on newer libraries over an older * MATLAB/Octave versions, this can cause empyt output. The flag use4bytedim defined * in the below test is 1 when this situation happens, and adapt output accordingly. */ if (sizeof(mwSize) == 8) { mwSize dims[2] = {0, 0}; unsigned int* intdims = (unsigned int*)dims; intdims[0] = 2; intdims[1] = 3; mxArray* tmp = mxCreateNumericArray(2, dims, mxUINT8_CLASS, mxREAL); use4bytedim = (mxGetNumberOfElements(tmp) == 6); mxDestroyArray(tmp); } flags.param.nthread = 1; flags.param.shuffle = 1; flags.param.typesize = 4; if (nrhs >= 2) { double* val = mxGetPr(prhs[1]); flags.param.clevel = (int)val[0]; } if (nrhs >= 3) { int len = mxGetNumberOfElements(prhs[2]); if (!mxIsChar(prhs[2]) || len == 0) { mexErrMsgTxt("the 'method' field must be a non-empty string"); } if ((zipid = (TZipMethod)zmat_keylookup((char*)mxArrayToString(prhs[2]), zipmethods)) < 0) { mexErrMsgTxt("the specified compression method is not supported"); } zipid = zipmethodid[(int)zipid]; } if (nrhs >= 4) { double* val = mxGetPr(prhs[3]); flags.param.nthread = val[0]; } if (nrhs >= 5) { double* val = mxGetPr(prhs[4]); flags.param.shuffle = val[0]; } if (nrhs >= 6) { double* val = mxGetPr(prhs[5]); flags.param.typesize = val[0]; } try { if (mxIsChar(prhs[0]) || (mxIsNumeric(prhs[0]) && !mxIsComplex(prhs[0])) || mxIsLogical(prhs[0])) { int ret = -1; mwSize inputsize = mxGetNumberOfElements(prhs[0]) * mxGetElementSize(prhs[0]); mwSize buflen[2] = {0}; unsigned char* outputbuf = NULL; size_t outputsize = 0; unsigned char* inputstr = (mxIsChar(prhs[0]) ? (unsigned char*)mxArrayToString(prhs[0]) : (unsigned char*)mxGetData(prhs[0])); int errcode = 0; // if input buffer is not empty, run main function zmat_run if (inputsize > 0) { errcode = zmat_run(inputsize, inputstr, &outputsize, &outputbuf, zipid, &ret, flags.iscompress); } // test error code if (errcode < 0) { if (outputbuf) { free(outputbuf); } outputbuf = NULL; outputsize = 0; } buflen[0] = 1; buflen[1] = outputsize; // if running on octave 4/matlab R2015 or older, dimensions are stored as 4-byte integers, determine this at runtime if (use4bytedim) { unsigned int intdims[4] = {0}; intdims[0] = 1; intdims[1] = (unsigned int)outputsize; plhs[0] = mxCreateNumericArray(2, (mwSize*)intdims, mxUINT8_CLASS, mxREAL); } else { plhs[0] = mxCreateNumericArray(2, buflen, mxUINT8_CLASS, mxREAL); } // return the compressed/decompressed buffer to plhs[0] if (outputbuf) { memcpy((unsigned char*)mxGetPr(plhs[0]), outputbuf, buflen[1]); free(outputbuf); } // return info struct if (nlhs > 1) { mwSize inputdim[2] = {1, 0}, *dims = (mwSize*)mxGetDimensions(prhs[0]); unsigned int* int64inputdim = NULL; plhs[1] = mxCreateStructMatrix(1, 1, 6, metadata); mxArray* val = mxCreateString(mxGetClassName(prhs[0])); mxSetFieldByNumber(plhs[1], 0, 0, val); inputdim[1] = mxGetNumberOfDimensions(prhs[0]); int64inputdim = (unsigned int*)malloc(inputdim[1] * sizeof(unsigned int)); if (use4bytedim) { unsigned int intinputdim[4] = {0}; unsigned int* intdims = (unsigned int*)(mxGetDimensions(prhs[0])); intinputdim[0] = 1; intinputdim[1] = (unsigned int)inputdim[1]; val = mxCreateNumericArray(2, (mwSize*)intinputdim, mxUINT32_CLASS, mxREAL); for (int i = 0; i < intinputdim[1]; i++) { int64inputdim[i] = intdims[i]; } memcpy(mxGetPr(val), int64inputdim, intinputdim[1]*sizeof(unsigned int)); } else { if (sizeof(mwSize) == 8) { val = mxCreateNumericArray(2, inputdim, mxUINT64_CLASS, mxREAL); memcpy(mxGetPr(val), dims, inputdim[1]*sizeof(mwSize)); } else { val = mxCreateNumericArray(2, inputdim, mxUINT32_CLASS, mxREAL); for (int i = 0; i < inputdim[1]; i++) { int64inputdim[i] = dims[i]; } memcpy(mxGetPr(val), int64inputdim, inputdim[1]*sizeof(unsigned int)); } } mxSetFieldByNumber(plhs[1], 0, 1, val); val = mxCreateDoubleMatrix(1, 1, mxREAL); *mxGetPr(val) = mxGetElementSize(prhs[0]); mxSetFieldByNumber(plhs[1], 0, 2, val); val = mxCreateString(zipmethods[zipid]); mxSetFieldByNumber(plhs[1], 0, 3, val); val = mxCreateDoubleMatrix(1, 1, mxREAL); *mxGetPr(val) = ret; mxSetFieldByNumber(plhs[1], 0, 4, val); val = mxCreateDoubleMatrix(1, 1, mxREAL); *mxGetPr(val) = flags.param.clevel; mxSetFieldByNumber(plhs[1], 0, 5, val); } if (errcode < 0) { mexWarnMsgTxt(zmat_error(-errcode)); } } else { mexErrMsgTxt("the input must be a char, non-complex numerical or logical array"); } } catch (const char* err) { mexPrintf("Error: %s\n", err); } catch (const std::exception& err) { mexPrintf("C++ Error: %s\n", err.what()); } catch (...) { mexPrintf("Unknown Exception\n"); } return; } /** * @brief Print a brief help information if nothing is provided */ void zmat_usage() { mexPrintf("ZMat (v0.9.9)\nUsage:\n\t[output,info]=zmat(input,iscompress,method);\n\nPlease run 'help zmat' for more details.\n"); } zmat-0.9.9/src/zmatlib.c0000644000175200007730000010425214515254731015340 0ustar rlaboissrlaboiss/***************************************************************************//** ** \mainpage ZMat - A portable C-library and MATLAB/Octave toolbox for inline data compression ** ** \author Qianqian Fang ** \copyright Qianqian Fang, 2019-2020 ** ** ZMat provides an easy-to-use interface for stream compression and decompression. ** ** It can be compiled as a MATLAB/Octave mex function (zipmat.mex/zmat.m) and compresses ** arrays and strings in MATLAB/Octave. It can also be compiled as a lightweight ** C-library (libzmat.a/libzmat.so) that can be called in C/C++/FORTRAN etc to ** provide stream-level compression and decompression. ** ** Currently, zmat/libzmat supports 6 different compression algorthms, including ** - zlib and gzip : the most widely used algorithm algorithms for .zip and .gz files ** - lzma and lzip : high compression ratio LZMA based algorithms for .lzma and .lzip files ** - lz4 and lz4hc : real-time compression based on LZ4 and LZ4HC algorithms ** - base64 : base64 encoding and decoding ** ** Depencency: ZLib library: https://www.zlib.net/ ** author: (C) 1995-2017 Jean-loup Gailly and Mark Adler ** ** Depencency: LZ4 library: https://lz4.github.io/lz4/ ** author: (C) 2011-2019, Yann Collet, ** ** Depencency: Original LZMA library ** author: Igor Pavlov ** ** Depencency: Eazylzma: https://github.com/lloyd/easylzma ** author: Lloyd Hilaiel (lloyd) ** ** Depencency: base64_encode()/base64_decode() ** \copyright 2005-2011, Jouni Malinen ** ** \section slicense License ** GPL v3, see LICENSE.txt for details *******************************************************************************/ /***************************************************************************//** \file zmatlib.c @brief Compression and decompression interfaces: zmat_run, zmat_encode, zmat_decode *******************************************************************************/ #include #include #include #include #include #include "zmatlib.h" #ifndef NO_ZLIB #include "zlib.h" #else #include "miniz.h" #define GZIP_HEADER_SIZE 10 #endif #ifndef NO_LZMA #include "easylzma/compress.h" #include "easylzma/decompress.h" #endif #ifndef NO_LZ4 #include "lz4/lz4.h" #include "lz4/lz4hc.h" #endif #ifndef NO_BLOSC2 #include "blosc2.h" #endif #ifndef NO_ZSTD #include "zstd.h" unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize); #endif #ifdef NO_ZLIB int miniz_gzip_uncompress(void* in_data, size_t in_len, void** out_data, size_t* out_len); #endif #ifndef NO_LZMA /** * @brief Easylzma interface to perform compression * * @param[in] format: output format (0 for lzip format, 1 for lzma-alone format) * @param[in] inData: input stream buffer pointer * @param[in] inLen: input stream buffer length * @param[in] outData: output stream buffer pointer * @param[in] outLen: output stream buffer length * @param[in] level: positive number: use default compression level (5); * negative interger: set compression level (-1, less, to -9, more compression) * @return return the fine grained lzma error code. */ int simpleCompress(elzma_file_format format, const unsigned char* inData, size_t inLen, unsigned char** outData, size_t* outLen, int level); /** * @brief Easylzma interface to perform decompression * * @param[in] format: output format (0 for lzip format, 1 for lzma-alone format) * @param[in] inData: input stream buffer pointer * @param[in] inLen: input stream buffer length * @param[in] outData: output stream buffer pointer * @param[in] outLen: output stream buffer length * @return return the fine grained lzma error code. */ int simpleDecompress(elzma_file_format format, const unsigned char* inData, size_t inLen, unsigned char** outData, size_t* outLen); #endif /** * @brief Coarse grained error messages (encoder-specific detailed error codes are in the status parameter) * */ const char* zmat_errcode[] = { "No error", /*0*/ "input can not be empty", /*-1*/ "failed to initialize zlib", /*-2*/ "zlib error, see info.status for error flag, often a result of mismatch in compression method", /*-3*/ "easylzma error, see info.status for error flag, often a result of mismatch in compression method",/*-4*/ "can not allocate output buffer",/*-5*/ "lz4 error, see info.status for error flag, often a result of mismatch in compression method",/*-6*/ "unsupported blosc2 codec",/*-7*/ "blosc2 error, see info.status for error flag, often a result of mismatch in compression method",/*-8*/ "zstd error, see info.status for error flag, often a result of mismatch in compression method",/*-9*/ "miniz error, see info.status for error flag, often a result of mismatch in compression method",/*-10*/ "unsupported method" /*-999*/ }; /** * @brief Convert error code to a string error message * * @param[in] id: zmat error code */ char* zmat_error(int id) { if (id >= 0 && id < (sizeof(zmat_errcode) / sizeof(zmat_errcode[0]))) { return (char*)(zmat_errcode[id]); } else { return "zmatlib: unknown error"; } } /** * @brief Main interface to perform compression/decompression * * @param[in] inputsize: input stream buffer length * @param[in] inputstr: input stream buffer pointer * @param[in, out] outputsize: output stream buffer length * @param[in, out] outputbuf: output stream buffer pointer * @param[out] ret: encoder/decoder specific detailed error code (if error occurs) * @param[in] iscompress: 0: decompression, 1: use default compression level; * negative interger: set compression level (-1, less, to -9, more compression) * @return return the coarse grained zmat error code; detailed error code is in ret. */ int zmat_run(const size_t inputsize, unsigned char* inputstr, size_t* outputsize, unsigned char** outputbuf, const int zipid, int* ret, const int iscompress) { z_stream zs; size_t buflen[2] = {0}; int clevel; union cflag { int iscompress; struct settings { char clevel; char nthread; char shuffle; char typesize; } param; } flags; *outputbuf = NULL; flags.iscompress = iscompress; zs.zalloc = Z_NULL; zs.zfree = Z_NULL; zs.opaque = Z_NULL; if (inputsize == 0) { return -1; } clevel = (flags.param.clevel == 0) ? 0 : flags.param.clevel; if (clevel) { /** * perform compression or encoding */ if (zipid == zmBase64) { /** * base64 encoding */ *outputbuf = base64_encode((const unsigned char*)inputstr, inputsize, outputsize, clevel); } else if (zipid == zmZlib || zipid == zmGzip) { /** * zlib (.zip) or gzip (.gz) compression */ if (zipid == zmZlib) { if (deflateInit(&zs, (clevel > 0) ? Z_DEFAULT_COMPRESSION : (-clevel)) != Z_OK) { return -2; } } else { #ifdef NO_ZLIB /* Initialize streaming buffer context */ memset(&zs, '\0', sizeof(zs)); zs.zalloc = Z_NULL; zs.zfree = Z_NULL; zs.opaque = Z_NULL; zs.next_in = inputstr; zs.avail_in = inputsize; zs.total_out = 0; if (deflateInit2(&zs, (clevel > 0) ? Z_DEFAULT_COMPRESSION : (-clevel), Z_DEFLATED, -Z_DEFAULT_WINDOW_BITS, 9, Z_DEFAULT_STRATEGY) != Z_OK) { return -2; } #else if (deflateInit2(&zs, (clevel > 0) ? Z_DEFAULT_COMPRESSION : (-clevel), Z_DEFLATED, 15 | 16, MAX_MEM_LEVEL, Z_DEFAULT_STRATEGY) != Z_OK) { return -2; } #endif } #ifdef NO_ZLIB if (zipid == zmGzip) { /* * miniz based gzip compression code was adapted based on the following * https://github.com/atheriel/fluent-bit/blob/8f0002b36601006240d50ea3c86769629d99b1e8/src/flb_gzip.c */ int flush = Z_NO_FLUSH; void* out_buf; size_t out_size = inputsize + 32; unsigned char* pb; const unsigned char gzip_magic_header [] = {0x1F, 0x8B, 8, 0, 0, 0, 0, 0, 0, 0xFF}; out_buf = (unsigned char*)malloc(out_size); memcpy(out_buf, gzip_magic_header, GZIP_HEADER_SIZE); pb = (unsigned char*) out_buf + GZIP_HEADER_SIZE; while (1) { zs.next_out = pb + zs.total_out; zs.avail_out = out_size - (pb - (unsigned char*) out_buf); if (zs.avail_in == 0) { flush = Z_FINISH; } *ret = deflate(&zs, flush); if (*ret == Z_STREAM_END) { break; } else if (*ret != Z_OK) { deflateEnd(&zs); return -3; } } if (deflateEnd(&zs) != Z_OK) { free(out_buf); return -3; } *outputsize = zs.total_out; /* Construct the gzip checksum (CRC32 footer) */ int footer_start = GZIP_HEADER_SIZE + *outputsize; pb = (unsigned char*) out_buf + footer_start; mz_ulong crc = mz_crc32(MZ_CRC32_INIT, inputstr, inputsize); *pb++ = crc & 0xFF; *pb++ = (crc >> 8) & 0xFF; *pb++ = (crc >> 16) & 0xFF; *pb++ = (crc >> 24) & 0xFF; *pb++ = inputsize & 0xFF; *pb++ = (inputsize >> 8) & 0xFF; *pb++ = (inputsize >> 16) & 0xFF; *pb++ = (inputsize >> 24) & 0xFF; /* update the final output buffer size */ *outputsize += GZIP_HEADER_SIZE + 8; *outputbuf = out_buf; } else { #endif buflen[0] = deflateBound(&zs, inputsize); *outputbuf = (unsigned char*)malloc(buflen[0]); zs.avail_in = inputsize; /* size of input, string + terminator*/ zs.next_in = (Bytef*)inputstr; /* input char array*/ zs.avail_out = buflen[0]; /* size of output*/ zs.next_out = (Bytef*)(*outputbuf); /*(Bytef *)(); // output char array*/ *ret = deflate(&zs, Z_FINISH); *outputsize = zs.total_out; if (*ret != Z_STREAM_END && *ret != Z_OK) { deflateEnd(&zs); return -3; } deflateEnd(&zs); #ifdef NO_ZLIB } #endif #ifndef NO_LZMA } else if (zipid == zmLzma || zipid == zmLzip) { /** * lzma (.lzma) or lzip (.lzip) compression */ *ret = simpleCompress((elzma_file_format)(zipid - 3), (unsigned char*)inputstr, inputsize, outputbuf, outputsize, clevel); if (*ret != ELZMA_E_OK) { return -4; } #endif #ifndef NO_LZ4 } else if (zipid == zmLz4 || zipid == zmLz4hc) { /** * lz4 or lz4hc compression */ *outputsize = LZ4_compressBound(inputsize); if (!(*outputbuf = (unsigned char*)malloc(*outputsize))) { return -5; } if (zipid == zmLz4) { *outputsize = LZ4_compress_default((const char*)inputstr, (char*)(*outputbuf), inputsize, *outputsize); } else { *outputsize = LZ4_compress_HC((const char*)inputstr, (char*)(*outputbuf), inputsize, *outputsize, (clevel > 0) ? 8 : (-clevel)); } *ret = *outputsize; if (*outputsize == 0) { return -6; } #endif #ifndef NO_ZSTD } else if (zipid == zmZstd) { /** * zstd compression */ *outputsize = ZSTD_compressBound(inputsize); if (!(*outputbuf = (unsigned char*)malloc(*outputsize))) { return -5; } *ret = ZSTD_compress((char*)(*outputbuf), *outputsize, (const char*)inputstr, inputsize, (clevel > 0) ? ZSTD_CLEVEL_DEFAULT : (-clevel)); if (ZSTD_isError(*ret)) { return -9; } *outputsize = *ret; if (!(*outputbuf = (unsigned char*)realloc(*outputbuf, *outputsize))) { return -5; } #endif #ifndef NO_BLOSC2 } else if (zipid >= zmBlosc2Blosclz && zipid <= zmBlosc2Zstd) { /** * blosc2 meta-compressor (support various filters and compression codecs) */ unsigned int nthread = 1, shuffle = 1, typesize = 4; const char* codecs[] = {"blosclz", "lz4", "lz4hc", "zlib", "zstd"}; nthread = (flags.param.nthread == 0 || flags.param.nthread == -1) ? 1 : flags.param.nthread; shuffle = (flags.param.shuffle == 0 || flags.param.shuffle == -1) ? 1 : flags.param.shuffle; typesize = (flags.param.typesize == 0 || flags.param.typesize == -1) ? 4 : flags.param.typesize; if (blosc1_set_compressor(codecs[zipid - zmBlosc2Blosclz]) == -1) { return -7; } blosc2_set_nthreads(nthread); *outputsize = inputsize + BLOSC2_MAX_OVERHEAD; /* blosc2 guarantees the compression will always succeed at this size */ if (!(*outputbuf = (unsigned char*)malloc(*outputsize))) { return -5; } *ret = blosc1_compress((clevel > 0) ? 5 : (-clevel), shuffle, typesize, inputsize, (const void*)inputstr, (void*)(*outputbuf), *outputsize); if (*ret < 0) { return -8; } *outputsize = *ret; if (!(*outputbuf = (unsigned char*)realloc(*outputbuf, *outputsize))) { return -5; } #endif } else { return -999; } } else { /** * perform decompression or decoding */ if (zipid == zmBase64) { /** * base64 decoding */ *outputbuf = base64_decode((const unsigned char*)inputstr, inputsize, outputsize); } else if (zipid == zmZlib || zipid == zmGzip) { /** * zlib (.zip) or gzip (.gz) decompression */ int count = 1; if (zipid == zmZlib) { if (inflateInit(&zs) != Z_OK) { return -2; } } else { #ifndef NO_ZLIB if (inflateInit2(&zs, 15 | 32) != Z_OK) { return -2; } #endif } buflen[0] = inputsize * 4; *outputbuf = (unsigned char*)malloc(buflen[0]); zs.avail_in = inputsize; /* size of input, string + terminator*/ zs.next_in = inputstr; /* input char array*/ zs.avail_out = buflen[0]; /* size of output*/ zs.next_out = (Bytef*)(*outputbuf); /* output char array*/ #ifdef NO_ZLIB if (zipid == zmZlib) { #endif while ((*ret = inflate(&zs, Z_SYNC_FLUSH)) != Z_STREAM_END && *ret != Z_DATA_ERROR && count <= 12) { *outputbuf = (unsigned char*)realloc(*outputbuf, (buflen[0] << count)); zs.next_out = (Bytef*)(*outputbuf + zs.total_out); zs.avail_out = (buflen[0] << count) - zs.total_out; /* size of output*/ count++; } *outputsize = zs.total_out; if (*ret != Z_STREAM_END && *ret != Z_OK) { inflateEnd(&zs); return -3; } inflateEnd(&zs); #ifdef NO_ZLIB } else { *ret = miniz_gzip_uncompress(inputstr, inputsize, (void**)outputbuf, outputsize); if (*ret != 0) { return -10; } } #endif #ifndef NO_LZMA } else if (zipid == zmLzma || zipid == zmLzip) { /** * lzma (.lzma) or lzip (.lzip) decompression */ *ret = simpleDecompress((elzma_file_format)(zipid - 3), (unsigned char*)inputstr, inputsize, outputbuf, outputsize); if (*ret != ELZMA_E_OK) { return -4; } #endif #ifndef NO_LZ4 } else if (zipid == zmLz4 || zipid == zmLz4hc) { /** * lz4 or lz4hc decompression */ int count = 2; *outputsize = (inputsize << count); if (!(*outputbuf = (unsigned char*)malloc(*outputsize))) { *ret = -5; return *ret; } while ((*ret = LZ4_decompress_safe((const char*)inputstr, (char*)(*outputbuf), inputsize, *outputsize)) <= 0 && count <= 10) { *outputsize = (inputsize << count); if (!(*outputbuf = (unsigned char*)realloc(*outputbuf, *outputsize))) { *ret = -5; return *ret; } count++; } *outputsize = *ret; if (*ret < 0) { return -6; } #endif #ifndef NO_ZSTD } else if (zipid == zmZstd) { /** * zstd decompression */ *outputsize = ZSTD_decompressBound(inputstr, inputsize); if (*outputsize == ZSTD_CONTENTSIZE_ERROR || !(*outputbuf = (unsigned char*)malloc(*outputsize))) { *ret = (*outputsize == ZSTD_CONTENTSIZE_ERROR) ? -9 : -5; return *ret; } *ret = ZSTD_decompress((void*)(*outputbuf), *outputsize, (const void*)inputstr, inputsize); *outputsize = *ret; if (ZSTD_isError(*ret)) { return -9; } #endif #ifndef NO_BLOSC2 } else if (zipid >= zmBlosc2Blosclz && zipid <= zmBlosc2Zstd) { /** * blosc2 meta-compressor (support various filters and compression codecs) */ int count = 2; *outputsize = (inputsize << count); if (!(*outputbuf = (unsigned char*)malloc(*outputsize))) { *ret = -5; return *ret; } while ((*ret = blosc1_decompress((const char*)inputstr, (char*)(*outputbuf), *outputsize)) <= 0 && count <= 16) { *outputsize = (inputsize << count); if (!(*outputbuf = (unsigned char*)realloc(*outputbuf, *outputsize))) { *ret = -5; return *ret; } count++; } *outputsize = *ret; if (*ret < 0) { return -8; } #endif } else { return -999; } } return 0; } /** * @brief Simplified interface to perform compression (use default compression level) * * @param[in] inputsize: input stream buffer length * @param[in] inputstr: input stream buffer pointer * @param[in] outputsize: output stream buffer length * @param[in] outputbuf: output stream buffer pointer * @param[in] ret: encoder/decoder specific detailed error code (if error occurs) * @return return the coarse grained zmat error code; detailed error code is in ret. */ int zmat_encode(const size_t inputsize, unsigned char* inputstr, size_t* outputsize, unsigned char** outputbuf, const int zipid, int* ret) { return zmat_run(inputsize, inputstr, outputsize, outputbuf, zipid, ret, 1); } /** * @brief Simplified interface to perform decompression * * @param[in] inputsize: input stream buffer length * @param[in] inputstr: input stream buffer pointer * @param[in] outputsize: output stream buffer length * @param[in] outputbuf: output stream buffer pointer * @param[in] ret: encoder/decoder specific detailed error code (if error occurs) * @return return the coarse grained zmat error code; detailed error code is in ret. */ int zmat_decode(const size_t inputsize, unsigned char* inputstr, size_t* outputsize, unsigned char** outputbuf, const int zipid, int* ret) { return zmat_run(inputsize, inputstr, outputsize, outputbuf, zipid, ret, 0); } /** * @brief Look up a string in a string list and return the index * * @param[in] origkey: string to be looked up * @param[out] table: the dictionary where the string is searched * @return if found, return the index of the string in the dictionary, otherwise -1. */ int zmat_keylookup(char* origkey, const char* table[]) { int i = 0; char* key = (char*)malloc(strlen(origkey) + 1); memcpy(key, origkey, strlen(origkey) + 1); while (key[i]) { key[i] = tolower(key[i]); i++; } i = 0; while (table[i] && table[i][0] != '\0') { if (strcmp(key, table[i]) == 0) { free(key); return i; } i++; } free(key); return -1; } /** * @brief Free the output buffer to facilitate use in fortran * * @param[in,out] outputbuf: the outputbuf buffer's initial address to be freed */ void zmat_free(unsigned char** outputbuf) { if (*outputbuf) { free(*outputbuf); } *outputbuf = NULL; } /* * @brief Base64 encoding/decoding (RFC1341) * @author Copyright (c) 2005-2011, Jouni Malinen * * This software may be distributed under the terms of the BSD license. * See README for more details. */ static const unsigned char base64_table[65] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; /** * @brief base64_encode - Base64 encode * @src: Data to be encoded * @len: Length of the data to be encoded * @out_len: Pointer to output length variable, or %NULL if not used * Returns: Allocated buffer of out_len bytes of encoded data, * or %NULL on failure * * Caller is responsible for freeing the returned buffer. Returned buffer is * nul terminated to make it easier to use as a C string. The nul terminator is * not included in out_len. */ unsigned char* base64_encode(const unsigned char* src, size_t len, size_t* out_len, int mode) { unsigned char* out, *pos; const unsigned char* end, *in; size_t olen; size_t line_len; olen = len * 4 / 3 + 4; /* 3-byte blocks to 4-byte */ olen += olen / 72; /* line feeds */ olen++; /* nul termination */ if (olen < len) { return NULL; /* integer overflow */ } out = (unsigned char*)malloc(olen); if (out == NULL) { return NULL; } end = src + len; in = src; pos = out; line_len = 0; while (end - in >= 3) { *pos++ = base64_table[in[0] >> 2]; *pos++ = base64_table[((in[0] & 0x03) << 4) | (in[1] >> 4)]; *pos++ = base64_table[((in[1] & 0x0f) << 2) | (in[2] >> 6)]; *pos++ = base64_table[in[2] & 0x3f]; in += 3; line_len += 4; if (mode > 1 && line_len >= 72) { *pos++ = '\n'; line_len = 0; } } if (end - in) { *pos++ = base64_table[in[0] >> 2]; if (end - in == 1) { *pos++ = base64_table[(in[0] & 0x03) << 4]; *pos++ = '='; } else { *pos++ = base64_table[((in[0] & 0x03) << 4) | (in[1] >> 4)]; *pos++ = base64_table[(in[1] & 0x0f) << 2]; } *pos++ = '='; line_len += 4; } if (mode > 2 && line_len) { *pos++ = '\n'; } *pos = '\0'; if (out_len) { *out_len = pos - out; } return out; } /** * base64_decode - Base64 decode * @src: Data to be decoded * @len: Length of the data to be decoded * @out_len: Pointer to output length variable * Returns: Allocated buffer of out_len bytes of decoded data, * or %NULL on failure * * Caller is responsible for freeing the returned buffer. */ unsigned char* base64_decode(const unsigned char* src, size_t len, size_t* out_len) { unsigned char dtable[256], *out, *pos, block[4], tmp; size_t i, count, olen; int pad = 0; memset(dtable, 0x80, 256); for (i = 0; i < sizeof(base64_table) - 1; i++) { dtable[base64_table[i]] = (unsigned char) i; } dtable['='] = 0; count = 0; for (i = 0; i < len; i++) { if (dtable[src[i]] != 0x80) { count++; } } if (count == 0 || count % 4) { return NULL; } olen = count / 4 * 3; pos = out = (unsigned char*)malloc(olen); if (out == NULL) { return NULL; } count = 0; for (i = 0; i < len; i++) { tmp = dtable[src[i]]; if (tmp == 0x80) { continue; } if (src[i] == '=') { pad++; } block[count] = tmp; count++; if (count == 4) { *pos++ = (block[0] << 2) | (block[1] >> 4); *pos++ = (block[1] << 4) | (block[2] >> 2); *pos++ = (block[2] << 6) | block[3]; count = 0; if (pad) { if (pad == 1) { pos--; } else if (pad == 2) { pos -= 2; } else { /* Invalid padding */ free(out); return NULL; } break; } } } *out_len = pos - out; return out; } #ifndef NO_LZMA /** * @brief Easylzma compression interface */ struct dataStream { const unsigned char* inData; size_t inLen; unsigned char* outData; size_t outLen; }; /** * @brief Easylzma input callback function */ static int inputCallback(void* ctx, void* buf, size_t* size) { size_t rd = 0; struct dataStream* ds = (struct dataStream*) ctx; assert(ds != NULL); rd = (ds->inLen < *size) ? ds->inLen : *size; if (rd > 0) { memcpy(buf, (void*) ds->inData, rd); ds->inData += rd; ds->inLen -= rd; } *size = rd; return 0; } /** * @brief Easylzma output callback function */ static size_t outputCallback(void* ctx, const void* buf, size_t size) { struct dataStream* ds = (struct dataStream*) ctx; assert(ds != NULL); if (size > 0) { ds->outData = (unsigned char*)realloc(ds->outData, ds->outLen + size); memcpy((void*) (ds->outData + ds->outLen), buf, size); ds->outLen += size; } return size; } /** * @brief Easylzma interface to perform compression * * @param[in] format: output format (0 for lzip format, 1 for lzma-alone format) * @param[in] inData: input stream buffer pointer * @param[in] inLen: input stream buffer length * @param[in] outData: output stream buffer pointer * @param[in] outLen: output stream buffer length * @param[in] level: positive number: use default compression level (5); * negative interger: set compression level (-1, less, to -9, more compression) * @return return the fine grained lzma error code. */ int simpleCompress(elzma_file_format format, const unsigned char* inData, size_t inLen, unsigned char** outData, size_t* outLen, int level) { int rc; elzma_compress_handle hand; /* allocate compression handle */ hand = elzma_compress_alloc(); assert(hand != NULL); rc = elzma_compress_config(hand, ELZMA_LC_DEFAULT, ELZMA_LP_DEFAULT, ELZMA_PB_DEFAULT, ((level > 0) ? 5 : -level), (1 << 20) /* 1mb */, format, inLen); if (rc != ELZMA_E_OK) { elzma_compress_free(&hand); return rc; } /* now run the compression */ { struct dataStream ds; ds.inData = inData; ds.inLen = inLen; ds.outData = NULL; ds.outLen = 0; rc = elzma_compress_run(hand, inputCallback, (void*) &ds, outputCallback, (void*) &ds, NULL, NULL); if (rc != ELZMA_E_OK) { if (ds.outData != NULL) { free(ds.outData); } elzma_compress_free(&hand); return rc; } *outData = ds.outData; *outLen = ds.outLen; } return rc; } /** * @brief Easylzma interface to perform decompression * * @param[in] format: output format (0 for lzip format, 1 for lzma-alone format) * @param[in] inData: input stream buffer pointer * @param[in] inLen: input stream buffer length * @param[in] outData: output stream buffer pointer * @param[in] outLen: output stream buffer length * @return return the fine grained lzma error code. */ int simpleDecompress(elzma_file_format format, const unsigned char* inData, size_t inLen, unsigned char** outData, size_t* outLen) { int rc; elzma_decompress_handle hand; hand = elzma_decompress_alloc(); /* now run the compression */ { struct dataStream ds; ds.inData = inData; ds.inLen = inLen; ds.outData = NULL; ds.outLen = 0; rc = elzma_decompress_run(hand, inputCallback, (void*) &ds, outputCallback, (void*) &ds, format); if (rc != ELZMA_E_OK) { if (ds.outData != NULL) { free(ds.outData); } elzma_decompress_free(&hand); return rc; } *outData = ds.outData; *outLen = ds.outLen; } return rc; } #ifdef NO_ZLIB /* * miniz based gzip compression code was adapted based on the following * https://github.com/atheriel/fluent-bit/blob/8f0002b36601006240d50ea3c86769629d99b1e8/src/flb_gzip.c */ /* Fluent Bit * ========== * Copyright (C) 2019-2020 The Fluent Bit Authors * Copyright (C) 2015-2018 Treasure Data Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ typedef enum { FTEXT = 1, FHCRC = 2, FEXTRA = 4, FNAME = 8, FCOMMENT = 16 } miniz_tinf_gzip_flag; static unsigned int read_le16(const unsigned char* p) { return ((unsigned int) p[0]) | ((unsigned int) p[1] << 8); } static unsigned int read_le32(const unsigned char* p) { return ((unsigned int) p[0]) | ((unsigned int) p[1] << 8) | ((unsigned int) p[2] << 16) | ((unsigned int) p[3] << 24); } /* Uncompress (inflate) GZip data */ int miniz_gzip_uncompress(void* in_data, size_t in_len, void** out_data, size_t* out_len) { int status; unsigned char* p; void* out_buf; size_t out_size = 0; void* zip_data; size_t zip_len; unsigned char flg; unsigned int xlen, hcrc; unsigned int dlen, crc; mz_ulong crc_out; mz_stream stream; const unsigned char* start; /* Minimal length: header + crc32 */ if (in_len < 18) { return -1; } /* Magic bytes */ p = in_data; if (p[0] != 0x1F || p[1] != 0x8B) { return -2; } if (p[2] != 8) { return -3; } /* Flag byte */ flg = p[3]; /* Reserved bits */ if (flg & 0xE0) { return -4; } /* Skip base header of 10 bytes */ start = p + GZIP_HEADER_SIZE; /* Skip extra data if present */ if (flg & FEXTRA) { xlen = read_le16(start); if (xlen > in_len - 12) { return -5; } start += xlen + 2; } /* Skip file name if present */ if (flg & FNAME) { do { if (start - p >= in_len) { return -6; } } while (*start++); } /* Skip file comment if present */ if (flg & FCOMMENT) { do { if (start - p >= in_len) { return -6; } } while (*start++); } /* Check header crc if present */ if (flg & FHCRC) { if (start - p > in_len - 2) { return -7; } hcrc = read_le16(start); crc = mz_crc32(MZ_CRC32_INIT, p, start - p) & 0x0000FFFF; if (hcrc != crc) { return -8; } start += 2; } /* Get decompressed length */ dlen = read_le32(&p[in_len - 4]); /* Get CRC32 checksum of original data */ crc = read_le32(&p[in_len - 8]); /* Decompress data */ if ((p + in_len) - p < 8) { return -9; } /* Allocate outgoing buffer */ out_buf = malloc(dlen); if (!out_buf) { return -10; } out_size = dlen; /* Map zip content */ zip_data = (unsigned char*) start; zip_len = (p + in_len) - start - 8; memset(&stream, 0, sizeof(stream)); stream.next_in = zip_data; stream.avail_in = zip_len; stream.next_out = out_buf; stream.avail_out = out_size; status = mz_inflateInit2(&stream, -Z_DEFAULT_WINDOW_BITS); if (status != MZ_OK) { free(out_buf); return -11; } status = mz_inflate(&stream, MZ_FINISH); if (status != MZ_STREAM_END) { mz_inflateEnd(&stream); free(out_buf); return -12; } if (stream.total_out != dlen) { mz_inflateEnd(&stream); free(out_buf); return -13; } /* terminate the stream, it's not longer required */ mz_inflateEnd(&stream); /* Validate message CRC vs inflated data CRC */ crc_out = mz_crc32(MZ_CRC32_INIT, out_buf, dlen); if (crc_out != crc) { free(out_buf); return -14; } /* set the uncompressed data */ *out_len = dlen; *out_data = out_buf; return 0; } #endif #endif zmat-0.9.9/src/miniz/0000755000175200007730000000000014515254731014654 5ustar rlaboissrlaboisszmat-0.9.9/src/miniz/ChangeLog.md0000644000175200007730000004100514515254731017025 0ustar rlaboissrlaboiss## Changelog ### 2.2.0 - Fix examples with amalgamation - Modified cmake script to support shared library mode and find_package - Fix for misleading doc comment on `mz_zip_reader_init_cfile` function - Add include location tolerance and stop forcing `_GNU_SOURCE` - Fix: mz_zip_reader_locate_file_v2 returns an mz_bool - Fix large file system checks - Add #elif to enable an external mz_crc32() to be linked in - Write with dynamic size (size of file/data to be added not known before adding) - Added uncompress2 for zlib compatibility - Add support for building as a Meson subproject - Added OSSFuzz support; Integrate with CIFuzz - Add pkg-config file - Fixed use-of-uninitialized value msan error when copying dist bytes with no output bytes written. - mz_zip_validate_file(): fix memory leak on errors - Fixed MSAN use-of-uninitialized in tinfl_decompress when invalid dist is decoded. In this instance dist was 31 which s_dist_base translates as 0 - Add flag to set (compressed) size in local file header - avoid use of uninitialized value in tdefl_record_literal ### 2.1.0 - More instances of memcpy instead of cast and use memcpy per default - Remove inline for c90 support - New function to read files via callback functions when adding them - Fix out of bounds read while reading Zip64 extended information - guard memcpy when n == 0 because buffer may be NULL - Implement inflateReset() function - Move comp/decomp alloc/free prototypes under guarding #ifndef MZ_NO_MALLOC - Fix large file support under Windows - Don't warn if _LARGEFILE64_SOURCE is not defined to 1 - Fixes for MSVC warnings - Remove check that path of file added to archive contains ':' or '\' - Add !defined check on MINIZ_USE_ALIGNED_LOADS_AND_STORES ### 2.0.8 - Remove unimplemented functions (mz_zip_locate_file and mz_zip_locate_file_v2) - Add license, changelog, readme and example files to release zip - Fix heap overflow to user buffer in tinfl_status tinfl_decompress - Fix corrupt archive if uncompressed file smaller than 4 byte and the file is added by mz_zip_writer_add_mem* ### 2.0.7 - Removed need in C++ compiler in cmake build - Fixed a lot of uninitialized value errors found with Valgrind by memsetting m_dict to 0 in tdefl_init - Fix resource leak in mz_zip_reader_init_file_v2 - Fix assert with mz_zip_writer_add_mem* w/MZ_DEFAULT_COMPRESSION - cmake build: install library and headers - Remove _LARGEFILE64_SOURCE requirement from apple defines for large files ### 2.0.6 - Improve MZ_ZIP_FLAG_WRITE_ZIP64 documentation - Remove check for cur_archive_file_ofs > UINT_MAX because cur_archive_file_ofs is not used after this point - Add cmake debug configuration - Fix PNG height when creating png files - Add "iterative" file extraction method based on mz_zip_reader_extract_to_callback. - Option to use memcpy for unaligned data access - Define processor/arch macros as zero if not set to one ### 2.0.4/2.0.5 - Fix compilation with the various omission compile definitions ### 2.0.3 - Fix GCC/clang compile warnings - Added callback for periodic flushes (for ZIP file streaming) - Use UTF-8 for file names in ZIP files per default ### 2.0.2 - Fix source backwards compatibility with 1.x - Fix a ZIP bit not being set correctly ### 2.0.1 - Added some tests - Added CI - Make source code ANSI C compatible ### 2.0.0 beta - Matthew Sitton merged miniz 1.x to Rich Geldreich's vogl ZIP64 changes. Miniz is now licensed as MIT since the vogl code base is MIT licensed - Miniz is now split into several files - Miniz does now not seek backwards when creating ZIP files. That is the ZIP files can be streamed - Miniz automatically switches to the ZIP64 format when the created ZIP files goes over ZIP file limits - Similar to [SQLite](https://www.sqlite.org/amalgamation.html) the Miniz source code is amalgamated into one miniz.c/miniz.h pair in a build step (amalgamate.sh). Please use miniz.c/miniz.h in your projects - Miniz 2 is only source back-compatible with miniz 1.x. It breaks binary compatibility because structures changed ### v1.16 BETA Oct 19, 2013 Still testing, this release is downloadable from [here](http://www.tenacioussoftware.com/miniz_v116_beta_r1.7z). Two key inflator-only robustness and streaming related changes. Also merged in tdefl_compressor_alloc(), tdefl_compressor_free() helpers to make script bindings easier for rustyzip. I would greatly appreciate any help with testing or any feedback. The inflator in raw (non-zlib) mode is now usable on gzip or similar streams that have a bunch of bytes following the raw deflate data (problem discovered by rustyzip author williamw520). This version should never read beyond the last byte of the raw deflate data independent of how many bytes you pass into the input buffer. The inflator now has a new failure status TINFL_STATUS_FAILED_CANNOT_MAKE_PROGRESS (-4). Previously, if the inflator was starved of bytes and could not make progress (because the input buffer was empty and the caller did not set the TINFL_FLAG_HAS_MORE_INPUT flag - say on truncated or corrupted compressed data stream) it would append all 0's to the input and try to soldier on. This is scary behavior if the caller didn't know when to stop accepting output (because it didn't know how much uncompressed data was expected, or didn't enforce a sane maximum). v1.16 will instead return TINFL_STATUS_FAILED_CANNOT_MAKE_PROGRESS immediately if it needs 1 or more bytes to make progress, the input buf is empty, and the caller has indicated that no more input is available. This is a "soft" failure, so you can call the inflator again with more input and it will try to continue, or you can give up and fail. This could be very useful in network streaming scenarios. - The inflator coroutine func. is subtle and complex so I'm being cautious about this release. I would greatly appreciate any help with testing or any feedback. I feel good about these changes, and they've been through several hours of automated testing, but they will probably not fix anything for the majority of prev. users so I'm going to mark this release as beta for a few weeks and continue testing it at work/home on various things. - The inflator in raw (non-zlib) mode is now usable on gzip or similar data streams that have a bunch of bytes following the raw deflate data (problem discovered by rustyzip author williamw520). This version should *never* read beyond the last byte of the raw deflate data independent of how many bytes you pass into the input buffer. This issue was caused by the various Huffman bitbuffer lookahead optimizations, and would not be an issue if the caller knew and enforced the precise size of the raw compressed data *or* if the compressed data was in zlib format (i.e. always followed by the byte aligned zlib adler32). So in other words, you can now call the inflator on deflate streams that are followed by arbitrary amounts of data and it's guaranteed that decompression will stop exactly on the last byte. - The inflator now has a new failure status: TINFL_STATUS_FAILED_CANNOT_MAKE_PROGRESS (-4). Previously, if the inflator was starved of bytes and could not make progress (because the input buffer was empty and the caller did not set the TINFL_FLAG_HAS_MORE_INPUT flag - say on truncated or corrupted compressed data stream) it would append all 0's to the input and try to soldier on. This is scary, because in the worst case, I believe it was possible for the prev. inflator to start outputting large amounts of literal data. If the caller didn't know when to stop accepting output (because it didn't know how much uncompressed data was expected, or didn't enforce a sane maximum) it could continue forever. v1.16 cannot fall into this failure mode, instead it'll return TINFL_STATUS_FAILED_CANNOT_MAKE_PROGRESS immediately if it needs 1 or more bytes to make progress, the input buf is empty, and the caller has indicated that no more input is available. This is a "soft" failure, so you can call the inflator again with more input and it will try to continue, or you can give up and fail. This could be very useful in network streaming scenarios. - Added documentation to all the tinfl return status codes, fixed miniz_tester so it accepts double minus params for Linux, tweaked example1.c, added a simple "follower bytes" test to miniz_tester.cpp. ### v1.15 r4 STABLE - Oct 13, 2013 Merged over a few very minor bug fixes that I fixed in the zip64 branch. This is downloadable from [here](http://code.google.com/p/miniz/downloads/list) and also in SVN head (as of 10/19/13). ### v1.15 - Oct. 13, 2013 Interim bugfix release while I work on the next major release with zip64 and streaming compression/decompression support. Fixed the MZ_ZIP_FLAG_DO_NOT_SORT_CENTRAL_DIRECTORY bug (thanks kahmyong.moon@hp.com), which could cause the locate files func to not find files when this flag was specified. Also fixed a bug in mz_zip_reader_extract_to_mem_no_alloc() with user provided read buffers (thanks kymoon). I also merged lots of compiler fixes from various github repo branches and Google Code issue reports. I finally added cmake support (only tested under for Linux so far), compiled and tested with clang v3.3 and gcc 4.6 (under Linux), added defl_write_image_to_png_file_in_memory_ex() (supports Y flipping for OpenGL use, real-time compression), added a new PNG example (example6.c - Mandelbrot), and I added 64-bit file I/O support (stat64(), etc.) for glibc. - Critical fix for the MZ_ZIP_FLAG_DO_NOT_SORT_CENTRAL_DIRECTORY bug (thanks kahmyong.moon@hp.com) which could cause locate files to not find files. This bug would only have occured in earlier versions if you explicitly used this flag, OR if you used mz_zip_extract_archive_file_to_heap() or mz_zip_add_mem_to_archive_file_in_place() (which used this flag). If you can't switch to v1.15 but want to fix this bug, just remove the uses of this flag from both helper funcs (and of course don't use the flag). - Bugfix in mz_zip_reader_extract_to_mem_no_alloc() from kymoon when pUser_read_buf is not NULL and compressed size is > uncompressed size - Fixing mz_zip_reader_extract_*() funcs so they don't try to extract compressed data from directory entries, to account for weird zipfiles which contain zero-size compressed data on dir entries. Hopefully this fix won't cause any issues on weird zip archives, because it assumes the low 16-bits of zip external attributes are DOS attributes (which I believe they always are in practice). - Fixing mz_zip_reader_is_file_a_directory() so it doesn't check the internal attributes, just the filename and external attributes - mz_zip_reader_init_file() - missing MZ_FCLOSE() call if the seek failed - Added cmake support for Linux builds which builds all the examples, tested with clang v3.3 and gcc v4.6. - Clang fix for tdefl_write_image_to_png_file_in_memory() from toffaletti - Merged MZ_FORCEINLINE fix from hdeanclark - Fix include before config #ifdef, thanks emil.brink - Added tdefl_write_image_to_png_file_in_memory_ex(): supports Y flipping (super useful for OpenGL apps), and explicit control over the compression level (so you can set it to 1 for real-time compression). - Merged in some compiler fixes from paulharris's github repro. - Retested this build under Windows (VS 2010, including static analysis), tcc 0.9.26, gcc v4.6 and clang v3.3. - Added example6.c, which dumps an image of the mandelbrot set to a PNG file. - Modified example2 to help test the MZ_ZIP_FLAG_DO_NOT_SORT_CENTRAL_DIRECTORY flag more. - In r3: Bugfix to mz_zip_writer_add_file() found during merge: Fix possible src file fclose() leak if alignment bytes+local header file write faiiled - In r4: Minor bugfix to mz_zip_writer_add_from_zip_reader(): Was pushing the wrong central dir header offset, appears harmless in this release, but it became a problem in the zip64 branch ### v1.14 - May 20, 2012 (SVN Only) Minor tweaks to get miniz.c compiling with the Tiny C Compiler, added #ifndef MINIZ_NO_TIME guards around utime.h includes. Adding mz_free() function, so the caller can free heap blocks returned by miniz using whatever heap functions it has been configured to use, MSVC specific fixes to use "safe" variants of several functions (localtime_s, fopen_s, freopen_s). MinGW32/64 GCC 4.6.1 compiler fixes: added MZ_FORCEINLINE, #include (thanks fermtect). Compiler specific fixes, some from fermtect. I upgraded to TDM GCC 4.6.1 and now static __forceinline is giving it fits, so I'm changing all usage of __forceinline to MZ_FORCEINLINE and forcing gcc to use __attribute__((__always_inline__)) (and MSVC to use __forceinline). Also various fixes from fermtect for MinGW32: added #include , 64-bit ftell/fseek fixes. ### v1.13 - May 19, 2012 From jason@cornsyrup.org and kelwert@mtu.edu - Most importantly, fixed mz_crc32() so it doesn't compute the wrong CRC-32's when mz_ulong is 64-bits. Temporarily/locally slammed in "typedef unsigned long mz_ulong" and re-ran a randomized regression test on ~500k files. Other stuff: Eliminated a bunch of warnings when compiling with GCC 32-bit/64. Ran all examples, miniz.c, and tinfl.c through MSVC 2008's /analyze (static analysis) option and fixed all warnings (except for the silly "Use of the comma-operator in a tested expression.." analysis warning, which I purposely use to work around a MSVC compiler warning). Created 32-bit and 64-bit Codeblocks projects/workspace. Built and tested Linux executables. The codeblocks workspace is compatible with Linux+Win32/x64. Added miniz_tester solution/project, which is a useful little app derived from LZHAM's tester app that I use as part of the regression test. Ran miniz.c and tinfl.c through another series of regression testing on ~500,000 files and archives. Modified example5.c so it purposely disables a bunch of high-level functionality (MINIZ_NO_STDIO, etc.). (Thanks to corysama for the MINIZ_NO_STDIO bug report.) Fix ftell() usage in a few of the examples so they exit with an error on files which are too large (a limitation of the examples, not miniz itself). Fix fail logic handling in mz_zip_add_mem_to_archive_file_in_place() so it always calls mz_zip_writer_finalize_archive() and mz_zip_writer_end(), even if the file add fails. - From jason@cornsyrup.org and kelwert@mtu.edu - Fix mz_crc32() so it doesn't compute the wrong CRC-32's when mz_ulong is 64-bit. - Temporarily/locally slammed in "typedef unsigned long mz_ulong" and re-ran a randomized regression test on ~500k files. - Eliminated a bunch of warnings when compiling with GCC 32-bit/64. - Ran all examples, miniz.c, and tinfl.c through MSVC 2008's /analyze (static analysis) option and fixed all warnings (except for the silly "Use of the comma-operator in a tested expression.." analysis warning, which I purposely use to work around a MSVC compiler warning). - Created 32-bit and 64-bit Codeblocks projects/workspace. Built and tested Linux executables. The codeblocks workspace is compatible with Linux+Win32/x64. - Added miniz_tester solution/project, which is a useful little app derived from LZHAM's tester app that I use as part of the regression test. - Ran miniz.c and tinfl.c through another series of regression testing on ~500,000 files and archives. - Modified example5.c so it purposely disables a bunch of high-level functionality (MINIZ_NO_STDIO, etc.). (Thanks to corysama for the MINIZ_NO_STDIO bug report.) - Fix ftell() usage in examples so they exit with an error on files which are too large (a limitation of the examples, not miniz itself). ### v1.12 - 4/12/12 More comments, added low-level example5.c, fixed a couple minor level_and_flags issues in the archive API's. level_and_flags can now be set to MZ_DEFAULT_COMPRESSION. Thanks to Bruce Dawson for the feedback/bug report. ### v1.11 - 5/28/11 Added statement from unlicense.org ### v1.10 - 5/27/11 - Substantial compressor optimizations: - Level 1 is now ~4x faster than before. The L1 compressor's throughput now varies between 70-110MB/sec. on a Core i7 (actual throughput varies depending on the type of data, and x64 vs. x86). - Improved baseline L2-L9 compression perf. Also, greatly improved compression perf. issues on some file types. - Refactored the compression code for better readability and maintainability. - Added level 10 compression level (L10 has slightly better ratio than level 9, but could have a potentially large drop in throughput on some files). ### v1.09 - 5/15/11 Initial stable release. zmat-0.9.9/src/miniz/miniz.c0000644000175200007730000116053714515254731016163 0ustar rlaboissrlaboiss#include "miniz.h" /************************************************************************** * * Copyright 2013-2014 RAD Game Tools and Valve Software * Copyright 2010-2014 Rich Geldreich and Tenacious Software LLC * All Rights Reserved. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal * in the Software without restriction, including without limitation the rights * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN * THE SOFTWARE. * **************************************************************************/ typedef unsigned char mz_validate_uint16[sizeof(mz_uint16) == 2 ? 1 : -1]; typedef unsigned char mz_validate_uint32[sizeof(mz_uint32) == 4 ? 1 : -1]; typedef unsigned char mz_validate_uint64[sizeof(mz_uint64) == 8 ? 1 : -1]; #ifdef __cplusplus extern "C" { #endif /* ------------------- zlib-style API's */ mz_ulong mz_adler32(mz_ulong adler, const unsigned char *ptr, size_t buf_len) { mz_uint32 i, s1 = (mz_uint32)(adler & 0xffff), s2 = (mz_uint32)(adler >> 16); size_t block_len = buf_len % 5552; if (!ptr) return MZ_ADLER32_INIT; while (buf_len) { for (i = 0; i + 7 < block_len; i += 8, ptr += 8) { s1 += ptr[0], s2 += s1; s1 += ptr[1], s2 += s1; s1 += ptr[2], s2 += s1; s1 += ptr[3], s2 += s1; s1 += ptr[4], s2 += s1; s1 += ptr[5], s2 += s1; s1 += ptr[6], s2 += s1; s1 += ptr[7], s2 += s1; } for (; i < block_len; ++i) s1 += *ptr++, s2 += s1; s1 %= 65521U, s2 %= 65521U; buf_len -= block_len; block_len = 5552; } return (s2 << 16) + s1; } /* Karl Malbrain's compact CRC-32. See "A compact CCITT crc16 and crc32 C implementation that balances processor cache usage against speed": http://www.geocities.com/malbrain/ */ #if 0 mz_ulong mz_crc32(mz_ulong crc, const mz_uint8 *ptr, size_t buf_len) { static const mz_uint32 s_crc32[16] = { 0, 0x1db71064, 0x3b6e20c8, 0x26d930ac, 0x76dc4190, 0x6b6b51f4, 0x4db26158, 0x5005713c, 0xedb88320, 0xf00f9344, 0xd6d6a3e8, 0xcb61b38c, 0x9b64c2b0, 0x86d3d2d4, 0xa00ae278, 0xbdbdf21c }; mz_uint32 crcu32 = (mz_uint32)crc; if (!ptr) return MZ_CRC32_INIT; crcu32 = ~crcu32; while (buf_len--) { mz_uint8 b = *ptr++; crcu32 = (crcu32 >> 4) ^ s_crc32[(crcu32 & 0xF) ^ (b & 0xF)]; crcu32 = (crcu32 >> 4) ^ s_crc32[(crcu32 & 0xF) ^ (b >> 4)]; } return ~crcu32; } #elif defined(USE_EXTERNAL_MZCRC) /* If USE_EXTERNAL_CRC is defined, an external module will export the * mz_crc32() symbol for us to use, e.g. an SSE-accelerated version. * Depending on the impl, it may be necessary to ~ the input/output crc values. */ mz_ulong mz_crc32(mz_ulong crc, const mz_uint8 *ptr, size_t buf_len); #else /* Faster, but larger CPU cache footprint. */ mz_ulong mz_crc32(mz_ulong crc, const mz_uint8 *ptr, size_t buf_len) { static const mz_uint32 s_crc_table[256] = { 0x00000000, 0x77073096, 0xEE0E612C, 0x990951BA, 0x076DC419, 0x706AF48F, 0xE963A535, 0x9E6495A3, 0x0EDB8832, 0x79DCB8A4, 0xE0D5E91E, 0x97D2D988, 0x09B64C2B, 0x7EB17CBD, 0xE7B82D07, 0x90BF1D91, 0x1DB71064, 0x6AB020F2, 0xF3B97148, 0x84BE41DE, 0x1ADAD47D, 0x6DDDE4EB, 0xF4D4B551, 0x83D385C7, 0x136C9856, 0x646BA8C0, 0xFD62F97A, 0x8A65C9EC, 0x14015C4F, 0x63066CD9, 0xFA0F3D63, 0x8D080DF5, 0x3B6E20C8, 0x4C69105E, 0xD56041E4, 0xA2677172, 0x3C03E4D1, 0x4B04D447, 0xD20D85FD, 0xA50AB56B, 0x35B5A8FA, 0x42B2986C, 0xDBBBC9D6, 0xACBCF940, 0x32D86CE3, 0x45DF5C75, 0xDCD60DCF, 0xABD13D59, 0x26D930AC, 0x51DE003A, 0xC8D75180, 0xBFD06116, 0x21B4F4B5, 0x56B3C423, 0xCFBA9599, 0xB8BDA50F, 0x2802B89E, 0x5F058808, 0xC60CD9B2, 0xB10BE924, 0x2F6F7C87, 0x58684C11, 0xC1611DAB, 0xB6662D3D, 0x76DC4190, 0x01DB7106, 0x98D220BC, 0xEFD5102A, 0x71B18589, 0x06B6B51F, 0x9FBFE4A5, 0xE8B8D433, 0x7807C9A2, 0x0F00F934, 0x9609A88E, 0xE10E9818, 0x7F6A0DBB, 0x086D3D2D, 0x91646C97, 0xE6635C01, 0x6B6B51F4, 0x1C6C6162, 0x856530D8, 0xF262004E, 0x6C0695ED, 0x1B01A57B, 0x8208F4C1, 0xF50FC457, 0x65B0D9C6, 0x12B7E950, 0x8BBEB8EA, 0xFCB9887C, 0x62DD1DDF, 0x15DA2D49, 0x8CD37CF3, 0xFBD44C65, 0x4DB26158, 0x3AB551CE, 0xA3BC0074, 0xD4BB30E2, 0x4ADFA541, 0x3DD895D7, 0xA4D1C46D, 0xD3D6F4FB, 0x4369E96A, 0x346ED9FC, 0xAD678846, 0xDA60B8D0, 0x44042D73, 0x33031DE5, 0xAA0A4C5F, 0xDD0D7CC9, 0x5005713C, 0x270241AA, 0xBE0B1010, 0xC90C2086, 0x5768B525, 0x206F85B3, 0xB966D409, 0xCE61E49F, 0x5EDEF90E, 0x29D9C998, 0xB0D09822, 0xC7D7A8B4, 0x59B33D17, 0x2EB40D81, 0xB7BD5C3B, 0xC0BA6CAD, 0xEDB88320, 0x9ABFB3B6, 0x03B6E20C, 0x74B1D29A, 0xEAD54739, 0x9DD277AF, 0x04DB2615, 0x73DC1683, 0xE3630B12, 0x94643B84, 0x0D6D6A3E, 0x7A6A5AA8, 0xE40ECF0B, 0x9309FF9D, 0x0A00AE27, 0x7D079EB1, 0xF00F9344, 0x8708A3D2, 0x1E01F268, 0x6906C2FE, 0xF762575D, 0x806567CB, 0x196C3671, 0x6E6B06E7, 0xFED41B76, 0x89D32BE0, 0x10DA7A5A, 0x67DD4ACC, 0xF9B9DF6F, 0x8EBEEFF9, 0x17B7BE43, 0x60B08ED5, 0xD6D6A3E8, 0xA1D1937E, 0x38D8C2C4, 0x4FDFF252, 0xD1BB67F1, 0xA6BC5767, 0x3FB506DD, 0x48B2364B, 0xD80D2BDA, 0xAF0A1B4C, 0x36034AF6, 0x41047A60, 0xDF60EFC3, 0xA867DF55, 0x316E8EEF, 0x4669BE79, 0xCB61B38C, 0xBC66831A, 0x256FD2A0, 0x5268E236, 0xCC0C7795, 0xBB0B4703, 0x220216B9, 0x5505262F, 0xC5BA3BBE, 0xB2BD0B28, 0x2BB45A92, 0x5CB36A04, 0xC2D7FFA7, 0xB5D0CF31, 0x2CD99E8B, 0x5BDEAE1D, 0x9B64C2B0, 0xEC63F226, 0x756AA39C, 0x026D930A, 0x9C0906A9, 0xEB0E363F, 0x72076785, 0x05005713, 0x95BF4A82, 0xE2B87A14, 0x7BB12BAE, 0x0CB61B38, 0x92D28E9B, 0xE5D5BE0D, 0x7CDCEFB7, 0x0BDBDF21, 0x86D3D2D4, 0xF1D4E242, 0x68DDB3F8, 0x1FDA836E, 0x81BE16CD, 0xF6B9265B, 0x6FB077E1, 0x18B74777, 0x88085AE6, 0xFF0F6A70, 0x66063BCA, 0x11010B5C, 0x8F659EFF, 0xF862AE69, 0x616BFFD3, 0x166CCF45, 0xA00AE278, 0xD70DD2EE, 0x4E048354, 0x3903B3C2, 0xA7672661, 0xD06016F7, 0x4969474D, 0x3E6E77DB, 0xAED16A4A, 0xD9D65ADC, 0x40DF0B66, 0x37D83BF0, 0xA9BCAE53, 0xDEBB9EC5, 0x47B2CF7F, 0x30B5FFE9, 0xBDBDF21C, 0xCABAC28A, 0x53B39330, 0x24B4A3A6, 0xBAD03605, 0xCDD70693, 0x54DE5729, 0x23D967BF, 0xB3667A2E, 0xC4614AB8, 0x5D681B02, 0x2A6F2B94, 0xB40BBE37, 0xC30C8EA1, 0x5A05DF1B, 0x2D02EF8D }; mz_uint32 crc32 = (mz_uint32)crc ^ 0xFFFFFFFF; const mz_uint8 *pByte_buf = (const mz_uint8 *)ptr; while (buf_len >= 4) { crc32 = (crc32 >> 8) ^ s_crc_table[(crc32 ^ pByte_buf[0]) & 0xFF]; crc32 = (crc32 >> 8) ^ s_crc_table[(crc32 ^ pByte_buf[1]) & 0xFF]; crc32 = (crc32 >> 8) ^ s_crc_table[(crc32 ^ pByte_buf[2]) & 0xFF]; crc32 = (crc32 >> 8) ^ s_crc_table[(crc32 ^ pByte_buf[3]) & 0xFF]; pByte_buf += 4; buf_len -= 4; } while (buf_len) { crc32 = (crc32 >> 8) ^ s_crc_table[(crc32 ^ pByte_buf[0]) & 0xFF]; ++pByte_buf; --buf_len; } return ~crc32; } #endif void mz_free(void *p) { MZ_FREE(p); } MINIZ_EXPORT void *miniz_def_alloc_func(void *opaque, size_t items, size_t size) { (void)opaque, (void)items, (void)size; return MZ_MALLOC(items * size); } MINIZ_EXPORT void miniz_def_free_func(void *opaque, void *address) { (void)opaque, (void)address; MZ_FREE(address); } MINIZ_EXPORT void *miniz_def_realloc_func(void *opaque, void *address, size_t items, size_t size) { (void)opaque, (void)address, (void)items, (void)size; return MZ_REALLOC(address, items * size); } const char *mz_version(void) { return MZ_VERSION; } #ifndef MINIZ_NO_ZLIB_APIS int mz_deflateInit(mz_streamp pStream, int level) { return mz_deflateInit2(pStream, level, MZ_DEFLATED, MZ_DEFAULT_WINDOW_BITS, 9, MZ_DEFAULT_STRATEGY); } int mz_deflateInit2(mz_streamp pStream, int level, int method, int window_bits, int mem_level, int strategy) { tdefl_compressor *pComp; mz_uint comp_flags = TDEFL_COMPUTE_ADLER32 | tdefl_create_comp_flags_from_zip_params(level, window_bits, strategy); if (!pStream) return MZ_STREAM_ERROR; if ((method != MZ_DEFLATED) || ((mem_level < 1) || (mem_level > 9)) || ((window_bits != MZ_DEFAULT_WINDOW_BITS) && (-window_bits != MZ_DEFAULT_WINDOW_BITS))) return MZ_PARAM_ERROR; pStream->data_type = 0; pStream->adler = MZ_ADLER32_INIT; pStream->msg = NULL; pStream->reserved = 0; pStream->total_in = 0; pStream->total_out = 0; if (!pStream->zalloc) pStream->zalloc = miniz_def_alloc_func; if (!pStream->zfree) pStream->zfree = miniz_def_free_func; pComp = (tdefl_compressor *)pStream->zalloc(pStream->opaque, 1, sizeof(tdefl_compressor)); if (!pComp) return MZ_MEM_ERROR; pStream->state = (struct mz_internal_state *)pComp; if (tdefl_init(pComp, NULL, NULL, comp_flags) != TDEFL_STATUS_OKAY) { mz_deflateEnd(pStream); return MZ_PARAM_ERROR; } return MZ_OK; } int mz_deflateReset(mz_streamp pStream) { if ((!pStream) || (!pStream->state) || (!pStream->zalloc) || (!pStream->zfree)) return MZ_STREAM_ERROR; pStream->total_in = pStream->total_out = 0; tdefl_init((tdefl_compressor *)pStream->state, NULL, NULL, ((tdefl_compressor *)pStream->state)->m_flags); return MZ_OK; } int mz_deflate(mz_streamp pStream, int flush) { size_t in_bytes, out_bytes; mz_ulong orig_total_in, orig_total_out; int mz_status = MZ_OK; if ((!pStream) || (!pStream->state) || (flush < 0) || (flush > MZ_FINISH) || (!pStream->next_out)) return MZ_STREAM_ERROR; if (!pStream->avail_out) return MZ_BUF_ERROR; if (flush == MZ_PARTIAL_FLUSH) flush = MZ_SYNC_FLUSH; if (((tdefl_compressor *)pStream->state)->m_prev_return_status == TDEFL_STATUS_DONE) return (flush == MZ_FINISH) ? MZ_STREAM_END : MZ_BUF_ERROR; orig_total_in = pStream->total_in; orig_total_out = pStream->total_out; for (;;) { tdefl_status defl_status; in_bytes = pStream->avail_in; out_bytes = pStream->avail_out; defl_status = tdefl_compress((tdefl_compressor *)pStream->state, pStream->next_in, &in_bytes, pStream->next_out, &out_bytes, (tdefl_flush)flush); pStream->next_in += (mz_uint)in_bytes; pStream->avail_in -= (mz_uint)in_bytes; pStream->total_in += (mz_uint)in_bytes; pStream->adler = tdefl_get_adler32((tdefl_compressor *)pStream->state); pStream->next_out += (mz_uint)out_bytes; pStream->avail_out -= (mz_uint)out_bytes; pStream->total_out += (mz_uint)out_bytes; if (defl_status < 0) { mz_status = MZ_STREAM_ERROR; break; } else if (defl_status == TDEFL_STATUS_DONE) { mz_status = MZ_STREAM_END; break; } else if (!pStream->avail_out) break; else if ((!pStream->avail_in) && (flush != MZ_FINISH)) { if ((flush) || (pStream->total_in != orig_total_in) || (pStream->total_out != orig_total_out)) break; return MZ_BUF_ERROR; /* Can't make forward progress without some input. */ } } return mz_status; } int mz_deflateEnd(mz_streamp pStream) { if (!pStream) return MZ_STREAM_ERROR; if (pStream->state) { pStream->zfree(pStream->opaque, pStream->state); pStream->state = NULL; } return MZ_OK; } mz_ulong mz_deflateBound(mz_streamp pStream, mz_ulong source_len) { (void)pStream; /* This is really over conservative. (And lame, but it's actually pretty tricky to compute a true upper bound given the way tdefl's blocking works.) */ return MZ_MAX(128 + (source_len * 110) / 100, 128 + source_len + ((source_len / (31 * 1024)) + 1) * 5); } int mz_compress2(unsigned char *pDest, mz_ulong *pDest_len, const unsigned char *pSource, mz_ulong source_len, int level) { int status; mz_stream stream; memset(&stream, 0, sizeof(stream)); /* In case mz_ulong is 64-bits (argh I hate longs). */ if ((source_len | *pDest_len) > 0xFFFFFFFFU) return MZ_PARAM_ERROR; stream.next_in = pSource; stream.avail_in = (mz_uint32)source_len; stream.next_out = pDest; stream.avail_out = (mz_uint32)*pDest_len; status = mz_deflateInit(&stream, level); if (status != MZ_OK) return status; status = mz_deflate(&stream, MZ_FINISH); if (status != MZ_STREAM_END) { mz_deflateEnd(&stream); return (status == MZ_OK) ? MZ_BUF_ERROR : status; } *pDest_len = stream.total_out; return mz_deflateEnd(&stream); } int mz_compress(unsigned char *pDest, mz_ulong *pDest_len, const unsigned char *pSource, mz_ulong source_len) { return mz_compress2(pDest, pDest_len, pSource, source_len, MZ_DEFAULT_COMPRESSION); } mz_ulong mz_compressBound(mz_ulong source_len) { return mz_deflateBound(NULL, source_len); } typedef struct { tinfl_decompressor m_decomp; mz_uint m_dict_ofs, m_dict_avail, m_first_call, m_has_flushed; int m_window_bits; mz_uint8 m_dict[TINFL_LZ_DICT_SIZE]; tinfl_status m_last_status; } inflate_state; int mz_inflateInit2(mz_streamp pStream, int window_bits) { inflate_state *pDecomp; if (!pStream) return MZ_STREAM_ERROR; if ((window_bits != MZ_DEFAULT_WINDOW_BITS) && (-window_bits != MZ_DEFAULT_WINDOW_BITS)) return MZ_PARAM_ERROR; pStream->data_type = 0; pStream->adler = 0; pStream->msg = NULL; pStream->total_in = 0; pStream->total_out = 0; pStream->reserved = 0; if (!pStream->zalloc) pStream->zalloc = miniz_def_alloc_func; if (!pStream->zfree) pStream->zfree = miniz_def_free_func; pDecomp = (inflate_state *)pStream->zalloc(pStream->opaque, 1, sizeof(inflate_state)); if (!pDecomp) return MZ_MEM_ERROR; pStream->state = (struct mz_internal_state *)pDecomp; tinfl_init(&pDecomp->m_decomp); pDecomp->m_dict_ofs = 0; pDecomp->m_dict_avail = 0; pDecomp->m_last_status = TINFL_STATUS_NEEDS_MORE_INPUT; pDecomp->m_first_call = 1; pDecomp->m_has_flushed = 0; pDecomp->m_window_bits = window_bits; return MZ_OK; } int mz_inflateInit(mz_streamp pStream) { return mz_inflateInit2(pStream, MZ_DEFAULT_WINDOW_BITS); } int mz_inflateReset(mz_streamp pStream) { inflate_state *pDecomp; if (!pStream) return MZ_STREAM_ERROR; pStream->data_type = 0; pStream->adler = 0; pStream->msg = NULL; pStream->total_in = 0; pStream->total_out = 0; pStream->reserved = 0; pDecomp = (inflate_state *)pStream->state; tinfl_init(&pDecomp->m_decomp); pDecomp->m_dict_ofs = 0; pDecomp->m_dict_avail = 0; pDecomp->m_last_status = TINFL_STATUS_NEEDS_MORE_INPUT; pDecomp->m_first_call = 1; pDecomp->m_has_flushed = 0; /* pDecomp->m_window_bits = window_bits */; return MZ_OK; } int mz_inflate(mz_streamp pStream, int flush) { inflate_state *pState; mz_uint n, first_call, decomp_flags = TINFL_FLAG_COMPUTE_ADLER32; size_t in_bytes, out_bytes, orig_avail_in; tinfl_status status; if ((!pStream) || (!pStream->state)) return MZ_STREAM_ERROR; if (flush == MZ_PARTIAL_FLUSH) flush = MZ_SYNC_FLUSH; if ((flush) && (flush != MZ_SYNC_FLUSH) && (flush != MZ_FINISH)) return MZ_STREAM_ERROR; pState = (inflate_state *)pStream->state; if (pState->m_window_bits > 0) decomp_flags |= TINFL_FLAG_PARSE_ZLIB_HEADER; orig_avail_in = pStream->avail_in; first_call = pState->m_first_call; pState->m_first_call = 0; if (pState->m_last_status < 0) return MZ_DATA_ERROR; if (pState->m_has_flushed && (flush != MZ_FINISH)) return MZ_STREAM_ERROR; pState->m_has_flushed |= (flush == MZ_FINISH); if ((flush == MZ_FINISH) && (first_call)) { /* MZ_FINISH on the first call implies that the input and output buffers are large enough to hold the entire compressed/decompressed file. */ decomp_flags |= TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF; in_bytes = pStream->avail_in; out_bytes = pStream->avail_out; status = tinfl_decompress(&pState->m_decomp, pStream->next_in, &in_bytes, pStream->next_out, pStream->next_out, &out_bytes, decomp_flags); pState->m_last_status = status; pStream->next_in += (mz_uint)in_bytes; pStream->avail_in -= (mz_uint)in_bytes; pStream->total_in += (mz_uint)in_bytes; pStream->adler = tinfl_get_adler32(&pState->m_decomp); pStream->next_out += (mz_uint)out_bytes; pStream->avail_out -= (mz_uint)out_bytes; pStream->total_out += (mz_uint)out_bytes; if (status < 0) return MZ_DATA_ERROR; else if (status != TINFL_STATUS_DONE) { pState->m_last_status = TINFL_STATUS_FAILED; return MZ_BUF_ERROR; } return MZ_STREAM_END; } /* flush != MZ_FINISH then we must assume there's more input. */ if (flush != MZ_FINISH) decomp_flags |= TINFL_FLAG_HAS_MORE_INPUT; if (pState->m_dict_avail) { n = MZ_MIN(pState->m_dict_avail, pStream->avail_out); memcpy(pStream->next_out, pState->m_dict + pState->m_dict_ofs, n); pStream->next_out += n; pStream->avail_out -= n; pStream->total_out += n; pState->m_dict_avail -= n; pState->m_dict_ofs = (pState->m_dict_ofs + n) & (TINFL_LZ_DICT_SIZE - 1); return ((pState->m_last_status == TINFL_STATUS_DONE) && (!pState->m_dict_avail)) ? MZ_STREAM_END : MZ_OK; } for (;;) { in_bytes = pStream->avail_in; out_bytes = TINFL_LZ_DICT_SIZE - pState->m_dict_ofs; status = tinfl_decompress(&pState->m_decomp, pStream->next_in, &in_bytes, pState->m_dict, pState->m_dict + pState->m_dict_ofs, &out_bytes, decomp_flags); pState->m_last_status = status; pStream->next_in += (mz_uint)in_bytes; pStream->avail_in -= (mz_uint)in_bytes; pStream->total_in += (mz_uint)in_bytes; pStream->adler = tinfl_get_adler32(&pState->m_decomp); pState->m_dict_avail = (mz_uint)out_bytes; n = MZ_MIN(pState->m_dict_avail, pStream->avail_out); memcpy(pStream->next_out, pState->m_dict + pState->m_dict_ofs, n); pStream->next_out += n; pStream->avail_out -= n; pStream->total_out += n; pState->m_dict_avail -= n; pState->m_dict_ofs = (pState->m_dict_ofs + n) & (TINFL_LZ_DICT_SIZE - 1); if (status < 0) return MZ_DATA_ERROR; /* Stream is corrupted (there could be some uncompressed data left in the output dictionary - oh well). */ else if ((status == TINFL_STATUS_NEEDS_MORE_INPUT) && (!orig_avail_in)) return MZ_BUF_ERROR; /* Signal caller that we can't make forward progress without supplying more input or by setting flush to MZ_FINISH. */ else if (flush == MZ_FINISH) { /* The output buffer MUST be large to hold the remaining uncompressed data when flush==MZ_FINISH. */ if (status == TINFL_STATUS_DONE) return pState->m_dict_avail ? MZ_BUF_ERROR : MZ_STREAM_END; /* status here must be TINFL_STATUS_HAS_MORE_OUTPUT, which means there's at least 1 more byte on the way. If there's no more room left in the output buffer then something is wrong. */ else if (!pStream->avail_out) return MZ_BUF_ERROR; } else if ((status == TINFL_STATUS_DONE) || (!pStream->avail_in) || (!pStream->avail_out) || (pState->m_dict_avail)) break; } return ((status == TINFL_STATUS_DONE) && (!pState->m_dict_avail)) ? MZ_STREAM_END : MZ_OK; } int mz_inflateEnd(mz_streamp pStream) { if (!pStream) return MZ_STREAM_ERROR; if (pStream->state) { pStream->zfree(pStream->opaque, pStream->state); pStream->state = NULL; } return MZ_OK; } int mz_uncompress2(unsigned char *pDest, mz_ulong *pDest_len, const unsigned char *pSource, mz_ulong *pSource_len) { mz_stream stream; int status; memset(&stream, 0, sizeof(stream)); /* In case mz_ulong is 64-bits (argh I hate longs). */ if ((*pSource_len | *pDest_len) > 0xFFFFFFFFU) return MZ_PARAM_ERROR; stream.next_in = pSource; stream.avail_in = (mz_uint32)*pSource_len; stream.next_out = pDest; stream.avail_out = (mz_uint32)*pDest_len; status = mz_inflateInit(&stream); if (status != MZ_OK) return status; status = mz_inflate(&stream, MZ_FINISH); *pSource_len = *pSource_len - stream.avail_in; if (status != MZ_STREAM_END) { mz_inflateEnd(&stream); return ((status == MZ_BUF_ERROR) && (!stream.avail_in)) ? MZ_DATA_ERROR : status; } *pDest_len = stream.total_out; return mz_inflateEnd(&stream); } int mz_uncompress(unsigned char *pDest, mz_ulong *pDest_len, const unsigned char *pSource, mz_ulong source_len) { return mz_uncompress2(pDest, pDest_len, pSource, &source_len); } const char *mz_error(int err) { static struct { int m_err; const char *m_pDesc; } s_error_descs[] = { { MZ_OK, "" }, { MZ_STREAM_END, "stream end" }, { MZ_NEED_DICT, "need dictionary" }, { MZ_ERRNO, "file error" }, { MZ_STREAM_ERROR, "stream error" }, { MZ_DATA_ERROR, "data error" }, { MZ_MEM_ERROR, "out of memory" }, { MZ_BUF_ERROR, "buf error" }, { MZ_VERSION_ERROR, "version error" }, { MZ_PARAM_ERROR, "parameter error" } }; mz_uint i; for (i = 0; i < sizeof(s_error_descs) / sizeof(s_error_descs[0]); ++i) if (s_error_descs[i].m_err == err) return s_error_descs[i].m_pDesc; return NULL; } #endif /*MINIZ_NO_ZLIB_APIS */ #ifdef __cplusplus } #endif /* This is free and unencumbered software released into the public domain. Anyone is free to copy, modify, publish, use, compile, sell, or distribute this software, either in source code form or as a compiled binary, for any purpose, commercial or non-commercial, and by any means. In jurisdictions that recognize copyright laws, the author or authors of this software dedicate any and all copyright interest in the software to the public domain. We make this dedication for the benefit of the public at large and to the detriment of our heirs and successors. We intend this dedication to be an overt act of relinquishment in perpetuity of all present and future rights to this software under copyright law. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. For more information, please refer to */ /************************************************************************** * * Copyright 2013-2014 RAD Game Tools and Valve Software * Copyright 2010-2014 Rich Geldreich and Tenacious Software LLC * All Rights Reserved. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal * in the Software without restriction, including without limitation the rights * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN * THE SOFTWARE. * **************************************************************************/ #ifdef __cplusplus extern "C" { #endif /* ------------------- Low-level Compression (independent from all decompression API's) */ /* Purposely making these tables static for faster init and thread safety. */ static const mz_uint16 s_tdefl_len_sym[256] = { 257, 258, 259, 260, 261, 262, 263, 264, 265, 265, 266, 266, 267, 267, 268, 268, 269, 269, 269, 269, 270, 270, 270, 270, 271, 271, 271, 271, 272, 272, 272, 272, 273, 273, 273, 273, 273, 273, 273, 273, 274, 274, 274, 274, 274, 274, 274, 274, 275, 275, 275, 275, 275, 275, 275, 275, 276, 276, 276, 276, 276, 276, 276, 276, 277, 277, 277, 277, 277, 277, 277, 277, 277, 277, 277, 277, 277, 277, 277, 277, 278, 278, 278, 278, 278, 278, 278, 278, 278, 278, 278, 278, 278, 278, 278, 278, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 285 }; static const mz_uint8 s_tdefl_len_extra[256] = { 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 0 }; static const mz_uint8 s_tdefl_small_dist_sym[512] = { 0, 1, 2, 3, 4, 4, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17 }; static const mz_uint8 s_tdefl_small_dist_extra[512] = { 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7 }; static const mz_uint8 s_tdefl_large_dist_sym[128] = { 0, 0, 18, 19, 20, 20, 21, 21, 22, 22, 22, 22, 23, 23, 23, 23, 24, 24, 24, 24, 24, 24, 24, 24, 25, 25, 25, 25, 25, 25, 25, 25, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29 }; static const mz_uint8 s_tdefl_large_dist_extra[128] = { 0, 0, 8, 8, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13 }; /* Radix sorts tdefl_sym_freq[] array by 16-bit key m_key. Returns ptr to sorted values. */ typedef struct { mz_uint16 m_key, m_sym_index; } tdefl_sym_freq; static tdefl_sym_freq *tdefl_radix_sort_syms(mz_uint num_syms, tdefl_sym_freq *pSyms0, tdefl_sym_freq *pSyms1) { mz_uint32 total_passes = 2, pass_shift, pass, i, hist[256 * 2]; tdefl_sym_freq *pCur_syms = pSyms0, *pNew_syms = pSyms1; MZ_CLEAR_OBJ(hist); for (i = 0; i < num_syms; i++) { mz_uint freq = pSyms0[i].m_key; hist[freq & 0xFF]++; hist[256 + ((freq >> 8) & 0xFF)]++; } while ((total_passes > 1) && (num_syms == hist[(total_passes - 1) * 256])) total_passes--; for (pass_shift = 0, pass = 0; pass < total_passes; pass++, pass_shift += 8) { const mz_uint32 *pHist = &hist[pass << 8]; mz_uint offsets[256], cur_ofs = 0; for (i = 0; i < 256; i++) { offsets[i] = cur_ofs; cur_ofs += pHist[i]; } for (i = 0; i < num_syms; i++) pNew_syms[offsets[(pCur_syms[i].m_key >> pass_shift) & 0xFF]++] = pCur_syms[i]; { tdefl_sym_freq *t = pCur_syms; pCur_syms = pNew_syms; pNew_syms = t; } } return pCur_syms; } /* tdefl_calculate_minimum_redundancy() originally written by: Alistair Moffat, alistair@cs.mu.oz.au, Jyrki Katajainen, jyrki@diku.dk, November 1996. */ static void tdefl_calculate_minimum_redundancy(tdefl_sym_freq *A, int n) { int root, leaf, next, avbl, used, dpth; if (n == 0) return; else if (n == 1) { A[0].m_key = 1; return; } A[0].m_key += A[1].m_key; root = 0; leaf = 2; for (next = 1; next < n - 1; next++) { if (leaf >= n || A[root].m_key < A[leaf].m_key) { A[next].m_key = A[root].m_key; A[root++].m_key = (mz_uint16)next; } else A[next].m_key = A[leaf++].m_key; if (leaf >= n || (root < next && A[root].m_key < A[leaf].m_key)) { A[next].m_key = (mz_uint16)(A[next].m_key + A[root].m_key); A[root++].m_key = (mz_uint16)next; } else A[next].m_key = (mz_uint16)(A[next].m_key + A[leaf++].m_key); } A[n - 2].m_key = 0; for (next = n - 3; next >= 0; next--) A[next].m_key = A[A[next].m_key].m_key + 1; avbl = 1; used = dpth = 0; root = n - 2; next = n - 1; while (avbl > 0) { while (root >= 0 && (int)A[root].m_key == dpth) { used++; root--; } while (avbl > used) { A[next--].m_key = (mz_uint16)(dpth); avbl--; } avbl = 2 * used; dpth++; used = 0; } } /* Limits canonical Huffman code table's max code size. */ enum { TDEFL_MAX_SUPPORTED_HUFF_CODESIZE = 32 }; static void tdefl_huffman_enforce_max_code_size(int *pNum_codes, int code_list_len, int max_code_size) { int i; mz_uint32 total = 0; if (code_list_len <= 1) return; for (i = max_code_size + 1; i <= TDEFL_MAX_SUPPORTED_HUFF_CODESIZE; i++) pNum_codes[max_code_size] += pNum_codes[i]; for (i = max_code_size; i > 0; i--) total += (((mz_uint32)pNum_codes[i]) << (max_code_size - i)); while (total != (1UL << max_code_size)) { pNum_codes[max_code_size]--; for (i = max_code_size - 1; i > 0; i--) if (pNum_codes[i]) { pNum_codes[i]--; pNum_codes[i + 1] += 2; break; } total--; } } static void tdefl_optimize_huffman_table(tdefl_compressor *d, int table_num, int table_len, int code_size_limit, int static_table) { int i, j, l, num_codes[1 + TDEFL_MAX_SUPPORTED_HUFF_CODESIZE]; mz_uint next_code[TDEFL_MAX_SUPPORTED_HUFF_CODESIZE + 1]; MZ_CLEAR_OBJ(num_codes); if (static_table) { for (i = 0; i < table_len; i++) num_codes[d->m_huff_code_sizes[table_num][i]]++; } else { tdefl_sym_freq syms0[TDEFL_MAX_HUFF_SYMBOLS], syms1[TDEFL_MAX_HUFF_SYMBOLS], *pSyms; int num_used_syms = 0; const mz_uint16 *pSym_count = &d->m_huff_count[table_num][0]; for (i = 0; i < table_len; i++) if (pSym_count[i]) { syms0[num_used_syms].m_key = (mz_uint16)pSym_count[i]; syms0[num_used_syms++].m_sym_index = (mz_uint16)i; } pSyms = tdefl_radix_sort_syms(num_used_syms, syms0, syms1); tdefl_calculate_minimum_redundancy(pSyms, num_used_syms); for (i = 0; i < num_used_syms; i++) num_codes[pSyms[i].m_key]++; tdefl_huffman_enforce_max_code_size(num_codes, num_used_syms, code_size_limit); MZ_CLEAR_OBJ(d->m_huff_code_sizes[table_num]); MZ_CLEAR_OBJ(d->m_huff_codes[table_num]); for (i = 1, j = num_used_syms; i <= code_size_limit; i++) for (l = num_codes[i]; l > 0; l--) d->m_huff_code_sizes[table_num][pSyms[--j].m_sym_index] = (mz_uint8)(i); } next_code[1] = 0; for (j = 0, i = 2; i <= code_size_limit; i++) next_code[i] = j = ((j + num_codes[i - 1]) << 1); for (i = 0; i < table_len; i++) { mz_uint rev_code = 0, code, code_size; if ((code_size = d->m_huff_code_sizes[table_num][i]) == 0) continue; code = next_code[code_size]++; for (l = code_size; l > 0; l--, code >>= 1) rev_code = (rev_code << 1) | (code & 1); d->m_huff_codes[table_num][i] = (mz_uint16)rev_code; } } #define TDEFL_PUT_BITS(b, l) \ do \ { \ mz_uint bits = b; \ mz_uint len = l; \ MZ_ASSERT(bits <= ((1U << len) - 1U)); \ d->m_bit_buffer |= (bits << d->m_bits_in); \ d->m_bits_in += len; \ while (d->m_bits_in >= 8) \ { \ if (d->m_pOutput_buf < d->m_pOutput_buf_end) \ *d->m_pOutput_buf++ = (mz_uint8)(d->m_bit_buffer); \ d->m_bit_buffer >>= 8; \ d->m_bits_in -= 8; \ } \ } \ MZ_MACRO_END #define TDEFL_RLE_PREV_CODE_SIZE() \ { \ if (rle_repeat_count) \ { \ if (rle_repeat_count < 3) \ { \ d->m_huff_count[2][prev_code_size] = (mz_uint16)(d->m_huff_count[2][prev_code_size] + rle_repeat_count); \ while (rle_repeat_count--) \ packed_code_sizes[num_packed_code_sizes++] = prev_code_size; \ } \ else \ { \ d->m_huff_count[2][16] = (mz_uint16)(d->m_huff_count[2][16] + 1); \ packed_code_sizes[num_packed_code_sizes++] = 16; \ packed_code_sizes[num_packed_code_sizes++] = (mz_uint8)(rle_repeat_count - 3); \ } \ rle_repeat_count = 0; \ } \ } #define TDEFL_RLE_ZERO_CODE_SIZE() \ { \ if (rle_z_count) \ { \ if (rle_z_count < 3) \ { \ d->m_huff_count[2][0] = (mz_uint16)(d->m_huff_count[2][0] + rle_z_count); \ while (rle_z_count--) \ packed_code_sizes[num_packed_code_sizes++] = 0; \ } \ else if (rle_z_count <= 10) \ { \ d->m_huff_count[2][17] = (mz_uint16)(d->m_huff_count[2][17] + 1); \ packed_code_sizes[num_packed_code_sizes++] = 17; \ packed_code_sizes[num_packed_code_sizes++] = (mz_uint8)(rle_z_count - 3); \ } \ else \ { \ d->m_huff_count[2][18] = (mz_uint16)(d->m_huff_count[2][18] + 1); \ packed_code_sizes[num_packed_code_sizes++] = 18; \ packed_code_sizes[num_packed_code_sizes++] = (mz_uint8)(rle_z_count - 11); \ } \ rle_z_count = 0; \ } \ } static mz_uint8 s_tdefl_packed_code_size_syms_swizzle[] = { 16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15 }; static void tdefl_start_dynamic_block(tdefl_compressor *d) { int num_lit_codes, num_dist_codes, num_bit_lengths; mz_uint i, total_code_sizes_to_pack, num_packed_code_sizes, rle_z_count, rle_repeat_count, packed_code_sizes_index; mz_uint8 code_sizes_to_pack[TDEFL_MAX_HUFF_SYMBOLS_0 + TDEFL_MAX_HUFF_SYMBOLS_1], packed_code_sizes[TDEFL_MAX_HUFF_SYMBOLS_0 + TDEFL_MAX_HUFF_SYMBOLS_1], prev_code_size = 0xFF; d->m_huff_count[0][256] = 1; tdefl_optimize_huffman_table(d, 0, TDEFL_MAX_HUFF_SYMBOLS_0, 15, MZ_FALSE); tdefl_optimize_huffman_table(d, 1, TDEFL_MAX_HUFF_SYMBOLS_1, 15, MZ_FALSE); for (num_lit_codes = 286; num_lit_codes > 257; num_lit_codes--) if (d->m_huff_code_sizes[0][num_lit_codes - 1]) break; for (num_dist_codes = 30; num_dist_codes > 1; num_dist_codes--) if (d->m_huff_code_sizes[1][num_dist_codes - 1]) break; memcpy(code_sizes_to_pack, &d->m_huff_code_sizes[0][0], num_lit_codes); memcpy(code_sizes_to_pack + num_lit_codes, &d->m_huff_code_sizes[1][0], num_dist_codes); total_code_sizes_to_pack = num_lit_codes + num_dist_codes; num_packed_code_sizes = 0; rle_z_count = 0; rle_repeat_count = 0; memset(&d->m_huff_count[2][0], 0, sizeof(d->m_huff_count[2][0]) * TDEFL_MAX_HUFF_SYMBOLS_2); for (i = 0; i < total_code_sizes_to_pack; i++) { mz_uint8 code_size = code_sizes_to_pack[i]; if (!code_size) { TDEFL_RLE_PREV_CODE_SIZE(); if (++rle_z_count == 138) { TDEFL_RLE_ZERO_CODE_SIZE(); } } else { TDEFL_RLE_ZERO_CODE_SIZE(); if (code_size != prev_code_size) { TDEFL_RLE_PREV_CODE_SIZE(); d->m_huff_count[2][code_size] = (mz_uint16)(d->m_huff_count[2][code_size] + 1); packed_code_sizes[num_packed_code_sizes++] = code_size; } else if (++rle_repeat_count == 6) { TDEFL_RLE_PREV_CODE_SIZE(); } } prev_code_size = code_size; } if (rle_repeat_count) { TDEFL_RLE_PREV_CODE_SIZE(); } else { TDEFL_RLE_ZERO_CODE_SIZE(); } tdefl_optimize_huffman_table(d, 2, TDEFL_MAX_HUFF_SYMBOLS_2, 7, MZ_FALSE); TDEFL_PUT_BITS(2, 2); TDEFL_PUT_BITS(num_lit_codes - 257, 5); TDEFL_PUT_BITS(num_dist_codes - 1, 5); for (num_bit_lengths = 18; num_bit_lengths >= 0; num_bit_lengths--) if (d->m_huff_code_sizes[2][s_tdefl_packed_code_size_syms_swizzle[num_bit_lengths]]) break; num_bit_lengths = MZ_MAX(4, (num_bit_lengths + 1)); TDEFL_PUT_BITS(num_bit_lengths - 4, 4); for (i = 0; (int)i < num_bit_lengths; i++) TDEFL_PUT_BITS(d->m_huff_code_sizes[2][s_tdefl_packed_code_size_syms_swizzle[i]], 3); for (packed_code_sizes_index = 0; packed_code_sizes_index < num_packed_code_sizes;) { mz_uint code = packed_code_sizes[packed_code_sizes_index++]; MZ_ASSERT(code < TDEFL_MAX_HUFF_SYMBOLS_2); TDEFL_PUT_BITS(d->m_huff_codes[2][code], d->m_huff_code_sizes[2][code]); if (code >= 16) TDEFL_PUT_BITS(packed_code_sizes[packed_code_sizes_index++], "\02\03\07"[code - 16]); } } static void tdefl_start_static_block(tdefl_compressor *d) { mz_uint i; mz_uint8 *p = &d->m_huff_code_sizes[0][0]; for (i = 0; i <= 143; ++i) *p++ = 8; for (; i <= 255; ++i) *p++ = 9; for (; i <= 279; ++i) *p++ = 7; for (; i <= 287; ++i) *p++ = 8; memset(d->m_huff_code_sizes[1], 5, 32); tdefl_optimize_huffman_table(d, 0, 288, 15, MZ_TRUE); tdefl_optimize_huffman_table(d, 1, 32, 15, MZ_TRUE); TDEFL_PUT_BITS(1, 2); } static const mz_uint mz_bitmasks[17] = { 0x0000, 0x0001, 0x0003, 0x0007, 0x000F, 0x001F, 0x003F, 0x007F, 0x00FF, 0x01FF, 0x03FF, 0x07FF, 0x0FFF, 0x1FFF, 0x3FFF, 0x7FFF, 0xFFFF }; #if MINIZ_USE_UNALIGNED_LOADS_AND_STORES && MINIZ_LITTLE_ENDIAN && MINIZ_HAS_64BIT_REGISTERS static mz_bool tdefl_compress_lz_codes(tdefl_compressor *d) { mz_uint flags; mz_uint8 *pLZ_codes; mz_uint8 *pOutput_buf = d->m_pOutput_buf; mz_uint8 *pLZ_code_buf_end = d->m_pLZ_code_buf; mz_uint64 bit_buffer = d->m_bit_buffer; mz_uint bits_in = d->m_bits_in; #define TDEFL_PUT_BITS_FAST(b, l) \ { \ bit_buffer |= (((mz_uint64)(b)) << bits_in); \ bits_in += (l); \ } flags = 1; for (pLZ_codes = d->m_lz_code_buf; pLZ_codes < pLZ_code_buf_end; flags >>= 1) { if (flags == 1) flags = *pLZ_codes++ | 0x100; if (flags & 1) { mz_uint s0, s1, n0, n1, sym, num_extra_bits; mz_uint match_len = pLZ_codes[0], match_dist = *(const mz_uint16 *)(pLZ_codes + 1); pLZ_codes += 3; MZ_ASSERT(d->m_huff_code_sizes[0][s_tdefl_len_sym[match_len]]); TDEFL_PUT_BITS_FAST(d->m_huff_codes[0][s_tdefl_len_sym[match_len]], d->m_huff_code_sizes[0][s_tdefl_len_sym[match_len]]); TDEFL_PUT_BITS_FAST(match_len & mz_bitmasks[s_tdefl_len_extra[match_len]], s_tdefl_len_extra[match_len]); /* This sequence coaxes MSVC into using cmov's vs. jmp's. */ s0 = s_tdefl_small_dist_sym[match_dist & 511]; n0 = s_tdefl_small_dist_extra[match_dist & 511]; s1 = s_tdefl_large_dist_sym[match_dist >> 8]; n1 = s_tdefl_large_dist_extra[match_dist >> 8]; sym = (match_dist < 512) ? s0 : s1; num_extra_bits = (match_dist < 512) ? n0 : n1; MZ_ASSERT(d->m_huff_code_sizes[1][sym]); TDEFL_PUT_BITS_FAST(d->m_huff_codes[1][sym], d->m_huff_code_sizes[1][sym]); TDEFL_PUT_BITS_FAST(match_dist & mz_bitmasks[num_extra_bits], num_extra_bits); } else { mz_uint lit = *pLZ_codes++; MZ_ASSERT(d->m_huff_code_sizes[0][lit]); TDEFL_PUT_BITS_FAST(d->m_huff_codes[0][lit], d->m_huff_code_sizes[0][lit]); if (((flags & 2) == 0) && (pLZ_codes < pLZ_code_buf_end)) { flags >>= 1; lit = *pLZ_codes++; MZ_ASSERT(d->m_huff_code_sizes[0][lit]); TDEFL_PUT_BITS_FAST(d->m_huff_codes[0][lit], d->m_huff_code_sizes[0][lit]); if (((flags & 2) == 0) && (pLZ_codes < pLZ_code_buf_end)) { flags >>= 1; lit = *pLZ_codes++; MZ_ASSERT(d->m_huff_code_sizes[0][lit]); TDEFL_PUT_BITS_FAST(d->m_huff_codes[0][lit], d->m_huff_code_sizes[0][lit]); } } } if (pOutput_buf >= d->m_pOutput_buf_end) return MZ_FALSE; *(mz_uint64 *)pOutput_buf = bit_buffer; pOutput_buf += (bits_in >> 3); bit_buffer >>= (bits_in & ~7); bits_in &= 7; } #undef TDEFL_PUT_BITS_FAST d->m_pOutput_buf = pOutput_buf; d->m_bits_in = 0; d->m_bit_buffer = 0; while (bits_in) { mz_uint32 n = MZ_MIN(bits_in, 16); TDEFL_PUT_BITS((mz_uint)bit_buffer & mz_bitmasks[n], n); bit_buffer >>= n; bits_in -= n; } TDEFL_PUT_BITS(d->m_huff_codes[0][256], d->m_huff_code_sizes[0][256]); return (d->m_pOutput_buf < d->m_pOutput_buf_end); } #else static mz_bool tdefl_compress_lz_codes(tdefl_compressor *d) { mz_uint flags; mz_uint8 *pLZ_codes; flags = 1; for (pLZ_codes = d->m_lz_code_buf; pLZ_codes < d->m_pLZ_code_buf; flags >>= 1) { if (flags == 1) flags = *pLZ_codes++ | 0x100; if (flags & 1) { mz_uint sym, num_extra_bits; mz_uint match_len = pLZ_codes[0], match_dist = (pLZ_codes[1] | (pLZ_codes[2] << 8)); pLZ_codes += 3; MZ_ASSERT(d->m_huff_code_sizes[0][s_tdefl_len_sym[match_len]]); TDEFL_PUT_BITS(d->m_huff_codes[0][s_tdefl_len_sym[match_len]], d->m_huff_code_sizes[0][s_tdefl_len_sym[match_len]]); TDEFL_PUT_BITS(match_len & mz_bitmasks[s_tdefl_len_extra[match_len]], s_tdefl_len_extra[match_len]); if (match_dist < 512) { sym = s_tdefl_small_dist_sym[match_dist]; num_extra_bits = s_tdefl_small_dist_extra[match_dist]; } else { sym = s_tdefl_large_dist_sym[match_dist >> 8]; num_extra_bits = s_tdefl_large_dist_extra[match_dist >> 8]; } MZ_ASSERT(d->m_huff_code_sizes[1][sym]); TDEFL_PUT_BITS(d->m_huff_codes[1][sym], d->m_huff_code_sizes[1][sym]); TDEFL_PUT_BITS(match_dist & mz_bitmasks[num_extra_bits], num_extra_bits); } else { mz_uint lit = *pLZ_codes++; MZ_ASSERT(d->m_huff_code_sizes[0][lit]); TDEFL_PUT_BITS(d->m_huff_codes[0][lit], d->m_huff_code_sizes[0][lit]); } } TDEFL_PUT_BITS(d->m_huff_codes[0][256], d->m_huff_code_sizes[0][256]); return (d->m_pOutput_buf < d->m_pOutput_buf_end); } #endif /* MINIZ_USE_UNALIGNED_LOADS_AND_STORES && MINIZ_LITTLE_ENDIAN && MINIZ_HAS_64BIT_REGISTERS */ static mz_bool tdefl_compress_block(tdefl_compressor *d, mz_bool static_block) { if (static_block) tdefl_start_static_block(d); else tdefl_start_dynamic_block(d); return tdefl_compress_lz_codes(d); } static int tdefl_flush_block(tdefl_compressor *d, int flush) { mz_uint saved_bit_buf, saved_bits_in; mz_uint8 *pSaved_output_buf; mz_bool comp_block_succeeded = MZ_FALSE; int n, use_raw_block = ((d->m_flags & TDEFL_FORCE_ALL_RAW_BLOCKS) != 0) && (d->m_lookahead_pos - d->m_lz_code_buf_dict_pos) <= d->m_dict_size; mz_uint8 *pOutput_buf_start = ((d->m_pPut_buf_func == NULL) && ((*d->m_pOut_buf_size - d->m_out_buf_ofs) >= TDEFL_OUT_BUF_SIZE)) ? ((mz_uint8 *)d->m_pOut_buf + d->m_out_buf_ofs) : d->m_output_buf; d->m_pOutput_buf = pOutput_buf_start; d->m_pOutput_buf_end = d->m_pOutput_buf + TDEFL_OUT_BUF_SIZE - 16; MZ_ASSERT(!d->m_output_flush_remaining); d->m_output_flush_ofs = 0; d->m_output_flush_remaining = 0; *d->m_pLZ_flags = (mz_uint8)(*d->m_pLZ_flags >> d->m_num_flags_left); d->m_pLZ_code_buf -= (d->m_num_flags_left == 8); if ((d->m_flags & TDEFL_WRITE_ZLIB_HEADER) && (!d->m_block_index)) { TDEFL_PUT_BITS(0x78, 8); TDEFL_PUT_BITS(0x01, 8); } TDEFL_PUT_BITS(flush == TDEFL_FINISH, 1); pSaved_output_buf = d->m_pOutput_buf; saved_bit_buf = d->m_bit_buffer; saved_bits_in = d->m_bits_in; if (!use_raw_block) comp_block_succeeded = tdefl_compress_block(d, (d->m_flags & TDEFL_FORCE_ALL_STATIC_BLOCKS) || (d->m_total_lz_bytes < 48)); /* If the block gets expanded, forget the current contents of the output buffer and send a raw block instead. */ if (((use_raw_block) || ((d->m_total_lz_bytes) && ((d->m_pOutput_buf - pSaved_output_buf + 1U) >= d->m_total_lz_bytes))) && ((d->m_lookahead_pos - d->m_lz_code_buf_dict_pos) <= d->m_dict_size)) { mz_uint i; d->m_pOutput_buf = pSaved_output_buf; d->m_bit_buffer = saved_bit_buf, d->m_bits_in = saved_bits_in; TDEFL_PUT_BITS(0, 2); if (d->m_bits_in) { TDEFL_PUT_BITS(0, 8 - d->m_bits_in); } for (i = 2; i; --i, d->m_total_lz_bytes ^= 0xFFFF) { TDEFL_PUT_BITS(d->m_total_lz_bytes & 0xFFFF, 16); } for (i = 0; i < d->m_total_lz_bytes; ++i) { TDEFL_PUT_BITS(d->m_dict[(d->m_lz_code_buf_dict_pos + i) & TDEFL_LZ_DICT_SIZE_MASK], 8); } } /* Check for the extremely unlikely (if not impossible) case of the compressed block not fitting into the output buffer when using dynamic codes. */ else if (!comp_block_succeeded) { d->m_pOutput_buf = pSaved_output_buf; d->m_bit_buffer = saved_bit_buf, d->m_bits_in = saved_bits_in; tdefl_compress_block(d, MZ_TRUE); } if (flush) { if (flush == TDEFL_FINISH) { if (d->m_bits_in) { TDEFL_PUT_BITS(0, 8 - d->m_bits_in); } if (d->m_flags & TDEFL_WRITE_ZLIB_HEADER) { mz_uint i, a = d->m_adler32; for (i = 0; i < 4; i++) { TDEFL_PUT_BITS((a >> 24) & 0xFF, 8); a <<= 8; } } } else { mz_uint i, z = 0; TDEFL_PUT_BITS(0, 3); if (d->m_bits_in) { TDEFL_PUT_BITS(0, 8 - d->m_bits_in); } for (i = 2; i; --i, z ^= 0xFFFF) { TDEFL_PUT_BITS(z & 0xFFFF, 16); } } } MZ_ASSERT(d->m_pOutput_buf < d->m_pOutput_buf_end); memset(&d->m_huff_count[0][0], 0, sizeof(d->m_huff_count[0][0]) * TDEFL_MAX_HUFF_SYMBOLS_0); memset(&d->m_huff_count[1][0], 0, sizeof(d->m_huff_count[1][0]) * TDEFL_MAX_HUFF_SYMBOLS_1); d->m_pLZ_code_buf = d->m_lz_code_buf + 1; d->m_pLZ_flags = d->m_lz_code_buf; d->m_num_flags_left = 8; d->m_lz_code_buf_dict_pos += d->m_total_lz_bytes; d->m_total_lz_bytes = 0; d->m_block_index++; if ((n = (int)(d->m_pOutput_buf - pOutput_buf_start)) != 0) { if (d->m_pPut_buf_func) { *d->m_pIn_buf_size = d->m_pSrc - (const mz_uint8 *)d->m_pIn_buf; if (!(*d->m_pPut_buf_func)(d->m_output_buf, n, d->m_pPut_buf_user)) return (d->m_prev_return_status = TDEFL_STATUS_PUT_BUF_FAILED); } else if (pOutput_buf_start == d->m_output_buf) { int bytes_to_copy = (int)MZ_MIN((size_t)n, (size_t)(*d->m_pOut_buf_size - d->m_out_buf_ofs)); memcpy((mz_uint8 *)d->m_pOut_buf + d->m_out_buf_ofs, d->m_output_buf, bytes_to_copy); d->m_out_buf_ofs += bytes_to_copy; if ((n -= bytes_to_copy) != 0) { d->m_output_flush_ofs = bytes_to_copy; d->m_output_flush_remaining = n; } } else { d->m_out_buf_ofs += n; } } return d->m_output_flush_remaining; } #if MINIZ_USE_UNALIGNED_LOADS_AND_STORES #ifdef MINIZ_UNALIGNED_USE_MEMCPY static mz_uint16 TDEFL_READ_UNALIGNED_WORD(const mz_uint8* p) { mz_uint16 ret; memcpy(&ret, p, sizeof(mz_uint16)); return ret; } static mz_uint16 TDEFL_READ_UNALIGNED_WORD2(const mz_uint16* p) { mz_uint16 ret; memcpy(&ret, p, sizeof(mz_uint16)); return ret; } #else #define TDEFL_READ_UNALIGNED_WORD(p) *(const mz_uint16 *)(p) #define TDEFL_READ_UNALIGNED_WORD2(p) *(const mz_uint16 *)(p) #endif static MZ_FORCEINLINE void tdefl_find_match(tdefl_compressor *d, mz_uint lookahead_pos, mz_uint max_dist, mz_uint max_match_len, mz_uint *pMatch_dist, mz_uint *pMatch_len) { mz_uint dist, pos = lookahead_pos & TDEFL_LZ_DICT_SIZE_MASK, match_len = *pMatch_len, probe_pos = pos, next_probe_pos, probe_len; mz_uint num_probes_left = d->m_max_probes[match_len >= 32]; const mz_uint16 *s = (const mz_uint16 *)(d->m_dict + pos), *p, *q; mz_uint16 c01 = TDEFL_READ_UNALIGNED_WORD(&d->m_dict[pos + match_len - 1]), s01 = TDEFL_READ_UNALIGNED_WORD2(s); MZ_ASSERT(max_match_len <= TDEFL_MAX_MATCH_LEN); if (max_match_len <= match_len) return; for (;;) { for (;;) { if (--num_probes_left == 0) return; #define TDEFL_PROBE \ next_probe_pos = d->m_next[probe_pos]; \ if ((!next_probe_pos) || ((dist = (mz_uint16)(lookahead_pos - next_probe_pos)) > max_dist)) \ return; \ probe_pos = next_probe_pos & TDEFL_LZ_DICT_SIZE_MASK; \ if (TDEFL_READ_UNALIGNED_WORD(&d->m_dict[probe_pos + match_len - 1]) == c01) \ break; TDEFL_PROBE; TDEFL_PROBE; TDEFL_PROBE; } if (!dist) break; q = (const mz_uint16 *)(d->m_dict + probe_pos); if (TDEFL_READ_UNALIGNED_WORD2(q) != s01) continue; p = s; probe_len = 32; do { } while ((TDEFL_READ_UNALIGNED_WORD2(++p) == TDEFL_READ_UNALIGNED_WORD2(++q)) && (TDEFL_READ_UNALIGNED_WORD2(++p) == TDEFL_READ_UNALIGNED_WORD2(++q)) && (TDEFL_READ_UNALIGNED_WORD2(++p) == TDEFL_READ_UNALIGNED_WORD2(++q)) && (TDEFL_READ_UNALIGNED_WORD2(++p) == TDEFL_READ_UNALIGNED_WORD2(++q)) && (--probe_len > 0)); if (!probe_len) { *pMatch_dist = dist; *pMatch_len = MZ_MIN(max_match_len, (mz_uint)TDEFL_MAX_MATCH_LEN); break; } else if ((probe_len = ((mz_uint)(p - s) * 2) + (mz_uint)(*(const mz_uint8 *)p == *(const mz_uint8 *)q)) > match_len) { *pMatch_dist = dist; if ((*pMatch_len = match_len = MZ_MIN(max_match_len, probe_len)) == max_match_len) break; c01 = TDEFL_READ_UNALIGNED_WORD(&d->m_dict[pos + match_len - 1]); } } } #else static MZ_FORCEINLINE void tdefl_find_match(tdefl_compressor *d, mz_uint lookahead_pos, mz_uint max_dist, mz_uint max_match_len, mz_uint *pMatch_dist, mz_uint *pMatch_len) { mz_uint dist, pos = lookahead_pos & TDEFL_LZ_DICT_SIZE_MASK, match_len = *pMatch_len, probe_pos = pos, next_probe_pos, probe_len; mz_uint num_probes_left = d->m_max_probes[match_len >= 32]; const mz_uint8 *s = d->m_dict + pos, *p, *q; mz_uint8 c0 = d->m_dict[pos + match_len], c1 = d->m_dict[pos + match_len - 1]; MZ_ASSERT(max_match_len <= TDEFL_MAX_MATCH_LEN); if (max_match_len <= match_len) return; for (;;) { for (;;) { if (--num_probes_left == 0) return; #define TDEFL_PROBE \ next_probe_pos = d->m_next[probe_pos]; \ if ((!next_probe_pos) || ((dist = (mz_uint16)(lookahead_pos - next_probe_pos)) > max_dist)) \ return; \ probe_pos = next_probe_pos & TDEFL_LZ_DICT_SIZE_MASK; \ if ((d->m_dict[probe_pos + match_len] == c0) && (d->m_dict[probe_pos + match_len - 1] == c1)) \ break; TDEFL_PROBE; TDEFL_PROBE; TDEFL_PROBE; } if (!dist) break; p = s; q = d->m_dict + probe_pos; for (probe_len = 0; probe_len < max_match_len; probe_len++) if (*p++ != *q++) break; if (probe_len > match_len) { *pMatch_dist = dist; if ((*pMatch_len = match_len = probe_len) == max_match_len) return; c0 = d->m_dict[pos + match_len]; c1 = d->m_dict[pos + match_len - 1]; } } } #endif /* #if MINIZ_USE_UNALIGNED_LOADS_AND_STORES */ #if MINIZ_USE_UNALIGNED_LOADS_AND_STORES && MINIZ_LITTLE_ENDIAN #ifdef MINIZ_UNALIGNED_USE_MEMCPY static mz_uint32 TDEFL_READ_UNALIGNED_WORD32(const mz_uint8* p) { mz_uint32 ret; memcpy(&ret, p, sizeof(mz_uint32)); return ret; } #else #define TDEFL_READ_UNALIGNED_WORD32(p) *(const mz_uint32 *)(p) #endif static mz_bool tdefl_compress_fast(tdefl_compressor *d) { /* Faster, minimally featured LZRW1-style match+parse loop with better register utilization. Intended for applications where raw throughput is valued more highly than ratio. */ mz_uint lookahead_pos = d->m_lookahead_pos, lookahead_size = d->m_lookahead_size, dict_size = d->m_dict_size, total_lz_bytes = d->m_total_lz_bytes, num_flags_left = d->m_num_flags_left; mz_uint8 *pLZ_code_buf = d->m_pLZ_code_buf, *pLZ_flags = d->m_pLZ_flags; mz_uint cur_pos = lookahead_pos & TDEFL_LZ_DICT_SIZE_MASK; while ((d->m_src_buf_left) || ((d->m_flush) && (lookahead_size))) { const mz_uint TDEFL_COMP_FAST_LOOKAHEAD_SIZE = 4096; mz_uint dst_pos = (lookahead_pos + lookahead_size) & TDEFL_LZ_DICT_SIZE_MASK; mz_uint num_bytes_to_process = (mz_uint)MZ_MIN(d->m_src_buf_left, TDEFL_COMP_FAST_LOOKAHEAD_SIZE - lookahead_size); d->m_src_buf_left -= num_bytes_to_process; lookahead_size += num_bytes_to_process; while (num_bytes_to_process) { mz_uint32 n = MZ_MIN(TDEFL_LZ_DICT_SIZE - dst_pos, num_bytes_to_process); memcpy(d->m_dict + dst_pos, d->m_pSrc, n); if (dst_pos < (TDEFL_MAX_MATCH_LEN - 1)) memcpy(d->m_dict + TDEFL_LZ_DICT_SIZE + dst_pos, d->m_pSrc, MZ_MIN(n, (TDEFL_MAX_MATCH_LEN - 1) - dst_pos)); d->m_pSrc += n; dst_pos = (dst_pos + n) & TDEFL_LZ_DICT_SIZE_MASK; num_bytes_to_process -= n; } dict_size = MZ_MIN(TDEFL_LZ_DICT_SIZE - lookahead_size, dict_size); if ((!d->m_flush) && (lookahead_size < TDEFL_COMP_FAST_LOOKAHEAD_SIZE)) break; while (lookahead_size >= 4) { mz_uint cur_match_dist, cur_match_len = 1; mz_uint8 *pCur_dict = d->m_dict + cur_pos; mz_uint first_trigram = TDEFL_READ_UNALIGNED_WORD32(pCur_dict) & 0xFFFFFF; mz_uint hash = (first_trigram ^ (first_trigram >> (24 - (TDEFL_LZ_HASH_BITS - 8)))) & TDEFL_LEVEL1_HASH_SIZE_MASK; mz_uint probe_pos = d->m_hash[hash]; d->m_hash[hash] = (mz_uint16)lookahead_pos; if (((cur_match_dist = (mz_uint16)(lookahead_pos - probe_pos)) <= dict_size) && ((TDEFL_READ_UNALIGNED_WORD32(d->m_dict + (probe_pos &= TDEFL_LZ_DICT_SIZE_MASK)) & 0xFFFFFF) == first_trigram)) { const mz_uint16 *p = (const mz_uint16 *)pCur_dict; const mz_uint16 *q = (const mz_uint16 *)(d->m_dict + probe_pos); mz_uint32 probe_len = 32; do { } while ((TDEFL_READ_UNALIGNED_WORD2(++p) == TDEFL_READ_UNALIGNED_WORD2(++q)) && (TDEFL_READ_UNALIGNED_WORD2(++p) == TDEFL_READ_UNALIGNED_WORD2(++q)) && (TDEFL_READ_UNALIGNED_WORD2(++p) == TDEFL_READ_UNALIGNED_WORD2(++q)) && (TDEFL_READ_UNALIGNED_WORD2(++p) == TDEFL_READ_UNALIGNED_WORD2(++q)) && (--probe_len > 0)); cur_match_len = ((mz_uint)(p - (const mz_uint16 *)pCur_dict) * 2) + (mz_uint)(*(const mz_uint8 *)p == *(const mz_uint8 *)q); if (!probe_len) cur_match_len = cur_match_dist ? TDEFL_MAX_MATCH_LEN : 0; if ((cur_match_len < TDEFL_MIN_MATCH_LEN) || ((cur_match_len == TDEFL_MIN_MATCH_LEN) && (cur_match_dist >= 8U * 1024U))) { cur_match_len = 1; *pLZ_code_buf++ = (mz_uint8)first_trigram; *pLZ_flags = (mz_uint8)(*pLZ_flags >> 1); d->m_huff_count[0][(mz_uint8)first_trigram]++; } else { mz_uint32 s0, s1; cur_match_len = MZ_MIN(cur_match_len, lookahead_size); MZ_ASSERT((cur_match_len >= TDEFL_MIN_MATCH_LEN) && (cur_match_dist >= 1) && (cur_match_dist <= TDEFL_LZ_DICT_SIZE)); cur_match_dist--; pLZ_code_buf[0] = (mz_uint8)(cur_match_len - TDEFL_MIN_MATCH_LEN); #ifdef MINIZ_UNALIGNED_USE_MEMCPY memcpy(&pLZ_code_buf[1], &cur_match_dist, sizeof(cur_match_dist)); #else *(mz_uint16 *)(&pLZ_code_buf[1]) = (mz_uint16)cur_match_dist; #endif pLZ_code_buf += 3; *pLZ_flags = (mz_uint8)((*pLZ_flags >> 1) | 0x80); s0 = s_tdefl_small_dist_sym[cur_match_dist & 511]; s1 = s_tdefl_large_dist_sym[cur_match_dist >> 8]; d->m_huff_count[1][(cur_match_dist < 512) ? s0 : s1]++; d->m_huff_count[0][s_tdefl_len_sym[cur_match_len - TDEFL_MIN_MATCH_LEN]]++; } } else { *pLZ_code_buf++ = (mz_uint8)first_trigram; *pLZ_flags = (mz_uint8)(*pLZ_flags >> 1); d->m_huff_count[0][(mz_uint8)first_trigram]++; } if (--num_flags_left == 0) { num_flags_left = 8; pLZ_flags = pLZ_code_buf++; } total_lz_bytes += cur_match_len; lookahead_pos += cur_match_len; dict_size = MZ_MIN(dict_size + cur_match_len, (mz_uint)TDEFL_LZ_DICT_SIZE); cur_pos = (cur_pos + cur_match_len) & TDEFL_LZ_DICT_SIZE_MASK; MZ_ASSERT(lookahead_size >= cur_match_len); lookahead_size -= cur_match_len; if (pLZ_code_buf > &d->m_lz_code_buf[TDEFL_LZ_CODE_BUF_SIZE - 8]) { int n; d->m_lookahead_pos = lookahead_pos; d->m_lookahead_size = lookahead_size; d->m_dict_size = dict_size; d->m_total_lz_bytes = total_lz_bytes; d->m_pLZ_code_buf = pLZ_code_buf; d->m_pLZ_flags = pLZ_flags; d->m_num_flags_left = num_flags_left; if ((n = tdefl_flush_block(d, 0)) != 0) return (n < 0) ? MZ_FALSE : MZ_TRUE; total_lz_bytes = d->m_total_lz_bytes; pLZ_code_buf = d->m_pLZ_code_buf; pLZ_flags = d->m_pLZ_flags; num_flags_left = d->m_num_flags_left; } } while (lookahead_size) { mz_uint8 lit = d->m_dict[cur_pos]; total_lz_bytes++; *pLZ_code_buf++ = lit; *pLZ_flags = (mz_uint8)(*pLZ_flags >> 1); if (--num_flags_left == 0) { num_flags_left = 8; pLZ_flags = pLZ_code_buf++; } d->m_huff_count[0][lit]++; lookahead_pos++; dict_size = MZ_MIN(dict_size + 1, (mz_uint)TDEFL_LZ_DICT_SIZE); cur_pos = (cur_pos + 1) & TDEFL_LZ_DICT_SIZE_MASK; lookahead_size--; if (pLZ_code_buf > &d->m_lz_code_buf[TDEFL_LZ_CODE_BUF_SIZE - 8]) { int n; d->m_lookahead_pos = lookahead_pos; d->m_lookahead_size = lookahead_size; d->m_dict_size = dict_size; d->m_total_lz_bytes = total_lz_bytes; d->m_pLZ_code_buf = pLZ_code_buf; d->m_pLZ_flags = pLZ_flags; d->m_num_flags_left = num_flags_left; if ((n = tdefl_flush_block(d, 0)) != 0) return (n < 0) ? MZ_FALSE : MZ_TRUE; total_lz_bytes = d->m_total_lz_bytes; pLZ_code_buf = d->m_pLZ_code_buf; pLZ_flags = d->m_pLZ_flags; num_flags_left = d->m_num_flags_left; } } } d->m_lookahead_pos = lookahead_pos; d->m_lookahead_size = lookahead_size; d->m_dict_size = dict_size; d->m_total_lz_bytes = total_lz_bytes; d->m_pLZ_code_buf = pLZ_code_buf; d->m_pLZ_flags = pLZ_flags; d->m_num_flags_left = num_flags_left; return MZ_TRUE; } #endif /* MINIZ_USE_UNALIGNED_LOADS_AND_STORES && MINIZ_LITTLE_ENDIAN */ static MZ_FORCEINLINE void tdefl_record_literal(tdefl_compressor *d, mz_uint8 lit) { d->m_total_lz_bytes++; *d->m_pLZ_code_buf++ = lit; *d->m_pLZ_flags = (mz_uint8)(*d->m_pLZ_flags >> 1); if (--d->m_num_flags_left == 0) { d->m_num_flags_left = 8; d->m_pLZ_flags = d->m_pLZ_code_buf++; } d->m_huff_count[0][lit]++; } static MZ_FORCEINLINE void tdefl_record_match(tdefl_compressor *d, mz_uint match_len, mz_uint match_dist) { mz_uint32 s0, s1; MZ_ASSERT((match_len >= TDEFL_MIN_MATCH_LEN) && (match_dist >= 1) && (match_dist <= TDEFL_LZ_DICT_SIZE)); d->m_total_lz_bytes += match_len; d->m_pLZ_code_buf[0] = (mz_uint8)(match_len - TDEFL_MIN_MATCH_LEN); match_dist -= 1; d->m_pLZ_code_buf[1] = (mz_uint8)(match_dist & 0xFF); d->m_pLZ_code_buf[2] = (mz_uint8)(match_dist >> 8); d->m_pLZ_code_buf += 3; *d->m_pLZ_flags = (mz_uint8)((*d->m_pLZ_flags >> 1) | 0x80); if (--d->m_num_flags_left == 0) { d->m_num_flags_left = 8; d->m_pLZ_flags = d->m_pLZ_code_buf++; } s0 = s_tdefl_small_dist_sym[match_dist & 511]; s1 = s_tdefl_large_dist_sym[(match_dist >> 8) & 127]; d->m_huff_count[1][(match_dist < 512) ? s0 : s1]++; d->m_huff_count[0][s_tdefl_len_sym[match_len - TDEFL_MIN_MATCH_LEN]]++; } static mz_bool tdefl_compress_normal(tdefl_compressor *d) { const mz_uint8 *pSrc = d->m_pSrc; size_t src_buf_left = d->m_src_buf_left; tdefl_flush flush = d->m_flush; while ((src_buf_left) || ((flush) && (d->m_lookahead_size))) { mz_uint len_to_move, cur_match_dist, cur_match_len, cur_pos; /* Update dictionary and hash chains. Keeps the lookahead size equal to TDEFL_MAX_MATCH_LEN. */ if ((d->m_lookahead_size + d->m_dict_size) >= (TDEFL_MIN_MATCH_LEN - 1)) { mz_uint dst_pos = (d->m_lookahead_pos + d->m_lookahead_size) & TDEFL_LZ_DICT_SIZE_MASK, ins_pos = d->m_lookahead_pos + d->m_lookahead_size - 2; mz_uint hash = (d->m_dict[ins_pos & TDEFL_LZ_DICT_SIZE_MASK] << TDEFL_LZ_HASH_SHIFT) ^ d->m_dict[(ins_pos + 1) & TDEFL_LZ_DICT_SIZE_MASK]; mz_uint num_bytes_to_process = (mz_uint)MZ_MIN(src_buf_left, TDEFL_MAX_MATCH_LEN - d->m_lookahead_size); const mz_uint8 *pSrc_end = pSrc + num_bytes_to_process; src_buf_left -= num_bytes_to_process; d->m_lookahead_size += num_bytes_to_process; while (pSrc != pSrc_end) { mz_uint8 c = *pSrc++; d->m_dict[dst_pos] = c; if (dst_pos < (TDEFL_MAX_MATCH_LEN - 1)) d->m_dict[TDEFL_LZ_DICT_SIZE + dst_pos] = c; hash = ((hash << TDEFL_LZ_HASH_SHIFT) ^ c) & (TDEFL_LZ_HASH_SIZE - 1); d->m_next[ins_pos & TDEFL_LZ_DICT_SIZE_MASK] = d->m_hash[hash]; d->m_hash[hash] = (mz_uint16)(ins_pos); dst_pos = (dst_pos + 1) & TDEFL_LZ_DICT_SIZE_MASK; ins_pos++; } } else { while ((src_buf_left) && (d->m_lookahead_size < TDEFL_MAX_MATCH_LEN)) { mz_uint8 c = *pSrc++; mz_uint dst_pos = (d->m_lookahead_pos + d->m_lookahead_size) & TDEFL_LZ_DICT_SIZE_MASK; src_buf_left--; d->m_dict[dst_pos] = c; if (dst_pos < (TDEFL_MAX_MATCH_LEN - 1)) d->m_dict[TDEFL_LZ_DICT_SIZE + dst_pos] = c; if ((++d->m_lookahead_size + d->m_dict_size) >= TDEFL_MIN_MATCH_LEN) { mz_uint ins_pos = d->m_lookahead_pos + (d->m_lookahead_size - 1) - 2; mz_uint hash = ((d->m_dict[ins_pos & TDEFL_LZ_DICT_SIZE_MASK] << (TDEFL_LZ_HASH_SHIFT * 2)) ^ (d->m_dict[(ins_pos + 1) & TDEFL_LZ_DICT_SIZE_MASK] << TDEFL_LZ_HASH_SHIFT) ^ c) & (TDEFL_LZ_HASH_SIZE - 1); d->m_next[ins_pos & TDEFL_LZ_DICT_SIZE_MASK] = d->m_hash[hash]; d->m_hash[hash] = (mz_uint16)(ins_pos); } } } d->m_dict_size = MZ_MIN(TDEFL_LZ_DICT_SIZE - d->m_lookahead_size, d->m_dict_size); if ((!flush) && (d->m_lookahead_size < TDEFL_MAX_MATCH_LEN)) break; /* Simple lazy/greedy parsing state machine. */ len_to_move = 1; cur_match_dist = 0; cur_match_len = d->m_saved_match_len ? d->m_saved_match_len : (TDEFL_MIN_MATCH_LEN - 1); cur_pos = d->m_lookahead_pos & TDEFL_LZ_DICT_SIZE_MASK; if (d->m_flags & (TDEFL_RLE_MATCHES | TDEFL_FORCE_ALL_RAW_BLOCKS)) { if ((d->m_dict_size) && (!(d->m_flags & TDEFL_FORCE_ALL_RAW_BLOCKS))) { mz_uint8 c = d->m_dict[(cur_pos - 1) & TDEFL_LZ_DICT_SIZE_MASK]; cur_match_len = 0; while (cur_match_len < d->m_lookahead_size) { if (d->m_dict[cur_pos + cur_match_len] != c) break; cur_match_len++; } if (cur_match_len < TDEFL_MIN_MATCH_LEN) cur_match_len = 0; else cur_match_dist = 1; } } else { tdefl_find_match(d, d->m_lookahead_pos, d->m_dict_size, d->m_lookahead_size, &cur_match_dist, &cur_match_len); } if (((cur_match_len == TDEFL_MIN_MATCH_LEN) && (cur_match_dist >= 8U * 1024U)) || (cur_pos == cur_match_dist) || ((d->m_flags & TDEFL_FILTER_MATCHES) && (cur_match_len <= 5))) { cur_match_dist = cur_match_len = 0; } if (d->m_saved_match_len) { if (cur_match_len > d->m_saved_match_len) { tdefl_record_literal(d, (mz_uint8)d->m_saved_lit); if (cur_match_len >= 128) { tdefl_record_match(d, cur_match_len, cur_match_dist); d->m_saved_match_len = 0; len_to_move = cur_match_len; } else { d->m_saved_lit = d->m_dict[cur_pos]; d->m_saved_match_dist = cur_match_dist; d->m_saved_match_len = cur_match_len; } } else { tdefl_record_match(d, d->m_saved_match_len, d->m_saved_match_dist); len_to_move = d->m_saved_match_len - 1; d->m_saved_match_len = 0; } } else if (!cur_match_dist) tdefl_record_literal(d, d->m_dict[MZ_MIN(cur_pos, sizeof(d->m_dict) - 1)]); else if ((d->m_greedy_parsing) || (d->m_flags & TDEFL_RLE_MATCHES) || (cur_match_len >= 128)) { tdefl_record_match(d, cur_match_len, cur_match_dist); len_to_move = cur_match_len; } else { d->m_saved_lit = d->m_dict[MZ_MIN(cur_pos, sizeof(d->m_dict) - 1)]; d->m_saved_match_dist = cur_match_dist; d->m_saved_match_len = cur_match_len; } /* Move the lookahead forward by len_to_move bytes. */ d->m_lookahead_pos += len_to_move; MZ_ASSERT(d->m_lookahead_size >= len_to_move); d->m_lookahead_size -= len_to_move; d->m_dict_size = MZ_MIN(d->m_dict_size + len_to_move, (mz_uint)TDEFL_LZ_DICT_SIZE); /* Check if it's time to flush the current LZ codes to the internal output buffer. */ if ((d->m_pLZ_code_buf > &d->m_lz_code_buf[TDEFL_LZ_CODE_BUF_SIZE - 8]) || ((d->m_total_lz_bytes > 31 * 1024) && (((((mz_uint)(d->m_pLZ_code_buf - d->m_lz_code_buf) * 115) >> 7) >= d->m_total_lz_bytes) || (d->m_flags & TDEFL_FORCE_ALL_RAW_BLOCKS)))) { int n; d->m_pSrc = pSrc; d->m_src_buf_left = src_buf_left; if ((n = tdefl_flush_block(d, 0)) != 0) return (n < 0) ? MZ_FALSE : MZ_TRUE; } } d->m_pSrc = pSrc; d->m_src_buf_left = src_buf_left; return MZ_TRUE; } static tdefl_status tdefl_flush_output_buffer(tdefl_compressor *d) { if (d->m_pIn_buf_size) { *d->m_pIn_buf_size = d->m_pSrc - (const mz_uint8 *)d->m_pIn_buf; } if (d->m_pOut_buf_size) { size_t n = MZ_MIN(*d->m_pOut_buf_size - d->m_out_buf_ofs, d->m_output_flush_remaining); memcpy((mz_uint8 *)d->m_pOut_buf + d->m_out_buf_ofs, d->m_output_buf + d->m_output_flush_ofs, n); d->m_output_flush_ofs += (mz_uint)n; d->m_output_flush_remaining -= (mz_uint)n; d->m_out_buf_ofs += n; *d->m_pOut_buf_size = d->m_out_buf_ofs; } return (d->m_finished && !d->m_output_flush_remaining) ? TDEFL_STATUS_DONE : TDEFL_STATUS_OKAY; } tdefl_status tdefl_compress(tdefl_compressor *d, const void *pIn_buf, size_t *pIn_buf_size, void *pOut_buf, size_t *pOut_buf_size, tdefl_flush flush) { if (!d) { if (pIn_buf_size) *pIn_buf_size = 0; if (pOut_buf_size) *pOut_buf_size = 0; return TDEFL_STATUS_BAD_PARAM; } d->m_pIn_buf = pIn_buf; d->m_pIn_buf_size = pIn_buf_size; d->m_pOut_buf = pOut_buf; d->m_pOut_buf_size = pOut_buf_size; d->m_pSrc = (const mz_uint8 *)(pIn_buf); d->m_src_buf_left = pIn_buf_size ? *pIn_buf_size : 0; d->m_out_buf_ofs = 0; d->m_flush = flush; if (((d->m_pPut_buf_func != NULL) == ((pOut_buf != NULL) || (pOut_buf_size != NULL))) || (d->m_prev_return_status != TDEFL_STATUS_OKAY) || (d->m_wants_to_finish && (flush != TDEFL_FINISH)) || (pIn_buf_size && *pIn_buf_size && !pIn_buf) || (pOut_buf_size && *pOut_buf_size && !pOut_buf)) { if (pIn_buf_size) *pIn_buf_size = 0; if (pOut_buf_size) *pOut_buf_size = 0; return (d->m_prev_return_status = TDEFL_STATUS_BAD_PARAM); } d->m_wants_to_finish |= (flush == TDEFL_FINISH); if ((d->m_output_flush_remaining) || (d->m_finished)) return (d->m_prev_return_status = tdefl_flush_output_buffer(d)); #if MINIZ_USE_UNALIGNED_LOADS_AND_STORES && MINIZ_LITTLE_ENDIAN if (((d->m_flags & TDEFL_MAX_PROBES_MASK) == 1) && ((d->m_flags & TDEFL_GREEDY_PARSING_FLAG) != 0) && ((d->m_flags & (TDEFL_FILTER_MATCHES | TDEFL_FORCE_ALL_RAW_BLOCKS | TDEFL_RLE_MATCHES)) == 0)) { if (!tdefl_compress_fast(d)) return d->m_prev_return_status; } else #endif /* #if MINIZ_USE_UNALIGNED_LOADS_AND_STORES && MINIZ_LITTLE_ENDIAN */ { if (!tdefl_compress_normal(d)) return d->m_prev_return_status; } if ((d->m_flags & (TDEFL_WRITE_ZLIB_HEADER | TDEFL_COMPUTE_ADLER32)) && (pIn_buf)) d->m_adler32 = (mz_uint32)mz_adler32(d->m_adler32, (const mz_uint8 *)pIn_buf, d->m_pSrc - (const mz_uint8 *)pIn_buf); if ((flush) && (!d->m_lookahead_size) && (!d->m_src_buf_left) && (!d->m_output_flush_remaining)) { if (tdefl_flush_block(d, flush) < 0) return d->m_prev_return_status; d->m_finished = (flush == TDEFL_FINISH); if (flush == TDEFL_FULL_FLUSH) { MZ_CLEAR_OBJ(d->m_hash); MZ_CLEAR_OBJ(d->m_next); d->m_dict_size = 0; } } return (d->m_prev_return_status = tdefl_flush_output_buffer(d)); } tdefl_status tdefl_compress_buffer(tdefl_compressor *d, const void *pIn_buf, size_t in_buf_size, tdefl_flush flush) { MZ_ASSERT(d->m_pPut_buf_func); return tdefl_compress(d, pIn_buf, &in_buf_size, NULL, NULL, flush); } tdefl_status tdefl_init(tdefl_compressor *d, tdefl_put_buf_func_ptr pPut_buf_func, void *pPut_buf_user, int flags) { d->m_pPut_buf_func = pPut_buf_func; d->m_pPut_buf_user = pPut_buf_user; d->m_flags = (mz_uint)(flags); d->m_max_probes[0] = 1 + ((flags & 0xFFF) + 2) / 3; d->m_greedy_parsing = (flags & TDEFL_GREEDY_PARSING_FLAG) != 0; d->m_max_probes[1] = 1 + (((flags & 0xFFF) >> 2) + 2) / 3; if (!(flags & TDEFL_NONDETERMINISTIC_PARSING_FLAG)) MZ_CLEAR_OBJ(d->m_hash); d->m_lookahead_pos = d->m_lookahead_size = d->m_dict_size = d->m_total_lz_bytes = d->m_lz_code_buf_dict_pos = d->m_bits_in = 0; d->m_output_flush_ofs = d->m_output_flush_remaining = d->m_finished = d->m_block_index = d->m_bit_buffer = d->m_wants_to_finish = 0; d->m_pLZ_code_buf = d->m_lz_code_buf + 1; d->m_pLZ_flags = d->m_lz_code_buf; *d->m_pLZ_flags = 0; d->m_num_flags_left = 8; d->m_pOutput_buf = d->m_output_buf; d->m_pOutput_buf_end = d->m_output_buf; d->m_prev_return_status = TDEFL_STATUS_OKAY; d->m_saved_match_dist = d->m_saved_match_len = d->m_saved_lit = 0; d->m_adler32 = 1; d->m_pIn_buf = NULL; d->m_pOut_buf = NULL; d->m_pIn_buf_size = NULL; d->m_pOut_buf_size = NULL; d->m_flush = TDEFL_NO_FLUSH; d->m_pSrc = NULL; d->m_src_buf_left = 0; d->m_out_buf_ofs = 0; if (!(flags & TDEFL_NONDETERMINISTIC_PARSING_FLAG)) MZ_CLEAR_OBJ(d->m_dict); memset(&d->m_huff_count[0][0], 0, sizeof(d->m_huff_count[0][0]) * TDEFL_MAX_HUFF_SYMBOLS_0); memset(&d->m_huff_count[1][0], 0, sizeof(d->m_huff_count[1][0]) * TDEFL_MAX_HUFF_SYMBOLS_1); return TDEFL_STATUS_OKAY; } tdefl_status tdefl_get_prev_return_status(tdefl_compressor *d) { return d->m_prev_return_status; } mz_uint32 tdefl_get_adler32(tdefl_compressor *d) { return d->m_adler32; } mz_bool tdefl_compress_mem_to_output(const void *pBuf, size_t buf_len, tdefl_put_buf_func_ptr pPut_buf_func, void *pPut_buf_user, int flags) { tdefl_compressor *pComp; mz_bool succeeded; if (((buf_len) && (!pBuf)) || (!pPut_buf_func)) return MZ_FALSE; pComp = (tdefl_compressor *)MZ_MALLOC(sizeof(tdefl_compressor)); if (!pComp) return MZ_FALSE; succeeded = (tdefl_init(pComp, pPut_buf_func, pPut_buf_user, flags) == TDEFL_STATUS_OKAY); succeeded = succeeded && (tdefl_compress_buffer(pComp, pBuf, buf_len, TDEFL_FINISH) == TDEFL_STATUS_DONE); MZ_FREE(pComp); return succeeded; } typedef struct { size_t m_size, m_capacity; mz_uint8 *m_pBuf; mz_bool m_expandable; } tdefl_output_buffer; static mz_bool tdefl_output_buffer_putter(const void *pBuf, int len, void *pUser) { tdefl_output_buffer *p = (tdefl_output_buffer *)pUser; size_t new_size = p->m_size + len; if (new_size > p->m_capacity) { size_t new_capacity = p->m_capacity; mz_uint8 *pNew_buf; if (!p->m_expandable) return MZ_FALSE; do { new_capacity = MZ_MAX(128U, new_capacity << 1U); } while (new_size > new_capacity); pNew_buf = (mz_uint8 *)MZ_REALLOC(p->m_pBuf, new_capacity); if (!pNew_buf) return MZ_FALSE; p->m_pBuf = pNew_buf; p->m_capacity = new_capacity; } memcpy((mz_uint8 *)p->m_pBuf + p->m_size, pBuf, len); p->m_size = new_size; return MZ_TRUE; } void *tdefl_compress_mem_to_heap(const void *pSrc_buf, size_t src_buf_len, size_t *pOut_len, int flags) { tdefl_output_buffer out_buf; MZ_CLEAR_OBJ(out_buf); if (!pOut_len) return MZ_FALSE; else *pOut_len = 0; out_buf.m_expandable = MZ_TRUE; if (!tdefl_compress_mem_to_output(pSrc_buf, src_buf_len, tdefl_output_buffer_putter, &out_buf, flags)) return NULL; *pOut_len = out_buf.m_size; return out_buf.m_pBuf; } size_t tdefl_compress_mem_to_mem(void *pOut_buf, size_t out_buf_len, const void *pSrc_buf, size_t src_buf_len, int flags) { tdefl_output_buffer out_buf; MZ_CLEAR_OBJ(out_buf); if (!pOut_buf) return 0; out_buf.m_pBuf = (mz_uint8 *)pOut_buf; out_buf.m_capacity = out_buf_len; if (!tdefl_compress_mem_to_output(pSrc_buf, src_buf_len, tdefl_output_buffer_putter, &out_buf, flags)) return 0; return out_buf.m_size; } static const mz_uint s_tdefl_num_probes[11] = { 0, 1, 6, 32, 16, 32, 128, 256, 512, 768, 1500 }; /* level may actually range from [0,10] (10 is a "hidden" max level, where we want a bit more compression and it's fine if throughput to fall off a cliff on some files). */ mz_uint tdefl_create_comp_flags_from_zip_params(int level, int window_bits, int strategy) { mz_uint comp_flags = s_tdefl_num_probes[(level >= 0) ? MZ_MIN(10, level) : MZ_DEFAULT_LEVEL] | ((level <= 3) ? TDEFL_GREEDY_PARSING_FLAG : 0); if (window_bits > 0) comp_flags |= TDEFL_WRITE_ZLIB_HEADER; if (!level) comp_flags |= TDEFL_FORCE_ALL_RAW_BLOCKS; else if (strategy == MZ_FILTERED) comp_flags |= TDEFL_FILTER_MATCHES; else if (strategy == MZ_HUFFMAN_ONLY) comp_flags &= ~TDEFL_MAX_PROBES_MASK; else if (strategy == MZ_FIXED) comp_flags |= TDEFL_FORCE_ALL_STATIC_BLOCKS; else if (strategy == MZ_RLE) comp_flags |= TDEFL_RLE_MATCHES; return comp_flags; } #ifdef _MSC_VER #pragma warning(push) #pragma warning(disable : 4204) /* nonstandard extension used : non-constant aggregate initializer (also supported by GNU C and C99, so no big deal) */ #endif /* Simple PNG writer function by Alex Evans, 2011. Released into the public domain: https://gist.github.com/908299, more context at http://altdevblogaday.org/2011/04/06/a-smaller-jpg-encoder/. This is actually a modification of Alex's original code so PNG files generated by this function pass pngcheck. */ void *tdefl_write_image_to_png_file_in_memory_ex(const void *pImage, int w, int h, int num_chans, size_t *pLen_out, mz_uint level, mz_bool flip) { /* Using a local copy of this array here in case MINIZ_NO_ZLIB_APIS was defined. */ static const mz_uint s_tdefl_png_num_probes[11] = { 0, 1, 6, 32, 16, 32, 128, 256, 512, 768, 1500 }; tdefl_compressor *pComp = (tdefl_compressor *)MZ_MALLOC(sizeof(tdefl_compressor)); tdefl_output_buffer out_buf; int i, bpl = w * num_chans, y, z; mz_uint32 c; *pLen_out = 0; if (!pComp) return NULL; MZ_CLEAR_OBJ(out_buf); out_buf.m_expandable = MZ_TRUE; out_buf.m_capacity = 57 + MZ_MAX(64, (1 + bpl) * h); if (NULL == (out_buf.m_pBuf = (mz_uint8 *)MZ_MALLOC(out_buf.m_capacity))) { MZ_FREE(pComp); return NULL; } /* write dummy header */ for (z = 41; z; --z) tdefl_output_buffer_putter(&z, 1, &out_buf); /* compress image data */ tdefl_init(pComp, tdefl_output_buffer_putter, &out_buf, s_tdefl_png_num_probes[MZ_MIN(10, level)] | TDEFL_WRITE_ZLIB_HEADER); for (y = 0; y < h; ++y) { tdefl_compress_buffer(pComp, &z, 1, TDEFL_NO_FLUSH); tdefl_compress_buffer(pComp, (mz_uint8 *)pImage + (flip ? (h - 1 - y) : y) * bpl, bpl, TDEFL_NO_FLUSH); } if (tdefl_compress_buffer(pComp, NULL, 0, TDEFL_FINISH) != TDEFL_STATUS_DONE) { MZ_FREE(pComp); MZ_FREE(out_buf.m_pBuf); return NULL; } /* write real header */ *pLen_out = out_buf.m_size - 41; { static const mz_uint8 chans[] = { 0x00, 0x00, 0x04, 0x02, 0x06 }; mz_uint8 pnghdr[41] = { 0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0x00, 0x00, 0x00, 0x0d, 0x49, 0x48, 0x44, 0x52, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x49, 0x44, 0x41, 0x54 }; pnghdr[18] = (mz_uint8)(w >> 8); pnghdr[19] = (mz_uint8)w; pnghdr[22] = (mz_uint8)(h >> 8); pnghdr[23] = (mz_uint8)h; pnghdr[25] = chans[num_chans]; pnghdr[33] = (mz_uint8)(*pLen_out >> 24); pnghdr[34] = (mz_uint8)(*pLen_out >> 16); pnghdr[35] = (mz_uint8)(*pLen_out >> 8); pnghdr[36] = (mz_uint8)*pLen_out; c = (mz_uint32)mz_crc32(MZ_CRC32_INIT, pnghdr + 12, 17); for (i = 0; i < 4; ++i, c <<= 8) ((mz_uint8 *)(pnghdr + 29))[i] = (mz_uint8)(c >> 24); memcpy(out_buf.m_pBuf, pnghdr, 41); } /* write footer (IDAT CRC-32, followed by IEND chunk) */ if (!tdefl_output_buffer_putter("\0\0\0\0\0\0\0\0\x49\x45\x4e\x44\xae\x42\x60\x82", 16, &out_buf)) { *pLen_out = 0; MZ_FREE(pComp); MZ_FREE(out_buf.m_pBuf); return NULL; } c = (mz_uint32)mz_crc32(MZ_CRC32_INIT, out_buf.m_pBuf + 41 - 4, *pLen_out + 4); for (i = 0; i < 4; ++i, c <<= 8) (out_buf.m_pBuf + out_buf.m_size - 16)[i] = (mz_uint8)(c >> 24); /* compute final size of file, grab compressed data buffer and return */ *pLen_out += 57; MZ_FREE(pComp); return out_buf.m_pBuf; } void *tdefl_write_image_to_png_file_in_memory(const void *pImage, int w, int h, int num_chans, size_t *pLen_out) { /* Level 6 corresponds to TDEFL_DEFAULT_MAX_PROBES or MZ_DEFAULT_LEVEL (but we can't depend on MZ_DEFAULT_LEVEL being available in case the zlib API's where #defined out) */ return tdefl_write_image_to_png_file_in_memory_ex(pImage, w, h, num_chans, pLen_out, 6, MZ_FALSE); } #ifndef MINIZ_NO_MALLOC /* Allocate the tdefl_compressor and tinfl_decompressor structures in C so that */ /* non-C language bindings to tdefL_ and tinfl_ API don't need to worry about */ /* structure size and allocation mechanism. */ tdefl_compressor *tdefl_compressor_alloc() { return (tdefl_compressor *)MZ_MALLOC(sizeof(tdefl_compressor)); } void tdefl_compressor_free(tdefl_compressor *pComp) { MZ_FREE(pComp); } #endif #ifdef _MSC_VER #pragma warning(pop) #endif #ifdef __cplusplus } #endif /************************************************************************** * * Copyright 2013-2014 RAD Game Tools and Valve Software * Copyright 2010-2014 Rich Geldreich and Tenacious Software LLC * All Rights Reserved. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal * in the Software without restriction, including without limitation the rights * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN * THE SOFTWARE. * **************************************************************************/ #ifdef __cplusplus extern "C" { #endif /* ------------------- Low-level Decompression (completely independent from all compression API's) */ #define TINFL_MEMCPY(d, s, l) memcpy(d, s, l) #define TINFL_MEMSET(p, c, l) memset(p, c, l) #define TINFL_CR_BEGIN \ switch (r->m_state) \ { \ case 0: #define TINFL_CR_RETURN(state_index, result) \ do \ { \ status = result; \ r->m_state = state_index; \ goto common_exit; \ case state_index:; \ } \ MZ_MACRO_END #define TINFL_CR_RETURN_FOREVER(state_index, result) \ do \ { \ for (;;) \ { \ TINFL_CR_RETURN(state_index, result); \ } \ } \ MZ_MACRO_END #define TINFL_CR_FINISH } #define TINFL_GET_BYTE(state_index, c) \ do \ { \ while (pIn_buf_cur >= pIn_buf_end) \ { \ TINFL_CR_RETURN(state_index, (decomp_flags & TINFL_FLAG_HAS_MORE_INPUT) ? TINFL_STATUS_NEEDS_MORE_INPUT : TINFL_STATUS_FAILED_CANNOT_MAKE_PROGRESS); \ } \ c = *pIn_buf_cur++; \ } \ MZ_MACRO_END #define TINFL_NEED_BITS(state_index, n) \ do \ { \ mz_uint c; \ TINFL_GET_BYTE(state_index, c); \ bit_buf |= (((tinfl_bit_buf_t)c) << num_bits); \ num_bits += 8; \ } while (num_bits < (mz_uint)(n)) #define TINFL_SKIP_BITS(state_index, n) \ do \ { \ if (num_bits < (mz_uint)(n)) \ { \ TINFL_NEED_BITS(state_index, n); \ } \ bit_buf >>= (n); \ num_bits -= (n); \ } \ MZ_MACRO_END #define TINFL_GET_BITS(state_index, b, n) \ do \ { \ if (num_bits < (mz_uint)(n)) \ { \ TINFL_NEED_BITS(state_index, n); \ } \ b = bit_buf & ((1 << (n)) - 1); \ bit_buf >>= (n); \ num_bits -= (n); \ } \ MZ_MACRO_END /* TINFL_HUFF_BITBUF_FILL() is only used rarely, when the number of bytes remaining in the input buffer falls below 2. */ /* It reads just enough bytes from the input stream that are needed to decode the next Huffman code (and absolutely no more). It works by trying to fully decode a */ /* Huffman code by using whatever bits are currently present in the bit buffer. If this fails, it reads another byte, and tries again until it succeeds or until the */ /* bit buffer contains >=15 bits (deflate's max. Huffman code size). */ #define TINFL_HUFF_BITBUF_FILL(state_index, pHuff) \ do \ { \ temp = (pHuff)->m_look_up[bit_buf & (TINFL_FAST_LOOKUP_SIZE - 1)]; \ if (temp >= 0) \ { \ code_len = temp >> 9; \ if ((code_len) && (num_bits >= code_len)) \ break; \ } \ else if (num_bits > TINFL_FAST_LOOKUP_BITS) \ { \ code_len = TINFL_FAST_LOOKUP_BITS; \ do \ { \ temp = (pHuff)->m_tree[~temp + ((bit_buf >> code_len++) & 1)]; \ } while ((temp < 0) && (num_bits >= (code_len + 1))); \ if (temp >= 0) \ break; \ } \ TINFL_GET_BYTE(state_index, c); \ bit_buf |= (((tinfl_bit_buf_t)c) << num_bits); \ num_bits += 8; \ } while (num_bits < 15); /* TINFL_HUFF_DECODE() decodes the next Huffman coded symbol. It's more complex than you would initially expect because the zlib API expects the decompressor to never read */ /* beyond the final byte of the deflate stream. (In other words, when this macro wants to read another byte from the input, it REALLY needs another byte in order to fully */ /* decode the next Huffman code.) Handling this properly is particularly important on raw deflate (non-zlib) streams, which aren't followed by a byte aligned adler-32. */ /* The slow path is only executed at the very end of the input buffer. */ /* v1.16: The original macro handled the case at the very end of the passed-in input buffer, but we also need to handle the case where the user passes in 1+zillion bytes */ /* following the deflate data and our non-conservative read-ahead path won't kick in here on this code. This is much trickier. */ #define TINFL_HUFF_DECODE(state_index, sym, pHuff) \ do \ { \ int temp; \ mz_uint code_len, c; \ if (num_bits < 15) \ { \ if ((pIn_buf_end - pIn_buf_cur) < 2) \ { \ TINFL_HUFF_BITBUF_FILL(state_index, pHuff); \ } \ else \ { \ bit_buf |= (((tinfl_bit_buf_t)pIn_buf_cur[0]) << num_bits) | (((tinfl_bit_buf_t)pIn_buf_cur[1]) << (num_bits + 8)); \ pIn_buf_cur += 2; \ num_bits += 16; \ } \ } \ if ((temp = (pHuff)->m_look_up[bit_buf & (TINFL_FAST_LOOKUP_SIZE - 1)]) >= 0) \ code_len = temp >> 9, temp &= 511; \ else \ { \ code_len = TINFL_FAST_LOOKUP_BITS; \ do \ { \ temp = (pHuff)->m_tree[~temp + ((bit_buf >> code_len++) & 1)]; \ } while (temp < 0); \ } \ sym = temp; \ bit_buf >>= code_len; \ num_bits -= code_len; \ } \ MZ_MACRO_END tinfl_status tinfl_decompress(tinfl_decompressor *r, const mz_uint8 *pIn_buf_next, size_t *pIn_buf_size, mz_uint8 *pOut_buf_start, mz_uint8 *pOut_buf_next, size_t *pOut_buf_size, const mz_uint32 decomp_flags) { static const int s_length_base[31] = { 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 17, 19, 23, 27, 31, 35, 43, 51, 59, 67, 83, 99, 115, 131, 163, 195, 227, 258, 0, 0 }; static const int s_length_extra[31] = { 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 0, 0, 0 }; static const int s_dist_base[32] = { 1, 2, 3, 4, 5, 7, 9, 13, 17, 25, 33, 49, 65, 97, 129, 193, 257, 385, 513, 769, 1025, 1537, 2049, 3073, 4097, 6145, 8193, 12289, 16385, 24577, 0, 0 }; static const int s_dist_extra[32] = { 0, 0, 0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9, 10, 10, 11, 11, 12, 12, 13, 13 }; static const mz_uint8 s_length_dezigzag[19] = { 16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15 }; static const int s_min_table_sizes[3] = { 257, 1, 4 }; tinfl_status status = TINFL_STATUS_FAILED; mz_uint32 num_bits, dist, counter, num_extra; tinfl_bit_buf_t bit_buf; const mz_uint8 *pIn_buf_cur = pIn_buf_next, *const pIn_buf_end = pIn_buf_next + *pIn_buf_size; mz_uint8 *pOut_buf_cur = pOut_buf_next, *const pOut_buf_end = pOut_buf_next + *pOut_buf_size; size_t out_buf_size_mask = (decomp_flags & TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF) ? (size_t)-1 : ((pOut_buf_next - pOut_buf_start) + *pOut_buf_size) - 1, dist_from_out_buf_start; /* Ensure the output buffer's size is a power of 2, unless the output buffer is large enough to hold the entire output file (in which case it doesn't matter). */ if (((out_buf_size_mask + 1) & out_buf_size_mask) || (pOut_buf_next < pOut_buf_start)) { *pIn_buf_size = *pOut_buf_size = 0; return TINFL_STATUS_BAD_PARAM; } num_bits = r->m_num_bits; bit_buf = r->m_bit_buf; dist = r->m_dist; counter = r->m_counter; num_extra = r->m_num_extra; dist_from_out_buf_start = r->m_dist_from_out_buf_start; TINFL_CR_BEGIN bit_buf = num_bits = dist = counter = num_extra = r->m_zhdr0 = r->m_zhdr1 = 0; r->m_z_adler32 = r->m_check_adler32 = 1; if (decomp_flags & TINFL_FLAG_PARSE_ZLIB_HEADER) { TINFL_GET_BYTE(1, r->m_zhdr0); TINFL_GET_BYTE(2, r->m_zhdr1); counter = (((r->m_zhdr0 * 256 + r->m_zhdr1) % 31 != 0) || (r->m_zhdr1 & 32) || ((r->m_zhdr0 & 15) != 8)); if (!(decomp_flags & TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF)) counter |= (((1U << (8U + (r->m_zhdr0 >> 4))) > 32768U) || ((out_buf_size_mask + 1) < (size_t)(1U << (8U + (r->m_zhdr0 >> 4))))); if (counter) { TINFL_CR_RETURN_FOREVER(36, TINFL_STATUS_FAILED); } } do { TINFL_GET_BITS(3, r->m_final, 3); r->m_type = r->m_final >> 1; if (r->m_type == 0) { TINFL_SKIP_BITS(5, num_bits & 7); for (counter = 0; counter < 4; ++counter) { if (num_bits) TINFL_GET_BITS(6, r->m_raw_header[counter], 8); else TINFL_GET_BYTE(7, r->m_raw_header[counter]); } if ((counter = (r->m_raw_header[0] | (r->m_raw_header[1] << 8))) != (mz_uint)(0xFFFF ^ (r->m_raw_header[2] | (r->m_raw_header[3] << 8)))) { TINFL_CR_RETURN_FOREVER(39, TINFL_STATUS_FAILED); } while ((counter) && (num_bits)) { TINFL_GET_BITS(51, dist, 8); while (pOut_buf_cur >= pOut_buf_end) { TINFL_CR_RETURN(52, TINFL_STATUS_HAS_MORE_OUTPUT); } *pOut_buf_cur++ = (mz_uint8)dist; counter--; } while (counter) { size_t n; while (pOut_buf_cur >= pOut_buf_end) { TINFL_CR_RETURN(9, TINFL_STATUS_HAS_MORE_OUTPUT); } while (pIn_buf_cur >= pIn_buf_end) { TINFL_CR_RETURN(38, (decomp_flags & TINFL_FLAG_HAS_MORE_INPUT) ? TINFL_STATUS_NEEDS_MORE_INPUT : TINFL_STATUS_FAILED_CANNOT_MAKE_PROGRESS); } n = MZ_MIN(MZ_MIN((size_t)(pOut_buf_end - pOut_buf_cur), (size_t)(pIn_buf_end - pIn_buf_cur)), counter); TINFL_MEMCPY(pOut_buf_cur, pIn_buf_cur, n); pIn_buf_cur += n; pOut_buf_cur += n; counter -= (mz_uint)n; } } else if (r->m_type == 3) { TINFL_CR_RETURN_FOREVER(10, TINFL_STATUS_FAILED); } else { if (r->m_type == 1) { mz_uint8 *p = r->m_tables[0].m_code_size; mz_uint i; r->m_table_sizes[0] = 288; r->m_table_sizes[1] = 32; TINFL_MEMSET(r->m_tables[1].m_code_size, 5, 32); for (i = 0; i <= 143; ++i) *p++ = 8; for (; i <= 255; ++i) *p++ = 9; for (; i <= 279; ++i) *p++ = 7; for (; i <= 287; ++i) *p++ = 8; } else { for (counter = 0; counter < 3; counter++) { TINFL_GET_BITS(11, r->m_table_sizes[counter], "\05\05\04"[counter]); r->m_table_sizes[counter] += s_min_table_sizes[counter]; } MZ_CLEAR_OBJ(r->m_tables[2].m_code_size); for (counter = 0; counter < r->m_table_sizes[2]; counter++) { mz_uint s; TINFL_GET_BITS(14, s, 3); r->m_tables[2].m_code_size[s_length_dezigzag[counter]] = (mz_uint8)s; } r->m_table_sizes[2] = 19; } for (; (int)r->m_type >= 0; r->m_type--) { int tree_next, tree_cur; tinfl_huff_table *pTable; mz_uint i, j, used_syms, total, sym_index, next_code[17], total_syms[16]; pTable = &r->m_tables[r->m_type]; MZ_CLEAR_OBJ(total_syms); MZ_CLEAR_OBJ(pTable->m_look_up); MZ_CLEAR_OBJ(pTable->m_tree); for (i = 0; i < r->m_table_sizes[r->m_type]; ++i) total_syms[pTable->m_code_size[i]]++; used_syms = 0, total = 0; next_code[0] = next_code[1] = 0; for (i = 1; i <= 15; ++i) { used_syms += total_syms[i]; next_code[i + 1] = (total = ((total + total_syms[i]) << 1)); } if ((65536 != total) && (used_syms > 1)) { TINFL_CR_RETURN_FOREVER(35, TINFL_STATUS_FAILED); } for (tree_next = -1, sym_index = 0; sym_index < r->m_table_sizes[r->m_type]; ++sym_index) { mz_uint rev_code = 0, l, cur_code, code_size = pTable->m_code_size[sym_index]; if (!code_size) continue; cur_code = next_code[code_size]++; for (l = code_size; l > 0; l--, cur_code >>= 1) rev_code = (rev_code << 1) | (cur_code & 1); if (code_size <= TINFL_FAST_LOOKUP_BITS) { mz_int16 k = (mz_int16)((code_size << 9) | sym_index); while (rev_code < TINFL_FAST_LOOKUP_SIZE) { pTable->m_look_up[rev_code] = k; rev_code += (1 << code_size); } continue; } if (0 == (tree_cur = pTable->m_look_up[rev_code & (TINFL_FAST_LOOKUP_SIZE - 1)])) { pTable->m_look_up[rev_code & (TINFL_FAST_LOOKUP_SIZE - 1)] = (mz_int16)tree_next; tree_cur = tree_next; tree_next -= 2; } rev_code >>= (TINFL_FAST_LOOKUP_BITS - 1); for (j = code_size; j > (TINFL_FAST_LOOKUP_BITS + 1); j--) { tree_cur -= ((rev_code >>= 1) & 1); if (!pTable->m_tree[-tree_cur - 1]) { pTable->m_tree[-tree_cur - 1] = (mz_int16)tree_next; tree_cur = tree_next; tree_next -= 2; } else tree_cur = pTable->m_tree[-tree_cur - 1]; } tree_cur -= ((rev_code >>= 1) & 1); pTable->m_tree[-tree_cur - 1] = (mz_int16)sym_index; } if (r->m_type == 2) { for (counter = 0; counter < (r->m_table_sizes[0] + r->m_table_sizes[1]);) { mz_uint s; TINFL_HUFF_DECODE(16, dist, &r->m_tables[2]); if (dist < 16) { r->m_len_codes[counter++] = (mz_uint8)dist; continue; } if ((dist == 16) && (!counter)) { TINFL_CR_RETURN_FOREVER(17, TINFL_STATUS_FAILED); } num_extra = "\02\03\07"[dist - 16]; TINFL_GET_BITS(18, s, num_extra); s += "\03\03\013"[dist - 16]; TINFL_MEMSET(r->m_len_codes + counter, (dist == 16) ? r->m_len_codes[counter - 1] : 0, s); counter += s; } if ((r->m_table_sizes[0] + r->m_table_sizes[1]) != counter) { TINFL_CR_RETURN_FOREVER(21, TINFL_STATUS_FAILED); } TINFL_MEMCPY(r->m_tables[0].m_code_size, r->m_len_codes, r->m_table_sizes[0]); TINFL_MEMCPY(r->m_tables[1].m_code_size, r->m_len_codes + r->m_table_sizes[0], r->m_table_sizes[1]); } } for (;;) { mz_uint8 *pSrc; for (;;) { if (((pIn_buf_end - pIn_buf_cur) < 4) || ((pOut_buf_end - pOut_buf_cur) < 2)) { TINFL_HUFF_DECODE(23, counter, &r->m_tables[0]); if (counter >= 256) break; while (pOut_buf_cur >= pOut_buf_end) { TINFL_CR_RETURN(24, TINFL_STATUS_HAS_MORE_OUTPUT); } *pOut_buf_cur++ = (mz_uint8)counter; } else { int sym2; mz_uint code_len; #if TINFL_USE_64BIT_BITBUF if (num_bits < 30) { bit_buf |= (((tinfl_bit_buf_t)MZ_READ_LE32(pIn_buf_cur)) << num_bits); pIn_buf_cur += 4; num_bits += 32; } #else if (num_bits < 15) { bit_buf |= (((tinfl_bit_buf_t)MZ_READ_LE16(pIn_buf_cur)) << num_bits); pIn_buf_cur += 2; num_bits += 16; } #endif if ((sym2 = r->m_tables[0].m_look_up[bit_buf & (TINFL_FAST_LOOKUP_SIZE - 1)]) >= 0) code_len = sym2 >> 9; else { code_len = TINFL_FAST_LOOKUP_BITS; do { sym2 = r->m_tables[0].m_tree[~sym2 + ((bit_buf >> code_len++) & 1)]; } while (sym2 < 0); } counter = sym2; bit_buf >>= code_len; num_bits -= code_len; if (counter & 256) break; #if !TINFL_USE_64BIT_BITBUF if (num_bits < 15) { bit_buf |= (((tinfl_bit_buf_t)MZ_READ_LE16(pIn_buf_cur)) << num_bits); pIn_buf_cur += 2; num_bits += 16; } #endif if ((sym2 = r->m_tables[0].m_look_up[bit_buf & (TINFL_FAST_LOOKUP_SIZE - 1)]) >= 0) code_len = sym2 >> 9; else { code_len = TINFL_FAST_LOOKUP_BITS; do { sym2 = r->m_tables[0].m_tree[~sym2 + ((bit_buf >> code_len++) & 1)]; } while (sym2 < 0); } bit_buf >>= code_len; num_bits -= code_len; pOut_buf_cur[0] = (mz_uint8)counter; if (sym2 & 256) { pOut_buf_cur++; counter = sym2; break; } pOut_buf_cur[1] = (mz_uint8)sym2; pOut_buf_cur += 2; } } if ((counter &= 511) == 256) break; num_extra = s_length_extra[counter - 257]; counter = s_length_base[counter - 257]; if (num_extra) { mz_uint extra_bits; TINFL_GET_BITS(25, extra_bits, num_extra); counter += extra_bits; } TINFL_HUFF_DECODE(26, dist, &r->m_tables[1]); num_extra = s_dist_extra[dist]; dist = s_dist_base[dist]; if (num_extra) { mz_uint extra_bits; TINFL_GET_BITS(27, extra_bits, num_extra); dist += extra_bits; } dist_from_out_buf_start = pOut_buf_cur - pOut_buf_start; if ((dist == 0 || dist > dist_from_out_buf_start || dist_from_out_buf_start == 0) && (decomp_flags & TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF)) { TINFL_CR_RETURN_FOREVER(37, TINFL_STATUS_FAILED); } pSrc = pOut_buf_start + ((dist_from_out_buf_start - dist) & out_buf_size_mask); if ((MZ_MAX(pOut_buf_cur, pSrc) + counter) > pOut_buf_end) { while (counter--) { while (pOut_buf_cur >= pOut_buf_end) { TINFL_CR_RETURN(53, TINFL_STATUS_HAS_MORE_OUTPUT); } *pOut_buf_cur++ = pOut_buf_start[(dist_from_out_buf_start++ - dist) & out_buf_size_mask]; } continue; } #if MINIZ_USE_UNALIGNED_LOADS_AND_STORES else if ((counter >= 9) && (counter <= dist)) { const mz_uint8 *pSrc_end = pSrc + (counter & ~7); do { #ifdef MINIZ_UNALIGNED_USE_MEMCPY memcpy(pOut_buf_cur, pSrc, sizeof(mz_uint32)*2); #else ((mz_uint32 *)pOut_buf_cur)[0] = ((const mz_uint32 *)pSrc)[0]; ((mz_uint32 *)pOut_buf_cur)[1] = ((const mz_uint32 *)pSrc)[1]; #endif pOut_buf_cur += 8; } while ((pSrc += 8) < pSrc_end); if ((counter &= 7) < 3) { if (counter) { pOut_buf_cur[0] = pSrc[0]; if (counter > 1) pOut_buf_cur[1] = pSrc[1]; pOut_buf_cur += counter; } continue; } } #endif while(counter>2) { pOut_buf_cur[0] = pSrc[0]; pOut_buf_cur[1] = pSrc[1]; pOut_buf_cur[2] = pSrc[2]; pOut_buf_cur += 3; pSrc += 3; counter -= 3; } if (counter > 0) { pOut_buf_cur[0] = pSrc[0]; if (counter > 1) pOut_buf_cur[1] = pSrc[1]; pOut_buf_cur += counter; } } } } while (!(r->m_final & 1)); /* Ensure byte alignment and put back any bytes from the bitbuf if we've looked ahead too far on gzip, or other Deflate streams followed by arbitrary data. */ /* I'm being super conservative here. A number of simplifications can be made to the byte alignment part, and the Adler32 check shouldn't ever need to worry about reading from the bitbuf now. */ TINFL_SKIP_BITS(32, num_bits & 7); while ((pIn_buf_cur > pIn_buf_next) && (num_bits >= 8)) { --pIn_buf_cur; num_bits -= 8; } bit_buf &= (tinfl_bit_buf_t)((((mz_uint64)1) << num_bits) - (mz_uint64)1); MZ_ASSERT(!num_bits); /* if this assert fires then we've read beyond the end of non-deflate/zlib streams with following data (such as gzip streams). */ if (decomp_flags & TINFL_FLAG_PARSE_ZLIB_HEADER) { for (counter = 0; counter < 4; ++counter) { mz_uint s; if (num_bits) TINFL_GET_BITS(41, s, 8); else TINFL_GET_BYTE(42, s); r->m_z_adler32 = (r->m_z_adler32 << 8) | s; } } TINFL_CR_RETURN_FOREVER(34, TINFL_STATUS_DONE); TINFL_CR_FINISH common_exit: /* As long as we aren't telling the caller that we NEED more input to make forward progress: */ /* Put back any bytes from the bitbuf in case we've looked ahead too far on gzip, or other Deflate streams followed by arbitrary data. */ /* We need to be very careful here to NOT push back any bytes we definitely know we need to make forward progress, though, or we'll lock the caller up into an inf loop. */ if ((status != TINFL_STATUS_NEEDS_MORE_INPUT) && (status != TINFL_STATUS_FAILED_CANNOT_MAKE_PROGRESS)) { while ((pIn_buf_cur > pIn_buf_next) && (num_bits >= 8)) { --pIn_buf_cur; num_bits -= 8; } } r->m_num_bits = num_bits; r->m_bit_buf = bit_buf & (tinfl_bit_buf_t)((((mz_uint64)1) << num_bits) - (mz_uint64)1); r->m_dist = dist; r->m_counter = counter; r->m_num_extra = num_extra; r->m_dist_from_out_buf_start = dist_from_out_buf_start; *pIn_buf_size = pIn_buf_cur - pIn_buf_next; *pOut_buf_size = pOut_buf_cur - pOut_buf_next; if ((decomp_flags & (TINFL_FLAG_PARSE_ZLIB_HEADER | TINFL_FLAG_COMPUTE_ADLER32)) && (status >= 0)) { const mz_uint8 *ptr = pOut_buf_next; size_t buf_len = *pOut_buf_size; mz_uint32 i, s1 = r->m_check_adler32 & 0xffff, s2 = r->m_check_adler32 >> 16; size_t block_len = buf_len % 5552; while (buf_len) { for (i = 0; i + 7 < block_len; i += 8, ptr += 8) { s1 += ptr[0], s2 += s1; s1 += ptr[1], s2 += s1; s1 += ptr[2], s2 += s1; s1 += ptr[3], s2 += s1; s1 += ptr[4], s2 += s1; s1 += ptr[5], s2 += s1; s1 += ptr[6], s2 += s1; s1 += ptr[7], s2 += s1; } for (; i < block_len; ++i) s1 += *ptr++, s2 += s1; s1 %= 65521U, s2 %= 65521U; buf_len -= block_len; block_len = 5552; } r->m_check_adler32 = (s2 << 16) + s1; if ((status == TINFL_STATUS_DONE) && (decomp_flags & TINFL_FLAG_PARSE_ZLIB_HEADER) && (r->m_check_adler32 != r->m_z_adler32)) status = TINFL_STATUS_ADLER32_MISMATCH; } return status; } /* Higher level helper functions. */ void *tinfl_decompress_mem_to_heap(const void *pSrc_buf, size_t src_buf_len, size_t *pOut_len, int flags) { tinfl_decompressor decomp; void *pBuf = NULL, *pNew_buf; size_t src_buf_ofs = 0, out_buf_capacity = 0; *pOut_len = 0; tinfl_init(&decomp); for (;;) { size_t src_buf_size = src_buf_len - src_buf_ofs, dst_buf_size = out_buf_capacity - *pOut_len, new_out_buf_capacity; tinfl_status status = tinfl_decompress(&decomp, (const mz_uint8 *)pSrc_buf + src_buf_ofs, &src_buf_size, (mz_uint8 *)pBuf, pBuf ? (mz_uint8 *)pBuf + *pOut_len : NULL, &dst_buf_size, (flags & ~TINFL_FLAG_HAS_MORE_INPUT) | TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF); if ((status < 0) || (status == TINFL_STATUS_NEEDS_MORE_INPUT)) { MZ_FREE(pBuf); *pOut_len = 0; return NULL; } src_buf_ofs += src_buf_size; *pOut_len += dst_buf_size; if (status == TINFL_STATUS_DONE) break; new_out_buf_capacity = out_buf_capacity * 2; if (new_out_buf_capacity < 128) new_out_buf_capacity = 128; pNew_buf = MZ_REALLOC(pBuf, new_out_buf_capacity); if (!pNew_buf) { MZ_FREE(pBuf); *pOut_len = 0; return NULL; } pBuf = pNew_buf; out_buf_capacity = new_out_buf_capacity; } return pBuf; } size_t tinfl_decompress_mem_to_mem(void *pOut_buf, size_t out_buf_len, const void *pSrc_buf, size_t src_buf_len, int flags) { tinfl_decompressor decomp; tinfl_status status; tinfl_init(&decomp); status = tinfl_decompress(&decomp, (const mz_uint8 *)pSrc_buf, &src_buf_len, (mz_uint8 *)pOut_buf, (mz_uint8 *)pOut_buf, &out_buf_len, (flags & ~TINFL_FLAG_HAS_MORE_INPUT) | TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF); return (status != TINFL_STATUS_DONE) ? TINFL_DECOMPRESS_MEM_TO_MEM_FAILED : out_buf_len; } int tinfl_decompress_mem_to_callback(const void *pIn_buf, size_t *pIn_buf_size, tinfl_put_buf_func_ptr pPut_buf_func, void *pPut_buf_user, int flags) { int result = 0; tinfl_decompressor decomp; mz_uint8 *pDict = (mz_uint8 *)MZ_MALLOC(TINFL_LZ_DICT_SIZE); size_t in_buf_ofs = 0, dict_ofs = 0; if (!pDict) return TINFL_STATUS_FAILED; tinfl_init(&decomp); for (;;) { size_t in_buf_size = *pIn_buf_size - in_buf_ofs, dst_buf_size = TINFL_LZ_DICT_SIZE - dict_ofs; tinfl_status status = tinfl_decompress(&decomp, (const mz_uint8 *)pIn_buf + in_buf_ofs, &in_buf_size, pDict, pDict + dict_ofs, &dst_buf_size, (flags & ~(TINFL_FLAG_HAS_MORE_INPUT | TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF))); in_buf_ofs += in_buf_size; if ((dst_buf_size) && (!(*pPut_buf_func)(pDict + dict_ofs, (int)dst_buf_size, pPut_buf_user))) break; if (status != TINFL_STATUS_HAS_MORE_OUTPUT) { result = (status == TINFL_STATUS_DONE); break; } dict_ofs = (dict_ofs + dst_buf_size) & (TINFL_LZ_DICT_SIZE - 1); } MZ_FREE(pDict); *pIn_buf_size = in_buf_ofs; return result; } #ifndef MINIZ_NO_MALLOC tinfl_decompressor *tinfl_decompressor_alloc() { tinfl_decompressor *pDecomp = (tinfl_decompressor *)MZ_MALLOC(sizeof(tinfl_decompressor)); if (pDecomp) tinfl_init(pDecomp); return pDecomp; } void tinfl_decompressor_free(tinfl_decompressor *pDecomp) { MZ_FREE(pDecomp); } #endif #ifdef __cplusplus } #endif /************************************************************************** * * Copyright 2013-2014 RAD Game Tools and Valve Software * Copyright 2010-2014 Rich Geldreich and Tenacious Software LLC * Copyright 2016 Martin Raiber * All Rights Reserved. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal * in the Software without restriction, including without limitation the rights * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN * THE SOFTWARE. * **************************************************************************/ #ifndef MINIZ_NO_ARCHIVE_APIS #ifdef __cplusplus extern "C" { #endif /* ------------------- .ZIP archive reading */ #ifdef MINIZ_NO_STDIO #define MZ_FILE void * #else #include #if defined(_MSC_VER) || defined(__MINGW64__) static FILE *mz_fopen(const char *pFilename, const char *pMode) { FILE *pFile = NULL; fopen_s(&pFile, pFilename, pMode); return pFile; } static FILE *mz_freopen(const char *pPath, const char *pMode, FILE *pStream) { FILE *pFile = NULL; if (freopen_s(&pFile, pPath, pMode, pStream)) return NULL; return pFile; } #ifndef MINIZ_NO_TIME #include #endif #define MZ_FOPEN mz_fopen #define MZ_FCLOSE fclose #define MZ_FREAD fread #define MZ_FWRITE fwrite #define MZ_FTELL64 _ftelli64 #define MZ_FSEEK64 _fseeki64 #define MZ_FILE_STAT_STRUCT _stat64 #define MZ_FILE_STAT _stat64 #define MZ_FFLUSH fflush #define MZ_FREOPEN mz_freopen #define MZ_DELETE_FILE remove #elif defined(__MINGW32__) #ifndef MINIZ_NO_TIME #include #endif #define MZ_FOPEN(f, m) fopen(f, m) #define MZ_FCLOSE fclose #define MZ_FREAD fread #define MZ_FWRITE fwrite #define MZ_FTELL64 ftello64 #define MZ_FSEEK64 fseeko64 #define MZ_FILE_STAT_STRUCT _stat #define MZ_FILE_STAT _stat #define MZ_FFLUSH fflush #define MZ_FREOPEN(f, m, s) freopen(f, m, s) #define MZ_DELETE_FILE remove #elif defined(__TINYC__) #ifndef MINIZ_NO_TIME #include #endif #define MZ_FOPEN(f, m) fopen(f, m) #define MZ_FCLOSE fclose #define MZ_FREAD fread #define MZ_FWRITE fwrite #define MZ_FTELL64 ftell #define MZ_FSEEK64 fseek #define MZ_FILE_STAT_STRUCT stat #define MZ_FILE_STAT stat #define MZ_FFLUSH fflush #define MZ_FREOPEN(f, m, s) freopen(f, m, s) #define MZ_DELETE_FILE remove #elif defined(__USE_LARGEFILE64) /* gcc, clang */ #ifndef MINIZ_NO_TIME #include #endif #define MZ_FOPEN(f, m) fopen64(f, m) #define MZ_FCLOSE fclose #define MZ_FREAD fread #define MZ_FWRITE fwrite #define MZ_FTELL64 ftello64 #define MZ_FSEEK64 fseeko64 #define MZ_FILE_STAT_STRUCT stat64 #define MZ_FILE_STAT stat64 #define MZ_FFLUSH fflush #define MZ_FREOPEN(p, m, s) freopen64(p, m, s) #define MZ_DELETE_FILE remove #elif defined(__APPLE__) #ifndef MINIZ_NO_TIME #include #endif #define MZ_FOPEN(f, m) fopen(f, m) #define MZ_FCLOSE fclose #define MZ_FREAD fread #define MZ_FWRITE fwrite #define MZ_FTELL64 ftello #define MZ_FSEEK64 fseeko #define MZ_FILE_STAT_STRUCT stat #define MZ_FILE_STAT stat #define MZ_FFLUSH fflush #define MZ_FREOPEN(p, m, s) freopen(p, m, s) #define MZ_DELETE_FILE remove #else #pragma message("Using fopen, ftello, fseeko, stat() etc. path for file I/O - this path may not support large files.") #ifndef MINIZ_NO_TIME #include #endif #define MZ_FOPEN(f, m) fopen(f, m) #define MZ_FCLOSE fclose #define MZ_FREAD fread #define MZ_FWRITE fwrite #ifdef __STRICT_ANSI__ #define MZ_FTELL64 ftell #define MZ_FSEEK64 fseek #else #define MZ_FTELL64 ftello #define MZ_FSEEK64 fseeko #endif #define MZ_FILE_STAT_STRUCT stat #define MZ_FILE_STAT stat #define MZ_FFLUSH fflush #define MZ_FREOPEN(f, m, s) freopen(f, m, s) #define MZ_DELETE_FILE remove #endif /* #ifdef _MSC_VER */ #endif /* #ifdef MINIZ_NO_STDIO */ #define MZ_TOLOWER(c) ((((c) >= 'A') && ((c) <= 'Z')) ? ((c) - 'A' + 'a') : (c)) /* Various ZIP archive enums. To completely avoid cross platform compiler alignment and platform endian issues, miniz.c doesn't use structs for any of this stuff. */ enum { /* ZIP archive identifiers and record sizes */ MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIG = 0x06054b50, MZ_ZIP_CENTRAL_DIR_HEADER_SIG = 0x02014b50, MZ_ZIP_LOCAL_DIR_HEADER_SIG = 0x04034b50, MZ_ZIP_LOCAL_DIR_HEADER_SIZE = 30, MZ_ZIP_CENTRAL_DIR_HEADER_SIZE = 46, MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE = 22, /* ZIP64 archive identifier and record sizes */ MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIG = 0x06064b50, MZ_ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIG = 0x07064b50, MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIZE = 56, MZ_ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIZE = 20, MZ_ZIP64_EXTENDED_INFORMATION_FIELD_HEADER_ID = 0x0001, MZ_ZIP_DATA_DESCRIPTOR_ID = 0x08074b50, MZ_ZIP_DATA_DESCRIPTER_SIZE64 = 24, MZ_ZIP_DATA_DESCRIPTER_SIZE32 = 16, /* Central directory header record offsets */ MZ_ZIP_CDH_SIG_OFS = 0, MZ_ZIP_CDH_VERSION_MADE_BY_OFS = 4, MZ_ZIP_CDH_VERSION_NEEDED_OFS = 6, MZ_ZIP_CDH_BIT_FLAG_OFS = 8, MZ_ZIP_CDH_METHOD_OFS = 10, MZ_ZIP_CDH_FILE_TIME_OFS = 12, MZ_ZIP_CDH_FILE_DATE_OFS = 14, MZ_ZIP_CDH_CRC32_OFS = 16, MZ_ZIP_CDH_COMPRESSED_SIZE_OFS = 20, MZ_ZIP_CDH_DECOMPRESSED_SIZE_OFS = 24, MZ_ZIP_CDH_FILENAME_LEN_OFS = 28, MZ_ZIP_CDH_EXTRA_LEN_OFS = 30, MZ_ZIP_CDH_COMMENT_LEN_OFS = 32, MZ_ZIP_CDH_DISK_START_OFS = 34, MZ_ZIP_CDH_INTERNAL_ATTR_OFS = 36, MZ_ZIP_CDH_EXTERNAL_ATTR_OFS = 38, MZ_ZIP_CDH_LOCAL_HEADER_OFS = 42, /* Local directory header offsets */ MZ_ZIP_LDH_SIG_OFS = 0, MZ_ZIP_LDH_VERSION_NEEDED_OFS = 4, MZ_ZIP_LDH_BIT_FLAG_OFS = 6, MZ_ZIP_LDH_METHOD_OFS = 8, MZ_ZIP_LDH_FILE_TIME_OFS = 10, MZ_ZIP_LDH_FILE_DATE_OFS = 12, MZ_ZIP_LDH_CRC32_OFS = 14, MZ_ZIP_LDH_COMPRESSED_SIZE_OFS = 18, MZ_ZIP_LDH_DECOMPRESSED_SIZE_OFS = 22, MZ_ZIP_LDH_FILENAME_LEN_OFS = 26, MZ_ZIP_LDH_EXTRA_LEN_OFS = 28, MZ_ZIP_LDH_BIT_FLAG_HAS_LOCATOR = 1 << 3, /* End of central directory offsets */ MZ_ZIP_ECDH_SIG_OFS = 0, MZ_ZIP_ECDH_NUM_THIS_DISK_OFS = 4, MZ_ZIP_ECDH_NUM_DISK_CDIR_OFS = 6, MZ_ZIP_ECDH_CDIR_NUM_ENTRIES_ON_DISK_OFS = 8, MZ_ZIP_ECDH_CDIR_TOTAL_ENTRIES_OFS = 10, MZ_ZIP_ECDH_CDIR_SIZE_OFS = 12, MZ_ZIP_ECDH_CDIR_OFS_OFS = 16, MZ_ZIP_ECDH_COMMENT_SIZE_OFS = 20, /* ZIP64 End of central directory locator offsets */ MZ_ZIP64_ECDL_SIG_OFS = 0, /* 4 bytes */ MZ_ZIP64_ECDL_NUM_DISK_CDIR_OFS = 4, /* 4 bytes */ MZ_ZIP64_ECDL_REL_OFS_TO_ZIP64_ECDR_OFS = 8, /* 8 bytes */ MZ_ZIP64_ECDL_TOTAL_NUMBER_OF_DISKS_OFS = 16, /* 4 bytes */ /* ZIP64 End of central directory header offsets */ MZ_ZIP64_ECDH_SIG_OFS = 0, /* 4 bytes */ MZ_ZIP64_ECDH_SIZE_OF_RECORD_OFS = 4, /* 8 bytes */ MZ_ZIP64_ECDH_VERSION_MADE_BY_OFS = 12, /* 2 bytes */ MZ_ZIP64_ECDH_VERSION_NEEDED_OFS = 14, /* 2 bytes */ MZ_ZIP64_ECDH_NUM_THIS_DISK_OFS = 16, /* 4 bytes */ MZ_ZIP64_ECDH_NUM_DISK_CDIR_OFS = 20, /* 4 bytes */ MZ_ZIP64_ECDH_CDIR_NUM_ENTRIES_ON_DISK_OFS = 24, /* 8 bytes */ MZ_ZIP64_ECDH_CDIR_TOTAL_ENTRIES_OFS = 32, /* 8 bytes */ MZ_ZIP64_ECDH_CDIR_SIZE_OFS = 40, /* 8 bytes */ MZ_ZIP64_ECDH_CDIR_OFS_OFS = 48, /* 8 bytes */ MZ_ZIP_VERSION_MADE_BY_DOS_FILESYSTEM_ID = 0, MZ_ZIP_DOS_DIR_ATTRIBUTE_BITFLAG = 0x10, MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_IS_ENCRYPTED = 1, MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_COMPRESSED_PATCH_FLAG = 32, MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_USES_STRONG_ENCRYPTION = 64, MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_LOCAL_DIR_IS_MASKED = 8192, MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_UTF8 = 1 << 11 }; typedef struct { void *m_p; size_t m_size, m_capacity; mz_uint m_element_size; } mz_zip_array; struct mz_zip_internal_state_tag { mz_zip_array m_central_dir; mz_zip_array m_central_dir_offsets; mz_zip_array m_sorted_central_dir_offsets; /* The flags passed in when the archive is initially opened. */ uint32_t m_init_flags; /* MZ_TRUE if the archive has a zip64 end of central directory headers, etc. */ mz_bool m_zip64; /* MZ_TRUE if we found zip64 extended info in the central directory (m_zip64 will also be slammed to true too, even if we didn't find a zip64 end of central dir header, etc.) */ mz_bool m_zip64_has_extended_info_fields; /* These fields are used by the file, FILE, memory, and memory/heap read/write helpers. */ MZ_FILE *m_pFile; mz_uint64 m_file_archive_start_ofs; void *m_pMem; size_t m_mem_size; size_t m_mem_capacity; }; #define MZ_ZIP_ARRAY_SET_ELEMENT_SIZE(array_ptr, element_size) (array_ptr)->m_element_size = element_size #if defined(DEBUG) || defined(_DEBUG) static MZ_FORCEINLINE mz_uint mz_zip_array_range_check(const mz_zip_array *pArray, mz_uint index) { MZ_ASSERT(index < pArray->m_size); return index; } #define MZ_ZIP_ARRAY_ELEMENT(array_ptr, element_type, index) ((element_type *)((array_ptr)->m_p))[mz_zip_array_range_check(array_ptr, index)] #else #define MZ_ZIP_ARRAY_ELEMENT(array_ptr, element_type, index) ((element_type *)((array_ptr)->m_p))[index] #endif static MZ_FORCEINLINE void mz_zip_array_init(mz_zip_array *pArray, mz_uint32 element_size) { memset(pArray, 0, sizeof(mz_zip_array)); pArray->m_element_size = element_size; } static MZ_FORCEINLINE void mz_zip_array_clear(mz_zip_archive *pZip, mz_zip_array *pArray) { pZip->m_pFree(pZip->m_pAlloc_opaque, pArray->m_p); memset(pArray, 0, sizeof(mz_zip_array)); } static mz_bool mz_zip_array_ensure_capacity(mz_zip_archive *pZip, mz_zip_array *pArray, size_t min_new_capacity, mz_uint growing) { void *pNew_p; size_t new_capacity = min_new_capacity; MZ_ASSERT(pArray->m_element_size); if (pArray->m_capacity >= min_new_capacity) return MZ_TRUE; if (growing) { new_capacity = MZ_MAX(1, pArray->m_capacity); while (new_capacity < min_new_capacity) new_capacity *= 2; } if (NULL == (pNew_p = pZip->m_pRealloc(pZip->m_pAlloc_opaque, pArray->m_p, pArray->m_element_size, new_capacity))) return MZ_FALSE; pArray->m_p = pNew_p; pArray->m_capacity = new_capacity; return MZ_TRUE; } static MZ_FORCEINLINE mz_bool mz_zip_array_reserve(mz_zip_archive *pZip, mz_zip_array *pArray, size_t new_capacity, mz_uint growing) { if (new_capacity > pArray->m_capacity) { if (!mz_zip_array_ensure_capacity(pZip, pArray, new_capacity, growing)) return MZ_FALSE; } return MZ_TRUE; } static MZ_FORCEINLINE mz_bool mz_zip_array_resize(mz_zip_archive *pZip, mz_zip_array *pArray, size_t new_size, mz_uint growing) { if (new_size > pArray->m_capacity) { if (!mz_zip_array_ensure_capacity(pZip, pArray, new_size, growing)) return MZ_FALSE; } pArray->m_size = new_size; return MZ_TRUE; } static MZ_FORCEINLINE mz_bool mz_zip_array_ensure_room(mz_zip_archive *pZip, mz_zip_array *pArray, size_t n) { return mz_zip_array_reserve(pZip, pArray, pArray->m_size + n, MZ_TRUE); } static MZ_FORCEINLINE mz_bool mz_zip_array_push_back(mz_zip_archive *pZip, mz_zip_array *pArray, const void *pElements, size_t n) { size_t orig_size = pArray->m_size; if (!mz_zip_array_resize(pZip, pArray, orig_size + n, MZ_TRUE)) return MZ_FALSE; if (n > 0) memcpy((mz_uint8 *)pArray->m_p + orig_size * pArray->m_element_size, pElements, n * pArray->m_element_size); return MZ_TRUE; } #ifndef MINIZ_NO_TIME static MZ_TIME_T mz_zip_dos_to_time_t(int dos_time, int dos_date) { struct tm tm; memset(&tm, 0, sizeof(tm)); tm.tm_isdst = -1; tm.tm_year = ((dos_date >> 9) & 127) + 1980 - 1900; tm.tm_mon = ((dos_date >> 5) & 15) - 1; tm.tm_mday = dos_date & 31; tm.tm_hour = (dos_time >> 11) & 31; tm.tm_min = (dos_time >> 5) & 63; tm.tm_sec = (dos_time << 1) & 62; return mktime(&tm); } #ifndef MINIZ_NO_ARCHIVE_WRITING_APIS static void mz_zip_time_t_to_dos_time(MZ_TIME_T time, mz_uint16 *pDOS_time, mz_uint16 *pDOS_date) { #ifdef _MSC_VER struct tm tm_struct; struct tm *tm = &tm_struct; errno_t err = localtime_s(tm, &time); if (err) { *pDOS_date = 0; *pDOS_time = 0; return; } #else struct tm *tm = localtime(&time); #endif /* #ifdef _MSC_VER */ *pDOS_time = (mz_uint16)(((tm->tm_hour) << 11) + ((tm->tm_min) << 5) + ((tm->tm_sec) >> 1)); *pDOS_date = (mz_uint16)(((tm->tm_year + 1900 - 1980) << 9) + ((tm->tm_mon + 1) << 5) + tm->tm_mday); } #endif /* MINIZ_NO_ARCHIVE_WRITING_APIS */ #ifndef MINIZ_NO_STDIO #ifndef MINIZ_NO_ARCHIVE_WRITING_APIS static mz_bool mz_zip_get_file_modified_time(const char *pFilename, MZ_TIME_T *pTime) { struct MZ_FILE_STAT_STRUCT file_stat; /* On Linux with x86 glibc, this call will fail on large files (I think >= 0x80000000 bytes) unless you compiled with _LARGEFILE64_SOURCE. Argh. */ if (MZ_FILE_STAT(pFilename, &file_stat) != 0) return MZ_FALSE; *pTime = file_stat.st_mtime; return MZ_TRUE; } #endif /* #ifndef MINIZ_NO_ARCHIVE_WRITING_APIS*/ static mz_bool mz_zip_set_file_times(const char *pFilename, MZ_TIME_T access_time, MZ_TIME_T modified_time) { struct utimbuf t; memset(&t, 0, sizeof(t)); t.actime = access_time; t.modtime = modified_time; return !utime(pFilename, &t); } #endif /* #ifndef MINIZ_NO_STDIO */ #endif /* #ifndef MINIZ_NO_TIME */ static MZ_FORCEINLINE mz_bool mz_zip_set_error(mz_zip_archive *pZip, mz_zip_error err_num) { if (pZip) pZip->m_last_error = err_num; return MZ_FALSE; } static mz_bool mz_zip_reader_init_internal(mz_zip_archive *pZip, mz_uint flags) { (void)flags; if ((!pZip) || (pZip->m_pState) || (pZip->m_zip_mode != MZ_ZIP_MODE_INVALID)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (!pZip->m_pAlloc) pZip->m_pAlloc = miniz_def_alloc_func; if (!pZip->m_pFree) pZip->m_pFree = miniz_def_free_func; if (!pZip->m_pRealloc) pZip->m_pRealloc = miniz_def_realloc_func; pZip->m_archive_size = 0; pZip->m_central_directory_file_ofs = 0; pZip->m_total_files = 0; pZip->m_last_error = MZ_ZIP_NO_ERROR; if (NULL == (pZip->m_pState = (mz_zip_internal_state *)pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, sizeof(mz_zip_internal_state)))) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); memset(pZip->m_pState, 0, sizeof(mz_zip_internal_state)); MZ_ZIP_ARRAY_SET_ELEMENT_SIZE(&pZip->m_pState->m_central_dir, sizeof(mz_uint8)); MZ_ZIP_ARRAY_SET_ELEMENT_SIZE(&pZip->m_pState->m_central_dir_offsets, sizeof(mz_uint32)); MZ_ZIP_ARRAY_SET_ELEMENT_SIZE(&pZip->m_pState->m_sorted_central_dir_offsets, sizeof(mz_uint32)); pZip->m_pState->m_init_flags = flags; pZip->m_pState->m_zip64 = MZ_FALSE; pZip->m_pState->m_zip64_has_extended_info_fields = MZ_FALSE; pZip->m_zip_mode = MZ_ZIP_MODE_READING; return MZ_TRUE; } static MZ_FORCEINLINE mz_bool mz_zip_reader_filename_less(const mz_zip_array *pCentral_dir_array, const mz_zip_array *pCentral_dir_offsets, mz_uint l_index, mz_uint r_index) { const mz_uint8 *pL = &MZ_ZIP_ARRAY_ELEMENT(pCentral_dir_array, mz_uint8, MZ_ZIP_ARRAY_ELEMENT(pCentral_dir_offsets, mz_uint32, l_index)), *pE; const mz_uint8 *pR = &MZ_ZIP_ARRAY_ELEMENT(pCentral_dir_array, mz_uint8, MZ_ZIP_ARRAY_ELEMENT(pCentral_dir_offsets, mz_uint32, r_index)); mz_uint l_len = MZ_READ_LE16(pL + MZ_ZIP_CDH_FILENAME_LEN_OFS), r_len = MZ_READ_LE16(pR + MZ_ZIP_CDH_FILENAME_LEN_OFS); mz_uint8 l = 0, r = 0; pL += MZ_ZIP_CENTRAL_DIR_HEADER_SIZE; pR += MZ_ZIP_CENTRAL_DIR_HEADER_SIZE; pE = pL + MZ_MIN(l_len, r_len); while (pL < pE) { if ((l = MZ_TOLOWER(*pL)) != (r = MZ_TOLOWER(*pR))) break; pL++; pR++; } return (pL == pE) ? (l_len < r_len) : (l < r); } #define MZ_SWAP_UINT32(a, b) \ do \ { \ mz_uint32 t = a; \ a = b; \ b = t; \ } \ MZ_MACRO_END /* Heap sort of lowercased filenames, used to help accelerate plain central directory searches by mz_zip_reader_locate_file(). (Could also use qsort(), but it could allocate memory.) */ static void mz_zip_reader_sort_central_dir_offsets_by_filename(mz_zip_archive *pZip) { mz_zip_internal_state *pState = pZip->m_pState; const mz_zip_array *pCentral_dir_offsets = &pState->m_central_dir_offsets; const mz_zip_array *pCentral_dir = &pState->m_central_dir; mz_uint32 *pIndices; mz_uint32 start, end; const mz_uint32 size = pZip->m_total_files; if (size <= 1U) return; pIndices = &MZ_ZIP_ARRAY_ELEMENT(&pState->m_sorted_central_dir_offsets, mz_uint32, 0); start = (size - 2U) >> 1U; for (;;) { mz_uint64 child, root = start; for (;;) { if ((child = (root << 1U) + 1U) >= size) break; child += (((child + 1U) < size) && (mz_zip_reader_filename_less(pCentral_dir, pCentral_dir_offsets, pIndices[child], pIndices[child + 1U]))); if (!mz_zip_reader_filename_less(pCentral_dir, pCentral_dir_offsets, pIndices[root], pIndices[child])) break; MZ_SWAP_UINT32(pIndices[root], pIndices[child]); root = child; } if (!start) break; start--; } end = size - 1; while (end > 0) { mz_uint64 child, root = 0; MZ_SWAP_UINT32(pIndices[end], pIndices[0]); for (;;) { if ((child = (root << 1U) + 1U) >= end) break; child += (((child + 1U) < end) && mz_zip_reader_filename_less(pCentral_dir, pCentral_dir_offsets, pIndices[child], pIndices[child + 1U])); if (!mz_zip_reader_filename_less(pCentral_dir, pCentral_dir_offsets, pIndices[root], pIndices[child])) break; MZ_SWAP_UINT32(pIndices[root], pIndices[child]); root = child; } end--; } } static mz_bool mz_zip_reader_locate_header_sig(mz_zip_archive *pZip, mz_uint32 record_sig, mz_uint32 record_size, mz_int64 *pOfs) { mz_int64 cur_file_ofs; mz_uint32 buf_u32[4096 / sizeof(mz_uint32)]; mz_uint8 *pBuf = (mz_uint8 *)buf_u32; /* Basic sanity checks - reject files which are too small */ if (pZip->m_archive_size < record_size) return MZ_FALSE; /* Find the record by scanning the file from the end towards the beginning. */ cur_file_ofs = MZ_MAX((mz_int64)pZip->m_archive_size - (mz_int64)sizeof(buf_u32), 0); for (;;) { int i, n = (int)MZ_MIN(sizeof(buf_u32), pZip->m_archive_size - cur_file_ofs); if (pZip->m_pRead(pZip->m_pIO_opaque, cur_file_ofs, pBuf, n) != (mz_uint)n) return MZ_FALSE; for (i = n - 4; i >= 0; --i) { mz_uint s = MZ_READ_LE32(pBuf + i); if (s == record_sig) { if ((pZip->m_archive_size - (cur_file_ofs + i)) >= record_size) break; } } if (i >= 0) { cur_file_ofs += i; break; } /* Give up if we've searched the entire file, or we've gone back "too far" (~64kb) */ if ((!cur_file_ofs) || ((pZip->m_archive_size - cur_file_ofs) >= (MZ_UINT16_MAX + record_size))) return MZ_FALSE; cur_file_ofs = MZ_MAX(cur_file_ofs - (sizeof(buf_u32) - 3), 0); } *pOfs = cur_file_ofs; return MZ_TRUE; } static mz_bool mz_zip_reader_read_central_dir(mz_zip_archive *pZip, mz_uint flags) { mz_uint cdir_size = 0, cdir_entries_on_this_disk = 0, num_this_disk = 0, cdir_disk_index = 0; mz_uint64 cdir_ofs = 0; mz_int64 cur_file_ofs = 0; const mz_uint8 *p; mz_uint32 buf_u32[4096 / sizeof(mz_uint32)]; mz_uint8 *pBuf = (mz_uint8 *)buf_u32; mz_bool sort_central_dir = ((flags & MZ_ZIP_FLAG_DO_NOT_SORT_CENTRAL_DIRECTORY) == 0); mz_uint32 zip64_end_of_central_dir_locator_u32[(MZ_ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIZE + sizeof(mz_uint32) - 1) / sizeof(mz_uint32)]; mz_uint8 *pZip64_locator = (mz_uint8 *)zip64_end_of_central_dir_locator_u32; mz_uint32 zip64_end_of_central_dir_header_u32[(MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIZE + sizeof(mz_uint32) - 1) / sizeof(mz_uint32)]; mz_uint8 *pZip64_end_of_central_dir = (mz_uint8 *)zip64_end_of_central_dir_header_u32; mz_uint64 zip64_end_of_central_dir_ofs = 0; /* Basic sanity checks - reject files which are too small, and check the first 4 bytes of the file to make sure a local header is there. */ if (pZip->m_archive_size < MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE) return mz_zip_set_error(pZip, MZ_ZIP_NOT_AN_ARCHIVE); if (!mz_zip_reader_locate_header_sig(pZip, MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIG, MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE, &cur_file_ofs)) return mz_zip_set_error(pZip, MZ_ZIP_FAILED_FINDING_CENTRAL_DIR); /* Read and verify the end of central directory record. */ if (pZip->m_pRead(pZip->m_pIO_opaque, cur_file_ofs, pBuf, MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE) != MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE) return mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); if (MZ_READ_LE32(pBuf + MZ_ZIP_ECDH_SIG_OFS) != MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIG) return mz_zip_set_error(pZip, MZ_ZIP_NOT_AN_ARCHIVE); if (cur_file_ofs >= (MZ_ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIZE + MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIZE)) { if (pZip->m_pRead(pZip->m_pIO_opaque, cur_file_ofs - MZ_ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIZE, pZip64_locator, MZ_ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIZE) == MZ_ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIZE) { if (MZ_READ_LE32(pZip64_locator + MZ_ZIP64_ECDL_SIG_OFS) == MZ_ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIG) { zip64_end_of_central_dir_ofs = MZ_READ_LE64(pZip64_locator + MZ_ZIP64_ECDL_REL_OFS_TO_ZIP64_ECDR_OFS); if (zip64_end_of_central_dir_ofs > (pZip->m_archive_size - MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIZE)) return mz_zip_set_error(pZip, MZ_ZIP_NOT_AN_ARCHIVE); if (pZip->m_pRead(pZip->m_pIO_opaque, zip64_end_of_central_dir_ofs, pZip64_end_of_central_dir, MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIZE) == MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIZE) { if (MZ_READ_LE32(pZip64_end_of_central_dir + MZ_ZIP64_ECDH_SIG_OFS) == MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIG) { pZip->m_pState->m_zip64 = MZ_TRUE; } } } } } pZip->m_total_files = MZ_READ_LE16(pBuf + MZ_ZIP_ECDH_CDIR_TOTAL_ENTRIES_OFS); cdir_entries_on_this_disk = MZ_READ_LE16(pBuf + MZ_ZIP_ECDH_CDIR_NUM_ENTRIES_ON_DISK_OFS); num_this_disk = MZ_READ_LE16(pBuf + MZ_ZIP_ECDH_NUM_THIS_DISK_OFS); cdir_disk_index = MZ_READ_LE16(pBuf + MZ_ZIP_ECDH_NUM_DISK_CDIR_OFS); cdir_size = MZ_READ_LE32(pBuf + MZ_ZIP_ECDH_CDIR_SIZE_OFS); cdir_ofs = MZ_READ_LE32(pBuf + MZ_ZIP_ECDH_CDIR_OFS_OFS); if (pZip->m_pState->m_zip64) { mz_uint32 zip64_total_num_of_disks = MZ_READ_LE32(pZip64_locator + MZ_ZIP64_ECDL_TOTAL_NUMBER_OF_DISKS_OFS); mz_uint64 zip64_cdir_total_entries = MZ_READ_LE64(pZip64_end_of_central_dir + MZ_ZIP64_ECDH_CDIR_TOTAL_ENTRIES_OFS); mz_uint64 zip64_cdir_total_entries_on_this_disk = MZ_READ_LE64(pZip64_end_of_central_dir + MZ_ZIP64_ECDH_CDIR_NUM_ENTRIES_ON_DISK_OFS); mz_uint64 zip64_size_of_end_of_central_dir_record = MZ_READ_LE64(pZip64_end_of_central_dir + MZ_ZIP64_ECDH_SIZE_OF_RECORD_OFS); mz_uint64 zip64_size_of_central_directory = MZ_READ_LE64(pZip64_end_of_central_dir + MZ_ZIP64_ECDH_CDIR_SIZE_OFS); if (zip64_size_of_end_of_central_dir_record < (MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIZE - 12)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); if (zip64_total_num_of_disks != 1U) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_MULTIDISK); /* Check for miniz's practical limits */ if (zip64_cdir_total_entries > MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_TOO_MANY_FILES); pZip->m_total_files = (mz_uint32)zip64_cdir_total_entries; if (zip64_cdir_total_entries_on_this_disk > MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_TOO_MANY_FILES); cdir_entries_on_this_disk = (mz_uint32)zip64_cdir_total_entries_on_this_disk; /* Check for miniz's current practical limits (sorry, this should be enough for millions of files) */ if (zip64_size_of_central_directory > MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_CDIR_SIZE); cdir_size = (mz_uint32)zip64_size_of_central_directory; num_this_disk = MZ_READ_LE32(pZip64_end_of_central_dir + MZ_ZIP64_ECDH_NUM_THIS_DISK_OFS); cdir_disk_index = MZ_READ_LE32(pZip64_end_of_central_dir + MZ_ZIP64_ECDH_NUM_DISK_CDIR_OFS); cdir_ofs = MZ_READ_LE64(pZip64_end_of_central_dir + MZ_ZIP64_ECDH_CDIR_OFS_OFS); } if (pZip->m_total_files != cdir_entries_on_this_disk) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_MULTIDISK); if (((num_this_disk | cdir_disk_index) != 0) && ((num_this_disk != 1) || (cdir_disk_index != 1))) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_MULTIDISK); if (cdir_size < pZip->m_total_files * MZ_ZIP_CENTRAL_DIR_HEADER_SIZE) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); if ((cdir_ofs + (mz_uint64)cdir_size) > pZip->m_archive_size) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); pZip->m_central_directory_file_ofs = cdir_ofs; if (pZip->m_total_files) { mz_uint i, n; /* Read the entire central directory into a heap block, and allocate another heap block to hold the unsorted central dir file record offsets, and possibly another to hold the sorted indices. */ if ((!mz_zip_array_resize(pZip, &pZip->m_pState->m_central_dir, cdir_size, MZ_FALSE)) || (!mz_zip_array_resize(pZip, &pZip->m_pState->m_central_dir_offsets, pZip->m_total_files, MZ_FALSE))) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); if (sort_central_dir) { if (!mz_zip_array_resize(pZip, &pZip->m_pState->m_sorted_central_dir_offsets, pZip->m_total_files, MZ_FALSE)) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } if (pZip->m_pRead(pZip->m_pIO_opaque, cdir_ofs, pZip->m_pState->m_central_dir.m_p, cdir_size) != cdir_size) return mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); /* Now create an index into the central directory file records, do some basic sanity checking on each record */ p = (const mz_uint8 *)pZip->m_pState->m_central_dir.m_p; for (n = cdir_size, i = 0; i < pZip->m_total_files; ++i) { mz_uint total_header_size, disk_index, bit_flags, filename_size, ext_data_size; mz_uint64 comp_size, decomp_size, local_header_ofs; if ((n < MZ_ZIP_CENTRAL_DIR_HEADER_SIZE) || (MZ_READ_LE32(p) != MZ_ZIP_CENTRAL_DIR_HEADER_SIG)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); MZ_ZIP_ARRAY_ELEMENT(&pZip->m_pState->m_central_dir_offsets, mz_uint32, i) = (mz_uint32)(p - (const mz_uint8 *)pZip->m_pState->m_central_dir.m_p); if (sort_central_dir) MZ_ZIP_ARRAY_ELEMENT(&pZip->m_pState->m_sorted_central_dir_offsets, mz_uint32, i) = i; comp_size = MZ_READ_LE32(p + MZ_ZIP_CDH_COMPRESSED_SIZE_OFS); decomp_size = MZ_READ_LE32(p + MZ_ZIP_CDH_DECOMPRESSED_SIZE_OFS); local_header_ofs = MZ_READ_LE32(p + MZ_ZIP_CDH_LOCAL_HEADER_OFS); filename_size = MZ_READ_LE16(p + MZ_ZIP_CDH_FILENAME_LEN_OFS); ext_data_size = MZ_READ_LE16(p + MZ_ZIP_CDH_EXTRA_LEN_OFS); if ((!pZip->m_pState->m_zip64_has_extended_info_fields) && (ext_data_size) && (MZ_MAX(MZ_MAX(comp_size, decomp_size), local_header_ofs) == MZ_UINT32_MAX)) { /* Attempt to find zip64 extended information field in the entry's extra data */ mz_uint32 extra_size_remaining = ext_data_size; if (extra_size_remaining) { const mz_uint8 *pExtra_data; void* buf = NULL; if (MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + filename_size + ext_data_size > n) { buf = MZ_MALLOC(ext_data_size); if(buf==NULL) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); if (pZip->m_pRead(pZip->m_pIO_opaque, cdir_ofs + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + filename_size, buf, ext_data_size) != ext_data_size) { MZ_FREE(buf); return mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); } pExtra_data = (mz_uint8*)buf; } else { pExtra_data = p + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + filename_size; } do { mz_uint32 field_id; mz_uint32 field_data_size; if (extra_size_remaining < (sizeof(mz_uint16) * 2)) { MZ_FREE(buf); return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); } field_id = MZ_READ_LE16(pExtra_data); field_data_size = MZ_READ_LE16(pExtra_data + sizeof(mz_uint16)); if ((field_data_size + sizeof(mz_uint16) * 2) > extra_size_remaining) { MZ_FREE(buf); return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); } if (field_id == MZ_ZIP64_EXTENDED_INFORMATION_FIELD_HEADER_ID) { /* Ok, the archive didn't have any zip64 headers but it uses a zip64 extended information field so mark it as zip64 anyway (this can occur with infozip's zip util when it reads compresses files from stdin). */ pZip->m_pState->m_zip64 = MZ_TRUE; pZip->m_pState->m_zip64_has_extended_info_fields = MZ_TRUE; break; } pExtra_data += sizeof(mz_uint16) * 2 + field_data_size; extra_size_remaining = extra_size_remaining - sizeof(mz_uint16) * 2 - field_data_size; } while (extra_size_remaining); MZ_FREE(buf); } } /* I've seen archives that aren't marked as zip64 that uses zip64 ext data, argh */ if ((comp_size != MZ_UINT32_MAX) && (decomp_size != MZ_UINT32_MAX)) { if (((!MZ_READ_LE32(p + MZ_ZIP_CDH_METHOD_OFS)) && (decomp_size != comp_size)) || (decomp_size && !comp_size)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); } disk_index = MZ_READ_LE16(p + MZ_ZIP_CDH_DISK_START_OFS); if ((disk_index == MZ_UINT16_MAX) || ((disk_index != num_this_disk) && (disk_index != 1))) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_MULTIDISK); if (comp_size != MZ_UINT32_MAX) { if (((mz_uint64)MZ_READ_LE32(p + MZ_ZIP_CDH_LOCAL_HEADER_OFS) + MZ_ZIP_LOCAL_DIR_HEADER_SIZE + comp_size) > pZip->m_archive_size) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); } bit_flags = MZ_READ_LE16(p + MZ_ZIP_CDH_BIT_FLAG_OFS); if (bit_flags & MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_LOCAL_DIR_IS_MASKED) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_ENCRYPTION); if ((total_header_size = MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + MZ_READ_LE16(p + MZ_ZIP_CDH_FILENAME_LEN_OFS) + MZ_READ_LE16(p + MZ_ZIP_CDH_EXTRA_LEN_OFS) + MZ_READ_LE16(p + MZ_ZIP_CDH_COMMENT_LEN_OFS)) > n) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); n -= total_header_size; p += total_header_size; } } if (sort_central_dir) mz_zip_reader_sort_central_dir_offsets_by_filename(pZip); return MZ_TRUE; } void mz_zip_zero_struct(mz_zip_archive *pZip) { if (pZip) MZ_CLEAR_OBJ(*pZip); } static mz_bool mz_zip_reader_end_internal(mz_zip_archive *pZip, mz_bool set_last_error) { mz_bool status = MZ_TRUE; if (!pZip) return MZ_FALSE; if ((!pZip->m_pState) || (!pZip->m_pAlloc) || (!pZip->m_pFree) || (pZip->m_zip_mode != MZ_ZIP_MODE_READING)) { if (set_last_error) pZip->m_last_error = MZ_ZIP_INVALID_PARAMETER; return MZ_FALSE; } if (pZip->m_pState) { mz_zip_internal_state *pState = pZip->m_pState; pZip->m_pState = NULL; mz_zip_array_clear(pZip, &pState->m_central_dir); mz_zip_array_clear(pZip, &pState->m_central_dir_offsets); mz_zip_array_clear(pZip, &pState->m_sorted_central_dir_offsets); #ifndef MINIZ_NO_STDIO if (pState->m_pFile) { if (pZip->m_zip_type == MZ_ZIP_TYPE_FILE) { if (MZ_FCLOSE(pState->m_pFile) == EOF) { if (set_last_error) pZip->m_last_error = MZ_ZIP_FILE_CLOSE_FAILED; status = MZ_FALSE; } } pState->m_pFile = NULL; } #endif /* #ifndef MINIZ_NO_STDIO */ pZip->m_pFree(pZip->m_pAlloc_opaque, pState); } pZip->m_zip_mode = MZ_ZIP_MODE_INVALID; return status; } mz_bool mz_zip_reader_end(mz_zip_archive *pZip) { return mz_zip_reader_end_internal(pZip, MZ_TRUE); } mz_bool mz_zip_reader_init(mz_zip_archive *pZip, mz_uint64 size, mz_uint flags) { if ((!pZip) || (!pZip->m_pRead)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (!mz_zip_reader_init_internal(pZip, flags)) return MZ_FALSE; pZip->m_zip_type = MZ_ZIP_TYPE_USER; pZip->m_archive_size = size; if (!mz_zip_reader_read_central_dir(pZip, flags)) { mz_zip_reader_end_internal(pZip, MZ_FALSE); return MZ_FALSE; } return MZ_TRUE; } static size_t mz_zip_mem_read_func(void *pOpaque, mz_uint64 file_ofs, void *pBuf, size_t n) { mz_zip_archive *pZip = (mz_zip_archive *)pOpaque; size_t s = (file_ofs >= pZip->m_archive_size) ? 0 : (size_t)MZ_MIN(pZip->m_archive_size - file_ofs, n); memcpy(pBuf, (const mz_uint8 *)pZip->m_pState->m_pMem + file_ofs, s); return s; } mz_bool mz_zip_reader_init_mem(mz_zip_archive *pZip, const void *pMem, size_t size, mz_uint flags) { if (!pMem) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (size < MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE) return mz_zip_set_error(pZip, MZ_ZIP_NOT_AN_ARCHIVE); if (!mz_zip_reader_init_internal(pZip, flags)) return MZ_FALSE; pZip->m_zip_type = MZ_ZIP_TYPE_MEMORY; pZip->m_archive_size = size; pZip->m_pRead = mz_zip_mem_read_func; pZip->m_pIO_opaque = pZip; pZip->m_pNeeds_keepalive = NULL; #ifdef __cplusplus pZip->m_pState->m_pMem = const_cast(pMem); #else pZip->m_pState->m_pMem = (void *)pMem; #endif pZip->m_pState->m_mem_size = size; if (!mz_zip_reader_read_central_dir(pZip, flags)) { mz_zip_reader_end_internal(pZip, MZ_FALSE); return MZ_FALSE; } return MZ_TRUE; } #ifndef MINIZ_NO_STDIO static size_t mz_zip_file_read_func(void *pOpaque, mz_uint64 file_ofs, void *pBuf, size_t n) { mz_zip_archive *pZip = (mz_zip_archive *)pOpaque; mz_int64 cur_ofs = MZ_FTELL64(pZip->m_pState->m_pFile); file_ofs += pZip->m_pState->m_file_archive_start_ofs; if (((mz_int64)file_ofs < 0) || (((cur_ofs != (mz_int64)file_ofs)) && (MZ_FSEEK64(pZip->m_pState->m_pFile, (mz_int64)file_ofs, SEEK_SET)))) return 0; return MZ_FREAD(pBuf, 1, n, pZip->m_pState->m_pFile); } mz_bool mz_zip_reader_init_file(mz_zip_archive *pZip, const char *pFilename, mz_uint32 flags) { return mz_zip_reader_init_file_v2(pZip, pFilename, flags, 0, 0); } mz_bool mz_zip_reader_init_file_v2(mz_zip_archive *pZip, const char *pFilename, mz_uint flags, mz_uint64 file_start_ofs, mz_uint64 archive_size) { mz_uint64 file_size; MZ_FILE *pFile; if ((!pZip) || (!pFilename) || ((archive_size) && (archive_size < MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE))) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); pFile = MZ_FOPEN(pFilename, "rb"); if (!pFile) return mz_zip_set_error(pZip, MZ_ZIP_FILE_OPEN_FAILED); file_size = archive_size; if (!file_size) { if (MZ_FSEEK64(pFile, 0, SEEK_END)) { MZ_FCLOSE(pFile); return mz_zip_set_error(pZip, MZ_ZIP_FILE_SEEK_FAILED); } file_size = MZ_FTELL64(pFile); } /* TODO: Better sanity check archive_size and the # of actual remaining bytes */ if (file_size < MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE) { MZ_FCLOSE(pFile); return mz_zip_set_error(pZip, MZ_ZIP_NOT_AN_ARCHIVE); } if (!mz_zip_reader_init_internal(pZip, flags)) { MZ_FCLOSE(pFile); return MZ_FALSE; } pZip->m_zip_type = MZ_ZIP_TYPE_FILE; pZip->m_pRead = mz_zip_file_read_func; pZip->m_pIO_opaque = pZip; pZip->m_pState->m_pFile = pFile; pZip->m_archive_size = file_size; pZip->m_pState->m_file_archive_start_ofs = file_start_ofs; if (!mz_zip_reader_read_central_dir(pZip, flags)) { mz_zip_reader_end_internal(pZip, MZ_FALSE); return MZ_FALSE; } return MZ_TRUE; } mz_bool mz_zip_reader_init_cfile(mz_zip_archive *pZip, MZ_FILE *pFile, mz_uint64 archive_size, mz_uint flags) { mz_uint64 cur_file_ofs; if ((!pZip) || (!pFile)) return mz_zip_set_error(pZip, MZ_ZIP_FILE_OPEN_FAILED); cur_file_ofs = MZ_FTELL64(pFile); if (!archive_size) { if (MZ_FSEEK64(pFile, 0, SEEK_END)) return mz_zip_set_error(pZip, MZ_ZIP_FILE_SEEK_FAILED); archive_size = MZ_FTELL64(pFile) - cur_file_ofs; if (archive_size < MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE) return mz_zip_set_error(pZip, MZ_ZIP_NOT_AN_ARCHIVE); } if (!mz_zip_reader_init_internal(pZip, flags)) return MZ_FALSE; pZip->m_zip_type = MZ_ZIP_TYPE_CFILE; pZip->m_pRead = mz_zip_file_read_func; pZip->m_pIO_opaque = pZip; pZip->m_pState->m_pFile = pFile; pZip->m_archive_size = archive_size; pZip->m_pState->m_file_archive_start_ofs = cur_file_ofs; if (!mz_zip_reader_read_central_dir(pZip, flags)) { mz_zip_reader_end_internal(pZip, MZ_FALSE); return MZ_FALSE; } return MZ_TRUE; } #endif /* #ifndef MINIZ_NO_STDIO */ static MZ_FORCEINLINE const mz_uint8 *mz_zip_get_cdh(mz_zip_archive *pZip, mz_uint file_index) { if ((!pZip) || (!pZip->m_pState) || (file_index >= pZip->m_total_files)) return NULL; return &MZ_ZIP_ARRAY_ELEMENT(&pZip->m_pState->m_central_dir, mz_uint8, MZ_ZIP_ARRAY_ELEMENT(&pZip->m_pState->m_central_dir_offsets, mz_uint32, file_index)); } mz_bool mz_zip_reader_is_file_encrypted(mz_zip_archive *pZip, mz_uint file_index) { mz_uint m_bit_flag; const mz_uint8 *p = mz_zip_get_cdh(pZip, file_index); if (!p) { mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); return MZ_FALSE; } m_bit_flag = MZ_READ_LE16(p + MZ_ZIP_CDH_BIT_FLAG_OFS); return (m_bit_flag & (MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_IS_ENCRYPTED | MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_USES_STRONG_ENCRYPTION)) != 0; } mz_bool mz_zip_reader_is_file_supported(mz_zip_archive *pZip, mz_uint file_index) { mz_uint bit_flag; mz_uint method; const mz_uint8 *p = mz_zip_get_cdh(pZip, file_index); if (!p) { mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); return MZ_FALSE; } method = MZ_READ_LE16(p + MZ_ZIP_CDH_METHOD_OFS); bit_flag = MZ_READ_LE16(p + MZ_ZIP_CDH_BIT_FLAG_OFS); if ((method != 0) && (method != MZ_DEFLATED)) { mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_METHOD); return MZ_FALSE; } if (bit_flag & (MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_IS_ENCRYPTED | MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_USES_STRONG_ENCRYPTION)) { mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_ENCRYPTION); return MZ_FALSE; } if (bit_flag & MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_COMPRESSED_PATCH_FLAG) { mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_FEATURE); return MZ_FALSE; } return MZ_TRUE; } mz_bool mz_zip_reader_is_file_a_directory(mz_zip_archive *pZip, mz_uint file_index) { mz_uint filename_len, attribute_mapping_id, external_attr; const mz_uint8 *p = mz_zip_get_cdh(pZip, file_index); if (!p) { mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); return MZ_FALSE; } filename_len = MZ_READ_LE16(p + MZ_ZIP_CDH_FILENAME_LEN_OFS); if (filename_len) { if (*(p + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + filename_len - 1) == '/') return MZ_TRUE; } /* Bugfix: This code was also checking if the internal attribute was non-zero, which wasn't correct. */ /* Most/all zip writers (hopefully) set DOS file/directory attributes in the low 16-bits, so check for the DOS directory flag and ignore the source OS ID in the created by field. */ /* FIXME: Remove this check? Is it necessary - we already check the filename. */ attribute_mapping_id = MZ_READ_LE16(p + MZ_ZIP_CDH_VERSION_MADE_BY_OFS) >> 8; (void)attribute_mapping_id; external_attr = MZ_READ_LE32(p + MZ_ZIP_CDH_EXTERNAL_ATTR_OFS); if ((external_attr & MZ_ZIP_DOS_DIR_ATTRIBUTE_BITFLAG) != 0) { return MZ_TRUE; } return MZ_FALSE; } static mz_bool mz_zip_file_stat_internal(mz_zip_archive *pZip, mz_uint file_index, const mz_uint8 *pCentral_dir_header, mz_zip_archive_file_stat *pStat, mz_bool *pFound_zip64_extra_data) { mz_uint n; const mz_uint8 *p = pCentral_dir_header; if (pFound_zip64_extra_data) *pFound_zip64_extra_data = MZ_FALSE; if ((!p) || (!pStat)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); /* Extract fields from the central directory record. */ pStat->m_file_index = file_index; pStat->m_central_dir_ofs = MZ_ZIP_ARRAY_ELEMENT(&pZip->m_pState->m_central_dir_offsets, mz_uint32, file_index); pStat->m_version_made_by = MZ_READ_LE16(p + MZ_ZIP_CDH_VERSION_MADE_BY_OFS); pStat->m_version_needed = MZ_READ_LE16(p + MZ_ZIP_CDH_VERSION_NEEDED_OFS); pStat->m_bit_flag = MZ_READ_LE16(p + MZ_ZIP_CDH_BIT_FLAG_OFS); pStat->m_method = MZ_READ_LE16(p + MZ_ZIP_CDH_METHOD_OFS); #ifndef MINIZ_NO_TIME pStat->m_time = mz_zip_dos_to_time_t(MZ_READ_LE16(p + MZ_ZIP_CDH_FILE_TIME_OFS), MZ_READ_LE16(p + MZ_ZIP_CDH_FILE_DATE_OFS)); #endif pStat->m_crc32 = MZ_READ_LE32(p + MZ_ZIP_CDH_CRC32_OFS); pStat->m_comp_size = MZ_READ_LE32(p + MZ_ZIP_CDH_COMPRESSED_SIZE_OFS); pStat->m_uncomp_size = MZ_READ_LE32(p + MZ_ZIP_CDH_DECOMPRESSED_SIZE_OFS); pStat->m_internal_attr = MZ_READ_LE16(p + MZ_ZIP_CDH_INTERNAL_ATTR_OFS); pStat->m_external_attr = MZ_READ_LE32(p + MZ_ZIP_CDH_EXTERNAL_ATTR_OFS); pStat->m_local_header_ofs = MZ_READ_LE32(p + MZ_ZIP_CDH_LOCAL_HEADER_OFS); /* Copy as much of the filename and comment as possible. */ n = MZ_READ_LE16(p + MZ_ZIP_CDH_FILENAME_LEN_OFS); n = MZ_MIN(n, MZ_ZIP_MAX_ARCHIVE_FILENAME_SIZE - 1); memcpy(pStat->m_filename, p + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE, n); pStat->m_filename[n] = '\0'; n = MZ_READ_LE16(p + MZ_ZIP_CDH_COMMENT_LEN_OFS); n = MZ_MIN(n, MZ_ZIP_MAX_ARCHIVE_FILE_COMMENT_SIZE - 1); pStat->m_comment_size = n; memcpy(pStat->m_comment, p + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + MZ_READ_LE16(p + MZ_ZIP_CDH_FILENAME_LEN_OFS) + MZ_READ_LE16(p + MZ_ZIP_CDH_EXTRA_LEN_OFS), n); pStat->m_comment[n] = '\0'; /* Set some flags for convienance */ pStat->m_is_directory = mz_zip_reader_is_file_a_directory(pZip, file_index); pStat->m_is_encrypted = mz_zip_reader_is_file_encrypted(pZip, file_index); pStat->m_is_supported = mz_zip_reader_is_file_supported(pZip, file_index); /* See if we need to read any zip64 extended information fields. */ /* Confusingly, these zip64 fields can be present even on non-zip64 archives (Debian zip on a huge files from stdin piped to stdout creates them). */ if (MZ_MAX(MZ_MAX(pStat->m_comp_size, pStat->m_uncomp_size), pStat->m_local_header_ofs) == MZ_UINT32_MAX) { /* Attempt to find zip64 extended information field in the entry's extra data */ mz_uint32 extra_size_remaining = MZ_READ_LE16(p + MZ_ZIP_CDH_EXTRA_LEN_OFS); if (extra_size_remaining) { const mz_uint8 *pExtra_data = p + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + MZ_READ_LE16(p + MZ_ZIP_CDH_FILENAME_LEN_OFS); do { mz_uint32 field_id; mz_uint32 field_data_size; if (extra_size_remaining < (sizeof(mz_uint16) * 2)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); field_id = MZ_READ_LE16(pExtra_data); field_data_size = MZ_READ_LE16(pExtra_data + sizeof(mz_uint16)); if ((field_data_size + sizeof(mz_uint16) * 2) > extra_size_remaining) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); if (field_id == MZ_ZIP64_EXTENDED_INFORMATION_FIELD_HEADER_ID) { const mz_uint8 *pField_data = pExtra_data + sizeof(mz_uint16) * 2; mz_uint32 field_data_remaining = field_data_size; if (pFound_zip64_extra_data) *pFound_zip64_extra_data = MZ_TRUE; if (pStat->m_uncomp_size == MZ_UINT32_MAX) { if (field_data_remaining < sizeof(mz_uint64)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); pStat->m_uncomp_size = MZ_READ_LE64(pField_data); pField_data += sizeof(mz_uint64); field_data_remaining -= sizeof(mz_uint64); } if (pStat->m_comp_size == MZ_UINT32_MAX) { if (field_data_remaining < sizeof(mz_uint64)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); pStat->m_comp_size = MZ_READ_LE64(pField_data); pField_data += sizeof(mz_uint64); field_data_remaining -= sizeof(mz_uint64); } if (pStat->m_local_header_ofs == MZ_UINT32_MAX) { if (field_data_remaining < sizeof(mz_uint64)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); pStat->m_local_header_ofs = MZ_READ_LE64(pField_data); pField_data += sizeof(mz_uint64); field_data_remaining -= sizeof(mz_uint64); } break; } pExtra_data += sizeof(mz_uint16) * 2 + field_data_size; extra_size_remaining = extra_size_remaining - sizeof(mz_uint16) * 2 - field_data_size; } while (extra_size_remaining); } } return MZ_TRUE; } static MZ_FORCEINLINE mz_bool mz_zip_string_equal(const char *pA, const char *pB, mz_uint len, mz_uint flags) { mz_uint i; if (flags & MZ_ZIP_FLAG_CASE_SENSITIVE) return 0 == memcmp(pA, pB, len); for (i = 0; i < len; ++i) if (MZ_TOLOWER(pA[i]) != MZ_TOLOWER(pB[i])) return MZ_FALSE; return MZ_TRUE; } static MZ_FORCEINLINE int mz_zip_filename_compare(const mz_zip_array *pCentral_dir_array, const mz_zip_array *pCentral_dir_offsets, mz_uint l_index, const char *pR, mz_uint r_len) { const mz_uint8 *pL = &MZ_ZIP_ARRAY_ELEMENT(pCentral_dir_array, mz_uint8, MZ_ZIP_ARRAY_ELEMENT(pCentral_dir_offsets, mz_uint32, l_index)), *pE; mz_uint l_len = MZ_READ_LE16(pL + MZ_ZIP_CDH_FILENAME_LEN_OFS); mz_uint8 l = 0, r = 0; pL += MZ_ZIP_CENTRAL_DIR_HEADER_SIZE; pE = pL + MZ_MIN(l_len, r_len); while (pL < pE) { if ((l = MZ_TOLOWER(*pL)) != (r = MZ_TOLOWER(*pR))) break; pL++; pR++; } return (pL == pE) ? (int)(l_len - r_len) : (l - r); } static mz_bool mz_zip_locate_file_binary_search(mz_zip_archive *pZip, const char *pFilename, mz_uint32 *pIndex) { mz_zip_internal_state *pState = pZip->m_pState; const mz_zip_array *pCentral_dir_offsets = &pState->m_central_dir_offsets; const mz_zip_array *pCentral_dir = &pState->m_central_dir; mz_uint32 *pIndices = &MZ_ZIP_ARRAY_ELEMENT(&pState->m_sorted_central_dir_offsets, mz_uint32, 0); const uint32_t size = pZip->m_total_files; const mz_uint filename_len = (mz_uint)strlen(pFilename); if (pIndex) *pIndex = 0; if (size) { /* yes I could use uint32_t's, but then we would have to add some special case checks in the loop, argh, and */ /* honestly the major expense here on 32-bit CPU's will still be the filename compare */ mz_int64 l = 0, h = (mz_int64)size - 1; while (l <= h) { mz_int64 m = l + ((h - l) >> 1); uint32_t file_index = pIndices[(uint32_t)m]; int comp = mz_zip_filename_compare(pCentral_dir, pCentral_dir_offsets, file_index, pFilename, filename_len); if (!comp) { if (pIndex) *pIndex = file_index; return MZ_TRUE; } else if (comp < 0) l = m + 1; else h = m - 1; } } return mz_zip_set_error(pZip, MZ_ZIP_FILE_NOT_FOUND); } int mz_zip_reader_locate_file(mz_zip_archive *pZip, const char *pName, const char *pComment, mz_uint flags) { mz_uint32 index; if (!mz_zip_reader_locate_file_v2(pZip, pName, pComment, flags, &index)) return -1; else return (int)index; } mz_bool mz_zip_reader_locate_file_v2(mz_zip_archive *pZip, const char *pName, const char *pComment, mz_uint flags, mz_uint32 *pIndex) { mz_uint file_index; size_t name_len, comment_len; if (pIndex) *pIndex = 0; if ((!pZip) || (!pZip->m_pState) || (!pName)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); /* See if we can use a binary search */ if (((pZip->m_pState->m_init_flags & MZ_ZIP_FLAG_DO_NOT_SORT_CENTRAL_DIRECTORY) == 0) && (pZip->m_zip_mode == MZ_ZIP_MODE_READING) && ((flags & (MZ_ZIP_FLAG_IGNORE_PATH | MZ_ZIP_FLAG_CASE_SENSITIVE)) == 0) && (!pComment) && (pZip->m_pState->m_sorted_central_dir_offsets.m_size)) { return mz_zip_locate_file_binary_search(pZip, pName, pIndex); } /* Locate the entry by scanning the entire central directory */ name_len = strlen(pName); if (name_len > MZ_UINT16_MAX) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); comment_len = pComment ? strlen(pComment) : 0; if (comment_len > MZ_UINT16_MAX) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); for (file_index = 0; file_index < pZip->m_total_files; file_index++) { const mz_uint8 *pHeader = &MZ_ZIP_ARRAY_ELEMENT(&pZip->m_pState->m_central_dir, mz_uint8, MZ_ZIP_ARRAY_ELEMENT(&pZip->m_pState->m_central_dir_offsets, mz_uint32, file_index)); mz_uint filename_len = MZ_READ_LE16(pHeader + MZ_ZIP_CDH_FILENAME_LEN_OFS); const char *pFilename = (const char *)pHeader + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE; if (filename_len < name_len) continue; if (comment_len) { mz_uint file_extra_len = MZ_READ_LE16(pHeader + MZ_ZIP_CDH_EXTRA_LEN_OFS), file_comment_len = MZ_READ_LE16(pHeader + MZ_ZIP_CDH_COMMENT_LEN_OFS); const char *pFile_comment = pFilename + filename_len + file_extra_len; if ((file_comment_len != comment_len) || (!mz_zip_string_equal(pComment, pFile_comment, file_comment_len, flags))) continue; } if ((flags & MZ_ZIP_FLAG_IGNORE_PATH) && (filename_len)) { int ofs = filename_len - 1; do { if ((pFilename[ofs] == '/') || (pFilename[ofs] == '\\') || (pFilename[ofs] == ':')) break; } while (--ofs >= 0); ofs++; pFilename += ofs; filename_len -= ofs; } if ((filename_len == name_len) && (mz_zip_string_equal(pName, pFilename, filename_len, flags))) { if (pIndex) *pIndex = file_index; return MZ_TRUE; } } return mz_zip_set_error(pZip, MZ_ZIP_FILE_NOT_FOUND); } mz_bool mz_zip_reader_extract_to_mem_no_alloc(mz_zip_archive *pZip, mz_uint file_index, void *pBuf, size_t buf_size, mz_uint flags, void *pUser_read_buf, size_t user_read_buf_size) { int status = TINFL_STATUS_DONE; mz_uint64 needed_size, cur_file_ofs, comp_remaining, out_buf_ofs = 0, read_buf_size, read_buf_ofs = 0, read_buf_avail; mz_zip_archive_file_stat file_stat; void *pRead_buf; mz_uint32 local_header_u32[(MZ_ZIP_LOCAL_DIR_HEADER_SIZE + sizeof(mz_uint32) - 1) / sizeof(mz_uint32)]; mz_uint8 *pLocal_header = (mz_uint8 *)local_header_u32; tinfl_decompressor inflator; if ((!pZip) || (!pZip->m_pState) || ((buf_size) && (!pBuf)) || ((user_read_buf_size) && (!pUser_read_buf)) || (!pZip->m_pRead)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (!mz_zip_reader_file_stat(pZip, file_index, &file_stat)) return MZ_FALSE; /* A directory or zero length file */ if ((file_stat.m_is_directory) || (!file_stat.m_comp_size)) return MZ_TRUE; /* Encryption and patch files are not supported. */ if (file_stat.m_bit_flag & (MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_IS_ENCRYPTED | MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_USES_STRONG_ENCRYPTION | MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_COMPRESSED_PATCH_FLAG)) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_ENCRYPTION); /* This function only supports decompressing stored and deflate. */ if ((!(flags & MZ_ZIP_FLAG_COMPRESSED_DATA)) && (file_stat.m_method != 0) && (file_stat.m_method != MZ_DEFLATED)) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_METHOD); /* Ensure supplied output buffer is large enough. */ needed_size = (flags & MZ_ZIP_FLAG_COMPRESSED_DATA) ? file_stat.m_comp_size : file_stat.m_uncomp_size; if (buf_size < needed_size) return mz_zip_set_error(pZip, MZ_ZIP_BUF_TOO_SMALL); /* Read and parse the local directory entry. */ cur_file_ofs = file_stat.m_local_header_ofs; if (pZip->m_pRead(pZip->m_pIO_opaque, cur_file_ofs, pLocal_header, MZ_ZIP_LOCAL_DIR_HEADER_SIZE) != MZ_ZIP_LOCAL_DIR_HEADER_SIZE) return mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); if (MZ_READ_LE32(pLocal_header) != MZ_ZIP_LOCAL_DIR_HEADER_SIG) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); cur_file_ofs += MZ_ZIP_LOCAL_DIR_HEADER_SIZE + MZ_READ_LE16(pLocal_header + MZ_ZIP_LDH_FILENAME_LEN_OFS) + MZ_READ_LE16(pLocal_header + MZ_ZIP_LDH_EXTRA_LEN_OFS); if ((cur_file_ofs + file_stat.m_comp_size) > pZip->m_archive_size) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); if ((flags & MZ_ZIP_FLAG_COMPRESSED_DATA) || (!file_stat.m_method)) { /* The file is stored or the caller has requested the compressed data. */ if (pZip->m_pRead(pZip->m_pIO_opaque, cur_file_ofs, pBuf, (size_t)needed_size) != needed_size) return mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); #ifndef MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS if ((flags & MZ_ZIP_FLAG_COMPRESSED_DATA) == 0) { if (mz_crc32(MZ_CRC32_INIT, (const mz_uint8 *)pBuf, (size_t)file_stat.m_uncomp_size) != file_stat.m_crc32) return mz_zip_set_error(pZip, MZ_ZIP_CRC_CHECK_FAILED); } #endif return MZ_TRUE; } /* Decompress the file either directly from memory or from a file input buffer. */ tinfl_init(&inflator); if (pZip->m_pState->m_pMem) { /* Read directly from the archive in memory. */ pRead_buf = (mz_uint8 *)pZip->m_pState->m_pMem + cur_file_ofs; read_buf_size = read_buf_avail = file_stat.m_comp_size; comp_remaining = 0; } else if (pUser_read_buf) { /* Use a user provided read buffer. */ if (!user_read_buf_size) return MZ_FALSE; pRead_buf = (mz_uint8 *)pUser_read_buf; read_buf_size = user_read_buf_size; read_buf_avail = 0; comp_remaining = file_stat.m_comp_size; } else { /* Temporarily allocate a read buffer. */ read_buf_size = MZ_MIN(file_stat.m_comp_size, (mz_uint64)MZ_ZIP_MAX_IO_BUF_SIZE); if (((sizeof(size_t) == sizeof(mz_uint32))) && (read_buf_size > 0x7FFFFFFF)) return mz_zip_set_error(pZip, MZ_ZIP_INTERNAL_ERROR); if (NULL == (pRead_buf = pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, (size_t)read_buf_size))) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); read_buf_avail = 0; comp_remaining = file_stat.m_comp_size; } do { /* The size_t cast here should be OK because we've verified that the output buffer is >= file_stat.m_uncomp_size above */ size_t in_buf_size, out_buf_size = (size_t)(file_stat.m_uncomp_size - out_buf_ofs); if ((!read_buf_avail) && (!pZip->m_pState->m_pMem)) { read_buf_avail = MZ_MIN(read_buf_size, comp_remaining); if (pZip->m_pRead(pZip->m_pIO_opaque, cur_file_ofs, pRead_buf, (size_t)read_buf_avail) != read_buf_avail) { status = TINFL_STATUS_FAILED; mz_zip_set_error(pZip, MZ_ZIP_DECOMPRESSION_FAILED); break; } cur_file_ofs += read_buf_avail; comp_remaining -= read_buf_avail; read_buf_ofs = 0; } in_buf_size = (size_t)read_buf_avail; status = tinfl_decompress(&inflator, (mz_uint8 *)pRead_buf + read_buf_ofs, &in_buf_size, (mz_uint8 *)pBuf, (mz_uint8 *)pBuf + out_buf_ofs, &out_buf_size, TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF | (comp_remaining ? TINFL_FLAG_HAS_MORE_INPUT : 0)); read_buf_avail -= in_buf_size; read_buf_ofs += in_buf_size; out_buf_ofs += out_buf_size; } while (status == TINFL_STATUS_NEEDS_MORE_INPUT); if (status == TINFL_STATUS_DONE) { /* Make sure the entire file was decompressed, and check its CRC. */ if (out_buf_ofs != file_stat.m_uncomp_size) { mz_zip_set_error(pZip, MZ_ZIP_UNEXPECTED_DECOMPRESSED_SIZE); status = TINFL_STATUS_FAILED; } #ifndef MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS else if (mz_crc32(MZ_CRC32_INIT, (const mz_uint8 *)pBuf, (size_t)file_stat.m_uncomp_size) != file_stat.m_crc32) { mz_zip_set_error(pZip, MZ_ZIP_CRC_CHECK_FAILED); status = TINFL_STATUS_FAILED; } #endif } if ((!pZip->m_pState->m_pMem) && (!pUser_read_buf)) pZip->m_pFree(pZip->m_pAlloc_opaque, pRead_buf); return status == TINFL_STATUS_DONE; } mz_bool mz_zip_reader_extract_file_to_mem_no_alloc(mz_zip_archive *pZip, const char *pFilename, void *pBuf, size_t buf_size, mz_uint flags, void *pUser_read_buf, size_t user_read_buf_size) { mz_uint32 file_index; if (!mz_zip_reader_locate_file_v2(pZip, pFilename, NULL, flags, &file_index)) return MZ_FALSE; return mz_zip_reader_extract_to_mem_no_alloc(pZip, file_index, pBuf, buf_size, flags, pUser_read_buf, user_read_buf_size); } mz_bool mz_zip_reader_extract_to_mem(mz_zip_archive *pZip, mz_uint file_index, void *pBuf, size_t buf_size, mz_uint flags) { return mz_zip_reader_extract_to_mem_no_alloc(pZip, file_index, pBuf, buf_size, flags, NULL, 0); } mz_bool mz_zip_reader_extract_file_to_mem(mz_zip_archive *pZip, const char *pFilename, void *pBuf, size_t buf_size, mz_uint flags) { return mz_zip_reader_extract_file_to_mem_no_alloc(pZip, pFilename, pBuf, buf_size, flags, NULL, 0); } void *mz_zip_reader_extract_to_heap(mz_zip_archive *pZip, mz_uint file_index, size_t *pSize, mz_uint flags) { mz_uint64 comp_size, uncomp_size, alloc_size; const mz_uint8 *p = mz_zip_get_cdh(pZip, file_index); void *pBuf; if (pSize) *pSize = 0; if (!p) { mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); return NULL; } comp_size = MZ_READ_LE32(p + MZ_ZIP_CDH_COMPRESSED_SIZE_OFS); uncomp_size = MZ_READ_LE32(p + MZ_ZIP_CDH_DECOMPRESSED_SIZE_OFS); alloc_size = (flags & MZ_ZIP_FLAG_COMPRESSED_DATA) ? comp_size : uncomp_size; if (((sizeof(size_t) == sizeof(mz_uint32))) && (alloc_size > 0x7FFFFFFF)) { mz_zip_set_error(pZip, MZ_ZIP_INTERNAL_ERROR); return NULL; } if (NULL == (pBuf = pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, (size_t)alloc_size))) { mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); return NULL; } if (!mz_zip_reader_extract_to_mem(pZip, file_index, pBuf, (size_t)alloc_size, flags)) { pZip->m_pFree(pZip->m_pAlloc_opaque, pBuf); return NULL; } if (pSize) *pSize = (size_t)alloc_size; return pBuf; } void *mz_zip_reader_extract_file_to_heap(mz_zip_archive *pZip, const char *pFilename, size_t *pSize, mz_uint flags) { mz_uint32 file_index; if (!mz_zip_reader_locate_file_v2(pZip, pFilename, NULL, flags, &file_index)) { if (pSize) *pSize = 0; return MZ_FALSE; } return mz_zip_reader_extract_to_heap(pZip, file_index, pSize, flags); } mz_bool mz_zip_reader_extract_to_callback(mz_zip_archive *pZip, mz_uint file_index, mz_file_write_func pCallback, void *pOpaque, mz_uint flags) { int status = TINFL_STATUS_DONE; #ifndef MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS mz_uint file_crc32 = MZ_CRC32_INIT; #endif mz_uint64 read_buf_size, read_buf_ofs = 0, read_buf_avail, comp_remaining, out_buf_ofs = 0, cur_file_ofs; mz_zip_archive_file_stat file_stat; void *pRead_buf = NULL; void *pWrite_buf = NULL; mz_uint32 local_header_u32[(MZ_ZIP_LOCAL_DIR_HEADER_SIZE + sizeof(mz_uint32) - 1) / sizeof(mz_uint32)]; mz_uint8 *pLocal_header = (mz_uint8 *)local_header_u32; if ((!pZip) || (!pZip->m_pState) || (!pCallback) || (!pZip->m_pRead)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (!mz_zip_reader_file_stat(pZip, file_index, &file_stat)) return MZ_FALSE; /* A directory or zero length file */ if ((file_stat.m_is_directory) || (!file_stat.m_comp_size)) return MZ_TRUE; /* Encryption and patch files are not supported. */ if (file_stat.m_bit_flag & (MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_IS_ENCRYPTED | MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_USES_STRONG_ENCRYPTION | MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_COMPRESSED_PATCH_FLAG)) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_ENCRYPTION); /* This function only supports decompressing stored and deflate. */ if ((!(flags & MZ_ZIP_FLAG_COMPRESSED_DATA)) && (file_stat.m_method != 0) && (file_stat.m_method != MZ_DEFLATED)) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_METHOD); /* Read and do some minimal validation of the local directory entry (this doesn't crack the zip64 stuff, which we already have from the central dir) */ cur_file_ofs = file_stat.m_local_header_ofs; if (pZip->m_pRead(pZip->m_pIO_opaque, cur_file_ofs, pLocal_header, MZ_ZIP_LOCAL_DIR_HEADER_SIZE) != MZ_ZIP_LOCAL_DIR_HEADER_SIZE) return mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); if (MZ_READ_LE32(pLocal_header) != MZ_ZIP_LOCAL_DIR_HEADER_SIG) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); cur_file_ofs += MZ_ZIP_LOCAL_DIR_HEADER_SIZE + MZ_READ_LE16(pLocal_header + MZ_ZIP_LDH_FILENAME_LEN_OFS) + MZ_READ_LE16(pLocal_header + MZ_ZIP_LDH_EXTRA_LEN_OFS); if ((cur_file_ofs + file_stat.m_comp_size) > pZip->m_archive_size) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); /* Decompress the file either directly from memory or from a file input buffer. */ if (pZip->m_pState->m_pMem) { pRead_buf = (mz_uint8 *)pZip->m_pState->m_pMem + cur_file_ofs; read_buf_size = read_buf_avail = file_stat.m_comp_size; comp_remaining = 0; } else { read_buf_size = MZ_MIN(file_stat.m_comp_size, (mz_uint64)MZ_ZIP_MAX_IO_BUF_SIZE); if (NULL == (pRead_buf = pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, (size_t)read_buf_size))) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); read_buf_avail = 0; comp_remaining = file_stat.m_comp_size; } if ((flags & MZ_ZIP_FLAG_COMPRESSED_DATA) || (!file_stat.m_method)) { /* The file is stored or the caller has requested the compressed data. */ if (pZip->m_pState->m_pMem) { if (((sizeof(size_t) == sizeof(mz_uint32))) && (file_stat.m_comp_size > MZ_UINT32_MAX)) return mz_zip_set_error(pZip, MZ_ZIP_INTERNAL_ERROR); if (pCallback(pOpaque, out_buf_ofs, pRead_buf, (size_t)file_stat.m_comp_size) != file_stat.m_comp_size) { mz_zip_set_error(pZip, MZ_ZIP_WRITE_CALLBACK_FAILED); status = TINFL_STATUS_FAILED; } else if (!(flags & MZ_ZIP_FLAG_COMPRESSED_DATA)) { #ifndef MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS file_crc32 = (mz_uint32)mz_crc32(file_crc32, (const mz_uint8 *)pRead_buf, (size_t)file_stat.m_comp_size); #endif } cur_file_ofs += file_stat.m_comp_size; out_buf_ofs += file_stat.m_comp_size; comp_remaining = 0; } else { while (comp_remaining) { read_buf_avail = MZ_MIN(read_buf_size, comp_remaining); if (pZip->m_pRead(pZip->m_pIO_opaque, cur_file_ofs, pRead_buf, (size_t)read_buf_avail) != read_buf_avail) { mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); status = TINFL_STATUS_FAILED; break; } #ifndef MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS if (!(flags & MZ_ZIP_FLAG_COMPRESSED_DATA)) { file_crc32 = (mz_uint32)mz_crc32(file_crc32, (const mz_uint8 *)pRead_buf, (size_t)read_buf_avail); } #endif if (pCallback(pOpaque, out_buf_ofs, pRead_buf, (size_t)read_buf_avail) != read_buf_avail) { mz_zip_set_error(pZip, MZ_ZIP_WRITE_CALLBACK_FAILED); status = TINFL_STATUS_FAILED; break; } cur_file_ofs += read_buf_avail; out_buf_ofs += read_buf_avail; comp_remaining -= read_buf_avail; } } } else { tinfl_decompressor inflator; tinfl_init(&inflator); if (NULL == (pWrite_buf = pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, TINFL_LZ_DICT_SIZE))) { mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); status = TINFL_STATUS_FAILED; } else { do { mz_uint8 *pWrite_buf_cur = (mz_uint8 *)pWrite_buf + (out_buf_ofs & (TINFL_LZ_DICT_SIZE - 1)); size_t in_buf_size, out_buf_size = TINFL_LZ_DICT_SIZE - (out_buf_ofs & (TINFL_LZ_DICT_SIZE - 1)); if ((!read_buf_avail) && (!pZip->m_pState->m_pMem)) { read_buf_avail = MZ_MIN(read_buf_size, comp_remaining); if (pZip->m_pRead(pZip->m_pIO_opaque, cur_file_ofs, pRead_buf, (size_t)read_buf_avail) != read_buf_avail) { mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); status = TINFL_STATUS_FAILED; break; } cur_file_ofs += read_buf_avail; comp_remaining -= read_buf_avail; read_buf_ofs = 0; } in_buf_size = (size_t)read_buf_avail; status = tinfl_decompress(&inflator, (const mz_uint8 *)pRead_buf + read_buf_ofs, &in_buf_size, (mz_uint8 *)pWrite_buf, pWrite_buf_cur, &out_buf_size, comp_remaining ? TINFL_FLAG_HAS_MORE_INPUT : 0); read_buf_avail -= in_buf_size; read_buf_ofs += in_buf_size; if (out_buf_size) { if (pCallback(pOpaque, out_buf_ofs, pWrite_buf_cur, out_buf_size) != out_buf_size) { mz_zip_set_error(pZip, MZ_ZIP_WRITE_CALLBACK_FAILED); status = TINFL_STATUS_FAILED; break; } #ifndef MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS file_crc32 = (mz_uint32)mz_crc32(file_crc32, pWrite_buf_cur, out_buf_size); #endif if ((out_buf_ofs += out_buf_size) > file_stat.m_uncomp_size) { mz_zip_set_error(pZip, MZ_ZIP_DECOMPRESSION_FAILED); status = TINFL_STATUS_FAILED; break; } } } while ((status == TINFL_STATUS_NEEDS_MORE_INPUT) || (status == TINFL_STATUS_HAS_MORE_OUTPUT)); } } if ((status == TINFL_STATUS_DONE) && (!(flags & MZ_ZIP_FLAG_COMPRESSED_DATA))) { /* Make sure the entire file was decompressed, and check its CRC. */ if (out_buf_ofs != file_stat.m_uncomp_size) { mz_zip_set_error(pZip, MZ_ZIP_UNEXPECTED_DECOMPRESSED_SIZE); status = TINFL_STATUS_FAILED; } #ifndef MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS else if (file_crc32 != file_stat.m_crc32) { mz_zip_set_error(pZip, MZ_ZIP_DECOMPRESSION_FAILED); status = TINFL_STATUS_FAILED; } #endif } if (!pZip->m_pState->m_pMem) pZip->m_pFree(pZip->m_pAlloc_opaque, pRead_buf); if (pWrite_buf) pZip->m_pFree(pZip->m_pAlloc_opaque, pWrite_buf); return status == TINFL_STATUS_DONE; } mz_bool mz_zip_reader_extract_file_to_callback(mz_zip_archive *pZip, const char *pFilename, mz_file_write_func pCallback, void *pOpaque, mz_uint flags) { mz_uint32 file_index; if (!mz_zip_reader_locate_file_v2(pZip, pFilename, NULL, flags, &file_index)) return MZ_FALSE; return mz_zip_reader_extract_to_callback(pZip, file_index, pCallback, pOpaque, flags); } mz_zip_reader_extract_iter_state* mz_zip_reader_extract_iter_new(mz_zip_archive *pZip, mz_uint file_index, mz_uint flags) { mz_zip_reader_extract_iter_state *pState; mz_uint32 local_header_u32[(MZ_ZIP_LOCAL_DIR_HEADER_SIZE + sizeof(mz_uint32) - 1) / sizeof(mz_uint32)]; mz_uint8 *pLocal_header = (mz_uint8 *)local_header_u32; /* Argument sanity check */ if ((!pZip) || (!pZip->m_pState)) return NULL; /* Allocate an iterator status structure */ pState = (mz_zip_reader_extract_iter_state*)pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, sizeof(mz_zip_reader_extract_iter_state)); if (!pState) { mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); return NULL; } /* Fetch file details */ if (!mz_zip_reader_file_stat(pZip, file_index, &pState->file_stat)) { pZip->m_pFree(pZip->m_pAlloc_opaque, pState); return NULL; } /* Encryption and patch files are not supported. */ if (pState->file_stat.m_bit_flag & (MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_IS_ENCRYPTED | MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_USES_STRONG_ENCRYPTION | MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_COMPRESSED_PATCH_FLAG)) { mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_ENCRYPTION); pZip->m_pFree(pZip->m_pAlloc_opaque, pState); return NULL; } /* This function only supports decompressing stored and deflate. */ if ((!(flags & MZ_ZIP_FLAG_COMPRESSED_DATA)) && (pState->file_stat.m_method != 0) && (pState->file_stat.m_method != MZ_DEFLATED)) { mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_METHOD); pZip->m_pFree(pZip->m_pAlloc_opaque, pState); return NULL; } /* Init state - save args */ pState->pZip = pZip; pState->flags = flags; /* Init state - reset variables to defaults */ pState->status = TINFL_STATUS_DONE; #ifndef MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS pState->file_crc32 = MZ_CRC32_INIT; #endif pState->read_buf_ofs = 0; pState->out_buf_ofs = 0; pState->pRead_buf = NULL; pState->pWrite_buf = NULL; pState->out_blk_remain = 0; /* Read and parse the local directory entry. */ pState->cur_file_ofs = pState->file_stat.m_local_header_ofs; if (pZip->m_pRead(pZip->m_pIO_opaque, pState->cur_file_ofs, pLocal_header, MZ_ZIP_LOCAL_DIR_HEADER_SIZE) != MZ_ZIP_LOCAL_DIR_HEADER_SIZE) { mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); pZip->m_pFree(pZip->m_pAlloc_opaque, pState); return NULL; } if (MZ_READ_LE32(pLocal_header) != MZ_ZIP_LOCAL_DIR_HEADER_SIG) { mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); pZip->m_pFree(pZip->m_pAlloc_opaque, pState); return NULL; } pState->cur_file_ofs += MZ_ZIP_LOCAL_DIR_HEADER_SIZE + MZ_READ_LE16(pLocal_header + MZ_ZIP_LDH_FILENAME_LEN_OFS) + MZ_READ_LE16(pLocal_header + MZ_ZIP_LDH_EXTRA_LEN_OFS); if ((pState->cur_file_ofs + pState->file_stat.m_comp_size) > pZip->m_archive_size) { mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); pZip->m_pFree(pZip->m_pAlloc_opaque, pState); return NULL; } /* Decompress the file either directly from memory or from a file input buffer. */ if (pZip->m_pState->m_pMem) { pState->pRead_buf = (mz_uint8 *)pZip->m_pState->m_pMem + pState->cur_file_ofs; pState->read_buf_size = pState->read_buf_avail = pState->file_stat.m_comp_size; pState->comp_remaining = pState->file_stat.m_comp_size; } else { if (!((flags & MZ_ZIP_FLAG_COMPRESSED_DATA) || (!pState->file_stat.m_method))) { /* Decompression required, therefore intermediate read buffer required */ pState->read_buf_size = MZ_MIN(pState->file_stat.m_comp_size, (mz_uint64)MZ_ZIP_MAX_IO_BUF_SIZE); if (NULL == (pState->pRead_buf = pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, (size_t)pState->read_buf_size))) { mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); pZip->m_pFree(pZip->m_pAlloc_opaque, pState); return NULL; } } else { /* Decompression not required - we will be reading directly into user buffer, no temp buf required */ pState->read_buf_size = 0; } pState->read_buf_avail = 0; pState->comp_remaining = pState->file_stat.m_comp_size; } if (!((flags & MZ_ZIP_FLAG_COMPRESSED_DATA) || (!pState->file_stat.m_method))) { /* Decompression required, init decompressor */ tinfl_init( &pState->inflator ); /* Allocate write buffer */ if (NULL == (pState->pWrite_buf = pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, TINFL_LZ_DICT_SIZE))) { mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); if (pState->pRead_buf) pZip->m_pFree(pZip->m_pAlloc_opaque, pState->pRead_buf); pZip->m_pFree(pZip->m_pAlloc_opaque, pState); return NULL; } } return pState; } mz_zip_reader_extract_iter_state* mz_zip_reader_extract_file_iter_new(mz_zip_archive *pZip, const char *pFilename, mz_uint flags) { mz_uint32 file_index; /* Locate file index by name */ if (!mz_zip_reader_locate_file_v2(pZip, pFilename, NULL, flags, &file_index)) return NULL; /* Construct iterator */ return mz_zip_reader_extract_iter_new(pZip, file_index, flags); } size_t mz_zip_reader_extract_iter_read(mz_zip_reader_extract_iter_state* pState, void* pvBuf, size_t buf_size) { size_t copied_to_caller = 0; /* Argument sanity check */ if ((!pState) || (!pState->pZip) || (!pState->pZip->m_pState) || (!pvBuf)) return 0; if ((pState->flags & MZ_ZIP_FLAG_COMPRESSED_DATA) || (!pState->file_stat.m_method)) { /* The file is stored or the caller has requested the compressed data, calc amount to return. */ copied_to_caller = (size_t)MZ_MIN( buf_size, pState->comp_remaining ); /* Zip is in memory....or requires reading from a file? */ if (pState->pZip->m_pState->m_pMem) { /* Copy data to caller's buffer */ memcpy( pvBuf, pState->pRead_buf, copied_to_caller ); pState->pRead_buf = ((mz_uint8*)pState->pRead_buf) + copied_to_caller; } else { /* Read directly into caller's buffer */ if (pState->pZip->m_pRead(pState->pZip->m_pIO_opaque, pState->cur_file_ofs, pvBuf, copied_to_caller) != copied_to_caller) { /* Failed to read all that was asked for, flag failure and alert user */ mz_zip_set_error(pState->pZip, MZ_ZIP_FILE_READ_FAILED); pState->status = TINFL_STATUS_FAILED; copied_to_caller = 0; } } #ifndef MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS /* Compute CRC if not returning compressed data only */ if (!(pState->flags & MZ_ZIP_FLAG_COMPRESSED_DATA)) pState->file_crc32 = (mz_uint32)mz_crc32(pState->file_crc32, (const mz_uint8 *)pvBuf, copied_to_caller); #endif /* Advance offsets, dec counters */ pState->cur_file_ofs += copied_to_caller; pState->out_buf_ofs += copied_to_caller; pState->comp_remaining -= copied_to_caller; } else { do { /* Calc ptr to write buffer - given current output pos and block size */ mz_uint8 *pWrite_buf_cur = (mz_uint8 *)pState->pWrite_buf + (pState->out_buf_ofs & (TINFL_LZ_DICT_SIZE - 1)); /* Calc max output size - given current output pos and block size */ size_t in_buf_size, out_buf_size = TINFL_LZ_DICT_SIZE - (pState->out_buf_ofs & (TINFL_LZ_DICT_SIZE - 1)); if (!pState->out_blk_remain) { /* Read more data from file if none available (and reading from file) */ if ((!pState->read_buf_avail) && (!pState->pZip->m_pState->m_pMem)) { /* Calc read size */ pState->read_buf_avail = MZ_MIN(pState->read_buf_size, pState->comp_remaining); if (pState->pZip->m_pRead(pState->pZip->m_pIO_opaque, pState->cur_file_ofs, pState->pRead_buf, (size_t)pState->read_buf_avail) != pState->read_buf_avail) { mz_zip_set_error(pState->pZip, MZ_ZIP_FILE_READ_FAILED); pState->status = TINFL_STATUS_FAILED; break; } /* Advance offsets, dec counters */ pState->cur_file_ofs += pState->read_buf_avail; pState->comp_remaining -= pState->read_buf_avail; pState->read_buf_ofs = 0; } /* Perform decompression */ in_buf_size = (size_t)pState->read_buf_avail; pState->status = tinfl_decompress(&pState->inflator, (const mz_uint8 *)pState->pRead_buf + pState->read_buf_ofs, &in_buf_size, (mz_uint8 *)pState->pWrite_buf, pWrite_buf_cur, &out_buf_size, pState->comp_remaining ? TINFL_FLAG_HAS_MORE_INPUT : 0); pState->read_buf_avail -= in_buf_size; pState->read_buf_ofs += in_buf_size; /* Update current output block size remaining */ pState->out_blk_remain = out_buf_size; } if (pState->out_blk_remain) { /* Calc amount to return. */ size_t to_copy = MZ_MIN( (buf_size - copied_to_caller), pState->out_blk_remain ); /* Copy data to caller's buffer */ memcpy( (uint8_t*)pvBuf + copied_to_caller, pWrite_buf_cur, to_copy ); #ifndef MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS /* Perform CRC */ pState->file_crc32 = (mz_uint32)mz_crc32(pState->file_crc32, pWrite_buf_cur, to_copy); #endif /* Decrement data consumed from block */ pState->out_blk_remain -= to_copy; /* Inc output offset, while performing sanity check */ if ((pState->out_buf_ofs += to_copy) > pState->file_stat.m_uncomp_size) { mz_zip_set_error(pState->pZip, MZ_ZIP_DECOMPRESSION_FAILED); pState->status = TINFL_STATUS_FAILED; break; } /* Increment counter of data copied to caller */ copied_to_caller += to_copy; } } while ( (copied_to_caller < buf_size) && ((pState->status == TINFL_STATUS_NEEDS_MORE_INPUT) || (pState->status == TINFL_STATUS_HAS_MORE_OUTPUT)) ); } /* Return how many bytes were copied into user buffer */ return copied_to_caller; } mz_bool mz_zip_reader_extract_iter_free(mz_zip_reader_extract_iter_state* pState) { int status; /* Argument sanity check */ if ((!pState) || (!pState->pZip) || (!pState->pZip->m_pState)) return MZ_FALSE; /* Was decompression completed and requested? */ if ((pState->status == TINFL_STATUS_DONE) && (!(pState->flags & MZ_ZIP_FLAG_COMPRESSED_DATA))) { /* Make sure the entire file was decompressed, and check its CRC. */ if (pState->out_buf_ofs != pState->file_stat.m_uncomp_size) { mz_zip_set_error(pState->pZip, MZ_ZIP_UNEXPECTED_DECOMPRESSED_SIZE); pState->status = TINFL_STATUS_FAILED; } #ifndef MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS else if (pState->file_crc32 != pState->file_stat.m_crc32) { mz_zip_set_error(pState->pZip, MZ_ZIP_DECOMPRESSION_FAILED); pState->status = TINFL_STATUS_FAILED; } #endif } /* Free buffers */ if (!pState->pZip->m_pState->m_pMem) pState->pZip->m_pFree(pState->pZip->m_pAlloc_opaque, pState->pRead_buf); if (pState->pWrite_buf) pState->pZip->m_pFree(pState->pZip->m_pAlloc_opaque, pState->pWrite_buf); /* Save status */ status = pState->status; /* Free context */ pState->pZip->m_pFree(pState->pZip->m_pAlloc_opaque, pState); return status == TINFL_STATUS_DONE; } #ifndef MINIZ_NO_STDIO static size_t mz_zip_file_write_callback(void *pOpaque, mz_uint64 ofs, const void *pBuf, size_t n) { (void)ofs; return MZ_FWRITE(pBuf, 1, n, (MZ_FILE *)pOpaque); } mz_bool mz_zip_reader_extract_to_file(mz_zip_archive *pZip, mz_uint file_index, const char *pDst_filename, mz_uint flags) { mz_bool status; mz_zip_archive_file_stat file_stat; MZ_FILE *pFile; if (!mz_zip_reader_file_stat(pZip, file_index, &file_stat)) return MZ_FALSE; if ((file_stat.m_is_directory) || (!file_stat.m_is_supported)) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_FEATURE); pFile = MZ_FOPEN(pDst_filename, "wb"); if (!pFile) return mz_zip_set_error(pZip, MZ_ZIP_FILE_OPEN_FAILED); status = mz_zip_reader_extract_to_callback(pZip, file_index, mz_zip_file_write_callback, pFile, flags); if (MZ_FCLOSE(pFile) == EOF) { if (status) mz_zip_set_error(pZip, MZ_ZIP_FILE_CLOSE_FAILED); status = MZ_FALSE; } #if !defined(MINIZ_NO_TIME) && !defined(MINIZ_NO_STDIO) if (status) mz_zip_set_file_times(pDst_filename, file_stat.m_time, file_stat.m_time); #endif return status; } mz_bool mz_zip_reader_extract_file_to_file(mz_zip_archive *pZip, const char *pArchive_filename, const char *pDst_filename, mz_uint flags) { mz_uint32 file_index; if (!mz_zip_reader_locate_file_v2(pZip, pArchive_filename, NULL, flags, &file_index)) return MZ_FALSE; return mz_zip_reader_extract_to_file(pZip, file_index, pDst_filename, flags); } mz_bool mz_zip_reader_extract_to_cfile(mz_zip_archive *pZip, mz_uint file_index, MZ_FILE *pFile, mz_uint flags) { mz_zip_archive_file_stat file_stat; if (!mz_zip_reader_file_stat(pZip, file_index, &file_stat)) return MZ_FALSE; if ((file_stat.m_is_directory) || (!file_stat.m_is_supported)) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_FEATURE); return mz_zip_reader_extract_to_callback(pZip, file_index, mz_zip_file_write_callback, pFile, flags); } mz_bool mz_zip_reader_extract_file_to_cfile(mz_zip_archive *pZip, const char *pArchive_filename, MZ_FILE *pFile, mz_uint flags) { mz_uint32 file_index; if (!mz_zip_reader_locate_file_v2(pZip, pArchive_filename, NULL, flags, &file_index)) return MZ_FALSE; return mz_zip_reader_extract_to_cfile(pZip, file_index, pFile, flags); } #endif /* #ifndef MINIZ_NO_STDIO */ static size_t mz_zip_compute_crc32_callback(void *pOpaque, mz_uint64 file_ofs, const void *pBuf, size_t n) { mz_uint32 *p = (mz_uint32 *)pOpaque; (void)file_ofs; *p = (mz_uint32)mz_crc32(*p, (const mz_uint8 *)pBuf, n); return n; } mz_bool mz_zip_validate_file(mz_zip_archive *pZip, mz_uint file_index, mz_uint flags) { mz_zip_archive_file_stat file_stat; mz_zip_internal_state *pState; const mz_uint8 *pCentral_dir_header; mz_bool found_zip64_ext_data_in_cdir = MZ_FALSE; mz_bool found_zip64_ext_data_in_ldir = MZ_FALSE; mz_uint32 local_header_u32[(MZ_ZIP_LOCAL_DIR_HEADER_SIZE + sizeof(mz_uint32) - 1) / sizeof(mz_uint32)]; mz_uint8 *pLocal_header = (mz_uint8 *)local_header_u32; mz_uint64 local_header_ofs = 0; mz_uint32 local_header_filename_len, local_header_extra_len, local_header_crc32; mz_uint64 local_header_comp_size, local_header_uncomp_size; mz_uint32 uncomp_crc32 = MZ_CRC32_INIT; mz_bool has_data_descriptor; mz_uint32 local_header_bit_flags; mz_zip_array file_data_array; mz_zip_array_init(&file_data_array, 1); if ((!pZip) || (!pZip->m_pState) || (!pZip->m_pAlloc) || (!pZip->m_pFree) || (!pZip->m_pRead)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (file_index > pZip->m_total_files) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); pState = pZip->m_pState; pCentral_dir_header = mz_zip_get_cdh(pZip, file_index); if (!mz_zip_file_stat_internal(pZip, file_index, pCentral_dir_header, &file_stat, &found_zip64_ext_data_in_cdir)) return MZ_FALSE; /* A directory or zero length file */ if ((file_stat.m_is_directory) || (!file_stat.m_uncomp_size)) return MZ_TRUE; /* Encryption and patch files are not supported. */ if (file_stat.m_is_encrypted) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_ENCRYPTION); /* This function only supports stored and deflate. */ if ((file_stat.m_method != 0) && (file_stat.m_method != MZ_DEFLATED)) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_METHOD); if (!file_stat.m_is_supported) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_FEATURE); /* Read and parse the local directory entry. */ local_header_ofs = file_stat.m_local_header_ofs; if (pZip->m_pRead(pZip->m_pIO_opaque, local_header_ofs, pLocal_header, MZ_ZIP_LOCAL_DIR_HEADER_SIZE) != MZ_ZIP_LOCAL_DIR_HEADER_SIZE) return mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); if (MZ_READ_LE32(pLocal_header) != MZ_ZIP_LOCAL_DIR_HEADER_SIG) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); local_header_filename_len = MZ_READ_LE16(pLocal_header + MZ_ZIP_LDH_FILENAME_LEN_OFS); local_header_extra_len = MZ_READ_LE16(pLocal_header + MZ_ZIP_LDH_EXTRA_LEN_OFS); local_header_comp_size = MZ_READ_LE32(pLocal_header + MZ_ZIP_LDH_COMPRESSED_SIZE_OFS); local_header_uncomp_size = MZ_READ_LE32(pLocal_header + MZ_ZIP_LDH_DECOMPRESSED_SIZE_OFS); local_header_crc32 = MZ_READ_LE32(pLocal_header + MZ_ZIP_LDH_CRC32_OFS); local_header_bit_flags = MZ_READ_LE16(pLocal_header + MZ_ZIP_LDH_BIT_FLAG_OFS); has_data_descriptor = (local_header_bit_flags & 8) != 0; if (local_header_filename_len != strlen(file_stat.m_filename)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); if ((local_header_ofs + MZ_ZIP_LOCAL_DIR_HEADER_SIZE + local_header_filename_len + local_header_extra_len + file_stat.m_comp_size) > pZip->m_archive_size) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); if (!mz_zip_array_resize(pZip, &file_data_array, MZ_MAX(local_header_filename_len, local_header_extra_len), MZ_FALSE)) { mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); goto handle_failure; } if (local_header_filename_len) { if (pZip->m_pRead(pZip->m_pIO_opaque, local_header_ofs + MZ_ZIP_LOCAL_DIR_HEADER_SIZE, file_data_array.m_p, local_header_filename_len) != local_header_filename_len) { mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); goto handle_failure; } /* I've seen 1 archive that had the same pathname, but used backslashes in the local dir and forward slashes in the central dir. Do we care about this? For now, this case will fail validation. */ if (memcmp(file_stat.m_filename, file_data_array.m_p, local_header_filename_len) != 0) { mz_zip_set_error(pZip, MZ_ZIP_VALIDATION_FAILED); goto handle_failure; } } if ((local_header_extra_len) && ((local_header_comp_size == MZ_UINT32_MAX) || (local_header_uncomp_size == MZ_UINT32_MAX))) { mz_uint32 extra_size_remaining = local_header_extra_len; const mz_uint8 *pExtra_data = (const mz_uint8 *)file_data_array.m_p; if (pZip->m_pRead(pZip->m_pIO_opaque, local_header_ofs + MZ_ZIP_LOCAL_DIR_HEADER_SIZE + local_header_filename_len, file_data_array.m_p, local_header_extra_len) != local_header_extra_len) { mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); goto handle_failure; } do { mz_uint32 field_id, field_data_size, field_total_size; if (extra_size_remaining < (sizeof(mz_uint16) * 2)) { mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); goto handle_failure; } field_id = MZ_READ_LE16(pExtra_data); field_data_size = MZ_READ_LE16(pExtra_data + sizeof(mz_uint16)); field_total_size = field_data_size + sizeof(mz_uint16) * 2; if (field_total_size > extra_size_remaining) { mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); goto handle_failure; } if (field_id == MZ_ZIP64_EXTENDED_INFORMATION_FIELD_HEADER_ID) { const mz_uint8 *pSrc_field_data = pExtra_data + sizeof(mz_uint32); if (field_data_size < sizeof(mz_uint64) * 2) { mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); goto handle_failure; } local_header_uncomp_size = MZ_READ_LE64(pSrc_field_data); local_header_comp_size = MZ_READ_LE64(pSrc_field_data + sizeof(mz_uint64)); found_zip64_ext_data_in_ldir = MZ_TRUE; break; } pExtra_data += field_total_size; extra_size_remaining -= field_total_size; } while (extra_size_remaining); } /* TODO: parse local header extra data when local_header_comp_size is 0xFFFFFFFF! (big_descriptor.zip) */ /* I've seen zips in the wild with the data descriptor bit set, but proper local header values and bogus data descriptors */ if ((has_data_descriptor) && (!local_header_comp_size) && (!local_header_crc32)) { mz_uint8 descriptor_buf[32]; mz_bool has_id; const mz_uint8 *pSrc; mz_uint32 file_crc32; mz_uint64 comp_size = 0, uncomp_size = 0; mz_uint32 num_descriptor_uint32s = ((pState->m_zip64) || (found_zip64_ext_data_in_ldir)) ? 6 : 4; if (pZip->m_pRead(pZip->m_pIO_opaque, local_header_ofs + MZ_ZIP_LOCAL_DIR_HEADER_SIZE + local_header_filename_len + local_header_extra_len + file_stat.m_comp_size, descriptor_buf, sizeof(mz_uint32) * num_descriptor_uint32s) != (sizeof(mz_uint32) * num_descriptor_uint32s)) { mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); goto handle_failure; } has_id = (MZ_READ_LE32(descriptor_buf) == MZ_ZIP_DATA_DESCRIPTOR_ID); pSrc = has_id ? (descriptor_buf + sizeof(mz_uint32)) : descriptor_buf; file_crc32 = MZ_READ_LE32(pSrc); if ((pState->m_zip64) || (found_zip64_ext_data_in_ldir)) { comp_size = MZ_READ_LE64(pSrc + sizeof(mz_uint32)); uncomp_size = MZ_READ_LE64(pSrc + sizeof(mz_uint32) + sizeof(mz_uint64)); } else { comp_size = MZ_READ_LE32(pSrc + sizeof(mz_uint32)); uncomp_size = MZ_READ_LE32(pSrc + sizeof(mz_uint32) + sizeof(mz_uint32)); } if ((file_crc32 != file_stat.m_crc32) || (comp_size != file_stat.m_comp_size) || (uncomp_size != file_stat.m_uncomp_size)) { mz_zip_set_error(pZip, MZ_ZIP_VALIDATION_FAILED); goto handle_failure; } } else { if ((local_header_crc32 != file_stat.m_crc32) || (local_header_comp_size != file_stat.m_comp_size) || (local_header_uncomp_size != file_stat.m_uncomp_size)) { mz_zip_set_error(pZip, MZ_ZIP_VALIDATION_FAILED); goto handle_failure; } } mz_zip_array_clear(pZip, &file_data_array); if ((flags & MZ_ZIP_FLAG_VALIDATE_HEADERS_ONLY) == 0) { if (!mz_zip_reader_extract_to_callback(pZip, file_index, mz_zip_compute_crc32_callback, &uncomp_crc32, 0)) return MZ_FALSE; /* 1 more check to be sure, although the extract checks too. */ if (uncomp_crc32 != file_stat.m_crc32) { mz_zip_set_error(pZip, MZ_ZIP_VALIDATION_FAILED); return MZ_FALSE; } } return MZ_TRUE; handle_failure: mz_zip_array_clear(pZip, &file_data_array); return MZ_FALSE; } mz_bool mz_zip_validate_archive(mz_zip_archive *pZip, mz_uint flags) { mz_zip_internal_state *pState; uint32_t i; if ((!pZip) || (!pZip->m_pState) || (!pZip->m_pAlloc) || (!pZip->m_pFree) || (!pZip->m_pRead)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); pState = pZip->m_pState; /* Basic sanity checks */ if (!pState->m_zip64) { if (pZip->m_total_files > MZ_UINT16_MAX) return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); if (pZip->m_archive_size > MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); } else { if (pZip->m_total_files >= MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); if (pState->m_central_dir.m_size >= MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); } for (i = 0; i < pZip->m_total_files; i++) { if (MZ_ZIP_FLAG_VALIDATE_LOCATE_FILE_FLAG & flags) { mz_uint32 found_index; mz_zip_archive_file_stat stat; if (!mz_zip_reader_file_stat(pZip, i, &stat)) return MZ_FALSE; if (!mz_zip_reader_locate_file_v2(pZip, stat.m_filename, NULL, 0, &found_index)) return MZ_FALSE; /* This check can fail if there are duplicate filenames in the archive (which we don't check for when writing - that's up to the user) */ if (found_index != i) return mz_zip_set_error(pZip, MZ_ZIP_VALIDATION_FAILED); } if (!mz_zip_validate_file(pZip, i, flags)) return MZ_FALSE; } return MZ_TRUE; } mz_bool mz_zip_validate_mem_archive(const void *pMem, size_t size, mz_uint flags, mz_zip_error *pErr) { mz_bool success = MZ_TRUE; mz_zip_archive zip; mz_zip_error actual_err = MZ_ZIP_NO_ERROR; if ((!pMem) || (!size)) { if (pErr) *pErr = MZ_ZIP_INVALID_PARAMETER; return MZ_FALSE; } mz_zip_zero_struct(&zip); if (!mz_zip_reader_init_mem(&zip, pMem, size, flags)) { if (pErr) *pErr = zip.m_last_error; return MZ_FALSE; } if (!mz_zip_validate_archive(&zip, flags)) { actual_err = zip.m_last_error; success = MZ_FALSE; } if (!mz_zip_reader_end_internal(&zip, success)) { if (!actual_err) actual_err = zip.m_last_error; success = MZ_FALSE; } if (pErr) *pErr = actual_err; return success; } #ifndef MINIZ_NO_STDIO mz_bool mz_zip_validate_file_archive(const char *pFilename, mz_uint flags, mz_zip_error *pErr) { mz_bool success = MZ_TRUE; mz_zip_archive zip; mz_zip_error actual_err = MZ_ZIP_NO_ERROR; if (!pFilename) { if (pErr) *pErr = MZ_ZIP_INVALID_PARAMETER; return MZ_FALSE; } mz_zip_zero_struct(&zip); if (!mz_zip_reader_init_file_v2(&zip, pFilename, flags, 0, 0)) { if (pErr) *pErr = zip.m_last_error; return MZ_FALSE; } if (!mz_zip_validate_archive(&zip, flags)) { actual_err = zip.m_last_error; success = MZ_FALSE; } if (!mz_zip_reader_end_internal(&zip, success)) { if (!actual_err) actual_err = zip.m_last_error; success = MZ_FALSE; } if (pErr) *pErr = actual_err; return success; } #endif /* #ifndef MINIZ_NO_STDIO */ /* ------------------- .ZIP archive writing */ #ifndef MINIZ_NO_ARCHIVE_WRITING_APIS static MZ_FORCEINLINE void mz_write_le16(mz_uint8 *p, mz_uint16 v) { p[0] = (mz_uint8)v; p[1] = (mz_uint8)(v >> 8); } static MZ_FORCEINLINE void mz_write_le32(mz_uint8 *p, mz_uint32 v) { p[0] = (mz_uint8)v; p[1] = (mz_uint8)(v >> 8); p[2] = (mz_uint8)(v >> 16); p[3] = (mz_uint8)(v >> 24); } static MZ_FORCEINLINE void mz_write_le64(mz_uint8 *p, mz_uint64 v) { mz_write_le32(p, (mz_uint32)v); mz_write_le32(p + sizeof(mz_uint32), (mz_uint32)(v >> 32)); } #define MZ_WRITE_LE16(p, v) mz_write_le16((mz_uint8 *)(p), (mz_uint16)(v)) #define MZ_WRITE_LE32(p, v) mz_write_le32((mz_uint8 *)(p), (mz_uint32)(v)) #define MZ_WRITE_LE64(p, v) mz_write_le64((mz_uint8 *)(p), (mz_uint64)(v)) static size_t mz_zip_heap_write_func(void *pOpaque, mz_uint64 file_ofs, const void *pBuf, size_t n) { mz_zip_archive *pZip = (mz_zip_archive *)pOpaque; mz_zip_internal_state *pState = pZip->m_pState; mz_uint64 new_size = MZ_MAX(file_ofs + n, pState->m_mem_size); if (!n) return 0; /* An allocation this big is likely to just fail on 32-bit systems, so don't even go there. */ if ((sizeof(size_t) == sizeof(mz_uint32)) && (new_size > 0x7FFFFFFF)) { mz_zip_set_error(pZip, MZ_ZIP_FILE_TOO_LARGE); return 0; } if (new_size > pState->m_mem_capacity) { void *pNew_block; size_t new_capacity = MZ_MAX(64, pState->m_mem_capacity); while (new_capacity < new_size) new_capacity *= 2; if (NULL == (pNew_block = pZip->m_pRealloc(pZip->m_pAlloc_opaque, pState->m_pMem, 1, new_capacity))) { mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); return 0; } pState->m_pMem = pNew_block; pState->m_mem_capacity = new_capacity; } memcpy((mz_uint8 *)pState->m_pMem + file_ofs, pBuf, n); pState->m_mem_size = (size_t)new_size; return n; } static mz_bool mz_zip_writer_end_internal(mz_zip_archive *pZip, mz_bool set_last_error) { mz_zip_internal_state *pState; mz_bool status = MZ_TRUE; if ((!pZip) || (!pZip->m_pState) || (!pZip->m_pAlloc) || (!pZip->m_pFree) || ((pZip->m_zip_mode != MZ_ZIP_MODE_WRITING) && (pZip->m_zip_mode != MZ_ZIP_MODE_WRITING_HAS_BEEN_FINALIZED))) { if (set_last_error) mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); return MZ_FALSE; } pState = pZip->m_pState; pZip->m_pState = NULL; mz_zip_array_clear(pZip, &pState->m_central_dir); mz_zip_array_clear(pZip, &pState->m_central_dir_offsets); mz_zip_array_clear(pZip, &pState->m_sorted_central_dir_offsets); #ifndef MINIZ_NO_STDIO if (pState->m_pFile) { if (pZip->m_zip_type == MZ_ZIP_TYPE_FILE) { if (MZ_FCLOSE(pState->m_pFile) == EOF) { if (set_last_error) mz_zip_set_error(pZip, MZ_ZIP_FILE_CLOSE_FAILED); status = MZ_FALSE; } } pState->m_pFile = NULL; } #endif /* #ifndef MINIZ_NO_STDIO */ if ((pZip->m_pWrite == mz_zip_heap_write_func) && (pState->m_pMem)) { pZip->m_pFree(pZip->m_pAlloc_opaque, pState->m_pMem); pState->m_pMem = NULL; } pZip->m_pFree(pZip->m_pAlloc_opaque, pState); pZip->m_zip_mode = MZ_ZIP_MODE_INVALID; return status; } mz_bool mz_zip_writer_init_v2(mz_zip_archive *pZip, mz_uint64 existing_size, mz_uint flags) { mz_bool zip64 = (flags & MZ_ZIP_FLAG_WRITE_ZIP64) != 0; if ((!pZip) || (pZip->m_pState) || (!pZip->m_pWrite) || (pZip->m_zip_mode != MZ_ZIP_MODE_INVALID)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (flags & MZ_ZIP_FLAG_WRITE_ALLOW_READING) { if (!pZip->m_pRead) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); } if (pZip->m_file_offset_alignment) { /* Ensure user specified file offset alignment is a power of 2. */ if (pZip->m_file_offset_alignment & (pZip->m_file_offset_alignment - 1)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); } if (!pZip->m_pAlloc) pZip->m_pAlloc = miniz_def_alloc_func; if (!pZip->m_pFree) pZip->m_pFree = miniz_def_free_func; if (!pZip->m_pRealloc) pZip->m_pRealloc = miniz_def_realloc_func; pZip->m_archive_size = existing_size; pZip->m_central_directory_file_ofs = 0; pZip->m_total_files = 0; if (NULL == (pZip->m_pState = (mz_zip_internal_state *)pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, sizeof(mz_zip_internal_state)))) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); memset(pZip->m_pState, 0, sizeof(mz_zip_internal_state)); MZ_ZIP_ARRAY_SET_ELEMENT_SIZE(&pZip->m_pState->m_central_dir, sizeof(mz_uint8)); MZ_ZIP_ARRAY_SET_ELEMENT_SIZE(&pZip->m_pState->m_central_dir_offsets, sizeof(mz_uint32)); MZ_ZIP_ARRAY_SET_ELEMENT_SIZE(&pZip->m_pState->m_sorted_central_dir_offsets, sizeof(mz_uint32)); pZip->m_pState->m_zip64 = zip64; pZip->m_pState->m_zip64_has_extended_info_fields = zip64; pZip->m_zip_type = MZ_ZIP_TYPE_USER; pZip->m_zip_mode = MZ_ZIP_MODE_WRITING; return MZ_TRUE; } mz_bool mz_zip_writer_init(mz_zip_archive *pZip, mz_uint64 existing_size) { return mz_zip_writer_init_v2(pZip, existing_size, 0); } mz_bool mz_zip_writer_init_heap_v2(mz_zip_archive *pZip, size_t size_to_reserve_at_beginning, size_t initial_allocation_size, mz_uint flags) { pZip->m_pWrite = mz_zip_heap_write_func; pZip->m_pNeeds_keepalive = NULL; if (flags & MZ_ZIP_FLAG_WRITE_ALLOW_READING) pZip->m_pRead = mz_zip_mem_read_func; pZip->m_pIO_opaque = pZip; if (!mz_zip_writer_init_v2(pZip, size_to_reserve_at_beginning, flags)) return MZ_FALSE; pZip->m_zip_type = MZ_ZIP_TYPE_HEAP; if (0 != (initial_allocation_size = MZ_MAX(initial_allocation_size, size_to_reserve_at_beginning))) { if (NULL == (pZip->m_pState->m_pMem = pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, initial_allocation_size))) { mz_zip_writer_end_internal(pZip, MZ_FALSE); return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } pZip->m_pState->m_mem_capacity = initial_allocation_size; } return MZ_TRUE; } mz_bool mz_zip_writer_init_heap(mz_zip_archive *pZip, size_t size_to_reserve_at_beginning, size_t initial_allocation_size) { return mz_zip_writer_init_heap_v2(pZip, size_to_reserve_at_beginning, initial_allocation_size, 0); } #ifndef MINIZ_NO_STDIO static size_t mz_zip_file_write_func(void *pOpaque, mz_uint64 file_ofs, const void *pBuf, size_t n) { mz_zip_archive *pZip = (mz_zip_archive *)pOpaque; mz_int64 cur_ofs = MZ_FTELL64(pZip->m_pState->m_pFile); file_ofs += pZip->m_pState->m_file_archive_start_ofs; if (((mz_int64)file_ofs < 0) || (((cur_ofs != (mz_int64)file_ofs)) && (MZ_FSEEK64(pZip->m_pState->m_pFile, (mz_int64)file_ofs, SEEK_SET)))) { mz_zip_set_error(pZip, MZ_ZIP_FILE_SEEK_FAILED); return 0; } return MZ_FWRITE(pBuf, 1, n, pZip->m_pState->m_pFile); } mz_bool mz_zip_writer_init_file(mz_zip_archive *pZip, const char *pFilename, mz_uint64 size_to_reserve_at_beginning) { return mz_zip_writer_init_file_v2(pZip, pFilename, size_to_reserve_at_beginning, 0); } mz_bool mz_zip_writer_init_file_v2(mz_zip_archive *pZip, const char *pFilename, mz_uint64 size_to_reserve_at_beginning, mz_uint flags) { MZ_FILE *pFile; pZip->m_pWrite = mz_zip_file_write_func; pZip->m_pNeeds_keepalive = NULL; if (flags & MZ_ZIP_FLAG_WRITE_ALLOW_READING) pZip->m_pRead = mz_zip_file_read_func; pZip->m_pIO_opaque = pZip; if (!mz_zip_writer_init_v2(pZip, size_to_reserve_at_beginning, flags)) return MZ_FALSE; if (NULL == (pFile = MZ_FOPEN(pFilename, (flags & MZ_ZIP_FLAG_WRITE_ALLOW_READING) ? "w+b" : "wb"))) { mz_zip_writer_end(pZip); return mz_zip_set_error(pZip, MZ_ZIP_FILE_OPEN_FAILED); } pZip->m_pState->m_pFile = pFile; pZip->m_zip_type = MZ_ZIP_TYPE_FILE; if (size_to_reserve_at_beginning) { mz_uint64 cur_ofs = 0; char buf[4096]; MZ_CLEAR_OBJ(buf); do { size_t n = (size_t)MZ_MIN(sizeof(buf), size_to_reserve_at_beginning); if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_ofs, buf, n) != n) { mz_zip_writer_end(pZip); return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); } cur_ofs += n; size_to_reserve_at_beginning -= n; } while (size_to_reserve_at_beginning); } return MZ_TRUE; } mz_bool mz_zip_writer_init_cfile(mz_zip_archive *pZip, MZ_FILE *pFile, mz_uint flags) { pZip->m_pWrite = mz_zip_file_write_func; pZip->m_pNeeds_keepalive = NULL; if (flags & MZ_ZIP_FLAG_WRITE_ALLOW_READING) pZip->m_pRead = mz_zip_file_read_func; pZip->m_pIO_opaque = pZip; if (!mz_zip_writer_init_v2(pZip, 0, flags)) return MZ_FALSE; pZip->m_pState->m_pFile = pFile; pZip->m_pState->m_file_archive_start_ofs = MZ_FTELL64(pZip->m_pState->m_pFile); pZip->m_zip_type = MZ_ZIP_TYPE_CFILE; return MZ_TRUE; } #endif /* #ifndef MINIZ_NO_STDIO */ mz_bool mz_zip_writer_init_from_reader_v2(mz_zip_archive *pZip, const char *pFilename, mz_uint flags) { mz_zip_internal_state *pState; if ((!pZip) || (!pZip->m_pState) || (pZip->m_zip_mode != MZ_ZIP_MODE_READING)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (flags & MZ_ZIP_FLAG_WRITE_ZIP64) { /* We don't support converting a non-zip64 file to zip64 - this seems like more trouble than it's worth. (What about the existing 32-bit data descriptors that could follow the compressed data?) */ if (!pZip->m_pState->m_zip64) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); } /* No sense in trying to write to an archive that's already at the support max size */ if (pZip->m_pState->m_zip64) { if (pZip->m_total_files == MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_TOO_MANY_FILES); } else { if (pZip->m_total_files == MZ_UINT16_MAX) return mz_zip_set_error(pZip, MZ_ZIP_TOO_MANY_FILES); if ((pZip->m_archive_size + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + MZ_ZIP_LOCAL_DIR_HEADER_SIZE) > MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_FILE_TOO_LARGE); } pState = pZip->m_pState; if (pState->m_pFile) { #ifdef MINIZ_NO_STDIO (void)pFilename; return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); #else if (pZip->m_pIO_opaque != pZip) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (pZip->m_zip_type == MZ_ZIP_TYPE_FILE) { if (!pFilename) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); /* Archive is being read from stdio and was originally opened only for reading. Try to reopen as writable. */ if (NULL == (pState->m_pFile = MZ_FREOPEN(pFilename, "r+b", pState->m_pFile))) { /* The mz_zip_archive is now in a bogus state because pState->m_pFile is NULL, so just close it. */ mz_zip_reader_end_internal(pZip, MZ_FALSE); return mz_zip_set_error(pZip, MZ_ZIP_FILE_OPEN_FAILED); } } pZip->m_pWrite = mz_zip_file_write_func; pZip->m_pNeeds_keepalive = NULL; #endif /* #ifdef MINIZ_NO_STDIO */ } else if (pState->m_pMem) { /* Archive lives in a memory block. Assume it's from the heap that we can resize using the realloc callback. */ if (pZip->m_pIO_opaque != pZip) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); pState->m_mem_capacity = pState->m_mem_size; pZip->m_pWrite = mz_zip_heap_write_func; pZip->m_pNeeds_keepalive = NULL; } /* Archive is being read via a user provided read function - make sure the user has specified a write function too. */ else if (!pZip->m_pWrite) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); /* Start writing new files at the archive's current central directory location. */ /* TODO: We could add a flag that lets the user start writing immediately AFTER the existing central dir - this would be safer. */ pZip->m_archive_size = pZip->m_central_directory_file_ofs; pZip->m_central_directory_file_ofs = 0; /* Clear the sorted central dir offsets, they aren't useful or maintained now. */ /* Even though we're now in write mode, files can still be extracted and verified, but file locates will be slow. */ /* TODO: We could easily maintain the sorted central directory offsets. */ mz_zip_array_clear(pZip, &pZip->m_pState->m_sorted_central_dir_offsets); pZip->m_zip_mode = MZ_ZIP_MODE_WRITING; return MZ_TRUE; } mz_bool mz_zip_writer_init_from_reader(mz_zip_archive *pZip, const char *pFilename) { return mz_zip_writer_init_from_reader_v2(pZip, pFilename, 0); } /* TODO: pArchive_name is a terrible name here! */ mz_bool mz_zip_writer_add_mem(mz_zip_archive *pZip, const char *pArchive_name, const void *pBuf, size_t buf_size, mz_uint level_and_flags) { return mz_zip_writer_add_mem_ex(pZip, pArchive_name, pBuf, buf_size, NULL, 0, level_and_flags, 0, 0); } typedef struct { mz_zip_archive *m_pZip; mz_uint64 m_cur_archive_file_ofs; mz_uint64 m_comp_size; } mz_zip_writer_add_state; static mz_bool mz_zip_writer_add_put_buf_callback(const void *pBuf, int len, void *pUser) { mz_zip_writer_add_state *pState = (mz_zip_writer_add_state *)pUser; if ((int)pState->m_pZip->m_pWrite(pState->m_pZip->m_pIO_opaque, pState->m_cur_archive_file_ofs, pBuf, len) != len) return MZ_FALSE; pState->m_cur_archive_file_ofs += len; pState->m_comp_size += len; return MZ_TRUE; } #define MZ_ZIP64_MAX_LOCAL_EXTRA_FIELD_SIZE (sizeof(mz_uint16) * 2 + sizeof(mz_uint64) * 2) #define MZ_ZIP64_MAX_CENTRAL_EXTRA_FIELD_SIZE (sizeof(mz_uint16) * 2 + sizeof(mz_uint64) * 3) static mz_uint32 mz_zip_writer_create_zip64_extra_data(mz_uint8 *pBuf, mz_uint64 *pUncomp_size, mz_uint64 *pComp_size, mz_uint64 *pLocal_header_ofs) { mz_uint8 *pDst = pBuf; mz_uint32 field_size = 0; MZ_WRITE_LE16(pDst + 0, MZ_ZIP64_EXTENDED_INFORMATION_FIELD_HEADER_ID); MZ_WRITE_LE16(pDst + 2, 0); pDst += sizeof(mz_uint16) * 2; if (pUncomp_size) { MZ_WRITE_LE64(pDst, *pUncomp_size); pDst += sizeof(mz_uint64); field_size += sizeof(mz_uint64); } if (pComp_size) { MZ_WRITE_LE64(pDst, *pComp_size); pDst += sizeof(mz_uint64); field_size += sizeof(mz_uint64); } if (pLocal_header_ofs) { MZ_WRITE_LE64(pDst, *pLocal_header_ofs); pDst += sizeof(mz_uint64); field_size += sizeof(mz_uint64); } MZ_WRITE_LE16(pBuf + 2, field_size); return (mz_uint32)(pDst - pBuf); } static mz_bool mz_zip_writer_create_local_dir_header(mz_zip_archive *pZip, mz_uint8 *pDst, mz_uint16 filename_size, mz_uint16 extra_size, mz_uint64 uncomp_size, mz_uint64 comp_size, mz_uint32 uncomp_crc32, mz_uint16 method, mz_uint16 bit_flags, mz_uint16 dos_time, mz_uint16 dos_date) { (void)pZip; memset(pDst, 0, MZ_ZIP_LOCAL_DIR_HEADER_SIZE); MZ_WRITE_LE32(pDst + MZ_ZIP_LDH_SIG_OFS, MZ_ZIP_LOCAL_DIR_HEADER_SIG); MZ_WRITE_LE16(pDst + MZ_ZIP_LDH_VERSION_NEEDED_OFS, method ? 20 : 0); MZ_WRITE_LE16(pDst + MZ_ZIP_LDH_BIT_FLAG_OFS, bit_flags); MZ_WRITE_LE16(pDst + MZ_ZIP_LDH_METHOD_OFS, method); MZ_WRITE_LE16(pDst + MZ_ZIP_LDH_FILE_TIME_OFS, dos_time); MZ_WRITE_LE16(pDst + MZ_ZIP_LDH_FILE_DATE_OFS, dos_date); MZ_WRITE_LE32(pDst + MZ_ZIP_LDH_CRC32_OFS, uncomp_crc32); MZ_WRITE_LE32(pDst + MZ_ZIP_LDH_COMPRESSED_SIZE_OFS, MZ_MIN(comp_size, MZ_UINT32_MAX)); MZ_WRITE_LE32(pDst + MZ_ZIP_LDH_DECOMPRESSED_SIZE_OFS, MZ_MIN(uncomp_size, MZ_UINT32_MAX)); MZ_WRITE_LE16(pDst + MZ_ZIP_LDH_FILENAME_LEN_OFS, filename_size); MZ_WRITE_LE16(pDst + MZ_ZIP_LDH_EXTRA_LEN_OFS, extra_size); return MZ_TRUE; } static mz_bool mz_zip_writer_create_central_dir_header(mz_zip_archive *pZip, mz_uint8 *pDst, mz_uint16 filename_size, mz_uint16 extra_size, mz_uint16 comment_size, mz_uint64 uncomp_size, mz_uint64 comp_size, mz_uint32 uncomp_crc32, mz_uint16 method, mz_uint16 bit_flags, mz_uint16 dos_time, mz_uint16 dos_date, mz_uint64 local_header_ofs, mz_uint32 ext_attributes) { (void)pZip; memset(pDst, 0, MZ_ZIP_CENTRAL_DIR_HEADER_SIZE); MZ_WRITE_LE32(pDst + MZ_ZIP_CDH_SIG_OFS, MZ_ZIP_CENTRAL_DIR_HEADER_SIG); MZ_WRITE_LE16(pDst + MZ_ZIP_CDH_VERSION_NEEDED_OFS, method ? 20 : 0); MZ_WRITE_LE16(pDst + MZ_ZIP_CDH_BIT_FLAG_OFS, bit_flags); MZ_WRITE_LE16(pDst + MZ_ZIP_CDH_METHOD_OFS, method); MZ_WRITE_LE16(pDst + MZ_ZIP_CDH_FILE_TIME_OFS, dos_time); MZ_WRITE_LE16(pDst + MZ_ZIP_CDH_FILE_DATE_OFS, dos_date); MZ_WRITE_LE32(pDst + MZ_ZIP_CDH_CRC32_OFS, uncomp_crc32); MZ_WRITE_LE32(pDst + MZ_ZIP_CDH_COMPRESSED_SIZE_OFS, MZ_MIN(comp_size, MZ_UINT32_MAX)); MZ_WRITE_LE32(pDst + MZ_ZIP_CDH_DECOMPRESSED_SIZE_OFS, MZ_MIN(uncomp_size, MZ_UINT32_MAX)); MZ_WRITE_LE16(pDst + MZ_ZIP_CDH_FILENAME_LEN_OFS, filename_size); MZ_WRITE_LE16(pDst + MZ_ZIP_CDH_EXTRA_LEN_OFS, extra_size); MZ_WRITE_LE16(pDst + MZ_ZIP_CDH_COMMENT_LEN_OFS, comment_size); MZ_WRITE_LE32(pDst + MZ_ZIP_CDH_EXTERNAL_ATTR_OFS, ext_attributes); MZ_WRITE_LE32(pDst + MZ_ZIP_CDH_LOCAL_HEADER_OFS, MZ_MIN(local_header_ofs, MZ_UINT32_MAX)); return MZ_TRUE; } static mz_bool mz_zip_writer_add_to_central_dir(mz_zip_archive *pZip, const char *pFilename, mz_uint16 filename_size, const void *pExtra, mz_uint16 extra_size, const void *pComment, mz_uint16 comment_size, mz_uint64 uncomp_size, mz_uint64 comp_size, mz_uint32 uncomp_crc32, mz_uint16 method, mz_uint16 bit_flags, mz_uint16 dos_time, mz_uint16 dos_date, mz_uint64 local_header_ofs, mz_uint32 ext_attributes, const char *user_extra_data, mz_uint user_extra_data_len) { mz_zip_internal_state *pState = pZip->m_pState; mz_uint32 central_dir_ofs = (mz_uint32)pState->m_central_dir.m_size; size_t orig_central_dir_size = pState->m_central_dir.m_size; mz_uint8 central_dir_header[MZ_ZIP_CENTRAL_DIR_HEADER_SIZE]; if (!pZip->m_pState->m_zip64) { if (local_header_ofs > 0xFFFFFFFF) return mz_zip_set_error(pZip, MZ_ZIP_FILE_TOO_LARGE); } /* miniz doesn't support central dirs >= MZ_UINT32_MAX bytes yet */ if (((mz_uint64)pState->m_central_dir.m_size + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + filename_size + extra_size + user_extra_data_len + comment_size) >= MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_CDIR_SIZE); if (!mz_zip_writer_create_central_dir_header(pZip, central_dir_header, filename_size, (mz_uint16)(extra_size + user_extra_data_len), comment_size, uncomp_size, comp_size, uncomp_crc32, method, bit_flags, dos_time, dos_date, local_header_ofs, ext_attributes)) return mz_zip_set_error(pZip, MZ_ZIP_INTERNAL_ERROR); if ((!mz_zip_array_push_back(pZip, &pState->m_central_dir, central_dir_header, MZ_ZIP_CENTRAL_DIR_HEADER_SIZE)) || (!mz_zip_array_push_back(pZip, &pState->m_central_dir, pFilename, filename_size)) || (!mz_zip_array_push_back(pZip, &pState->m_central_dir, pExtra, extra_size)) || (!mz_zip_array_push_back(pZip, &pState->m_central_dir, user_extra_data, user_extra_data_len)) || (!mz_zip_array_push_back(pZip, &pState->m_central_dir, pComment, comment_size)) || (!mz_zip_array_push_back(pZip, &pState->m_central_dir_offsets, ¢ral_dir_ofs, 1))) { /* Try to resize the central directory array back into its original state. */ mz_zip_array_resize(pZip, &pState->m_central_dir, orig_central_dir_size, MZ_FALSE); return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } return MZ_TRUE; } static mz_bool mz_zip_writer_validate_archive_name(const char *pArchive_name) { /* Basic ZIP archive filename validity checks: Valid filenames cannot start with a forward slash, cannot contain a drive letter, and cannot use DOS-style backward slashes. */ if (*pArchive_name == '/') return MZ_FALSE; /* Making sure the name does not contain drive letters or DOS style backward slashes is the responsibility of the program using miniz*/ return MZ_TRUE; } static mz_uint mz_zip_writer_compute_padding_needed_for_file_alignment(mz_zip_archive *pZip) { mz_uint32 n; if (!pZip->m_file_offset_alignment) return 0; n = (mz_uint32)(pZip->m_archive_size & (pZip->m_file_offset_alignment - 1)); return (mz_uint)((pZip->m_file_offset_alignment - n) & (pZip->m_file_offset_alignment - 1)); } static mz_bool mz_zip_writer_write_zeros(mz_zip_archive *pZip, mz_uint64 cur_file_ofs, mz_uint32 n) { char buf[4096]; memset(buf, 0, MZ_MIN(sizeof(buf), n)); while (n) { mz_uint32 s = MZ_MIN(sizeof(buf), n); if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_file_ofs, buf, s) != s) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); cur_file_ofs += s; n -= s; } return MZ_TRUE; } mz_bool mz_zip_writer_add_mem_ex(mz_zip_archive *pZip, const char *pArchive_name, const void *pBuf, size_t buf_size, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags, mz_uint64 uncomp_size, mz_uint32 uncomp_crc32) { return mz_zip_writer_add_mem_ex_v2(pZip, pArchive_name, pBuf, buf_size, pComment, comment_size, level_and_flags, uncomp_size, uncomp_crc32, NULL, NULL, 0, NULL, 0); } mz_bool mz_zip_writer_add_mem_ex_v2(mz_zip_archive *pZip, const char *pArchive_name, const void *pBuf, size_t buf_size, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags, mz_uint64 uncomp_size, mz_uint32 uncomp_crc32, MZ_TIME_T *last_modified, const char *user_extra_data, mz_uint user_extra_data_len, const char *user_extra_data_central, mz_uint user_extra_data_central_len) { mz_uint16 method = 0, dos_time = 0, dos_date = 0; mz_uint level, ext_attributes = 0, num_alignment_padding_bytes; mz_uint64 local_dir_header_ofs = pZip->m_archive_size, cur_archive_file_ofs = pZip->m_archive_size, comp_size = 0; size_t archive_name_size; mz_uint8 local_dir_header[MZ_ZIP_LOCAL_DIR_HEADER_SIZE]; tdefl_compressor *pComp = NULL; mz_bool store_data_uncompressed; mz_zip_internal_state *pState; mz_uint8 *pExtra_data = NULL; mz_uint32 extra_size = 0; mz_uint8 extra_data[MZ_ZIP64_MAX_CENTRAL_EXTRA_FIELD_SIZE]; mz_uint16 bit_flags = 0; if ((int)level_and_flags < 0) level_and_flags = MZ_DEFAULT_LEVEL; if (uncomp_size || (buf_size && !(level_and_flags & MZ_ZIP_FLAG_COMPRESSED_DATA))) bit_flags |= MZ_ZIP_LDH_BIT_FLAG_HAS_LOCATOR; if (!(level_and_flags & MZ_ZIP_FLAG_ASCII_FILENAME)) bit_flags |= MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_UTF8; level = level_and_flags & 0xF; store_data_uncompressed = ((!level) || (level_and_flags & MZ_ZIP_FLAG_COMPRESSED_DATA)); if ((!pZip) || (!pZip->m_pState) || (pZip->m_zip_mode != MZ_ZIP_MODE_WRITING) || ((buf_size) && (!pBuf)) || (!pArchive_name) || ((comment_size) && (!pComment)) || (level > MZ_UBER_COMPRESSION)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); pState = pZip->m_pState; if (pState->m_zip64) { if (pZip->m_total_files == MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_TOO_MANY_FILES); } else { if (pZip->m_total_files == MZ_UINT16_MAX) { pState->m_zip64 = MZ_TRUE; /*return mz_zip_set_error(pZip, MZ_ZIP_TOO_MANY_FILES); */ } if ((buf_size > 0xFFFFFFFF) || (uncomp_size > 0xFFFFFFFF)) { pState->m_zip64 = MZ_TRUE; /*return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); */ } } if ((!(level_and_flags & MZ_ZIP_FLAG_COMPRESSED_DATA)) && (uncomp_size)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (!mz_zip_writer_validate_archive_name(pArchive_name)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_FILENAME); #ifndef MINIZ_NO_TIME if (last_modified != NULL) { mz_zip_time_t_to_dos_time(*last_modified, &dos_time, &dos_date); } else { MZ_TIME_T cur_time; time(&cur_time); mz_zip_time_t_to_dos_time(cur_time, &dos_time, &dos_date); } #endif /* #ifndef MINIZ_NO_TIME */ if (!(level_and_flags & MZ_ZIP_FLAG_COMPRESSED_DATA)) { uncomp_crc32 = (mz_uint32)mz_crc32(MZ_CRC32_INIT, (const mz_uint8 *)pBuf, buf_size); uncomp_size = buf_size; if (uncomp_size <= 3) { level = 0; store_data_uncompressed = MZ_TRUE; } } archive_name_size = strlen(pArchive_name); if (archive_name_size > MZ_UINT16_MAX) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_FILENAME); num_alignment_padding_bytes = mz_zip_writer_compute_padding_needed_for_file_alignment(pZip); /* miniz doesn't support central dirs >= MZ_UINT32_MAX bytes yet */ if (((mz_uint64)pState->m_central_dir.m_size + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + archive_name_size + MZ_ZIP64_MAX_CENTRAL_EXTRA_FIELD_SIZE + comment_size) >= MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_CDIR_SIZE); if (!pState->m_zip64) { /* Bail early if the archive would obviously become too large */ if ((pZip->m_archive_size + num_alignment_padding_bytes + MZ_ZIP_LOCAL_DIR_HEADER_SIZE + archive_name_size + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + archive_name_size + comment_size + user_extra_data_len + pState->m_central_dir.m_size + MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE + user_extra_data_central_len + MZ_ZIP_DATA_DESCRIPTER_SIZE32) > 0xFFFFFFFF) { pState->m_zip64 = MZ_TRUE; /*return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); */ } } if ((archive_name_size) && (pArchive_name[archive_name_size - 1] == '/')) { /* Set DOS Subdirectory attribute bit. */ ext_attributes |= MZ_ZIP_DOS_DIR_ATTRIBUTE_BITFLAG; /* Subdirectories cannot contain data. */ if ((buf_size) || (uncomp_size)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); } /* Try to do any allocations before writing to the archive, so if an allocation fails the file remains unmodified. (A good idea if we're doing an in-place modification.) */ if ((!mz_zip_array_ensure_room(pZip, &pState->m_central_dir, MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + archive_name_size + comment_size + (pState->m_zip64 ? MZ_ZIP64_MAX_CENTRAL_EXTRA_FIELD_SIZE : 0))) || (!mz_zip_array_ensure_room(pZip, &pState->m_central_dir_offsets, 1))) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); if ((!store_data_uncompressed) && (buf_size)) { if (NULL == (pComp = (tdefl_compressor *)pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, sizeof(tdefl_compressor)))) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } if (!mz_zip_writer_write_zeros(pZip, cur_archive_file_ofs, num_alignment_padding_bytes)) { pZip->m_pFree(pZip->m_pAlloc_opaque, pComp); return MZ_FALSE; } local_dir_header_ofs += num_alignment_padding_bytes; if (pZip->m_file_offset_alignment) { MZ_ASSERT((local_dir_header_ofs & (pZip->m_file_offset_alignment - 1)) == 0); } cur_archive_file_ofs += num_alignment_padding_bytes; MZ_CLEAR_OBJ(local_dir_header); if (!store_data_uncompressed || (level_and_flags & MZ_ZIP_FLAG_COMPRESSED_DATA)) { method = MZ_DEFLATED; } if (pState->m_zip64) { if (uncomp_size >= MZ_UINT32_MAX || local_dir_header_ofs >= MZ_UINT32_MAX) { pExtra_data = extra_data; extra_size = mz_zip_writer_create_zip64_extra_data(extra_data, (uncomp_size >= MZ_UINT32_MAX) ? &uncomp_size : NULL, (uncomp_size >= MZ_UINT32_MAX) ? &comp_size : NULL, (local_dir_header_ofs >= MZ_UINT32_MAX) ? &local_dir_header_ofs : NULL); } if (!mz_zip_writer_create_local_dir_header(pZip, local_dir_header, (mz_uint16)archive_name_size, (mz_uint16)(extra_size + user_extra_data_len), 0, 0, 0, method, bit_flags, dos_time, dos_date)) return mz_zip_set_error(pZip, MZ_ZIP_INTERNAL_ERROR); if (pZip->m_pWrite(pZip->m_pIO_opaque, local_dir_header_ofs, local_dir_header, sizeof(local_dir_header)) != sizeof(local_dir_header)) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); cur_archive_file_ofs += sizeof(local_dir_header); if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, pArchive_name, archive_name_size) != archive_name_size) { pZip->m_pFree(pZip->m_pAlloc_opaque, pComp); return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); } cur_archive_file_ofs += archive_name_size; if (pExtra_data != NULL) { if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, extra_data, extra_size) != extra_size) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); cur_archive_file_ofs += extra_size; } } else { if ((comp_size > MZ_UINT32_MAX) || (cur_archive_file_ofs > MZ_UINT32_MAX)) return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); if (!mz_zip_writer_create_local_dir_header(pZip, local_dir_header, (mz_uint16)archive_name_size, (mz_uint16)user_extra_data_len, 0, 0, 0, method, bit_flags, dos_time, dos_date)) return mz_zip_set_error(pZip, MZ_ZIP_INTERNAL_ERROR); if (pZip->m_pWrite(pZip->m_pIO_opaque, local_dir_header_ofs, local_dir_header, sizeof(local_dir_header)) != sizeof(local_dir_header)) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); cur_archive_file_ofs += sizeof(local_dir_header); if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, pArchive_name, archive_name_size) != archive_name_size) { pZip->m_pFree(pZip->m_pAlloc_opaque, pComp); return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); } cur_archive_file_ofs += archive_name_size; } if (user_extra_data_len > 0) { if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, user_extra_data, user_extra_data_len) != user_extra_data_len) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); cur_archive_file_ofs += user_extra_data_len; } if (store_data_uncompressed) { if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, pBuf, buf_size) != buf_size) { pZip->m_pFree(pZip->m_pAlloc_opaque, pComp); return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); } cur_archive_file_ofs += buf_size; comp_size = buf_size; } else if (buf_size) { mz_zip_writer_add_state state; state.m_pZip = pZip; state.m_cur_archive_file_ofs = cur_archive_file_ofs; state.m_comp_size = 0; if ((tdefl_init(pComp, mz_zip_writer_add_put_buf_callback, &state, tdefl_create_comp_flags_from_zip_params(level, -15, MZ_DEFAULT_STRATEGY)) != TDEFL_STATUS_OKAY) || (tdefl_compress_buffer(pComp, pBuf, buf_size, TDEFL_FINISH) != TDEFL_STATUS_DONE)) { pZip->m_pFree(pZip->m_pAlloc_opaque, pComp); return mz_zip_set_error(pZip, MZ_ZIP_COMPRESSION_FAILED); } comp_size = state.m_comp_size; cur_archive_file_ofs = state.m_cur_archive_file_ofs; } pZip->m_pFree(pZip->m_pAlloc_opaque, pComp); pComp = NULL; if (uncomp_size) { mz_uint8 local_dir_footer[MZ_ZIP_DATA_DESCRIPTER_SIZE64]; mz_uint32 local_dir_footer_size = MZ_ZIP_DATA_DESCRIPTER_SIZE32; MZ_ASSERT(bit_flags & MZ_ZIP_LDH_BIT_FLAG_HAS_LOCATOR); MZ_WRITE_LE32(local_dir_footer + 0, MZ_ZIP_DATA_DESCRIPTOR_ID); MZ_WRITE_LE32(local_dir_footer + 4, uncomp_crc32); if (pExtra_data == NULL) { if (comp_size > MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); MZ_WRITE_LE32(local_dir_footer + 8, comp_size); MZ_WRITE_LE32(local_dir_footer + 12, uncomp_size); } else { MZ_WRITE_LE64(local_dir_footer + 8, comp_size); MZ_WRITE_LE64(local_dir_footer + 16, uncomp_size); local_dir_footer_size = MZ_ZIP_DATA_DESCRIPTER_SIZE64; } if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, local_dir_footer, local_dir_footer_size) != local_dir_footer_size) return MZ_FALSE; cur_archive_file_ofs += local_dir_footer_size; } if (pExtra_data != NULL) { extra_size = mz_zip_writer_create_zip64_extra_data(extra_data, (uncomp_size >= MZ_UINT32_MAX) ? &uncomp_size : NULL, (uncomp_size >= MZ_UINT32_MAX) ? &comp_size : NULL, (local_dir_header_ofs >= MZ_UINT32_MAX) ? &local_dir_header_ofs : NULL); } if (!mz_zip_writer_add_to_central_dir(pZip, pArchive_name, (mz_uint16)archive_name_size, pExtra_data, (mz_uint16)extra_size, pComment, comment_size, uncomp_size, comp_size, uncomp_crc32, method, bit_flags, dos_time, dos_date, local_dir_header_ofs, ext_attributes, user_extra_data_central, user_extra_data_central_len)) return MZ_FALSE; pZip->m_total_files++; pZip->m_archive_size = cur_archive_file_ofs; return MZ_TRUE; } mz_bool mz_zip_writer_add_read_buf_callback(mz_zip_archive *pZip, const char *pArchive_name, mz_file_read_func read_callback, void* callback_opaque, mz_uint64 max_size, const MZ_TIME_T *pFile_time, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags, const char *user_extra_data, mz_uint user_extra_data_len, const char *user_extra_data_central, mz_uint user_extra_data_central_len) { mz_uint16 gen_flags = (level_and_flags & MZ_ZIP_FLAG_WRITE_HEADER_SET_SIZE) ? 0 : MZ_ZIP_LDH_BIT_FLAG_HAS_LOCATOR; mz_uint uncomp_crc32 = MZ_CRC32_INIT, level, num_alignment_padding_bytes; mz_uint16 method = 0, dos_time = 0, dos_date = 0, ext_attributes = 0; mz_uint64 local_dir_header_ofs, cur_archive_file_ofs = pZip->m_archive_size, uncomp_size = 0, comp_size = 0; size_t archive_name_size; mz_uint8 local_dir_header[MZ_ZIP_LOCAL_DIR_HEADER_SIZE]; mz_uint8 *pExtra_data = NULL; mz_uint32 extra_size = 0; mz_uint8 extra_data[MZ_ZIP64_MAX_CENTRAL_EXTRA_FIELD_SIZE]; mz_zip_internal_state *pState; mz_uint64 file_ofs = 0, cur_archive_header_file_ofs; if (!(level_and_flags & MZ_ZIP_FLAG_ASCII_FILENAME)) gen_flags |= MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_UTF8; if ((int)level_and_flags < 0) level_and_flags = MZ_DEFAULT_LEVEL; level = level_and_flags & 0xF; /* Sanity checks */ if ((!pZip) || (!pZip->m_pState) || (pZip->m_zip_mode != MZ_ZIP_MODE_WRITING) || (!pArchive_name) || ((comment_size) && (!pComment)) || (level > MZ_UBER_COMPRESSION)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); pState = pZip->m_pState; if ((!pState->m_zip64) && (max_size > MZ_UINT32_MAX)) { /* Source file is too large for non-zip64 */ /*return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); */ pState->m_zip64 = MZ_TRUE; } /* We could support this, but why? */ if (level_and_flags & MZ_ZIP_FLAG_COMPRESSED_DATA) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (!mz_zip_writer_validate_archive_name(pArchive_name)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_FILENAME); if (pState->m_zip64) { if (pZip->m_total_files == MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_TOO_MANY_FILES); } else { if (pZip->m_total_files == MZ_UINT16_MAX) { pState->m_zip64 = MZ_TRUE; /*return mz_zip_set_error(pZip, MZ_ZIP_TOO_MANY_FILES); */ } } archive_name_size = strlen(pArchive_name); if (archive_name_size > MZ_UINT16_MAX) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_FILENAME); num_alignment_padding_bytes = mz_zip_writer_compute_padding_needed_for_file_alignment(pZip); /* miniz doesn't support central dirs >= MZ_UINT32_MAX bytes yet */ if (((mz_uint64)pState->m_central_dir.m_size + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + archive_name_size + MZ_ZIP64_MAX_CENTRAL_EXTRA_FIELD_SIZE + comment_size) >= MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_CDIR_SIZE); if (!pState->m_zip64) { /* Bail early if the archive would obviously become too large */ if ((pZip->m_archive_size + num_alignment_padding_bytes + MZ_ZIP_LOCAL_DIR_HEADER_SIZE + archive_name_size + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + archive_name_size + comment_size + user_extra_data_len + pState->m_central_dir.m_size + MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE + 1024 + MZ_ZIP_DATA_DESCRIPTER_SIZE32 + user_extra_data_central_len) > 0xFFFFFFFF) { pState->m_zip64 = MZ_TRUE; /*return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); */ } } #ifndef MINIZ_NO_TIME if (pFile_time) { mz_zip_time_t_to_dos_time(*pFile_time, &dos_time, &dos_date); } #endif if (max_size <= 3) level = 0; if (!mz_zip_writer_write_zeros(pZip, cur_archive_file_ofs, num_alignment_padding_bytes)) { return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); } cur_archive_file_ofs += num_alignment_padding_bytes; local_dir_header_ofs = cur_archive_file_ofs; if (pZip->m_file_offset_alignment) { MZ_ASSERT((cur_archive_file_ofs & (pZip->m_file_offset_alignment - 1)) == 0); } if (max_size && level) { method = MZ_DEFLATED; } MZ_CLEAR_OBJ(local_dir_header); if (pState->m_zip64) { if (max_size >= MZ_UINT32_MAX || local_dir_header_ofs >= MZ_UINT32_MAX) { pExtra_data = extra_data; if (level_and_flags & MZ_ZIP_FLAG_WRITE_HEADER_SET_SIZE) extra_size = mz_zip_writer_create_zip64_extra_data(extra_data, (max_size >= MZ_UINT32_MAX) ? &uncomp_size : NULL, (max_size >= MZ_UINT32_MAX) ? &comp_size : NULL, (local_dir_header_ofs >= MZ_UINT32_MAX) ? &local_dir_header_ofs : NULL); else extra_size = mz_zip_writer_create_zip64_extra_data(extra_data, NULL, NULL, (local_dir_header_ofs >= MZ_UINT32_MAX) ? &local_dir_header_ofs : NULL); } if (!mz_zip_writer_create_local_dir_header(pZip, local_dir_header, (mz_uint16)archive_name_size, (mz_uint16)(extra_size + user_extra_data_len), 0, 0, 0, method, gen_flags, dos_time, dos_date)) return mz_zip_set_error(pZip, MZ_ZIP_INTERNAL_ERROR); if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, local_dir_header, sizeof(local_dir_header)) != sizeof(local_dir_header)) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); cur_archive_file_ofs += sizeof(local_dir_header); if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, pArchive_name, archive_name_size) != archive_name_size) { return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); } cur_archive_file_ofs += archive_name_size; if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, extra_data, extra_size) != extra_size) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); cur_archive_file_ofs += extra_size; } else { if ((comp_size > MZ_UINT32_MAX) || (cur_archive_file_ofs > MZ_UINT32_MAX)) return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); if (!mz_zip_writer_create_local_dir_header(pZip, local_dir_header, (mz_uint16)archive_name_size, (mz_uint16)user_extra_data_len, 0, 0, 0, method, gen_flags, dos_time, dos_date)) return mz_zip_set_error(pZip, MZ_ZIP_INTERNAL_ERROR); if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, local_dir_header, sizeof(local_dir_header)) != sizeof(local_dir_header)) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); cur_archive_file_ofs += sizeof(local_dir_header); if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, pArchive_name, archive_name_size) != archive_name_size) { return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); } cur_archive_file_ofs += archive_name_size; } if (user_extra_data_len > 0) { if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, user_extra_data, user_extra_data_len) != user_extra_data_len) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); cur_archive_file_ofs += user_extra_data_len; } if (max_size) { void *pRead_buf = pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, MZ_ZIP_MAX_IO_BUF_SIZE); if (!pRead_buf) { return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } if (!level) { while (1) { size_t n = read_callback(callback_opaque, file_ofs, pRead_buf, MZ_ZIP_MAX_IO_BUF_SIZE); if (n == 0) break; if ((n > MZ_ZIP_MAX_IO_BUF_SIZE) || (file_ofs + n > max_size)) { pZip->m_pFree(pZip->m_pAlloc_opaque, pRead_buf); return mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); } if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, pRead_buf, n) != n) { pZip->m_pFree(pZip->m_pAlloc_opaque, pRead_buf); return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); } file_ofs += n; uncomp_crc32 = (mz_uint32)mz_crc32(uncomp_crc32, (const mz_uint8 *)pRead_buf, n); cur_archive_file_ofs += n; } uncomp_size = file_ofs; comp_size = uncomp_size; } else { mz_bool result = MZ_FALSE; mz_zip_writer_add_state state; tdefl_compressor *pComp = (tdefl_compressor *)pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, sizeof(tdefl_compressor)); if (!pComp) { pZip->m_pFree(pZip->m_pAlloc_opaque, pRead_buf); return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } state.m_pZip = pZip; state.m_cur_archive_file_ofs = cur_archive_file_ofs; state.m_comp_size = 0; if (tdefl_init(pComp, mz_zip_writer_add_put_buf_callback, &state, tdefl_create_comp_flags_from_zip_params(level, -15, MZ_DEFAULT_STRATEGY)) != TDEFL_STATUS_OKAY) { pZip->m_pFree(pZip->m_pAlloc_opaque, pComp); pZip->m_pFree(pZip->m_pAlloc_opaque, pRead_buf); return mz_zip_set_error(pZip, MZ_ZIP_INTERNAL_ERROR); } for (;;) { tdefl_status status; tdefl_flush flush = TDEFL_NO_FLUSH; size_t n = read_callback(callback_opaque, file_ofs, pRead_buf, MZ_ZIP_MAX_IO_BUF_SIZE); if ((n > MZ_ZIP_MAX_IO_BUF_SIZE) || (file_ofs + n > max_size)) { mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); break; } file_ofs += n; uncomp_crc32 = (mz_uint32)mz_crc32(uncomp_crc32, (const mz_uint8 *)pRead_buf, n); if (pZip->m_pNeeds_keepalive != NULL && pZip->m_pNeeds_keepalive(pZip->m_pIO_opaque)) flush = TDEFL_FULL_FLUSH; if (n == 0) flush = TDEFL_FINISH; status = tdefl_compress_buffer(pComp, pRead_buf, n, flush); if (status == TDEFL_STATUS_DONE) { result = MZ_TRUE; break; } else if (status != TDEFL_STATUS_OKAY) { mz_zip_set_error(pZip, MZ_ZIP_COMPRESSION_FAILED); break; } } pZip->m_pFree(pZip->m_pAlloc_opaque, pComp); if (!result) { pZip->m_pFree(pZip->m_pAlloc_opaque, pRead_buf); return MZ_FALSE; } uncomp_size = file_ofs; comp_size = state.m_comp_size; cur_archive_file_ofs = state.m_cur_archive_file_ofs; } pZip->m_pFree(pZip->m_pAlloc_opaque, pRead_buf); } if (!(level_and_flags & MZ_ZIP_FLAG_WRITE_HEADER_SET_SIZE)) { mz_uint8 local_dir_footer[MZ_ZIP_DATA_DESCRIPTER_SIZE64]; mz_uint32 local_dir_footer_size = MZ_ZIP_DATA_DESCRIPTER_SIZE32; MZ_WRITE_LE32(local_dir_footer + 0, MZ_ZIP_DATA_DESCRIPTOR_ID); MZ_WRITE_LE32(local_dir_footer + 4, uncomp_crc32); if (pExtra_data == NULL) { if (comp_size > MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); MZ_WRITE_LE32(local_dir_footer + 8, comp_size); MZ_WRITE_LE32(local_dir_footer + 12, uncomp_size); } else { MZ_WRITE_LE64(local_dir_footer + 8, comp_size); MZ_WRITE_LE64(local_dir_footer + 16, uncomp_size); local_dir_footer_size = MZ_ZIP_DATA_DESCRIPTER_SIZE64; } if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_file_ofs, local_dir_footer, local_dir_footer_size) != local_dir_footer_size) return MZ_FALSE; cur_archive_file_ofs += local_dir_footer_size; } if (level_and_flags & MZ_ZIP_FLAG_WRITE_HEADER_SET_SIZE) { if (pExtra_data != NULL) { extra_size = mz_zip_writer_create_zip64_extra_data(extra_data, (max_size >= MZ_UINT32_MAX) ? &uncomp_size : NULL, (max_size >= MZ_UINT32_MAX) ? &comp_size : NULL, (local_dir_header_ofs >= MZ_UINT32_MAX) ? &local_dir_header_ofs : NULL); } if (!mz_zip_writer_create_local_dir_header(pZip, local_dir_header, (mz_uint16)archive_name_size, (mz_uint16)(extra_size + user_extra_data_len), (max_size >= MZ_UINT32_MAX) ? MZ_UINT32_MAX : uncomp_size, (max_size >= MZ_UINT32_MAX) ? MZ_UINT32_MAX : comp_size, uncomp_crc32, method, gen_flags, dos_time, dos_date)) return mz_zip_set_error(pZip, MZ_ZIP_INTERNAL_ERROR); cur_archive_header_file_ofs = local_dir_header_ofs; if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_header_file_ofs, local_dir_header, sizeof(local_dir_header)) != sizeof(local_dir_header)) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); if (pExtra_data != NULL) { cur_archive_header_file_ofs += sizeof(local_dir_header); if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_header_file_ofs, pArchive_name, archive_name_size) != archive_name_size) { return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); } cur_archive_header_file_ofs += archive_name_size; if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_archive_header_file_ofs, extra_data, extra_size) != extra_size) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); cur_archive_header_file_ofs += extra_size; } } if (pExtra_data != NULL) { extra_size = mz_zip_writer_create_zip64_extra_data(extra_data, (uncomp_size >= MZ_UINT32_MAX) ? &uncomp_size : NULL, (uncomp_size >= MZ_UINT32_MAX) ? &comp_size : NULL, (local_dir_header_ofs >= MZ_UINT32_MAX) ? &local_dir_header_ofs : NULL); } if (!mz_zip_writer_add_to_central_dir(pZip, pArchive_name, (mz_uint16)archive_name_size, pExtra_data, (mz_uint16)extra_size, pComment, comment_size, uncomp_size, comp_size, uncomp_crc32, method, gen_flags, dos_time, dos_date, local_dir_header_ofs, ext_attributes, user_extra_data_central, user_extra_data_central_len)) return MZ_FALSE; pZip->m_total_files++; pZip->m_archive_size = cur_archive_file_ofs; return MZ_TRUE; } #ifndef MINIZ_NO_STDIO static size_t mz_file_read_func_stdio(void *pOpaque, mz_uint64 file_ofs, void *pBuf, size_t n) { MZ_FILE *pSrc_file = (MZ_FILE *)pOpaque; mz_int64 cur_ofs = MZ_FTELL64(pSrc_file); if (((mz_int64)file_ofs < 0) || (((cur_ofs != (mz_int64)file_ofs)) && (MZ_FSEEK64(pSrc_file, (mz_int64)file_ofs, SEEK_SET)))) return 0; return MZ_FREAD(pBuf, 1, n, pSrc_file); } mz_bool mz_zip_writer_add_cfile(mz_zip_archive *pZip, const char *pArchive_name, MZ_FILE *pSrc_file, mz_uint64 max_size, const MZ_TIME_T *pFile_time, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags, const char *user_extra_data, mz_uint user_extra_data_len, const char *user_extra_data_central, mz_uint user_extra_data_central_len) { return mz_zip_writer_add_read_buf_callback(pZip, pArchive_name, mz_file_read_func_stdio, pSrc_file, max_size, pFile_time, pComment, comment_size, level_and_flags, user_extra_data, user_extra_data_len, user_extra_data_central, user_extra_data_central_len); } mz_bool mz_zip_writer_add_file(mz_zip_archive *pZip, const char *pArchive_name, const char *pSrc_filename, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags) { MZ_FILE *pSrc_file = NULL; mz_uint64 uncomp_size = 0; MZ_TIME_T file_modified_time; MZ_TIME_T *pFile_time = NULL; mz_bool status; memset(&file_modified_time, 0, sizeof(file_modified_time)); #if !defined(MINIZ_NO_TIME) && !defined(MINIZ_NO_STDIO) pFile_time = &file_modified_time; if (!mz_zip_get_file_modified_time(pSrc_filename, &file_modified_time)) return mz_zip_set_error(pZip, MZ_ZIP_FILE_STAT_FAILED); #endif pSrc_file = MZ_FOPEN(pSrc_filename, "rb"); if (!pSrc_file) return mz_zip_set_error(pZip, MZ_ZIP_FILE_OPEN_FAILED); MZ_FSEEK64(pSrc_file, 0, SEEK_END); uncomp_size = MZ_FTELL64(pSrc_file); MZ_FSEEK64(pSrc_file, 0, SEEK_SET); status = mz_zip_writer_add_cfile(pZip, pArchive_name, pSrc_file, uncomp_size, pFile_time, pComment, comment_size, level_and_flags, NULL, 0, NULL, 0); MZ_FCLOSE(pSrc_file); return status; } #endif /* #ifndef MINIZ_NO_STDIO */ static mz_bool mz_zip_writer_update_zip64_extension_block(mz_zip_array *pNew_ext, mz_zip_archive *pZip, const mz_uint8 *pExt, uint32_t ext_len, mz_uint64 *pComp_size, mz_uint64 *pUncomp_size, mz_uint64 *pLocal_header_ofs, mz_uint32 *pDisk_start) { /* + 64 should be enough for any new zip64 data */ if (!mz_zip_array_reserve(pZip, pNew_ext, ext_len + 64, MZ_FALSE)) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); mz_zip_array_resize(pZip, pNew_ext, 0, MZ_FALSE); if ((pUncomp_size) || (pComp_size) || (pLocal_header_ofs) || (pDisk_start)) { mz_uint8 new_ext_block[64]; mz_uint8 *pDst = new_ext_block; mz_write_le16(pDst, MZ_ZIP64_EXTENDED_INFORMATION_FIELD_HEADER_ID); mz_write_le16(pDst + sizeof(mz_uint16), 0); pDst += sizeof(mz_uint16) * 2; if (pUncomp_size) { mz_write_le64(pDst, *pUncomp_size); pDst += sizeof(mz_uint64); } if (pComp_size) { mz_write_le64(pDst, *pComp_size); pDst += sizeof(mz_uint64); } if (pLocal_header_ofs) { mz_write_le64(pDst, *pLocal_header_ofs); pDst += sizeof(mz_uint64); } if (pDisk_start) { mz_write_le32(pDst, *pDisk_start); pDst += sizeof(mz_uint32); } mz_write_le16(new_ext_block + sizeof(mz_uint16), (mz_uint16)((pDst - new_ext_block) - sizeof(mz_uint16) * 2)); if (!mz_zip_array_push_back(pZip, pNew_ext, new_ext_block, pDst - new_ext_block)) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } if ((pExt) && (ext_len)) { mz_uint32 extra_size_remaining = ext_len; const mz_uint8 *pExtra_data = pExt; do { mz_uint32 field_id, field_data_size, field_total_size; if (extra_size_remaining < (sizeof(mz_uint16) * 2)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); field_id = MZ_READ_LE16(pExtra_data); field_data_size = MZ_READ_LE16(pExtra_data + sizeof(mz_uint16)); field_total_size = field_data_size + sizeof(mz_uint16) * 2; if (field_total_size > extra_size_remaining) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); if (field_id != MZ_ZIP64_EXTENDED_INFORMATION_FIELD_HEADER_ID) { if (!mz_zip_array_push_back(pZip, pNew_ext, pExtra_data, field_total_size)) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } pExtra_data += field_total_size; extra_size_remaining -= field_total_size; } while (extra_size_remaining); } return MZ_TRUE; } /* TODO: This func is now pretty freakin complex due to zip64, split it up? */ mz_bool mz_zip_writer_add_from_zip_reader(mz_zip_archive *pZip, mz_zip_archive *pSource_zip, mz_uint src_file_index) { mz_uint n, bit_flags, num_alignment_padding_bytes, src_central_dir_following_data_size; mz_uint64 src_archive_bytes_remaining, local_dir_header_ofs; mz_uint64 cur_src_file_ofs, cur_dst_file_ofs; mz_uint32 local_header_u32[(MZ_ZIP_LOCAL_DIR_HEADER_SIZE + sizeof(mz_uint32) - 1) / sizeof(mz_uint32)]; mz_uint8 *pLocal_header = (mz_uint8 *)local_header_u32; mz_uint8 new_central_header[MZ_ZIP_CENTRAL_DIR_HEADER_SIZE]; size_t orig_central_dir_size; mz_zip_internal_state *pState; void *pBuf; const mz_uint8 *pSrc_central_header; mz_zip_archive_file_stat src_file_stat; mz_uint32 src_filename_len, src_comment_len, src_ext_len; mz_uint32 local_header_filename_size, local_header_extra_len; mz_uint64 local_header_comp_size, local_header_uncomp_size; mz_bool found_zip64_ext_data_in_ldir = MZ_FALSE; /* Sanity checks */ if ((!pZip) || (!pZip->m_pState) || (pZip->m_zip_mode != MZ_ZIP_MODE_WRITING) || (!pSource_zip->m_pRead)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); pState = pZip->m_pState; /* Don't support copying files from zip64 archives to non-zip64, even though in some cases this is possible */ if ((pSource_zip->m_pState->m_zip64) && (!pZip->m_pState->m_zip64)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); /* Get pointer to the source central dir header and crack it */ if (NULL == (pSrc_central_header = mz_zip_get_cdh(pSource_zip, src_file_index))) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (MZ_READ_LE32(pSrc_central_header + MZ_ZIP_CDH_SIG_OFS) != MZ_ZIP_CENTRAL_DIR_HEADER_SIG) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); src_filename_len = MZ_READ_LE16(pSrc_central_header + MZ_ZIP_CDH_FILENAME_LEN_OFS); src_comment_len = MZ_READ_LE16(pSrc_central_header + MZ_ZIP_CDH_COMMENT_LEN_OFS); src_ext_len = MZ_READ_LE16(pSrc_central_header + MZ_ZIP_CDH_EXTRA_LEN_OFS); src_central_dir_following_data_size = src_filename_len + src_ext_len + src_comment_len; /* TODO: We don't support central dir's >= MZ_UINT32_MAX bytes right now (+32 fudge factor in case we need to add more extra data) */ if ((pState->m_central_dir.m_size + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + src_central_dir_following_data_size + 32) >= MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_CDIR_SIZE); num_alignment_padding_bytes = mz_zip_writer_compute_padding_needed_for_file_alignment(pZip); if (!pState->m_zip64) { if (pZip->m_total_files == MZ_UINT16_MAX) return mz_zip_set_error(pZip, MZ_ZIP_TOO_MANY_FILES); } else { /* TODO: Our zip64 support still has some 32-bit limits that may not be worth fixing. */ if (pZip->m_total_files == MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_TOO_MANY_FILES); } if (!mz_zip_file_stat_internal(pSource_zip, src_file_index, pSrc_central_header, &src_file_stat, NULL)) return MZ_FALSE; cur_src_file_ofs = src_file_stat.m_local_header_ofs; cur_dst_file_ofs = pZip->m_archive_size; /* Read the source archive's local dir header */ if (pSource_zip->m_pRead(pSource_zip->m_pIO_opaque, cur_src_file_ofs, pLocal_header, MZ_ZIP_LOCAL_DIR_HEADER_SIZE) != MZ_ZIP_LOCAL_DIR_HEADER_SIZE) return mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); if (MZ_READ_LE32(pLocal_header) != MZ_ZIP_LOCAL_DIR_HEADER_SIG) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); cur_src_file_ofs += MZ_ZIP_LOCAL_DIR_HEADER_SIZE; /* Compute the total size we need to copy (filename+extra data+compressed data) */ local_header_filename_size = MZ_READ_LE16(pLocal_header + MZ_ZIP_LDH_FILENAME_LEN_OFS); local_header_extra_len = MZ_READ_LE16(pLocal_header + MZ_ZIP_LDH_EXTRA_LEN_OFS); local_header_comp_size = MZ_READ_LE32(pLocal_header + MZ_ZIP_LDH_COMPRESSED_SIZE_OFS); local_header_uncomp_size = MZ_READ_LE32(pLocal_header + MZ_ZIP_LDH_DECOMPRESSED_SIZE_OFS); src_archive_bytes_remaining = local_header_filename_size + local_header_extra_len + src_file_stat.m_comp_size; /* Try to find a zip64 extended information field */ if ((local_header_extra_len) && ((local_header_comp_size == MZ_UINT32_MAX) || (local_header_uncomp_size == MZ_UINT32_MAX))) { mz_zip_array file_data_array; const mz_uint8 *pExtra_data; mz_uint32 extra_size_remaining = local_header_extra_len; mz_zip_array_init(&file_data_array, 1); if (!mz_zip_array_resize(pZip, &file_data_array, local_header_extra_len, MZ_FALSE)) { return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } if (pSource_zip->m_pRead(pSource_zip->m_pIO_opaque, src_file_stat.m_local_header_ofs + MZ_ZIP_LOCAL_DIR_HEADER_SIZE + local_header_filename_size, file_data_array.m_p, local_header_extra_len) != local_header_extra_len) { mz_zip_array_clear(pZip, &file_data_array); return mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); } pExtra_data = (const mz_uint8 *)file_data_array.m_p; do { mz_uint32 field_id, field_data_size, field_total_size; if (extra_size_remaining < (sizeof(mz_uint16) * 2)) { mz_zip_array_clear(pZip, &file_data_array); return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); } field_id = MZ_READ_LE16(pExtra_data); field_data_size = MZ_READ_LE16(pExtra_data + sizeof(mz_uint16)); field_total_size = field_data_size + sizeof(mz_uint16) * 2; if (field_total_size > extra_size_remaining) { mz_zip_array_clear(pZip, &file_data_array); return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); } if (field_id == MZ_ZIP64_EXTENDED_INFORMATION_FIELD_HEADER_ID) { const mz_uint8 *pSrc_field_data = pExtra_data + sizeof(mz_uint32); if (field_data_size < sizeof(mz_uint64) * 2) { mz_zip_array_clear(pZip, &file_data_array); return mz_zip_set_error(pZip, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED); } local_header_uncomp_size = MZ_READ_LE64(pSrc_field_data); local_header_comp_size = MZ_READ_LE64(pSrc_field_data + sizeof(mz_uint64)); /* may be 0 if there's a descriptor */ found_zip64_ext_data_in_ldir = MZ_TRUE; break; } pExtra_data += field_total_size; extra_size_remaining -= field_total_size; } while (extra_size_remaining); mz_zip_array_clear(pZip, &file_data_array); } if (!pState->m_zip64) { /* Try to detect if the new archive will most likely wind up too big and bail early (+(sizeof(mz_uint32) * 4) is for the optional descriptor which could be present, +64 is a fudge factor). */ /* We also check when the archive is finalized so this doesn't need to be perfect. */ mz_uint64 approx_new_archive_size = cur_dst_file_ofs + num_alignment_padding_bytes + MZ_ZIP_LOCAL_DIR_HEADER_SIZE + src_archive_bytes_remaining + (sizeof(mz_uint32) * 4) + pState->m_central_dir.m_size + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + src_central_dir_following_data_size + MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE + 64; if (approx_new_archive_size >= MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); } /* Write dest archive padding */ if (!mz_zip_writer_write_zeros(pZip, cur_dst_file_ofs, num_alignment_padding_bytes)) return MZ_FALSE; cur_dst_file_ofs += num_alignment_padding_bytes; local_dir_header_ofs = cur_dst_file_ofs; if (pZip->m_file_offset_alignment) { MZ_ASSERT((local_dir_header_ofs & (pZip->m_file_offset_alignment - 1)) == 0); } /* The original zip's local header+ext block doesn't change, even with zip64, so we can just copy it over to the dest zip */ if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_dst_file_ofs, pLocal_header, MZ_ZIP_LOCAL_DIR_HEADER_SIZE) != MZ_ZIP_LOCAL_DIR_HEADER_SIZE) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); cur_dst_file_ofs += MZ_ZIP_LOCAL_DIR_HEADER_SIZE; /* Copy over the source archive bytes to the dest archive, also ensure we have enough buf space to handle optional data descriptor */ if (NULL == (pBuf = pZip->m_pAlloc(pZip->m_pAlloc_opaque, 1, (size_t)MZ_MAX(32U, MZ_MIN((mz_uint64)MZ_ZIP_MAX_IO_BUF_SIZE, src_archive_bytes_remaining))))) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); while (src_archive_bytes_remaining) { n = (mz_uint)MZ_MIN((mz_uint64)MZ_ZIP_MAX_IO_BUF_SIZE, src_archive_bytes_remaining); if (pSource_zip->m_pRead(pSource_zip->m_pIO_opaque, cur_src_file_ofs, pBuf, n) != n) { pZip->m_pFree(pZip->m_pAlloc_opaque, pBuf); return mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); } cur_src_file_ofs += n; if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_dst_file_ofs, pBuf, n) != n) { pZip->m_pFree(pZip->m_pAlloc_opaque, pBuf); return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); } cur_dst_file_ofs += n; src_archive_bytes_remaining -= n; } /* Now deal with the optional data descriptor */ bit_flags = MZ_READ_LE16(pLocal_header + MZ_ZIP_LDH_BIT_FLAG_OFS); if (bit_flags & 8) { /* Copy data descriptor */ if ((pSource_zip->m_pState->m_zip64) || (found_zip64_ext_data_in_ldir)) { /* src is zip64, dest must be zip64 */ /* name uint32_t's */ /* id 1 (optional in zip64?) */ /* crc 1 */ /* comp_size 2 */ /* uncomp_size 2 */ if (pSource_zip->m_pRead(pSource_zip->m_pIO_opaque, cur_src_file_ofs, pBuf, (sizeof(mz_uint32) * 6)) != (sizeof(mz_uint32) * 6)) { pZip->m_pFree(pZip->m_pAlloc_opaque, pBuf); return mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); } n = sizeof(mz_uint32) * ((MZ_READ_LE32(pBuf) == MZ_ZIP_DATA_DESCRIPTOR_ID) ? 6 : 5); } else { /* src is NOT zip64 */ mz_bool has_id; if (pSource_zip->m_pRead(pSource_zip->m_pIO_opaque, cur_src_file_ofs, pBuf, sizeof(mz_uint32) * 4) != sizeof(mz_uint32) * 4) { pZip->m_pFree(pZip->m_pAlloc_opaque, pBuf); return mz_zip_set_error(pZip, MZ_ZIP_FILE_READ_FAILED); } has_id = (MZ_READ_LE32(pBuf) == MZ_ZIP_DATA_DESCRIPTOR_ID); if (pZip->m_pState->m_zip64) { /* dest is zip64, so upgrade the data descriptor */ const mz_uint32 *pSrc_descriptor = (const mz_uint32 *)((const mz_uint8 *)pBuf + (has_id ? sizeof(mz_uint32) : 0)); const mz_uint32 src_crc32 = pSrc_descriptor[0]; const mz_uint64 src_comp_size = pSrc_descriptor[1]; const mz_uint64 src_uncomp_size = pSrc_descriptor[2]; mz_write_le32((mz_uint8 *)pBuf, MZ_ZIP_DATA_DESCRIPTOR_ID); mz_write_le32((mz_uint8 *)pBuf + sizeof(mz_uint32) * 1, src_crc32); mz_write_le64((mz_uint8 *)pBuf + sizeof(mz_uint32) * 2, src_comp_size); mz_write_le64((mz_uint8 *)pBuf + sizeof(mz_uint32) * 4, src_uncomp_size); n = sizeof(mz_uint32) * 6; } else { /* dest is NOT zip64, just copy it as-is */ n = sizeof(mz_uint32) * (has_id ? 4 : 3); } } if (pZip->m_pWrite(pZip->m_pIO_opaque, cur_dst_file_ofs, pBuf, n) != n) { pZip->m_pFree(pZip->m_pAlloc_opaque, pBuf); return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); } cur_src_file_ofs += n; cur_dst_file_ofs += n; } pZip->m_pFree(pZip->m_pAlloc_opaque, pBuf); /* Finally, add the new central dir header */ orig_central_dir_size = pState->m_central_dir.m_size; memcpy(new_central_header, pSrc_central_header, MZ_ZIP_CENTRAL_DIR_HEADER_SIZE); if (pState->m_zip64) { /* This is the painful part: We need to write a new central dir header + ext block with updated zip64 fields, and ensure the old fields (if any) are not included. */ const mz_uint8 *pSrc_ext = pSrc_central_header + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + src_filename_len; mz_zip_array new_ext_block; mz_zip_array_init(&new_ext_block, sizeof(mz_uint8)); MZ_WRITE_LE32(new_central_header + MZ_ZIP_CDH_COMPRESSED_SIZE_OFS, MZ_UINT32_MAX); MZ_WRITE_LE32(new_central_header + MZ_ZIP_CDH_DECOMPRESSED_SIZE_OFS, MZ_UINT32_MAX); MZ_WRITE_LE32(new_central_header + MZ_ZIP_CDH_LOCAL_HEADER_OFS, MZ_UINT32_MAX); if (!mz_zip_writer_update_zip64_extension_block(&new_ext_block, pZip, pSrc_ext, src_ext_len, &src_file_stat.m_comp_size, &src_file_stat.m_uncomp_size, &local_dir_header_ofs, NULL)) { mz_zip_array_clear(pZip, &new_ext_block); return MZ_FALSE; } MZ_WRITE_LE16(new_central_header + MZ_ZIP_CDH_EXTRA_LEN_OFS, new_ext_block.m_size); if (!mz_zip_array_push_back(pZip, &pState->m_central_dir, new_central_header, MZ_ZIP_CENTRAL_DIR_HEADER_SIZE)) { mz_zip_array_clear(pZip, &new_ext_block); return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } if (!mz_zip_array_push_back(pZip, &pState->m_central_dir, pSrc_central_header + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE, src_filename_len)) { mz_zip_array_clear(pZip, &new_ext_block); mz_zip_array_resize(pZip, &pState->m_central_dir, orig_central_dir_size, MZ_FALSE); return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } if (!mz_zip_array_push_back(pZip, &pState->m_central_dir, new_ext_block.m_p, new_ext_block.m_size)) { mz_zip_array_clear(pZip, &new_ext_block); mz_zip_array_resize(pZip, &pState->m_central_dir, orig_central_dir_size, MZ_FALSE); return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } if (!mz_zip_array_push_back(pZip, &pState->m_central_dir, pSrc_central_header + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE + src_filename_len + src_ext_len, src_comment_len)) { mz_zip_array_clear(pZip, &new_ext_block); mz_zip_array_resize(pZip, &pState->m_central_dir, orig_central_dir_size, MZ_FALSE); return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } mz_zip_array_clear(pZip, &new_ext_block); } else { /* sanity checks */ if (cur_dst_file_ofs > MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); if (local_dir_header_ofs >= MZ_UINT32_MAX) return mz_zip_set_error(pZip, MZ_ZIP_ARCHIVE_TOO_LARGE); MZ_WRITE_LE32(new_central_header + MZ_ZIP_CDH_LOCAL_HEADER_OFS, local_dir_header_ofs); if (!mz_zip_array_push_back(pZip, &pState->m_central_dir, new_central_header, MZ_ZIP_CENTRAL_DIR_HEADER_SIZE)) return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); if (!mz_zip_array_push_back(pZip, &pState->m_central_dir, pSrc_central_header + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE, src_central_dir_following_data_size)) { mz_zip_array_resize(pZip, &pState->m_central_dir, orig_central_dir_size, MZ_FALSE); return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } } /* This shouldn't trigger unless we screwed up during the initial sanity checks */ if (pState->m_central_dir.m_size >= MZ_UINT32_MAX) { /* TODO: Support central dirs >= 32-bits in size */ mz_zip_array_resize(pZip, &pState->m_central_dir, orig_central_dir_size, MZ_FALSE); return mz_zip_set_error(pZip, MZ_ZIP_UNSUPPORTED_CDIR_SIZE); } n = (mz_uint32)orig_central_dir_size; if (!mz_zip_array_push_back(pZip, &pState->m_central_dir_offsets, &n, 1)) { mz_zip_array_resize(pZip, &pState->m_central_dir, orig_central_dir_size, MZ_FALSE); return mz_zip_set_error(pZip, MZ_ZIP_ALLOC_FAILED); } pZip->m_total_files++; pZip->m_archive_size = cur_dst_file_ofs; return MZ_TRUE; } mz_bool mz_zip_writer_finalize_archive(mz_zip_archive *pZip) { mz_zip_internal_state *pState; mz_uint64 central_dir_ofs, central_dir_size; mz_uint8 hdr[256]; if ((!pZip) || (!pZip->m_pState) || (pZip->m_zip_mode != MZ_ZIP_MODE_WRITING)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); pState = pZip->m_pState; if (pState->m_zip64) { if ((pZip->m_total_files > MZ_UINT32_MAX) || (pState->m_central_dir.m_size >= MZ_UINT32_MAX)) return mz_zip_set_error(pZip, MZ_ZIP_TOO_MANY_FILES); } else { if ((pZip->m_total_files > MZ_UINT16_MAX) || ((pZip->m_archive_size + pState->m_central_dir.m_size + MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE) > MZ_UINT32_MAX)) return mz_zip_set_error(pZip, MZ_ZIP_TOO_MANY_FILES); } central_dir_ofs = 0; central_dir_size = 0; if (pZip->m_total_files) { /* Write central directory */ central_dir_ofs = pZip->m_archive_size; central_dir_size = pState->m_central_dir.m_size; pZip->m_central_directory_file_ofs = central_dir_ofs; if (pZip->m_pWrite(pZip->m_pIO_opaque, central_dir_ofs, pState->m_central_dir.m_p, (size_t)central_dir_size) != central_dir_size) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); pZip->m_archive_size += central_dir_size; } if (pState->m_zip64) { /* Write zip64 end of central directory header */ mz_uint64 rel_ofs_to_zip64_ecdr = pZip->m_archive_size; MZ_CLEAR_OBJ(hdr); MZ_WRITE_LE32(hdr + MZ_ZIP64_ECDH_SIG_OFS, MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIG); MZ_WRITE_LE64(hdr + MZ_ZIP64_ECDH_SIZE_OF_RECORD_OFS, MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIZE - sizeof(mz_uint32) - sizeof(mz_uint64)); MZ_WRITE_LE16(hdr + MZ_ZIP64_ECDH_VERSION_MADE_BY_OFS, 0x031E); /* TODO: always Unix */ MZ_WRITE_LE16(hdr + MZ_ZIP64_ECDH_VERSION_NEEDED_OFS, 0x002D); MZ_WRITE_LE64(hdr + MZ_ZIP64_ECDH_CDIR_NUM_ENTRIES_ON_DISK_OFS, pZip->m_total_files); MZ_WRITE_LE64(hdr + MZ_ZIP64_ECDH_CDIR_TOTAL_ENTRIES_OFS, pZip->m_total_files); MZ_WRITE_LE64(hdr + MZ_ZIP64_ECDH_CDIR_SIZE_OFS, central_dir_size); MZ_WRITE_LE64(hdr + MZ_ZIP64_ECDH_CDIR_OFS_OFS, central_dir_ofs); if (pZip->m_pWrite(pZip->m_pIO_opaque, pZip->m_archive_size, hdr, MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIZE) != MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIZE) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); pZip->m_archive_size += MZ_ZIP64_END_OF_CENTRAL_DIR_HEADER_SIZE; /* Write zip64 end of central directory locator */ MZ_CLEAR_OBJ(hdr); MZ_WRITE_LE32(hdr + MZ_ZIP64_ECDL_SIG_OFS, MZ_ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIG); MZ_WRITE_LE64(hdr + MZ_ZIP64_ECDL_REL_OFS_TO_ZIP64_ECDR_OFS, rel_ofs_to_zip64_ecdr); MZ_WRITE_LE32(hdr + MZ_ZIP64_ECDL_TOTAL_NUMBER_OF_DISKS_OFS, 1); if (pZip->m_pWrite(pZip->m_pIO_opaque, pZip->m_archive_size, hdr, MZ_ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIZE) != MZ_ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIZE) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); pZip->m_archive_size += MZ_ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIZE; } /* Write end of central directory record */ MZ_CLEAR_OBJ(hdr); MZ_WRITE_LE32(hdr + MZ_ZIP_ECDH_SIG_OFS, MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIG); MZ_WRITE_LE16(hdr + MZ_ZIP_ECDH_CDIR_NUM_ENTRIES_ON_DISK_OFS, MZ_MIN(MZ_UINT16_MAX, pZip->m_total_files)); MZ_WRITE_LE16(hdr + MZ_ZIP_ECDH_CDIR_TOTAL_ENTRIES_OFS, MZ_MIN(MZ_UINT16_MAX, pZip->m_total_files)); MZ_WRITE_LE32(hdr + MZ_ZIP_ECDH_CDIR_SIZE_OFS, MZ_MIN(MZ_UINT32_MAX, central_dir_size)); MZ_WRITE_LE32(hdr + MZ_ZIP_ECDH_CDIR_OFS_OFS, MZ_MIN(MZ_UINT32_MAX, central_dir_ofs)); if (pZip->m_pWrite(pZip->m_pIO_opaque, pZip->m_archive_size, hdr, MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE) != MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE) return mz_zip_set_error(pZip, MZ_ZIP_FILE_WRITE_FAILED); #ifndef MINIZ_NO_STDIO if ((pState->m_pFile) && (MZ_FFLUSH(pState->m_pFile) == EOF)) return mz_zip_set_error(pZip, MZ_ZIP_FILE_CLOSE_FAILED); #endif /* #ifndef MINIZ_NO_STDIO */ pZip->m_archive_size += MZ_ZIP_END_OF_CENTRAL_DIR_HEADER_SIZE; pZip->m_zip_mode = MZ_ZIP_MODE_WRITING_HAS_BEEN_FINALIZED; return MZ_TRUE; } mz_bool mz_zip_writer_finalize_heap_archive(mz_zip_archive *pZip, void **ppBuf, size_t *pSize) { if ((!ppBuf) || (!pSize)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); *ppBuf = NULL; *pSize = 0; if ((!pZip) || (!pZip->m_pState)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (pZip->m_pWrite != mz_zip_heap_write_func) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); if (!mz_zip_writer_finalize_archive(pZip)) return MZ_FALSE; *ppBuf = pZip->m_pState->m_pMem; *pSize = pZip->m_pState->m_mem_size; pZip->m_pState->m_pMem = NULL; pZip->m_pState->m_mem_size = pZip->m_pState->m_mem_capacity = 0; return MZ_TRUE; } mz_bool mz_zip_writer_end(mz_zip_archive *pZip) { return mz_zip_writer_end_internal(pZip, MZ_TRUE); } #ifndef MINIZ_NO_STDIO mz_bool mz_zip_add_mem_to_archive_file_in_place(const char *pZip_filename, const char *pArchive_name, const void *pBuf, size_t buf_size, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags) { return mz_zip_add_mem_to_archive_file_in_place_v2(pZip_filename, pArchive_name, pBuf, buf_size, pComment, comment_size, level_and_flags, NULL); } mz_bool mz_zip_add_mem_to_archive_file_in_place_v2(const char *pZip_filename, const char *pArchive_name, const void *pBuf, size_t buf_size, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags, mz_zip_error *pErr) { mz_bool status, created_new_archive = MZ_FALSE; mz_zip_archive zip_archive; struct MZ_FILE_STAT_STRUCT file_stat; mz_zip_error actual_err = MZ_ZIP_NO_ERROR; mz_zip_zero_struct(&zip_archive); if ((int)level_and_flags < 0) level_and_flags = MZ_DEFAULT_LEVEL; if ((!pZip_filename) || (!pArchive_name) || ((buf_size) && (!pBuf)) || ((comment_size) && (!pComment)) || ((level_and_flags & 0xF) > MZ_UBER_COMPRESSION)) { if (pErr) *pErr = MZ_ZIP_INVALID_PARAMETER; return MZ_FALSE; } if (!mz_zip_writer_validate_archive_name(pArchive_name)) { if (pErr) *pErr = MZ_ZIP_INVALID_FILENAME; return MZ_FALSE; } /* Important: The regular non-64 bit version of stat() can fail here if the file is very large, which could cause the archive to be overwritten. */ /* So be sure to compile with _LARGEFILE64_SOURCE 1 */ if (MZ_FILE_STAT(pZip_filename, &file_stat) != 0) { /* Create a new archive. */ if (!mz_zip_writer_init_file_v2(&zip_archive, pZip_filename, 0, level_and_flags)) { if (pErr) *pErr = zip_archive.m_last_error; return MZ_FALSE; } created_new_archive = MZ_TRUE; } else { /* Append to an existing archive. */ if (!mz_zip_reader_init_file_v2(&zip_archive, pZip_filename, level_and_flags | MZ_ZIP_FLAG_DO_NOT_SORT_CENTRAL_DIRECTORY, 0, 0)) { if (pErr) *pErr = zip_archive.m_last_error; return MZ_FALSE; } if (!mz_zip_writer_init_from_reader_v2(&zip_archive, pZip_filename, level_and_flags)) { if (pErr) *pErr = zip_archive.m_last_error; mz_zip_reader_end_internal(&zip_archive, MZ_FALSE); return MZ_FALSE; } } status = mz_zip_writer_add_mem_ex(&zip_archive, pArchive_name, pBuf, buf_size, pComment, comment_size, level_and_flags, 0, 0); actual_err = zip_archive.m_last_error; /* Always finalize, even if adding failed for some reason, so we have a valid central directory. (This may not always succeed, but we can try.) */ if (!mz_zip_writer_finalize_archive(&zip_archive)) { if (!actual_err) actual_err = zip_archive.m_last_error; status = MZ_FALSE; } if (!mz_zip_writer_end_internal(&zip_archive, status)) { if (!actual_err) actual_err = zip_archive.m_last_error; status = MZ_FALSE; } if ((!status) && (created_new_archive)) { /* It's a new archive and something went wrong, so just delete it. */ int ignoredStatus = MZ_DELETE_FILE(pZip_filename); (void)ignoredStatus; } if (pErr) *pErr = actual_err; return status; } void *mz_zip_extract_archive_file_to_heap_v2(const char *pZip_filename, const char *pArchive_name, const char *pComment, size_t *pSize, mz_uint flags, mz_zip_error *pErr) { mz_uint32 file_index; mz_zip_archive zip_archive; void *p = NULL; if (pSize) *pSize = 0; if ((!pZip_filename) || (!pArchive_name)) { if (pErr) *pErr = MZ_ZIP_INVALID_PARAMETER; return NULL; } mz_zip_zero_struct(&zip_archive); if (!mz_zip_reader_init_file_v2(&zip_archive, pZip_filename, flags | MZ_ZIP_FLAG_DO_NOT_SORT_CENTRAL_DIRECTORY, 0, 0)) { if (pErr) *pErr = zip_archive.m_last_error; return NULL; } if (mz_zip_reader_locate_file_v2(&zip_archive, pArchive_name, pComment, flags, &file_index)) { p = mz_zip_reader_extract_to_heap(&zip_archive, file_index, pSize, flags); } mz_zip_reader_end_internal(&zip_archive, p != NULL); if (pErr) *pErr = zip_archive.m_last_error; return p; } void *mz_zip_extract_archive_file_to_heap(const char *pZip_filename, const char *pArchive_name, size_t *pSize, mz_uint flags) { return mz_zip_extract_archive_file_to_heap_v2(pZip_filename, pArchive_name, NULL, pSize, flags, NULL); } #endif /* #ifndef MINIZ_NO_STDIO */ #endif /* #ifndef MINIZ_NO_ARCHIVE_WRITING_APIS */ /* ------------------- Misc utils */ mz_zip_mode mz_zip_get_mode(mz_zip_archive *pZip) { return pZip ? pZip->m_zip_mode : MZ_ZIP_MODE_INVALID; } mz_zip_type mz_zip_get_type(mz_zip_archive *pZip) { return pZip ? pZip->m_zip_type : MZ_ZIP_TYPE_INVALID; } mz_zip_error mz_zip_set_last_error(mz_zip_archive *pZip, mz_zip_error err_num) { mz_zip_error prev_err; if (!pZip) return MZ_ZIP_INVALID_PARAMETER; prev_err = pZip->m_last_error; pZip->m_last_error = err_num; return prev_err; } mz_zip_error mz_zip_peek_last_error(mz_zip_archive *pZip) { if (!pZip) return MZ_ZIP_INVALID_PARAMETER; return pZip->m_last_error; } mz_zip_error mz_zip_clear_last_error(mz_zip_archive *pZip) { return mz_zip_set_last_error(pZip, MZ_ZIP_NO_ERROR); } mz_zip_error mz_zip_get_last_error(mz_zip_archive *pZip) { mz_zip_error prev_err; if (!pZip) return MZ_ZIP_INVALID_PARAMETER; prev_err = pZip->m_last_error; pZip->m_last_error = MZ_ZIP_NO_ERROR; return prev_err; } const char *mz_zip_get_error_string(mz_zip_error mz_err) { switch (mz_err) { case MZ_ZIP_NO_ERROR: return "no error"; case MZ_ZIP_UNDEFINED_ERROR: return "undefined error"; case MZ_ZIP_TOO_MANY_FILES: return "too many files"; case MZ_ZIP_FILE_TOO_LARGE: return "file too large"; case MZ_ZIP_UNSUPPORTED_METHOD: return "unsupported method"; case MZ_ZIP_UNSUPPORTED_ENCRYPTION: return "unsupported encryption"; case MZ_ZIP_UNSUPPORTED_FEATURE: return "unsupported feature"; case MZ_ZIP_FAILED_FINDING_CENTRAL_DIR: return "failed finding central directory"; case MZ_ZIP_NOT_AN_ARCHIVE: return "not a ZIP archive"; case MZ_ZIP_INVALID_HEADER_OR_CORRUPTED: return "invalid header or archive is corrupted"; case MZ_ZIP_UNSUPPORTED_MULTIDISK: return "unsupported multidisk archive"; case MZ_ZIP_DECOMPRESSION_FAILED: return "decompression failed or archive is corrupted"; case MZ_ZIP_COMPRESSION_FAILED: return "compression failed"; case MZ_ZIP_UNEXPECTED_DECOMPRESSED_SIZE: return "unexpected decompressed size"; case MZ_ZIP_CRC_CHECK_FAILED: return "CRC-32 check failed"; case MZ_ZIP_UNSUPPORTED_CDIR_SIZE: return "unsupported central directory size"; case MZ_ZIP_ALLOC_FAILED: return "allocation failed"; case MZ_ZIP_FILE_OPEN_FAILED: return "file open failed"; case MZ_ZIP_FILE_CREATE_FAILED: return "file create failed"; case MZ_ZIP_FILE_WRITE_FAILED: return "file write failed"; case MZ_ZIP_FILE_READ_FAILED: return "file read failed"; case MZ_ZIP_FILE_CLOSE_FAILED: return "file close failed"; case MZ_ZIP_FILE_SEEK_FAILED: return "file seek failed"; case MZ_ZIP_FILE_STAT_FAILED: return "file stat failed"; case MZ_ZIP_INVALID_PARAMETER: return "invalid parameter"; case MZ_ZIP_INVALID_FILENAME: return "invalid filename"; case MZ_ZIP_BUF_TOO_SMALL: return "buffer too small"; case MZ_ZIP_INTERNAL_ERROR: return "internal error"; case MZ_ZIP_FILE_NOT_FOUND: return "file not found"; case MZ_ZIP_ARCHIVE_TOO_LARGE: return "archive is too large"; case MZ_ZIP_VALIDATION_FAILED: return "validation failed"; case MZ_ZIP_WRITE_CALLBACK_FAILED: return "write calledback failed"; default: break; } return "unknown error"; } /* Note: Just because the archive is not zip64 doesn't necessarily mean it doesn't have Zip64 extended information extra field, argh. */ mz_bool mz_zip_is_zip64(mz_zip_archive *pZip) { if ((!pZip) || (!pZip->m_pState)) return MZ_FALSE; return pZip->m_pState->m_zip64; } size_t mz_zip_get_central_dir_size(mz_zip_archive *pZip) { if ((!pZip) || (!pZip->m_pState)) return 0; return pZip->m_pState->m_central_dir.m_size; } mz_uint mz_zip_reader_get_num_files(mz_zip_archive *pZip) { return pZip ? pZip->m_total_files : 0; } mz_uint64 mz_zip_get_archive_size(mz_zip_archive *pZip) { if (!pZip) return 0; return pZip->m_archive_size; } mz_uint64 mz_zip_get_archive_file_start_offset(mz_zip_archive *pZip) { if ((!pZip) || (!pZip->m_pState)) return 0; return pZip->m_pState->m_file_archive_start_ofs; } MZ_FILE *mz_zip_get_cfile(mz_zip_archive *pZip) { if ((!pZip) || (!pZip->m_pState)) return 0; return pZip->m_pState->m_pFile; } size_t mz_zip_read_archive_data(mz_zip_archive *pZip, mz_uint64 file_ofs, void *pBuf, size_t n) { if ((!pZip) || (!pZip->m_pState) || (!pBuf) || (!pZip->m_pRead)) return mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); return pZip->m_pRead(pZip->m_pIO_opaque, file_ofs, pBuf, n); } mz_uint mz_zip_reader_get_filename(mz_zip_archive *pZip, mz_uint file_index, char *pFilename, mz_uint filename_buf_size) { mz_uint n; const mz_uint8 *p = mz_zip_get_cdh(pZip, file_index); if (!p) { if (filename_buf_size) pFilename[0] = '\0'; mz_zip_set_error(pZip, MZ_ZIP_INVALID_PARAMETER); return 0; } n = MZ_READ_LE16(p + MZ_ZIP_CDH_FILENAME_LEN_OFS); if (filename_buf_size) { n = MZ_MIN(n, filename_buf_size - 1); memcpy(pFilename, p + MZ_ZIP_CENTRAL_DIR_HEADER_SIZE, n); pFilename[n] = '\0'; } return n + 1; } mz_bool mz_zip_reader_file_stat(mz_zip_archive *pZip, mz_uint file_index, mz_zip_archive_file_stat *pStat) { return mz_zip_file_stat_internal(pZip, file_index, mz_zip_get_cdh(pZip, file_index), pStat, NULL); } mz_bool mz_zip_end(mz_zip_archive *pZip) { if (!pZip) return MZ_FALSE; if (pZip->m_zip_mode == MZ_ZIP_MODE_READING) return mz_zip_reader_end(pZip); #ifndef MINIZ_NO_ARCHIVE_WRITING_APIS else if ((pZip->m_zip_mode == MZ_ZIP_MODE_WRITING) || (pZip->m_zip_mode == MZ_ZIP_MODE_WRITING_HAS_BEEN_FINALIZED)) return mz_zip_writer_end(pZip); #endif return MZ_FALSE; } #ifdef __cplusplus } #endif #endif /*#ifndef MINIZ_NO_ARCHIVE_APIS*/ zmat-0.9.9/src/miniz/zlib.h0000777000175200007730000000000014515254731017261 2miniz.hustar rlaboissrlaboisszmat-0.9.9/src/miniz/miniz.h0000644000175200007730000020750114515254731016160 0ustar rlaboissrlaboiss#define MINIZ_EXPORT /* miniz.c 2.2.0 - public domain deflate/inflate, zlib-subset, ZIP reading/writing/appending, PNG writing See "unlicense" statement at the end of this file. Rich Geldreich , last updated Oct. 13, 2013 Implements RFC 1950: http://www.ietf.org/rfc/rfc1950.txt and RFC 1951: http://www.ietf.org/rfc/rfc1951.txt Most API's defined in miniz.c are optional. For example, to disable the archive related functions just define MINIZ_NO_ARCHIVE_APIS, or to get rid of all stdio usage define MINIZ_NO_STDIO (see the list below for more macros). * Low-level Deflate/Inflate implementation notes: Compression: Use the "tdefl" API's. The compressor supports raw, static, and dynamic blocks, lazy or greedy parsing, match length filtering, RLE-only, and Huffman-only streams. It performs and compresses approximately as well as zlib. Decompression: Use the "tinfl" API's. The entire decompressor is implemented as a single function coroutine: see tinfl_decompress(). It supports decompression into a 32KB (or larger power of 2) wrapping buffer, or into a memory block large enough to hold the entire file. The low-level tdefl/tinfl API's do not make any use of dynamic memory allocation. * zlib-style API notes: miniz.c implements a fairly large subset of zlib. There's enough functionality present for it to be a drop-in zlib replacement in many apps: The z_stream struct, optional memory allocation callbacks deflateInit/deflateInit2/deflate/deflateReset/deflateEnd/deflateBound inflateInit/inflateInit2/inflate/inflateReset/inflateEnd compress, compress2, compressBound, uncompress CRC-32, Adler-32 - Using modern, minimal code size, CPU cache friendly routines. Supports raw deflate streams or standard zlib streams with adler-32 checking. Limitations: The callback API's are not implemented yet. No support for gzip headers or zlib static dictionaries. I've tried to closely emulate zlib's various flavors of stream flushing and return status codes, but there are no guarantees that miniz.c pulls this off perfectly. * PNG writing: See the tdefl_write_image_to_png_file_in_memory() function, originally written by Alex Evans. Supports 1-4 bytes/pixel images. * ZIP archive API notes: The ZIP archive API's where designed with simplicity and efficiency in mind, with just enough abstraction to get the job done with minimal fuss. There are simple API's to retrieve file information, read files from existing archives, create new archives, append new files to existing archives, or clone archive data from one archive to another. It supports archives located in memory or the heap, on disk (using stdio.h), or you can specify custom file read/write callbacks. - Archive reading: Just call this function to read a single file from a disk archive: void *mz_zip_extract_archive_file_to_heap(const char *pZip_filename, const char *pArchive_name, size_t *pSize, mz_uint zip_flags); For more complex cases, use the "mz_zip_reader" functions. Upon opening an archive, the entire central directory is located and read as-is into memory, and subsequent file access only occurs when reading individual files. - Archives file scanning: The simple way is to use this function to scan a loaded archive for a specific file: int mz_zip_reader_locate_file(mz_zip_archive *pZip, const char *pName, const char *pComment, mz_uint flags); The locate operation can optionally check file comments too, which (as one example) can be used to identify multiple versions of the same file in an archive. This function uses a simple linear search through the central directory, so it's not very fast. Alternately, you can iterate through all the files in an archive (using mz_zip_reader_get_num_files()) and retrieve detailed info on each file by calling mz_zip_reader_file_stat(). - Archive creation: Use the "mz_zip_writer" functions. The ZIP writer immediately writes compressed file data to disk and builds an exact image of the central directory in memory. The central directory image is written all at once at the end of the archive file when the archive is finalized. The archive writer can optionally align each file's local header and file data to any power of 2 alignment, which can be useful when the archive will be read from optical media. Also, the writer supports placing arbitrary data blobs at the very beginning of ZIP archives. Archives written using either feature are still readable by any ZIP tool. - Archive appending: The simple way to add a single file to an archive is to call this function: mz_bool mz_zip_add_mem_to_archive_file_in_place(const char *pZip_filename, const char *pArchive_name, const void *pBuf, size_t buf_size, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags); The archive will be created if it doesn't already exist, otherwise it'll be appended to. Note the appending is done in-place and is not an atomic operation, so if something goes wrong during the operation it's possible the archive could be left without a central directory (although the local file headers and file data will be fine, so the archive will be recoverable). For more complex archive modification scenarios: 1. The safest way is to use a mz_zip_reader to read the existing archive, cloning only those bits you want to preserve into a new archive using using the mz_zip_writer_add_from_zip_reader() function (which compiles the compressed file data as-is). When you're done, delete the old archive and rename the newly written archive, and you're done. This is safe but requires a bunch of temporary disk space or heap memory. 2. Or, you can convert an mz_zip_reader in-place to an mz_zip_writer using mz_zip_writer_init_from_reader(), append new files as needed, then finalize the archive which will write an updated central directory to the original archive. (This is basically what mz_zip_add_mem_to_archive_file_in_place() does.) There's a possibility that the archive's central directory could be lost with this method if anything goes wrong, though. - ZIP archive support limitations: No spanning support. Extraction functions can only handle unencrypted, stored or deflated files. Requires streams capable of seeking. * This is a header file library, like stb_image.c. To get only a header file, either cut and paste the below header, or create miniz.h, #define MINIZ_HEADER_FILE_ONLY, and then include miniz.c from it. * Important: For best perf. be sure to customize the below macros for your target platform: #define MINIZ_USE_UNALIGNED_LOADS_AND_STORES 1 #define MINIZ_LITTLE_ENDIAN 1 #define MINIZ_HAS_64BIT_REGISTERS 1 * On platforms using glibc, Be sure to "#define _LARGEFILE64_SOURCE 1" before including miniz.c to ensure miniz uses the 64-bit variants: fopen64(), stat64(), etc. Otherwise you won't be able to process large files (i.e. 32-bit stat() fails for me on files > 0x7FFFFFFF bytes). */ #pragma once /* Defines to completely disable specific portions of miniz.c: If all macros here are defined the only functionality remaining will be CRC-32, adler-32, tinfl, and tdefl. */ /* Define MINIZ_NO_STDIO to disable all usage and any functions which rely on stdio for file I/O. */ /*#define MINIZ_NO_STDIO */ /* If MINIZ_NO_TIME is specified then the ZIP archive functions will not be able to get the current time, or */ /* get/set file times, and the C run-time funcs that get/set times won't be called. */ /* The current downside is the times written to your archives will be from 1979. */ /*#define MINIZ_NO_TIME */ /* Define MINIZ_NO_ARCHIVE_APIS to disable all ZIP archive API's. */ /*#define MINIZ_NO_ARCHIVE_APIS */ /* Define MINIZ_NO_ARCHIVE_WRITING_APIS to disable all writing related ZIP archive API's. */ /*#define MINIZ_NO_ARCHIVE_WRITING_APIS */ /* Define MINIZ_NO_ZLIB_APIS to remove all ZLIB-style compression/decompression API's. */ /*#define MINIZ_NO_ZLIB_APIS */ /* Define MINIZ_NO_ZLIB_COMPATIBLE_NAME to disable zlib names, to prevent conflicts against stock zlib. */ /*#define MINIZ_NO_ZLIB_COMPATIBLE_NAMES */ /* Define MINIZ_NO_MALLOC to disable all calls to malloc, free, and realloc. Note if MINIZ_NO_MALLOC is defined then the user must always provide custom user alloc/free/realloc callbacks to the zlib and archive API's, and a few stand-alone helper API's which don't provide custom user functions (such as tdefl_compress_mem_to_heap() and tinfl_decompress_mem_to_heap()) won't work. */ /*#define MINIZ_NO_MALLOC */ #if defined(__TINYC__) && (defined(__linux) || defined(__linux__)) /* TODO: Work around "error: include file 'sys\utime.h' when compiling with tcc on Linux */ #define MINIZ_NO_TIME #endif #include #if !defined(MINIZ_NO_TIME) && !defined(MINIZ_NO_ARCHIVE_APIS) #include #endif #if defined(_M_IX86) || defined(_M_X64) || defined(__i386__) || defined(__i386) || defined(__i486__) || defined(__i486) || defined(i386) || defined(__ia64__) || defined(__x86_64__) /* MINIZ_X86_OR_X64_CPU is only used to help set the below macros. */ #define MINIZ_X86_OR_X64_CPU 1 #else #define MINIZ_X86_OR_X64_CPU 0 #endif #if (__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) || MINIZ_X86_OR_X64_CPU /* Set MINIZ_LITTLE_ENDIAN to 1 if the processor is little endian. */ #define MINIZ_LITTLE_ENDIAN 1 #else #define MINIZ_LITTLE_ENDIAN 0 #endif /* Set MINIZ_USE_UNALIGNED_LOADS_AND_STORES only if not set */ #if !defined(MINIZ_USE_UNALIGNED_LOADS_AND_STORES) #if MINIZ_X86_OR_X64_CPU /* Set MINIZ_USE_UNALIGNED_LOADS_AND_STORES to 1 on CPU's that permit efficient integer loads and stores from unaligned addresses. */ #define MINIZ_USE_UNALIGNED_LOADS_AND_STORES 1 #define MINIZ_UNALIGNED_USE_MEMCPY #else #define MINIZ_USE_UNALIGNED_LOADS_AND_STORES 0 #endif #endif #if defined(_M_X64) || defined(_WIN64) || defined(__MINGW64__) || defined(_LP64) || defined(__LP64__) || defined(__ia64__) || defined(__x86_64__) /* Set MINIZ_HAS_64BIT_REGISTERS to 1 if operations on 64-bit integers are reasonably fast (and don't involve compiler generated calls to helper functions). */ #define MINIZ_HAS_64BIT_REGISTERS 1 #else #define MINIZ_HAS_64BIT_REGISTERS 0 #endif #ifdef __cplusplus extern "C" { #endif /* ------------------- zlib-style API Definitions. */ /* For more compatibility with zlib, miniz.c uses unsigned long for some parameters/struct members. Beware: mz_ulong can be either 32 or 64-bits! */ typedef unsigned long mz_ulong; /* mz_free() internally uses the MZ_FREE() macro (which by default calls free() unless you've modified the MZ_MALLOC macro) to release a block allocated from the heap. */ MINIZ_EXPORT void mz_free(void *p); #define MZ_ADLER32_INIT (1) /* mz_adler32() returns the initial adler-32 value to use when called with ptr==NULL. */ MINIZ_EXPORT mz_ulong mz_adler32(mz_ulong adler, const unsigned char *ptr, size_t buf_len); #define MZ_CRC32_INIT (0) /* mz_crc32() returns the initial CRC-32 value to use when called with ptr==NULL. */ MINIZ_EXPORT mz_ulong mz_crc32(mz_ulong crc, const unsigned char *ptr, size_t buf_len); /* Compression strategies. */ enum { MZ_DEFAULT_STRATEGY = 0, MZ_FILTERED = 1, MZ_HUFFMAN_ONLY = 2, MZ_RLE = 3, MZ_FIXED = 4 }; /* Method */ #define MZ_DEFLATED 8 /* Heap allocation callbacks. Note that mz_alloc_func parameter types purposely differ from zlib's: items/size is size_t, not unsigned long. */ typedef void *(*mz_alloc_func)(void *opaque, size_t items, size_t size); typedef void (*mz_free_func)(void *opaque, void *address); typedef void *(*mz_realloc_func)(void *opaque, void *address, size_t items, size_t size); /* Compression levels: 0-9 are the standard zlib-style levels, 10 is best possible compression (not zlib compatible, and may be very slow), MZ_DEFAULT_COMPRESSION=MZ_DEFAULT_LEVEL. */ enum { MZ_NO_COMPRESSION = 0, MZ_BEST_SPEED = 1, MZ_BEST_COMPRESSION = 9, MZ_UBER_COMPRESSION = 10, MZ_DEFAULT_LEVEL = 6, MZ_DEFAULT_COMPRESSION = -1 }; #define MZ_VERSION "10.2.0" #define MZ_VERNUM 0xA100 #define MZ_VER_MAJOR 10 #define MZ_VER_MINOR 2 #define MZ_VER_REVISION 0 #define MZ_VER_SUBREVISION 0 #ifndef MINIZ_NO_ZLIB_APIS /* Flush values. For typical usage you only need MZ_NO_FLUSH and MZ_FINISH. The other values are for advanced use (refer to the zlib docs). */ enum { MZ_NO_FLUSH = 0, MZ_PARTIAL_FLUSH = 1, MZ_SYNC_FLUSH = 2, MZ_FULL_FLUSH = 3, MZ_FINISH = 4, MZ_BLOCK = 5 }; /* Return status codes. MZ_PARAM_ERROR is non-standard. */ enum { MZ_OK = 0, MZ_STREAM_END = 1, MZ_NEED_DICT = 2, MZ_ERRNO = -1, MZ_STREAM_ERROR = -2, MZ_DATA_ERROR = -3, MZ_MEM_ERROR = -4, MZ_BUF_ERROR = -5, MZ_VERSION_ERROR = -6, MZ_PARAM_ERROR = -10000 }; /* Window bits */ #define MZ_DEFAULT_WINDOW_BITS 15 struct mz_internal_state; /* Compression/decompression stream struct. */ typedef struct mz_stream_s { const unsigned char *next_in; /* pointer to next byte to read */ unsigned int avail_in; /* number of bytes available at next_in */ mz_ulong total_in; /* total number of bytes consumed so far */ unsigned char *next_out; /* pointer to next byte to write */ unsigned int avail_out; /* number of bytes that can be written to next_out */ mz_ulong total_out; /* total number of bytes produced so far */ char *msg; /* error msg (unused) */ struct mz_internal_state *state; /* internal state, allocated by zalloc/zfree */ mz_alloc_func zalloc; /* optional heap allocation function (defaults to malloc) */ mz_free_func zfree; /* optional heap free function (defaults to free) */ void *opaque; /* heap alloc function user pointer */ int data_type; /* data_type (unused) */ mz_ulong adler; /* adler32 of the source or uncompressed data */ mz_ulong reserved; /* not used */ } mz_stream; typedef mz_stream *mz_streamp; /* Returns the version string of miniz.c. */ MINIZ_EXPORT const char *mz_version(void); /* mz_deflateInit() initializes a compressor with default options: */ /* Parameters: */ /* pStream must point to an initialized mz_stream struct. */ /* level must be between [MZ_NO_COMPRESSION, MZ_BEST_COMPRESSION]. */ /* level 1 enables a specially optimized compression function that's been optimized purely for performance, not ratio. */ /* (This special func. is currently only enabled when MINIZ_USE_UNALIGNED_LOADS_AND_STORES and MINIZ_LITTLE_ENDIAN are defined.) */ /* Return values: */ /* MZ_OK on success. */ /* MZ_STREAM_ERROR if the stream is bogus. */ /* MZ_PARAM_ERROR if the input parameters are bogus. */ /* MZ_MEM_ERROR on out of memory. */ MINIZ_EXPORT int mz_deflateInit(mz_streamp pStream, int level); /* mz_deflateInit2() is like mz_deflate(), except with more control: */ /* Additional parameters: */ /* method must be MZ_DEFLATED */ /* window_bits must be MZ_DEFAULT_WINDOW_BITS (to wrap the deflate stream with zlib header/adler-32 footer) or -MZ_DEFAULT_WINDOW_BITS (raw deflate/no header or footer) */ /* mem_level must be between [1, 9] (it's checked but ignored by miniz.c) */ MINIZ_EXPORT int mz_deflateInit2(mz_streamp pStream, int level, int method, int window_bits, int mem_level, int strategy); /* Quickly resets a compressor without having to reallocate anything. Same as calling mz_deflateEnd() followed by mz_deflateInit()/mz_deflateInit2(). */ MINIZ_EXPORT int mz_deflateReset(mz_streamp pStream); /* mz_deflate() compresses the input to output, consuming as much of the input and producing as much output as possible. */ /* Parameters: */ /* pStream is the stream to read from and write to. You must initialize/update the next_in, avail_in, next_out, and avail_out members. */ /* flush may be MZ_NO_FLUSH, MZ_PARTIAL_FLUSH/MZ_SYNC_FLUSH, MZ_FULL_FLUSH, or MZ_FINISH. */ /* Return values: */ /* MZ_OK on success (when flushing, or if more input is needed but not available, and/or there's more output to be written but the output buffer is full). */ /* MZ_STREAM_END if all input has been consumed and all output bytes have been written. Don't call mz_deflate() on the stream anymore. */ /* MZ_STREAM_ERROR if the stream is bogus. */ /* MZ_PARAM_ERROR if one of the parameters is invalid. */ /* MZ_BUF_ERROR if no forward progress is possible because the input and/or output buffers are empty. (Fill up the input buffer or free up some output space and try again.) */ MINIZ_EXPORT int mz_deflate(mz_streamp pStream, int flush); /* mz_deflateEnd() deinitializes a compressor: */ /* Return values: */ /* MZ_OK on success. */ /* MZ_STREAM_ERROR if the stream is bogus. */ MINIZ_EXPORT int mz_deflateEnd(mz_streamp pStream); /* mz_deflateBound() returns a (very) conservative upper bound on the amount of data that could be generated by deflate(), assuming flush is set to only MZ_NO_FLUSH or MZ_FINISH. */ MINIZ_EXPORT mz_ulong mz_deflateBound(mz_streamp pStream, mz_ulong source_len); /* Single-call compression functions mz_compress() and mz_compress2(): */ /* Returns MZ_OK on success, or one of the error codes from mz_deflate() on failure. */ MINIZ_EXPORT int mz_compress(unsigned char *pDest, mz_ulong *pDest_len, const unsigned char *pSource, mz_ulong source_len); MINIZ_EXPORT int mz_compress2(unsigned char *pDest, mz_ulong *pDest_len, const unsigned char *pSource, mz_ulong source_len, int level); /* mz_compressBound() returns a (very) conservative upper bound on the amount of data that could be generated by calling mz_compress(). */ MINIZ_EXPORT mz_ulong mz_compressBound(mz_ulong source_len); /* Initializes a decompressor. */ MINIZ_EXPORT int mz_inflateInit(mz_streamp pStream); /* mz_inflateInit2() is like mz_inflateInit() with an additional option that controls the window size and whether or not the stream has been wrapped with a zlib header/footer: */ /* window_bits must be MZ_DEFAULT_WINDOW_BITS (to parse zlib header/footer) or -MZ_DEFAULT_WINDOW_BITS (raw deflate). */ MINIZ_EXPORT int mz_inflateInit2(mz_streamp pStream, int window_bits); /* Quickly resets a compressor without having to reallocate anything. Same as calling mz_inflateEnd() followed by mz_inflateInit()/mz_inflateInit2(). */ MINIZ_EXPORT int mz_inflateReset(mz_streamp pStream); /* Decompresses the input stream to the output, consuming only as much of the input as needed, and writing as much to the output as possible. */ /* Parameters: */ /* pStream is the stream to read from and write to. You must initialize/update the next_in, avail_in, next_out, and avail_out members. */ /* flush may be MZ_NO_FLUSH, MZ_SYNC_FLUSH, or MZ_FINISH. */ /* On the first call, if flush is MZ_FINISH it's assumed the input and output buffers are both sized large enough to decompress the entire stream in a single call (this is slightly faster). */ /* MZ_FINISH implies that there are no more source bytes available beside what's already in the input buffer, and that the output buffer is large enough to hold the rest of the decompressed data. */ /* Return values: */ /* MZ_OK on success. Either more input is needed but not available, and/or there's more output to be written but the output buffer is full. */ /* MZ_STREAM_END if all needed input has been consumed and all output bytes have been written. For zlib streams, the adler-32 of the decompressed data has also been verified. */ /* MZ_STREAM_ERROR if the stream is bogus. */ /* MZ_DATA_ERROR if the deflate stream is invalid. */ /* MZ_PARAM_ERROR if one of the parameters is invalid. */ /* MZ_BUF_ERROR if no forward progress is possible because the input buffer is empty but the inflater needs more input to continue, or if the output buffer is not large enough. Call mz_inflate() again */ /* with more input data, or with more room in the output buffer (except when using single call decompression, described above). */ MINIZ_EXPORT int mz_inflate(mz_streamp pStream, int flush); /* Deinitializes a decompressor. */ MINIZ_EXPORT int mz_inflateEnd(mz_streamp pStream); /* Single-call decompression. */ /* Returns MZ_OK on success, or one of the error codes from mz_inflate() on failure. */ MINIZ_EXPORT int mz_uncompress(unsigned char *pDest, mz_ulong *pDest_len, const unsigned char *pSource, mz_ulong source_len); MINIZ_EXPORT int mz_uncompress2(unsigned char *pDest, mz_ulong *pDest_len, const unsigned char *pSource, mz_ulong *pSource_len); /* Returns a string description of the specified error code, or NULL if the error code is invalid. */ MINIZ_EXPORT const char *mz_error(int err); /* Redefine zlib-compatible names to miniz equivalents, so miniz.c can be used as a drop-in replacement for the subset of zlib that miniz.c supports. */ /* Define MINIZ_NO_ZLIB_COMPATIBLE_NAMES to disable zlib-compatibility if you use zlib in the same project. */ #ifndef MINIZ_NO_ZLIB_COMPATIBLE_NAMES typedef unsigned char Byte; typedef unsigned int uInt; typedef mz_ulong uLong; typedef Byte Bytef; typedef uInt uIntf; typedef char charf; typedef int intf; typedef void *voidpf; typedef uLong uLongf; typedef void *voidp; typedef void *const voidpc; #define Z_NULL 0 #define Z_NO_FLUSH MZ_NO_FLUSH #define Z_PARTIAL_FLUSH MZ_PARTIAL_FLUSH #define Z_SYNC_FLUSH MZ_SYNC_FLUSH #define Z_FULL_FLUSH MZ_FULL_FLUSH #define Z_FINISH MZ_FINISH #define Z_BLOCK MZ_BLOCK #define Z_OK MZ_OK #define Z_STREAM_END MZ_STREAM_END #define Z_NEED_DICT MZ_NEED_DICT #define Z_ERRNO MZ_ERRNO #define Z_STREAM_ERROR MZ_STREAM_ERROR #define Z_DATA_ERROR MZ_DATA_ERROR #define Z_MEM_ERROR MZ_MEM_ERROR #define Z_BUF_ERROR MZ_BUF_ERROR #define Z_VERSION_ERROR MZ_VERSION_ERROR #define Z_PARAM_ERROR MZ_PARAM_ERROR #define Z_NO_COMPRESSION MZ_NO_COMPRESSION #define Z_BEST_SPEED MZ_BEST_SPEED #define Z_BEST_COMPRESSION MZ_BEST_COMPRESSION #define Z_DEFAULT_COMPRESSION MZ_DEFAULT_COMPRESSION #define Z_DEFAULT_STRATEGY MZ_DEFAULT_STRATEGY #define Z_FILTERED MZ_FILTERED #define Z_HUFFMAN_ONLY MZ_HUFFMAN_ONLY #define Z_RLE MZ_RLE #define Z_FIXED MZ_FIXED #define Z_DEFLATED MZ_DEFLATED #define Z_DEFAULT_WINDOW_BITS MZ_DEFAULT_WINDOW_BITS #define alloc_func mz_alloc_func #define free_func mz_free_func #define internal_state mz_internal_state #define z_stream mz_stream #define deflateInit mz_deflateInit #define deflateInit2 mz_deflateInit2 #define deflateReset mz_deflateReset #define deflate mz_deflate #define deflateEnd mz_deflateEnd #define deflateBound mz_deflateBound #define compress mz_compress #define compress2 mz_compress2 #define compressBound mz_compressBound #define inflateInit mz_inflateInit #define inflateInit2 mz_inflateInit2 #define inflateReset mz_inflateReset #define inflate mz_inflate #define inflateEnd mz_inflateEnd #define uncompress mz_uncompress #define uncompress2 mz_uncompress2 #define crc32 mz_crc32 #define adler32 mz_adler32 #define MAX_WBITS 15 #define MAX_MEM_LEVEL 9 #define zError mz_error #define ZLIB_VERSION MZ_VERSION #define ZLIB_VERNUM MZ_VERNUM #define ZLIB_VER_MAJOR MZ_VER_MAJOR #define ZLIB_VER_MINOR MZ_VER_MINOR #define ZLIB_VER_REVISION MZ_VER_REVISION #define ZLIB_VER_SUBREVISION MZ_VER_SUBREVISION #define zlibVersion mz_version #define zlib_version mz_version() #endif /* #ifndef MINIZ_NO_ZLIB_COMPATIBLE_NAMES */ #endif /* MINIZ_NO_ZLIB_APIS */ #ifdef __cplusplus } #endif #pragma once #include #include #include #include /* ------------------- Types and macros */ typedef unsigned char mz_uint8; typedef signed short mz_int16; typedef unsigned short mz_uint16; typedef unsigned int mz_uint32; typedef unsigned int mz_uint; typedef int64_t mz_int64; typedef uint64_t mz_uint64; typedef int mz_bool; #define MZ_FALSE (0) #define MZ_TRUE (1) /* Works around MSVC's spammy "warning C4127: conditional expression is constant" message. */ #ifdef _MSC_VER #define MZ_MACRO_END while (0, 0) #else #define MZ_MACRO_END while (0) #endif #ifdef MINIZ_NO_STDIO #define MZ_FILE void * #else #include #define MZ_FILE FILE #endif /* #ifdef MINIZ_NO_STDIO */ #ifdef MINIZ_NO_TIME typedef struct mz_dummy_time_t_tag { int m_dummy; } mz_dummy_time_t; #define MZ_TIME_T mz_dummy_time_t #else #define MZ_TIME_T time_t #endif #define MZ_ASSERT(x) assert(x) #ifdef MINIZ_NO_MALLOC #define MZ_MALLOC(x) NULL #define MZ_FREE(x) (void)x, ((void)0) #define MZ_REALLOC(p, x) NULL #else #define MZ_MALLOC(x) malloc(x) #define MZ_FREE(x) free(x) #define MZ_REALLOC(p, x) realloc(p, x) #endif #define MZ_MAX(a, b) (((a) > (b)) ? (a) : (b)) #define MZ_MIN(a, b) (((a) < (b)) ? (a) : (b)) #define MZ_CLEAR_OBJ(obj) memset(&(obj), 0, sizeof(obj)) #if MINIZ_USE_UNALIGNED_LOADS_AND_STORES && MINIZ_LITTLE_ENDIAN #define MZ_READ_LE16(p) *((const mz_uint16 *)(p)) #define MZ_READ_LE32(p) *((const mz_uint32 *)(p)) #else #define MZ_READ_LE16(p) ((mz_uint32)(((const mz_uint8 *)(p))[0]) | ((mz_uint32)(((const mz_uint8 *)(p))[1]) << 8U)) #define MZ_READ_LE32(p) ((mz_uint32)(((const mz_uint8 *)(p))[0]) | ((mz_uint32)(((const mz_uint8 *)(p))[1]) << 8U) | ((mz_uint32)(((const mz_uint8 *)(p))[2]) << 16U) | ((mz_uint32)(((const mz_uint8 *)(p))[3]) << 24U)) #endif #define MZ_READ_LE64(p) (((mz_uint64)MZ_READ_LE32(p)) | (((mz_uint64)MZ_READ_LE32((const mz_uint8 *)(p) + sizeof(mz_uint32))) << 32U)) #ifdef _MSC_VER #define MZ_FORCEINLINE __forceinline #elif defined(__GNUC__) #define MZ_FORCEINLINE __inline__ __attribute__((__always_inline__)) #else #define MZ_FORCEINLINE inline #endif #ifdef __cplusplus extern "C" { #endif extern MINIZ_EXPORT void *miniz_def_alloc_func(void *opaque, size_t items, size_t size); extern MINIZ_EXPORT void miniz_def_free_func(void *opaque, void *address); extern MINIZ_EXPORT void *miniz_def_realloc_func(void *opaque, void *address, size_t items, size_t size); #define MZ_UINT16_MAX (0xFFFFU) #define MZ_UINT32_MAX (0xFFFFFFFFU) #ifdef __cplusplus } #endif #pragma once #ifdef __cplusplus extern "C" { #endif /* ------------------- Low-level Compression API Definitions */ /* Set TDEFL_LESS_MEMORY to 1 to use less memory (compression will be slightly slower, and raw/dynamic blocks will be output more frequently). */ #define TDEFL_LESS_MEMORY 0 /* tdefl_init() compression flags logically OR'd together (low 12 bits contain the max. number of probes per dictionary search): */ /* TDEFL_DEFAULT_MAX_PROBES: The compressor defaults to 128 dictionary probes per dictionary search. 0=Huffman only, 1=Huffman+LZ (fastest/crap compression), 4095=Huffman+LZ (slowest/best compression). */ enum { TDEFL_HUFFMAN_ONLY = 0, TDEFL_DEFAULT_MAX_PROBES = 128, TDEFL_MAX_PROBES_MASK = 0xFFF }; /* TDEFL_WRITE_ZLIB_HEADER: If set, the compressor outputs a zlib header before the deflate data, and the Adler-32 of the source data at the end. Otherwise, you'll get raw deflate data. */ /* TDEFL_COMPUTE_ADLER32: Always compute the adler-32 of the input data (even when not writing zlib headers). */ /* TDEFL_GREEDY_PARSING_FLAG: Set to use faster greedy parsing, instead of more efficient lazy parsing. */ /* TDEFL_NONDETERMINISTIC_PARSING_FLAG: Enable to decrease the compressor's initialization time to the minimum, but the output may vary from run to run given the same input (depending on the contents of memory). */ /* TDEFL_RLE_MATCHES: Only look for RLE matches (matches with a distance of 1) */ /* TDEFL_FILTER_MATCHES: Discards matches <= 5 chars if enabled. */ /* TDEFL_FORCE_ALL_STATIC_BLOCKS: Disable usage of optimized Huffman tables. */ /* TDEFL_FORCE_ALL_RAW_BLOCKS: Only use raw (uncompressed) deflate blocks. */ /* The low 12 bits are reserved to control the max # of hash probes per dictionary lookup (see TDEFL_MAX_PROBES_MASK). */ enum { TDEFL_WRITE_ZLIB_HEADER = 0x01000, TDEFL_COMPUTE_ADLER32 = 0x02000, TDEFL_GREEDY_PARSING_FLAG = 0x04000, TDEFL_NONDETERMINISTIC_PARSING_FLAG = 0x08000, TDEFL_RLE_MATCHES = 0x10000, TDEFL_FILTER_MATCHES = 0x20000, TDEFL_FORCE_ALL_STATIC_BLOCKS = 0x40000, TDEFL_FORCE_ALL_RAW_BLOCKS = 0x80000 }; /* High level compression functions: */ /* tdefl_compress_mem_to_heap() compresses a block in memory to a heap block allocated via malloc(). */ /* On entry: */ /* pSrc_buf, src_buf_len: Pointer and size of source block to compress. */ /* flags: The max match finder probes (default is 128) logically OR'd against the above flags. Higher probes are slower but improve compression. */ /* On return: */ /* Function returns a pointer to the compressed data, or NULL on failure. */ /* *pOut_len will be set to the compressed data's size, which could be larger than src_buf_len on uncompressible data. */ /* The caller must free() the returned block when it's no longer needed. */ MINIZ_EXPORT void *tdefl_compress_mem_to_heap(const void *pSrc_buf, size_t src_buf_len, size_t *pOut_len, int flags); /* tdefl_compress_mem_to_mem() compresses a block in memory to another block in memory. */ /* Returns 0 on failure. */ MINIZ_EXPORT size_t tdefl_compress_mem_to_mem(void *pOut_buf, size_t out_buf_len, const void *pSrc_buf, size_t src_buf_len, int flags); /* Compresses an image to a compressed PNG file in memory. */ /* On entry: */ /* pImage, w, h, and num_chans describe the image to compress. num_chans may be 1, 2, 3, or 4. */ /* The image pitch in bytes per scanline will be w*num_chans. The leftmost pixel on the top scanline is stored first in memory. */ /* level may range from [0,10], use MZ_NO_COMPRESSION, MZ_BEST_SPEED, MZ_BEST_COMPRESSION, etc. or a decent default is MZ_DEFAULT_LEVEL */ /* If flip is true, the image will be flipped on the Y axis (useful for OpenGL apps). */ /* On return: */ /* Function returns a pointer to the compressed data, or NULL on failure. */ /* *pLen_out will be set to the size of the PNG image file. */ /* The caller must mz_free() the returned heap block (which will typically be larger than *pLen_out) when it's no longer needed. */ MINIZ_EXPORT void *tdefl_write_image_to_png_file_in_memory_ex(const void *pImage, int w, int h, int num_chans, size_t *pLen_out, mz_uint level, mz_bool flip); MINIZ_EXPORT void *tdefl_write_image_to_png_file_in_memory(const void *pImage, int w, int h, int num_chans, size_t *pLen_out); /* Output stream interface. The compressor uses this interface to write compressed data. It'll typically be called TDEFL_OUT_BUF_SIZE at a time. */ typedef mz_bool (*tdefl_put_buf_func_ptr)(const void *pBuf, int len, void *pUser); /* tdefl_compress_mem_to_output() compresses a block to an output stream. The above helpers use this function internally. */ MINIZ_EXPORT mz_bool tdefl_compress_mem_to_output(const void *pBuf, size_t buf_len, tdefl_put_buf_func_ptr pPut_buf_func, void *pPut_buf_user, int flags); enum { TDEFL_MAX_HUFF_TABLES = 3, TDEFL_MAX_HUFF_SYMBOLS_0 = 288, TDEFL_MAX_HUFF_SYMBOLS_1 = 32, TDEFL_MAX_HUFF_SYMBOLS_2 = 19, TDEFL_LZ_DICT_SIZE = 32768, TDEFL_LZ_DICT_SIZE_MASK = TDEFL_LZ_DICT_SIZE - 1, TDEFL_MIN_MATCH_LEN = 3, TDEFL_MAX_MATCH_LEN = 258 }; /* TDEFL_OUT_BUF_SIZE MUST be large enough to hold a single entire compressed output block (using static/fixed Huffman codes). */ #if TDEFL_LESS_MEMORY enum { TDEFL_LZ_CODE_BUF_SIZE = 24 * 1024, TDEFL_OUT_BUF_SIZE = (TDEFL_LZ_CODE_BUF_SIZE * 13) / 10, TDEFL_MAX_HUFF_SYMBOLS = 288, TDEFL_LZ_HASH_BITS = 12, TDEFL_LEVEL1_HASH_SIZE_MASK = 4095, TDEFL_LZ_HASH_SHIFT = (TDEFL_LZ_HASH_BITS + 2) / 3, TDEFL_LZ_HASH_SIZE = 1 << TDEFL_LZ_HASH_BITS }; #else enum { TDEFL_LZ_CODE_BUF_SIZE = 64 * 1024, TDEFL_OUT_BUF_SIZE = (TDEFL_LZ_CODE_BUF_SIZE * 13) / 10, TDEFL_MAX_HUFF_SYMBOLS = 288, TDEFL_LZ_HASH_BITS = 15, TDEFL_LEVEL1_HASH_SIZE_MASK = 4095, TDEFL_LZ_HASH_SHIFT = (TDEFL_LZ_HASH_BITS + 2) / 3, TDEFL_LZ_HASH_SIZE = 1 << TDEFL_LZ_HASH_BITS }; #endif /* The low-level tdefl functions below may be used directly if the above helper functions aren't flexible enough. The low-level functions don't make any heap allocations, unlike the above helper functions. */ typedef enum { TDEFL_STATUS_BAD_PARAM = -2, TDEFL_STATUS_PUT_BUF_FAILED = -1, TDEFL_STATUS_OKAY = 0, TDEFL_STATUS_DONE = 1 } tdefl_status; /* Must map to MZ_NO_FLUSH, MZ_SYNC_FLUSH, etc. enums */ typedef enum { TDEFL_NO_FLUSH = 0, TDEFL_SYNC_FLUSH = 2, TDEFL_FULL_FLUSH = 3, TDEFL_FINISH = 4 } tdefl_flush; /* tdefl's compression state structure. */ typedef struct { tdefl_put_buf_func_ptr m_pPut_buf_func; void *m_pPut_buf_user; mz_uint m_flags, m_max_probes[2]; int m_greedy_parsing; mz_uint m_adler32, m_lookahead_pos, m_lookahead_size, m_dict_size; mz_uint8 *m_pLZ_code_buf, *m_pLZ_flags, *m_pOutput_buf, *m_pOutput_buf_end; mz_uint m_num_flags_left, m_total_lz_bytes, m_lz_code_buf_dict_pos, m_bits_in, m_bit_buffer; mz_uint m_saved_match_dist, m_saved_match_len, m_saved_lit, m_output_flush_ofs, m_output_flush_remaining, m_finished, m_block_index, m_wants_to_finish; tdefl_status m_prev_return_status; const void *m_pIn_buf; void *m_pOut_buf; size_t *m_pIn_buf_size, *m_pOut_buf_size; tdefl_flush m_flush; const mz_uint8 *m_pSrc; size_t m_src_buf_left, m_out_buf_ofs; mz_uint8 m_dict[TDEFL_LZ_DICT_SIZE + TDEFL_MAX_MATCH_LEN - 1]; mz_uint16 m_huff_count[TDEFL_MAX_HUFF_TABLES][TDEFL_MAX_HUFF_SYMBOLS]; mz_uint16 m_huff_codes[TDEFL_MAX_HUFF_TABLES][TDEFL_MAX_HUFF_SYMBOLS]; mz_uint8 m_huff_code_sizes[TDEFL_MAX_HUFF_TABLES][TDEFL_MAX_HUFF_SYMBOLS]; mz_uint8 m_lz_code_buf[TDEFL_LZ_CODE_BUF_SIZE]; mz_uint16 m_next[TDEFL_LZ_DICT_SIZE]; mz_uint16 m_hash[TDEFL_LZ_HASH_SIZE]; mz_uint8 m_output_buf[TDEFL_OUT_BUF_SIZE]; } tdefl_compressor; /* Initializes the compressor. */ /* There is no corresponding deinit() function because the tdefl API's do not dynamically allocate memory. */ /* pBut_buf_func: If NULL, output data will be supplied to the specified callback. In this case, the user should call the tdefl_compress_buffer() API for compression. */ /* If pBut_buf_func is NULL the user should always call the tdefl_compress() API. */ /* flags: See the above enums (TDEFL_HUFFMAN_ONLY, TDEFL_WRITE_ZLIB_HEADER, etc.) */ MINIZ_EXPORT tdefl_status tdefl_init(tdefl_compressor *d, tdefl_put_buf_func_ptr pPut_buf_func, void *pPut_buf_user, int flags); /* Compresses a block of data, consuming as much of the specified input buffer as possible, and writing as much compressed data to the specified output buffer as possible. */ MINIZ_EXPORT tdefl_status tdefl_compress(tdefl_compressor *d, const void *pIn_buf, size_t *pIn_buf_size, void *pOut_buf, size_t *pOut_buf_size, tdefl_flush flush); /* tdefl_compress_buffer() is only usable when the tdefl_init() is called with a non-NULL tdefl_put_buf_func_ptr. */ /* tdefl_compress_buffer() always consumes the entire input buffer. */ MINIZ_EXPORT tdefl_status tdefl_compress_buffer(tdefl_compressor *d, const void *pIn_buf, size_t in_buf_size, tdefl_flush flush); MINIZ_EXPORT tdefl_status tdefl_get_prev_return_status(tdefl_compressor *d); MINIZ_EXPORT mz_uint32 tdefl_get_adler32(tdefl_compressor *d); /* Create tdefl_compress() flags given zlib-style compression parameters. */ /* level may range from [0,10] (where 10 is absolute max compression, but may be much slower on some files) */ /* window_bits may be -15 (raw deflate) or 15 (zlib) */ /* strategy may be either MZ_DEFAULT_STRATEGY, MZ_FILTERED, MZ_HUFFMAN_ONLY, MZ_RLE, or MZ_FIXED */ MINIZ_EXPORT mz_uint tdefl_create_comp_flags_from_zip_params(int level, int window_bits, int strategy); #ifndef MINIZ_NO_MALLOC /* Allocate the tdefl_compressor structure in C so that */ /* non-C language bindings to tdefl_ API don't need to worry about */ /* structure size and allocation mechanism. */ MINIZ_EXPORT tdefl_compressor *tdefl_compressor_alloc(void); MINIZ_EXPORT void tdefl_compressor_free(tdefl_compressor *pComp); #endif #ifdef __cplusplus } #endif #pragma once /* ------------------- Low-level Decompression API Definitions */ #ifdef __cplusplus extern "C" { #endif /* Decompression flags used by tinfl_decompress(). */ /* TINFL_FLAG_PARSE_ZLIB_HEADER: If set, the input has a valid zlib header and ends with an adler32 checksum (it's a valid zlib stream). Otherwise, the input is a raw deflate stream. */ /* TINFL_FLAG_HAS_MORE_INPUT: If set, there are more input bytes available beyond the end of the supplied input buffer. If clear, the input buffer contains all remaining input. */ /* TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF: If set, the output buffer is large enough to hold the entire decompressed stream. If clear, the output buffer is at least the size of the dictionary (typically 32KB). */ /* TINFL_FLAG_COMPUTE_ADLER32: Force adler-32 checksum computation of the decompressed bytes. */ enum { TINFL_FLAG_PARSE_ZLIB_HEADER = 1, TINFL_FLAG_HAS_MORE_INPUT = 2, TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF = 4, TINFL_FLAG_COMPUTE_ADLER32 = 8 }; /* High level decompression functions: */ /* tinfl_decompress_mem_to_heap() decompresses a block in memory to a heap block allocated via malloc(). */ /* On entry: */ /* pSrc_buf, src_buf_len: Pointer and size of the Deflate or zlib source data to decompress. */ /* On return: */ /* Function returns a pointer to the decompressed data, or NULL on failure. */ /* *pOut_len will be set to the decompressed data's size, which could be larger than src_buf_len on uncompressible data. */ /* The caller must call mz_free() on the returned block when it's no longer needed. */ MINIZ_EXPORT void *tinfl_decompress_mem_to_heap(const void *pSrc_buf, size_t src_buf_len, size_t *pOut_len, int flags); /* tinfl_decompress_mem_to_mem() decompresses a block in memory to another block in memory. */ /* Returns TINFL_DECOMPRESS_MEM_TO_MEM_FAILED on failure, or the number of bytes written on success. */ #define TINFL_DECOMPRESS_MEM_TO_MEM_FAILED ((size_t)(-1)) MINIZ_EXPORT size_t tinfl_decompress_mem_to_mem(void *pOut_buf, size_t out_buf_len, const void *pSrc_buf, size_t src_buf_len, int flags); /* tinfl_decompress_mem_to_callback() decompresses a block in memory to an internal 32KB buffer, and a user provided callback function will be called to flush the buffer. */ /* Returns 1 on success or 0 on failure. */ typedef int (*tinfl_put_buf_func_ptr)(const void *pBuf, int len, void *pUser); MINIZ_EXPORT int tinfl_decompress_mem_to_callback(const void *pIn_buf, size_t *pIn_buf_size, tinfl_put_buf_func_ptr pPut_buf_func, void *pPut_buf_user, int flags); struct tinfl_decompressor_tag; typedef struct tinfl_decompressor_tag tinfl_decompressor; #ifndef MINIZ_NO_MALLOC /* Allocate the tinfl_decompressor structure in C so that */ /* non-C language bindings to tinfl_ API don't need to worry about */ /* structure size and allocation mechanism. */ MINIZ_EXPORT tinfl_decompressor *tinfl_decompressor_alloc(void); MINIZ_EXPORT void tinfl_decompressor_free(tinfl_decompressor *pDecomp); #endif /* Max size of LZ dictionary. */ #define TINFL_LZ_DICT_SIZE 32768 /* Return status. */ typedef enum { /* This flags indicates the inflator needs 1 or more input bytes to make forward progress, but the caller is indicating that no more are available. The compressed data */ /* is probably corrupted. If you call the inflator again with more bytes it'll try to continue processing the input but this is a BAD sign (either the data is corrupted or you called it incorrectly). */ /* If you call it again with no input you'll just get TINFL_STATUS_FAILED_CANNOT_MAKE_PROGRESS again. */ TINFL_STATUS_FAILED_CANNOT_MAKE_PROGRESS = -4, /* This flag indicates that one or more of the input parameters was obviously bogus. (You can try calling it again, but if you get this error the calling code is wrong.) */ TINFL_STATUS_BAD_PARAM = -3, /* This flags indicate the inflator is finished but the adler32 check of the uncompressed data didn't match. If you call it again it'll return TINFL_STATUS_DONE. */ TINFL_STATUS_ADLER32_MISMATCH = -2, /* This flags indicate the inflator has somehow failed (bad code, corrupted input, etc.). If you call it again without resetting via tinfl_init() it it'll just keep on returning the same status failure code. */ TINFL_STATUS_FAILED = -1, /* Any status code less than TINFL_STATUS_DONE must indicate a failure. */ /* This flag indicates the inflator has returned every byte of uncompressed data that it can, has consumed every byte that it needed, has successfully reached the end of the deflate stream, and */ /* if zlib headers and adler32 checking enabled that it has successfully checked the uncompressed data's adler32. If you call it again you'll just get TINFL_STATUS_DONE over and over again. */ TINFL_STATUS_DONE = 0, /* This flag indicates the inflator MUST have more input data (even 1 byte) before it can make any more forward progress, or you need to clear the TINFL_FLAG_HAS_MORE_INPUT */ /* flag on the next call if you don't have any more source data. If the source data was somehow corrupted it's also possible (but unlikely) for the inflator to keep on demanding input to */ /* proceed, so be sure to properly set the TINFL_FLAG_HAS_MORE_INPUT flag. */ TINFL_STATUS_NEEDS_MORE_INPUT = 1, /* This flag indicates the inflator definitely has 1 or more bytes of uncompressed data available, but it cannot write this data into the output buffer. */ /* Note if the source compressed data was corrupted it's possible for the inflator to return a lot of uncompressed data to the caller. I've been assuming you know how much uncompressed data to expect */ /* (either exact or worst case) and will stop calling the inflator and fail after receiving too much. In pure streaming scenarios where you have no idea how many bytes to expect this may not be possible */ /* so I may need to add some code to address this. */ TINFL_STATUS_HAS_MORE_OUTPUT = 2 } tinfl_status; /* Initializes the decompressor to its initial state. */ #define tinfl_init(r) \ do \ { \ (r)->m_state = 0; \ } \ MZ_MACRO_END #define tinfl_get_adler32(r) (r)->m_check_adler32 /* Main low-level decompressor coroutine function. This is the only function actually needed for decompression. All the other functions are just high-level helpers for improved usability. */ /* This is a universal API, i.e. it can be used as a building block to build any desired higher level decompression API. In the limit case, it can be called once per every byte input or output. */ MINIZ_EXPORT tinfl_status tinfl_decompress(tinfl_decompressor *r, const mz_uint8 *pIn_buf_next, size_t *pIn_buf_size, mz_uint8 *pOut_buf_start, mz_uint8 *pOut_buf_next, size_t *pOut_buf_size, const mz_uint32 decomp_flags); /* Internal/private bits follow. */ enum { TINFL_MAX_HUFF_TABLES = 3, TINFL_MAX_HUFF_SYMBOLS_0 = 288, TINFL_MAX_HUFF_SYMBOLS_1 = 32, TINFL_MAX_HUFF_SYMBOLS_2 = 19, TINFL_FAST_LOOKUP_BITS = 10, TINFL_FAST_LOOKUP_SIZE = 1 << TINFL_FAST_LOOKUP_BITS }; typedef struct { mz_uint8 m_code_size[TINFL_MAX_HUFF_SYMBOLS_0]; mz_int16 m_look_up[TINFL_FAST_LOOKUP_SIZE], m_tree[TINFL_MAX_HUFF_SYMBOLS_0 * 2]; } tinfl_huff_table; #if MINIZ_HAS_64BIT_REGISTERS #define TINFL_USE_64BIT_BITBUF 1 #else #define TINFL_USE_64BIT_BITBUF 0 #endif #if TINFL_USE_64BIT_BITBUF typedef mz_uint64 tinfl_bit_buf_t; #define TINFL_BITBUF_SIZE (64) #else typedef mz_uint32 tinfl_bit_buf_t; #define TINFL_BITBUF_SIZE (32) #endif struct tinfl_decompressor_tag { mz_uint32 m_state, m_num_bits, m_zhdr0, m_zhdr1, m_z_adler32, m_final, m_type, m_check_adler32, m_dist, m_counter, m_num_extra, m_table_sizes[TINFL_MAX_HUFF_TABLES]; tinfl_bit_buf_t m_bit_buf; size_t m_dist_from_out_buf_start; tinfl_huff_table m_tables[TINFL_MAX_HUFF_TABLES]; mz_uint8 m_raw_header[4], m_len_codes[TINFL_MAX_HUFF_SYMBOLS_0 + TINFL_MAX_HUFF_SYMBOLS_1 + 137]; }; #ifdef __cplusplus } #endif #pragma once /* ------------------- ZIP archive reading/writing */ #ifndef MINIZ_NO_ARCHIVE_APIS #ifdef __cplusplus extern "C" { #endif enum { /* Note: These enums can be reduced as needed to save memory or stack space - they are pretty conservative. */ MZ_ZIP_MAX_IO_BUF_SIZE = 64 * 1024, MZ_ZIP_MAX_ARCHIVE_FILENAME_SIZE = 512, MZ_ZIP_MAX_ARCHIVE_FILE_COMMENT_SIZE = 512 }; typedef struct { /* Central directory file index. */ mz_uint32 m_file_index; /* Byte offset of this entry in the archive's central directory. Note we currently only support up to UINT_MAX or less bytes in the central dir. */ mz_uint64 m_central_dir_ofs; /* These fields are copied directly from the zip's central dir. */ mz_uint16 m_version_made_by; mz_uint16 m_version_needed; mz_uint16 m_bit_flag; mz_uint16 m_method; #ifndef MINIZ_NO_TIME MZ_TIME_T m_time; #endif /* CRC-32 of uncompressed data. */ mz_uint32 m_crc32; /* File's compressed size. */ mz_uint64 m_comp_size; /* File's uncompressed size. Note, I've seen some old archives where directory entries had 512 bytes for their uncompressed sizes, but when you try to unpack them you actually get 0 bytes. */ mz_uint64 m_uncomp_size; /* Zip internal and external file attributes. */ mz_uint16 m_internal_attr; mz_uint32 m_external_attr; /* Entry's local header file offset in bytes. */ mz_uint64 m_local_header_ofs; /* Size of comment in bytes. */ mz_uint32 m_comment_size; /* MZ_TRUE if the entry appears to be a directory. */ mz_bool m_is_directory; /* MZ_TRUE if the entry uses encryption/strong encryption (which miniz_zip doesn't support) */ mz_bool m_is_encrypted; /* MZ_TRUE if the file is not encrypted, a patch file, and if it uses a compression method we support. */ mz_bool m_is_supported; /* Filename. If string ends in '/' it's a subdirectory entry. */ /* Guaranteed to be zero terminated, may be truncated to fit. */ char m_filename[MZ_ZIP_MAX_ARCHIVE_FILENAME_SIZE]; /* Comment field. */ /* Guaranteed to be zero terminated, may be truncated to fit. */ char m_comment[MZ_ZIP_MAX_ARCHIVE_FILE_COMMENT_SIZE]; } mz_zip_archive_file_stat; typedef size_t (*mz_file_read_func)(void *pOpaque, mz_uint64 file_ofs, void *pBuf, size_t n); typedef size_t (*mz_file_write_func)(void *pOpaque, mz_uint64 file_ofs, const void *pBuf, size_t n); typedef mz_bool (*mz_file_needs_keepalive)(void *pOpaque); struct mz_zip_internal_state_tag; typedef struct mz_zip_internal_state_tag mz_zip_internal_state; typedef enum { MZ_ZIP_MODE_INVALID = 0, MZ_ZIP_MODE_READING = 1, MZ_ZIP_MODE_WRITING = 2, MZ_ZIP_MODE_WRITING_HAS_BEEN_FINALIZED = 3 } mz_zip_mode; typedef enum { MZ_ZIP_FLAG_CASE_SENSITIVE = 0x0100, MZ_ZIP_FLAG_IGNORE_PATH = 0x0200, MZ_ZIP_FLAG_COMPRESSED_DATA = 0x0400, MZ_ZIP_FLAG_DO_NOT_SORT_CENTRAL_DIRECTORY = 0x0800, MZ_ZIP_FLAG_VALIDATE_LOCATE_FILE_FLAG = 0x1000, /* if enabled, mz_zip_reader_locate_file() will be called on each file as its validated to ensure the func finds the file in the central dir (intended for testing) */ MZ_ZIP_FLAG_VALIDATE_HEADERS_ONLY = 0x2000, /* validate the local headers, but don't decompress the entire file and check the crc32 */ MZ_ZIP_FLAG_WRITE_ZIP64 = 0x4000, /* always use the zip64 file format, instead of the original zip file format with automatic switch to zip64. Use as flags parameter with mz_zip_writer_init*_v2 */ MZ_ZIP_FLAG_WRITE_ALLOW_READING = 0x8000, MZ_ZIP_FLAG_ASCII_FILENAME = 0x10000, /*After adding a compressed file, seek back to local file header and set the correct sizes*/ MZ_ZIP_FLAG_WRITE_HEADER_SET_SIZE = 0x20000 } mz_zip_flags; typedef enum { MZ_ZIP_TYPE_INVALID = 0, MZ_ZIP_TYPE_USER, MZ_ZIP_TYPE_MEMORY, MZ_ZIP_TYPE_HEAP, MZ_ZIP_TYPE_FILE, MZ_ZIP_TYPE_CFILE, MZ_ZIP_TOTAL_TYPES } mz_zip_type; /* miniz error codes. Be sure to update mz_zip_get_error_string() if you add or modify this enum. */ typedef enum { MZ_ZIP_NO_ERROR = 0, MZ_ZIP_UNDEFINED_ERROR, MZ_ZIP_TOO_MANY_FILES, MZ_ZIP_FILE_TOO_LARGE, MZ_ZIP_UNSUPPORTED_METHOD, MZ_ZIP_UNSUPPORTED_ENCRYPTION, MZ_ZIP_UNSUPPORTED_FEATURE, MZ_ZIP_FAILED_FINDING_CENTRAL_DIR, MZ_ZIP_NOT_AN_ARCHIVE, MZ_ZIP_INVALID_HEADER_OR_CORRUPTED, MZ_ZIP_UNSUPPORTED_MULTIDISK, MZ_ZIP_DECOMPRESSION_FAILED, MZ_ZIP_COMPRESSION_FAILED, MZ_ZIP_UNEXPECTED_DECOMPRESSED_SIZE, MZ_ZIP_CRC_CHECK_FAILED, MZ_ZIP_UNSUPPORTED_CDIR_SIZE, MZ_ZIP_ALLOC_FAILED, MZ_ZIP_FILE_OPEN_FAILED, MZ_ZIP_FILE_CREATE_FAILED, MZ_ZIP_FILE_WRITE_FAILED, MZ_ZIP_FILE_READ_FAILED, MZ_ZIP_FILE_CLOSE_FAILED, MZ_ZIP_FILE_SEEK_FAILED, MZ_ZIP_FILE_STAT_FAILED, MZ_ZIP_INVALID_PARAMETER, MZ_ZIP_INVALID_FILENAME, MZ_ZIP_BUF_TOO_SMALL, MZ_ZIP_INTERNAL_ERROR, MZ_ZIP_FILE_NOT_FOUND, MZ_ZIP_ARCHIVE_TOO_LARGE, MZ_ZIP_VALIDATION_FAILED, MZ_ZIP_WRITE_CALLBACK_FAILED, MZ_ZIP_TOTAL_ERRORS } mz_zip_error; typedef struct { mz_uint64 m_archive_size; mz_uint64 m_central_directory_file_ofs; /* We only support up to UINT32_MAX files in zip64 mode. */ mz_uint32 m_total_files; mz_zip_mode m_zip_mode; mz_zip_type m_zip_type; mz_zip_error m_last_error; mz_uint64 m_file_offset_alignment; mz_alloc_func m_pAlloc; mz_free_func m_pFree; mz_realloc_func m_pRealloc; void *m_pAlloc_opaque; mz_file_read_func m_pRead; mz_file_write_func m_pWrite; mz_file_needs_keepalive m_pNeeds_keepalive; void *m_pIO_opaque; mz_zip_internal_state *m_pState; } mz_zip_archive; typedef struct { mz_zip_archive *pZip; mz_uint flags; int status; #ifndef MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS mz_uint file_crc32; #endif mz_uint64 read_buf_size, read_buf_ofs, read_buf_avail, comp_remaining, out_buf_ofs, cur_file_ofs; mz_zip_archive_file_stat file_stat; void *pRead_buf; void *pWrite_buf; size_t out_blk_remain; tinfl_decompressor inflator; } mz_zip_reader_extract_iter_state; /* -------- ZIP reading */ /* Inits a ZIP archive reader. */ /* These functions read and validate the archive's central directory. */ MINIZ_EXPORT mz_bool mz_zip_reader_init(mz_zip_archive *pZip, mz_uint64 size, mz_uint flags); MINIZ_EXPORT mz_bool mz_zip_reader_init_mem(mz_zip_archive *pZip, const void *pMem, size_t size, mz_uint flags); #ifndef MINIZ_NO_STDIO /* Read a archive from a disk file. */ /* file_start_ofs is the file offset where the archive actually begins, or 0. */ /* actual_archive_size is the true total size of the archive, which may be smaller than the file's actual size on disk. If zero the entire file is treated as the archive. */ MINIZ_EXPORT mz_bool mz_zip_reader_init_file(mz_zip_archive *pZip, const char *pFilename, mz_uint32 flags); MINIZ_EXPORT mz_bool mz_zip_reader_init_file_v2(mz_zip_archive *pZip, const char *pFilename, mz_uint flags, mz_uint64 file_start_ofs, mz_uint64 archive_size); /* Read an archive from an already opened FILE, beginning at the current file position. */ /* The archive is assumed to be archive_size bytes long. If archive_size is 0, then the entire rest of the file is assumed to contain the archive. */ /* The FILE will NOT be closed when mz_zip_reader_end() is called. */ MINIZ_EXPORT mz_bool mz_zip_reader_init_cfile(mz_zip_archive *pZip, MZ_FILE *pFile, mz_uint64 archive_size, mz_uint flags); #endif /* Ends archive reading, freeing all allocations, and closing the input archive file if mz_zip_reader_init_file() was used. */ MINIZ_EXPORT mz_bool mz_zip_reader_end(mz_zip_archive *pZip); /* -------- ZIP reading or writing */ /* Clears a mz_zip_archive struct to all zeros. */ /* Important: This must be done before passing the struct to any mz_zip functions. */ MINIZ_EXPORT void mz_zip_zero_struct(mz_zip_archive *pZip); MINIZ_EXPORT mz_zip_mode mz_zip_get_mode(mz_zip_archive *pZip); MINIZ_EXPORT mz_zip_type mz_zip_get_type(mz_zip_archive *pZip); /* Returns the total number of files in the archive. */ MINIZ_EXPORT mz_uint mz_zip_reader_get_num_files(mz_zip_archive *pZip); MINIZ_EXPORT mz_uint64 mz_zip_get_archive_size(mz_zip_archive *pZip); MINIZ_EXPORT mz_uint64 mz_zip_get_archive_file_start_offset(mz_zip_archive *pZip); MINIZ_EXPORT MZ_FILE *mz_zip_get_cfile(mz_zip_archive *pZip); /* Reads n bytes of raw archive data, starting at file offset file_ofs, to pBuf. */ MINIZ_EXPORT size_t mz_zip_read_archive_data(mz_zip_archive *pZip, mz_uint64 file_ofs, void *pBuf, size_t n); /* All mz_zip funcs set the m_last_error field in the mz_zip_archive struct. These functions retrieve/manipulate this field. */ /* Note that the m_last_error functionality is not thread safe. */ MINIZ_EXPORT mz_zip_error mz_zip_set_last_error(mz_zip_archive *pZip, mz_zip_error err_num); MINIZ_EXPORT mz_zip_error mz_zip_peek_last_error(mz_zip_archive *pZip); MINIZ_EXPORT mz_zip_error mz_zip_clear_last_error(mz_zip_archive *pZip); MINIZ_EXPORT mz_zip_error mz_zip_get_last_error(mz_zip_archive *pZip); MINIZ_EXPORT const char *mz_zip_get_error_string(mz_zip_error mz_err); /* MZ_TRUE if the archive file entry is a directory entry. */ MINIZ_EXPORT mz_bool mz_zip_reader_is_file_a_directory(mz_zip_archive *pZip, mz_uint file_index); /* MZ_TRUE if the file is encrypted/strong encrypted. */ MINIZ_EXPORT mz_bool mz_zip_reader_is_file_encrypted(mz_zip_archive *pZip, mz_uint file_index); /* MZ_TRUE if the compression method is supported, and the file is not encrypted, and the file is not a compressed patch file. */ MINIZ_EXPORT mz_bool mz_zip_reader_is_file_supported(mz_zip_archive *pZip, mz_uint file_index); /* Retrieves the filename of an archive file entry. */ /* Returns the number of bytes written to pFilename, or if filename_buf_size is 0 this function returns the number of bytes needed to fully store the filename. */ MINIZ_EXPORT mz_uint mz_zip_reader_get_filename(mz_zip_archive *pZip, mz_uint file_index, char *pFilename, mz_uint filename_buf_size); /* Attempts to locates a file in the archive's central directory. */ /* Valid flags: MZ_ZIP_FLAG_CASE_SENSITIVE, MZ_ZIP_FLAG_IGNORE_PATH */ /* Returns -1 if the file cannot be found. */ MINIZ_EXPORT int mz_zip_reader_locate_file(mz_zip_archive *pZip, const char *pName, const char *pComment, mz_uint flags); MINIZ_EXPORT mz_bool mz_zip_reader_locate_file_v2(mz_zip_archive *pZip, const char *pName, const char *pComment, mz_uint flags, mz_uint32 *file_index); /* Returns detailed information about an archive file entry. */ MINIZ_EXPORT mz_bool mz_zip_reader_file_stat(mz_zip_archive *pZip, mz_uint file_index, mz_zip_archive_file_stat *pStat); /* MZ_TRUE if the file is in zip64 format. */ /* A file is considered zip64 if it contained a zip64 end of central directory marker, or if it contained any zip64 extended file information fields in the central directory. */ MINIZ_EXPORT mz_bool mz_zip_is_zip64(mz_zip_archive *pZip); /* Returns the total central directory size in bytes. */ /* The current max supported size is <= MZ_UINT32_MAX. */ MINIZ_EXPORT size_t mz_zip_get_central_dir_size(mz_zip_archive *pZip); /* Extracts a archive file to a memory buffer using no memory allocation. */ /* There must be at least enough room on the stack to store the inflator's state (~34KB or so). */ MINIZ_EXPORT mz_bool mz_zip_reader_extract_to_mem_no_alloc(mz_zip_archive *pZip, mz_uint file_index, void *pBuf, size_t buf_size, mz_uint flags, void *pUser_read_buf, size_t user_read_buf_size); MINIZ_EXPORT mz_bool mz_zip_reader_extract_file_to_mem_no_alloc(mz_zip_archive *pZip, const char *pFilename, void *pBuf, size_t buf_size, mz_uint flags, void *pUser_read_buf, size_t user_read_buf_size); /* Extracts a archive file to a memory buffer. */ MINIZ_EXPORT mz_bool mz_zip_reader_extract_to_mem(mz_zip_archive *pZip, mz_uint file_index, void *pBuf, size_t buf_size, mz_uint flags); MINIZ_EXPORT mz_bool mz_zip_reader_extract_file_to_mem(mz_zip_archive *pZip, const char *pFilename, void *pBuf, size_t buf_size, mz_uint flags); /* Extracts a archive file to a dynamically allocated heap buffer. */ /* The memory will be allocated via the mz_zip_archive's alloc/realloc functions. */ /* Returns NULL and sets the last error on failure. */ MINIZ_EXPORT void *mz_zip_reader_extract_to_heap(mz_zip_archive *pZip, mz_uint file_index, size_t *pSize, mz_uint flags); MINIZ_EXPORT void *mz_zip_reader_extract_file_to_heap(mz_zip_archive *pZip, const char *pFilename, size_t *pSize, mz_uint flags); /* Extracts a archive file using a callback function to output the file's data. */ MINIZ_EXPORT mz_bool mz_zip_reader_extract_to_callback(mz_zip_archive *pZip, mz_uint file_index, mz_file_write_func pCallback, void *pOpaque, mz_uint flags); MINIZ_EXPORT mz_bool mz_zip_reader_extract_file_to_callback(mz_zip_archive *pZip, const char *pFilename, mz_file_write_func pCallback, void *pOpaque, mz_uint flags); /* Extract a file iteratively */ MINIZ_EXPORT mz_zip_reader_extract_iter_state* mz_zip_reader_extract_iter_new(mz_zip_archive *pZip, mz_uint file_index, mz_uint flags); MINIZ_EXPORT mz_zip_reader_extract_iter_state* mz_zip_reader_extract_file_iter_new(mz_zip_archive *pZip, const char *pFilename, mz_uint flags); MINIZ_EXPORT size_t mz_zip_reader_extract_iter_read(mz_zip_reader_extract_iter_state* pState, void* pvBuf, size_t buf_size); MINIZ_EXPORT mz_bool mz_zip_reader_extract_iter_free(mz_zip_reader_extract_iter_state* pState); #ifndef MINIZ_NO_STDIO /* Extracts a archive file to a disk file and sets its last accessed and modified times. */ /* This function only extracts files, not archive directory records. */ MINIZ_EXPORT mz_bool mz_zip_reader_extract_to_file(mz_zip_archive *pZip, mz_uint file_index, const char *pDst_filename, mz_uint flags); MINIZ_EXPORT mz_bool mz_zip_reader_extract_file_to_file(mz_zip_archive *pZip, const char *pArchive_filename, const char *pDst_filename, mz_uint flags); /* Extracts a archive file starting at the current position in the destination FILE stream. */ MINIZ_EXPORT mz_bool mz_zip_reader_extract_to_cfile(mz_zip_archive *pZip, mz_uint file_index, MZ_FILE *File, mz_uint flags); MINIZ_EXPORT mz_bool mz_zip_reader_extract_file_to_cfile(mz_zip_archive *pZip, const char *pArchive_filename, MZ_FILE *pFile, mz_uint flags); #endif #if 0 /* TODO */ typedef void *mz_zip_streaming_extract_state_ptr; mz_zip_streaming_extract_state_ptr mz_zip_streaming_extract_begin(mz_zip_archive *pZip, mz_uint file_index, mz_uint flags); uint64_t mz_zip_streaming_extract_get_size(mz_zip_archive *pZip, mz_zip_streaming_extract_state_ptr pState); uint64_t mz_zip_streaming_extract_get_cur_ofs(mz_zip_archive *pZip, mz_zip_streaming_extract_state_ptr pState); mz_bool mz_zip_streaming_extract_seek(mz_zip_archive *pZip, mz_zip_streaming_extract_state_ptr pState, uint64_t new_ofs); size_t mz_zip_streaming_extract_read(mz_zip_archive *pZip, mz_zip_streaming_extract_state_ptr pState, void *pBuf, size_t buf_size); mz_bool mz_zip_streaming_extract_end(mz_zip_archive *pZip, mz_zip_streaming_extract_state_ptr pState); #endif /* This function compares the archive's local headers, the optional local zip64 extended information block, and the optional descriptor following the compressed data vs. the data in the central directory. */ /* It also validates that each file can be successfully uncompressed unless the MZ_ZIP_FLAG_VALIDATE_HEADERS_ONLY is specified. */ MINIZ_EXPORT mz_bool mz_zip_validate_file(mz_zip_archive *pZip, mz_uint file_index, mz_uint flags); /* Validates an entire archive by calling mz_zip_validate_file() on each file. */ MINIZ_EXPORT mz_bool mz_zip_validate_archive(mz_zip_archive *pZip, mz_uint flags); /* Misc utils/helpers, valid for ZIP reading or writing */ MINIZ_EXPORT mz_bool mz_zip_validate_mem_archive(const void *pMem, size_t size, mz_uint flags, mz_zip_error *pErr); MINIZ_EXPORT mz_bool mz_zip_validate_file_archive(const char *pFilename, mz_uint flags, mz_zip_error *pErr); /* Universal end function - calls either mz_zip_reader_end() or mz_zip_writer_end(). */ MINIZ_EXPORT mz_bool mz_zip_end(mz_zip_archive *pZip); /* -------- ZIP writing */ #ifndef MINIZ_NO_ARCHIVE_WRITING_APIS /* Inits a ZIP archive writer. */ /*Set pZip->m_pWrite (and pZip->m_pIO_opaque) before calling mz_zip_writer_init or mz_zip_writer_init_v2*/ /*The output is streamable, i.e. file_ofs in mz_file_write_func always increases only by n*/ MINIZ_EXPORT mz_bool mz_zip_writer_init(mz_zip_archive *pZip, mz_uint64 existing_size); MINIZ_EXPORT mz_bool mz_zip_writer_init_v2(mz_zip_archive *pZip, mz_uint64 existing_size, mz_uint flags); MINIZ_EXPORT mz_bool mz_zip_writer_init_heap(mz_zip_archive *pZip, size_t size_to_reserve_at_beginning, size_t initial_allocation_size); MINIZ_EXPORT mz_bool mz_zip_writer_init_heap_v2(mz_zip_archive *pZip, size_t size_to_reserve_at_beginning, size_t initial_allocation_size, mz_uint flags); #ifndef MINIZ_NO_STDIO MINIZ_EXPORT mz_bool mz_zip_writer_init_file(mz_zip_archive *pZip, const char *pFilename, mz_uint64 size_to_reserve_at_beginning); MINIZ_EXPORT mz_bool mz_zip_writer_init_file_v2(mz_zip_archive *pZip, const char *pFilename, mz_uint64 size_to_reserve_at_beginning, mz_uint flags); MINIZ_EXPORT mz_bool mz_zip_writer_init_cfile(mz_zip_archive *pZip, MZ_FILE *pFile, mz_uint flags); #endif /* Converts a ZIP archive reader object into a writer object, to allow efficient in-place file appends to occur on an existing archive. */ /* For archives opened using mz_zip_reader_init_file, pFilename must be the archive's filename so it can be reopened for writing. If the file can't be reopened, mz_zip_reader_end() will be called. */ /* For archives opened using mz_zip_reader_init_mem, the memory block must be growable using the realloc callback (which defaults to realloc unless you've overridden it). */ /* Finally, for archives opened using mz_zip_reader_init, the mz_zip_archive's user provided m_pWrite function cannot be NULL. */ /* Note: In-place archive modification is not recommended unless you know what you're doing, because if execution stops or something goes wrong before */ /* the archive is finalized the file's central directory will be hosed. */ MINIZ_EXPORT mz_bool mz_zip_writer_init_from_reader(mz_zip_archive *pZip, const char *pFilename); MINIZ_EXPORT mz_bool mz_zip_writer_init_from_reader_v2(mz_zip_archive *pZip, const char *pFilename, mz_uint flags); /* Adds the contents of a memory buffer to an archive. These functions record the current local time into the archive. */ /* To add a directory entry, call this method with an archive name ending in a forwardslash with an empty buffer. */ /* level_and_flags - compression level (0-10, see MZ_BEST_SPEED, MZ_BEST_COMPRESSION, etc.) logically OR'd with zero or more mz_zip_flags, or just set to MZ_DEFAULT_COMPRESSION. */ MINIZ_EXPORT mz_bool mz_zip_writer_add_mem(mz_zip_archive *pZip, const char *pArchive_name, const void *pBuf, size_t buf_size, mz_uint level_and_flags); /* Like mz_zip_writer_add_mem(), except you can specify a file comment field, and optionally supply the function with already compressed data. */ /* uncomp_size/uncomp_crc32 are only used if the MZ_ZIP_FLAG_COMPRESSED_DATA flag is specified. */ MINIZ_EXPORT mz_bool mz_zip_writer_add_mem_ex(mz_zip_archive *pZip, const char *pArchive_name, const void *pBuf, size_t buf_size, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags, mz_uint64 uncomp_size, mz_uint32 uncomp_crc32); MINIZ_EXPORT mz_bool mz_zip_writer_add_mem_ex_v2(mz_zip_archive *pZip, const char *pArchive_name, const void *pBuf, size_t buf_size, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags, mz_uint64 uncomp_size, mz_uint32 uncomp_crc32, MZ_TIME_T *last_modified, const char *user_extra_data_local, mz_uint user_extra_data_local_len, const char *user_extra_data_central, mz_uint user_extra_data_central_len); /* Adds the contents of a file to an archive. This function also records the disk file's modified time into the archive. */ /* File data is supplied via a read callback function. User mz_zip_writer_add_(c)file to add a file directly.*/ MINIZ_EXPORT mz_bool mz_zip_writer_add_read_buf_callback(mz_zip_archive *pZip, const char *pArchive_name, mz_file_read_func read_callback, void* callback_opaque, mz_uint64 max_size, const MZ_TIME_T *pFile_time, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags, const char *user_extra_data_local, mz_uint user_extra_data_local_len, const char *user_extra_data_central, mz_uint user_extra_data_central_len); #ifndef MINIZ_NO_STDIO /* Adds the contents of a disk file to an archive. This function also records the disk file's modified time into the archive. */ /* level_and_flags - compression level (0-10, see MZ_BEST_SPEED, MZ_BEST_COMPRESSION, etc.) logically OR'd with zero or more mz_zip_flags, or just set to MZ_DEFAULT_COMPRESSION. */ MINIZ_EXPORT mz_bool mz_zip_writer_add_file(mz_zip_archive *pZip, const char *pArchive_name, const char *pSrc_filename, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags); /* Like mz_zip_writer_add_file(), except the file data is read from the specified FILE stream. */ MINIZ_EXPORT mz_bool mz_zip_writer_add_cfile(mz_zip_archive *pZip, const char *pArchive_name, MZ_FILE *pSrc_file, mz_uint64 max_size, const MZ_TIME_T *pFile_time, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags, const char *user_extra_data_local, mz_uint user_extra_data_local_len, const char *user_extra_data_central, mz_uint user_extra_data_central_len); #endif /* Adds a file to an archive by fully cloning the data from another archive. */ /* This function fully clones the source file's compressed data (no recompression), along with its full filename, extra data (it may add or modify the zip64 local header extra data field), and the optional descriptor following the compressed data. */ MINIZ_EXPORT mz_bool mz_zip_writer_add_from_zip_reader(mz_zip_archive *pZip, mz_zip_archive *pSource_zip, mz_uint src_file_index); /* Finalizes the archive by writing the central directory records followed by the end of central directory record. */ /* After an archive is finalized, the only valid call on the mz_zip_archive struct is mz_zip_writer_end(). */ /* An archive must be manually finalized by calling this function for it to be valid. */ MINIZ_EXPORT mz_bool mz_zip_writer_finalize_archive(mz_zip_archive *pZip); /* Finalizes a heap archive, returning a poiner to the heap block and its size. */ /* The heap block will be allocated using the mz_zip_archive's alloc/realloc callbacks. */ MINIZ_EXPORT mz_bool mz_zip_writer_finalize_heap_archive(mz_zip_archive *pZip, void **ppBuf, size_t *pSize); /* Ends archive writing, freeing all allocations, and closing the output file if mz_zip_writer_init_file() was used. */ /* Note for the archive to be valid, it *must* have been finalized before ending (this function will not do it for you). */ MINIZ_EXPORT mz_bool mz_zip_writer_end(mz_zip_archive *pZip); /* -------- Misc. high-level helper functions: */ /* mz_zip_add_mem_to_archive_file_in_place() efficiently (but not atomically) appends a memory blob to a ZIP archive. */ /* Note this is NOT a fully safe operation. If it crashes or dies in some way your archive can be left in a screwed up state (without a central directory). */ /* level_and_flags - compression level (0-10, see MZ_BEST_SPEED, MZ_BEST_COMPRESSION, etc.) logically OR'd with zero or more mz_zip_flags, or just set to MZ_DEFAULT_COMPRESSION. */ /* TODO: Perhaps add an option to leave the existing central dir in place in case the add dies? We could then truncate the file (so the old central dir would be at the end) if something goes wrong. */ MINIZ_EXPORT mz_bool mz_zip_add_mem_to_archive_file_in_place(const char *pZip_filename, const char *pArchive_name, const void *pBuf, size_t buf_size, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags); MINIZ_EXPORT mz_bool mz_zip_add_mem_to_archive_file_in_place_v2(const char *pZip_filename, const char *pArchive_name, const void *pBuf, size_t buf_size, const void *pComment, mz_uint16 comment_size, mz_uint level_and_flags, mz_zip_error *pErr); /* Reads a single file from an archive into a heap block. */ /* If pComment is not NULL, only the file with the specified comment will be extracted. */ /* Returns NULL on failure. */ MINIZ_EXPORT void *mz_zip_extract_archive_file_to_heap(const char *pZip_filename, const char *pArchive_name, size_t *pSize, mz_uint flags); MINIZ_EXPORT void *mz_zip_extract_archive_file_to_heap_v2(const char *pZip_filename, const char *pArchive_name, const char *pComment, size_t *pSize, mz_uint flags, mz_zip_error *pErr); #endif /* #ifndef MINIZ_NO_ARCHIVE_WRITING_APIS */ #ifdef __cplusplus } #endif #endif /* MINIZ_NO_ARCHIVE_APIS */ zmat-0.9.9/src/miniz/readme.md0000644000175200007730000001023614515254731016435 0ustar rlaboissrlaboiss## Miniz Miniz is a lossless, high performance data compression library in a single source file that implements the zlib (RFC 1950) and Deflate (RFC 1951) compressed data format specification standards. It supports the most commonly used functions exported by the zlib library, but is a completely independent implementation so zlib's licensing requirements do not apply. Miniz also contains simple to use functions for writing .PNG format image files and reading/writing/appending .ZIP format archives. Miniz's compression speed has been tuned to be comparable to zlib's, and it also has a specialized real-time compressor function designed to compare well against fastlz/minilzo. ## Usage Please use the files from the [releases page](https://github.com/richgel999/miniz/releases) in your projects. Do not use the git checkout directly! The different source and header files are [amalgamated](https://www.sqlite.org/amalgamation.html) into one `miniz.c`/`miniz.h` pair in a build step (`amalgamate.sh`). Include `miniz.c` and `miniz.h` in your project to use Miniz. ## Features * MIT licensed * A portable, single source and header file library written in plain C. Tested with GCC, clang and Visual Studio. * Easily tuned and trimmed down by defines * A drop-in replacement for zlib's most used API's (tested in several open source projects that use zlib, such as libpng and libzip). * Fills a single threaded performance vs. compression ratio gap between several popular real-time compressors and zlib. For example, at level 1, miniz.c compresses around 5-9% better than minilzo, but is approx. 35% slower. At levels 2-9, miniz.c is designed to compare favorably against zlib's ratio and speed. See the miniz performance comparison page for example timings. * Not a block based compressor: miniz.c fully supports stream based processing using a coroutine-style implementation. The zlib-style API functions can be called a single byte at a time if that's all you've got. * Easy to use. The low-level compressor (tdefl) and decompressor (tinfl) have simple state structs which can be saved/restored as needed with simple memcpy's. The low-level codec API's don't use the heap in any way. * Entire inflater (including optional zlib header parsing and Adler-32 checking) is implemented in a single function as a coroutine, which is separately available in a small (~550 line) source file: miniz_tinfl.c * A fairly complete (but totally optional) set of .ZIP archive manipulation and extraction API's. The archive functionality is intended to solve common problems encountered in embedded, mobile, or game development situations. (The archive API's are purposely just powerful enough to write an entire archiver given a bit of additional higher-level logic.) ## Known Problems * No support for encrypted archives. Not sure how useful this stuff is in practice. * Minimal documentation. The assumption is that the user is already familiar with the basic zlib API. I need to write an API wiki - for now I've tried to place key comments before each enum/API, and I've included 6 examples that demonstrate how to use the module's major features. ## Special Thanks Thanks to Alex Evans for the PNG writer function. Also, thanks to Paul Holden and Thorsten Scheuermann for feedback and testing, Matt Pritchard for all his encouragement, and Sean Barrett's various public domain libraries for inspiration (and encouraging me to write miniz.c in C, which was much more enjoyable and less painful than I thought it would be considering I've been programming in C++ for so long). Thanks to Bruce Dawson for reporting a problem with the level_and_flags archive API parameter (which is fixed in v1.12) and general feedback, and Janez Zemva for indirectly encouraging me into writing more examples. ## Patents I was recently asked if miniz avoids patent issues. miniz purposely uses the same core algorithms as the ones used by zlib. The compressor uses vanilla hash chaining as described [here](https://datatracker.ietf.org/doc/html/rfc1951#section-4). Also see the [gzip FAQ](https://web.archive.org/web/20160308045258/http://www.gzip.org/#faq11). In my opinion, if miniz falls prey to a patent attack then zlib/gzip are likely to be at serious risk too. zmat-0.9.9/src/miniz/LICENSE0000644000175200007730000000224014515254731015657 0ustar rlaboissrlaboissCopyright 2013-2014 RAD Game Tools and Valve Software Copyright 2010-2014 Rich Geldreich and Tenacious Software LLC All Rights Reserved. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. zmat-0.9.9/src/easylzma/0000755000175200007730000000000014515254731015353 5ustar rlaboissrlaboisszmat-0.9.9/src/easylzma/Makefile0000644000175200007730000010111214515254731017007 0ustar rlaboissrlaboiss# CMAKE generated file: DO NOT EDIT! # Generated by "Unix Makefiles" Generator, CMake Version 3.10 # Default target executed when no arguments are given to make. default_target: all .PHONY : default_target # Allow only one "make -f Makefile2" at a time, but pass parallelism. .NOTPARALLEL: #============================================================================= # Special targets provided by cmake. # Disable implicit rules so canonical targets will work. .SUFFIXES: # Remove some rules from gmake that .SUFFIXES does not remove. SUFFIXES = .SUFFIXES: .hpux_make_needs_suffix_list # Suppress display of executed commands. $(VERBOSE).SILENT: # A target that is always out of date. cmake_force: .PHONY : cmake_force #============================================================================= # Set environment variables for the build. # The shell in which to execute make rules. SHELL = /bin/sh # The CMake executable. CMAKE_COMMAND = /usr/bin/cmake # The command to remove a file. RM = /usr/bin/cmake -E remove -f # Escaping for special characters. EQUALS = = # The top-level source directory on which CMake was run. CMAKE_SOURCE_DIR = /home/fangq/space/git/Project/github/zmat/src/easylzma # The top-level build directory on which CMake was run. CMAKE_BINARY_DIR = /home/fangq/space/git/Project/github/zmat/src/easylzma #============================================================================= # Targets provided globally by CMake. # Special rule for the target rebuild_cache rebuild_cache: @$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --cyan "Running CMake to regenerate build system..." /usr/bin/cmake -H$(CMAKE_SOURCE_DIR) -B$(CMAKE_BINARY_DIR) .PHONY : rebuild_cache # Special rule for the target rebuild_cache rebuild_cache/fast: rebuild_cache .PHONY : rebuild_cache/fast # Special rule for the target edit_cache edit_cache: @$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --cyan "No interactive CMake dialog available..." /usr/bin/cmake -E echo No\ interactive\ CMake\ dialog\ available. .PHONY : edit_cache # Special rule for the target edit_cache edit_cache/fast: edit_cache .PHONY : edit_cache/fast # The main all target all: cmake_check_build_system cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(CMAKE_COMMAND) -E cmake_progress_start /home/fangq/space/git/Project/github/zmat/src/easylzma/CMakeFiles /home/fangq/space/git/Project/github/zmat/src/easylzma/src/CMakeFiles/progress.marks cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f CMakeFiles/Makefile2 src/all $(CMAKE_COMMAND) -E cmake_progress_start /home/fangq/space/git/Project/github/zmat/src/easylzma/CMakeFiles 0 .PHONY : all # The main clean target clean: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f CMakeFiles/Makefile2 src/clean .PHONY : clean # The main clean target clean/fast: clean .PHONY : clean/fast # Prepare targets for installation. preinstall: all cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f CMakeFiles/Makefile2 src/preinstall .PHONY : preinstall # Prepare targets for installation. preinstall/fast: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f CMakeFiles/Makefile2 src/preinstall .PHONY : preinstall/fast # clear depends depend: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(CMAKE_COMMAND) -H$(CMAKE_SOURCE_DIR) -B$(CMAKE_BINARY_DIR) --check-build-system CMakeFiles/Makefile.cmake 1 .PHONY : depend # Convenience name for target. src/CMakeFiles/easylzma_s.dir/rule: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f CMakeFiles/Makefile2 src/CMakeFiles/easylzma_s.dir/rule .PHONY : src/CMakeFiles/easylzma_s.dir/rule # Convenience name for target. easylzma_s: src/CMakeFiles/easylzma_s.dir/rule .PHONY : easylzma_s # fast build rule for target. easylzma_s/fast: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/build .PHONY : easylzma_s/fast # Convenience name for target. src/CMakeFiles/easylzma.dir/rule: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f CMakeFiles/Makefile2 src/CMakeFiles/easylzma.dir/rule .PHONY : src/CMakeFiles/easylzma.dir/rule # Convenience name for target. easylzma: src/CMakeFiles/easylzma.dir/rule .PHONY : easylzma # fast build rule for target. easylzma/fast: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/build .PHONY : easylzma/fast common_internal.o: common_internal.c.o .PHONY : common_internal.o # target to build an object file common_internal.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/common_internal.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/common_internal.c.o .PHONY : common_internal.c.o common_internal.i: common_internal.c.i .PHONY : common_internal.i # target to preprocess a source file common_internal.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/common_internal.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/common_internal.c.i .PHONY : common_internal.c.i common_internal.s: common_internal.c.s .PHONY : common_internal.s # target to generate assembly for a file common_internal.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/common_internal.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/common_internal.c.s .PHONY : common_internal.c.s compress.o: compress.c.o .PHONY : compress.o # target to build an object file compress.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/compress.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/compress.c.o .PHONY : compress.c.o compress.i: compress.c.i .PHONY : compress.i # target to preprocess a source file compress.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/compress.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/compress.c.i .PHONY : compress.c.i compress.s: compress.c.s .PHONY : compress.s # target to generate assembly for a file compress.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/compress.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/compress.c.s .PHONY : compress.c.s decompress.o: decompress.c.o .PHONY : decompress.o # target to build an object file decompress.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/decompress.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/decompress.c.o .PHONY : decompress.c.o decompress.i: decompress.c.i .PHONY : decompress.i # target to preprocess a source file decompress.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/decompress.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/decompress.c.i .PHONY : decompress.c.i decompress.s: decompress.c.s .PHONY : decompress.s # target to generate assembly for a file decompress.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/decompress.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/decompress.c.s .PHONY : decompress.c.s lzip_header.o: lzip_header.c.o .PHONY : lzip_header.o # target to build an object file lzip_header.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/lzip_header.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/lzip_header.c.o .PHONY : lzip_header.c.o lzip_header.i: lzip_header.c.i .PHONY : lzip_header.i # target to preprocess a source file lzip_header.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/lzip_header.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/lzip_header.c.i .PHONY : lzip_header.c.i lzip_header.s: lzip_header.c.s .PHONY : lzip_header.s # target to generate assembly for a file lzip_header.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/lzip_header.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/lzip_header.c.s .PHONY : lzip_header.c.s lzma_header.o: lzma_header.c.o .PHONY : lzma_header.o # target to build an object file lzma_header.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/lzma_header.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/lzma_header.c.o .PHONY : lzma_header.c.o lzma_header.i: lzma_header.c.i .PHONY : lzma_header.i # target to preprocess a source file lzma_header.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/lzma_header.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/lzma_header.c.i .PHONY : lzma_header.c.i lzma_header.s: lzma_header.c.s .PHONY : lzma_header.s # target to generate assembly for a file lzma_header.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/lzma_header.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/lzma_header.c.s .PHONY : lzma_header.c.s pavlov/7zBuf.o: pavlov/7zBuf.c.o .PHONY : pavlov/7zBuf.o # target to build an object file pavlov/7zBuf.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zBuf.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zBuf.c.o .PHONY : pavlov/7zBuf.c.o pavlov/7zBuf.i: pavlov/7zBuf.c.i .PHONY : pavlov/7zBuf.i # target to preprocess a source file pavlov/7zBuf.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zBuf.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zBuf.c.i .PHONY : pavlov/7zBuf.c.i pavlov/7zBuf.s: pavlov/7zBuf.c.s .PHONY : pavlov/7zBuf.s # target to generate assembly for a file pavlov/7zBuf.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zBuf.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zBuf.c.s .PHONY : pavlov/7zBuf.c.s pavlov/7zBuf2.o: pavlov/7zBuf2.c.o .PHONY : pavlov/7zBuf2.o # target to build an object file pavlov/7zBuf2.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zBuf2.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zBuf2.c.o .PHONY : pavlov/7zBuf2.c.o pavlov/7zBuf2.i: pavlov/7zBuf2.c.i .PHONY : pavlov/7zBuf2.i # target to preprocess a source file pavlov/7zBuf2.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zBuf2.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zBuf2.c.i .PHONY : pavlov/7zBuf2.c.i pavlov/7zBuf2.s: pavlov/7zBuf2.c.s .PHONY : pavlov/7zBuf2.s # target to generate assembly for a file pavlov/7zBuf2.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zBuf2.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zBuf2.c.s .PHONY : pavlov/7zBuf2.c.s pavlov/7zCrc.o: pavlov/7zCrc.c.o .PHONY : pavlov/7zCrc.o # target to build an object file pavlov/7zCrc.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zCrc.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zCrc.c.o .PHONY : pavlov/7zCrc.c.o pavlov/7zCrc.i: pavlov/7zCrc.c.i .PHONY : pavlov/7zCrc.i # target to preprocess a source file pavlov/7zCrc.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zCrc.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zCrc.c.i .PHONY : pavlov/7zCrc.c.i pavlov/7zCrc.s: pavlov/7zCrc.c.s .PHONY : pavlov/7zCrc.s # target to generate assembly for a file pavlov/7zCrc.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zCrc.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zCrc.c.s .PHONY : pavlov/7zCrc.c.s pavlov/7zFile.o: pavlov/7zFile.c.o .PHONY : pavlov/7zFile.o # target to build an object file pavlov/7zFile.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zFile.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zFile.c.o .PHONY : pavlov/7zFile.c.o pavlov/7zFile.i: pavlov/7zFile.c.i .PHONY : pavlov/7zFile.i # target to preprocess a source file pavlov/7zFile.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zFile.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zFile.c.i .PHONY : pavlov/7zFile.c.i pavlov/7zFile.s: pavlov/7zFile.c.s .PHONY : pavlov/7zFile.s # target to generate assembly for a file pavlov/7zFile.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zFile.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zFile.c.s .PHONY : pavlov/7zFile.c.s pavlov/7zStream.o: pavlov/7zStream.c.o .PHONY : pavlov/7zStream.o # target to build an object file pavlov/7zStream.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zStream.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zStream.c.o .PHONY : pavlov/7zStream.c.o pavlov/7zStream.i: pavlov/7zStream.c.i .PHONY : pavlov/7zStream.i # target to preprocess a source file pavlov/7zStream.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zStream.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zStream.c.i .PHONY : pavlov/7zStream.c.i pavlov/7zStream.s: pavlov/7zStream.c.s .PHONY : pavlov/7zStream.s # target to generate assembly for a file pavlov/7zStream.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/7zStream.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/7zStream.c.s .PHONY : pavlov/7zStream.c.s pavlov/Alloc.o: pavlov/Alloc.c.o .PHONY : pavlov/Alloc.o # target to build an object file pavlov/Alloc.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/Alloc.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/Alloc.c.o .PHONY : pavlov/Alloc.c.o pavlov/Alloc.i: pavlov/Alloc.c.i .PHONY : pavlov/Alloc.i # target to preprocess a source file pavlov/Alloc.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/Alloc.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/Alloc.c.i .PHONY : pavlov/Alloc.c.i pavlov/Alloc.s: pavlov/Alloc.c.s .PHONY : pavlov/Alloc.s # target to generate assembly for a file pavlov/Alloc.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/Alloc.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/Alloc.c.s .PHONY : pavlov/Alloc.c.s pavlov/Bcj2.o: pavlov/Bcj2.c.o .PHONY : pavlov/Bcj2.o # target to build an object file pavlov/Bcj2.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/Bcj2.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/Bcj2.c.o .PHONY : pavlov/Bcj2.c.o pavlov/Bcj2.i: pavlov/Bcj2.c.i .PHONY : pavlov/Bcj2.i # target to preprocess a source file pavlov/Bcj2.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/Bcj2.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/Bcj2.c.i .PHONY : pavlov/Bcj2.c.i pavlov/Bcj2.s: pavlov/Bcj2.c.s .PHONY : pavlov/Bcj2.s # target to generate assembly for a file pavlov/Bcj2.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/Bcj2.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/Bcj2.c.s .PHONY : pavlov/Bcj2.c.s pavlov/Bra.o: pavlov/Bra.c.o .PHONY : pavlov/Bra.o # target to build an object file pavlov/Bra.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/Bra.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/Bra.c.o .PHONY : pavlov/Bra.c.o pavlov/Bra.i: pavlov/Bra.c.i .PHONY : pavlov/Bra.i # target to preprocess a source file pavlov/Bra.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/Bra.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/Bra.c.i .PHONY : pavlov/Bra.c.i pavlov/Bra.s: pavlov/Bra.c.s .PHONY : pavlov/Bra.s # target to generate assembly for a file pavlov/Bra.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/Bra.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/Bra.c.s .PHONY : pavlov/Bra.c.s pavlov/Bra86.o: pavlov/Bra86.c.o .PHONY : pavlov/Bra86.o # target to build an object file pavlov/Bra86.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/Bra86.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/Bra86.c.o .PHONY : pavlov/Bra86.c.o pavlov/Bra86.i: pavlov/Bra86.c.i .PHONY : pavlov/Bra86.i # target to preprocess a source file pavlov/Bra86.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/Bra86.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/Bra86.c.i .PHONY : pavlov/Bra86.c.i pavlov/Bra86.s: pavlov/Bra86.c.s .PHONY : pavlov/Bra86.s # target to generate assembly for a file pavlov/Bra86.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/Bra86.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/Bra86.c.s .PHONY : pavlov/Bra86.c.s pavlov/BraIA64.o: pavlov/BraIA64.c.o .PHONY : pavlov/BraIA64.o # target to build an object file pavlov/BraIA64.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/BraIA64.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/BraIA64.c.o .PHONY : pavlov/BraIA64.c.o pavlov/BraIA64.i: pavlov/BraIA64.c.i .PHONY : pavlov/BraIA64.i # target to preprocess a source file pavlov/BraIA64.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/BraIA64.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/BraIA64.c.i .PHONY : pavlov/BraIA64.c.i pavlov/BraIA64.s: pavlov/BraIA64.c.s .PHONY : pavlov/BraIA64.s # target to generate assembly for a file pavlov/BraIA64.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/BraIA64.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/BraIA64.c.s .PHONY : pavlov/BraIA64.c.s pavlov/LzFind.o: pavlov/LzFind.c.o .PHONY : pavlov/LzFind.o # target to build an object file pavlov/LzFind.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/LzFind.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/LzFind.c.o .PHONY : pavlov/LzFind.c.o pavlov/LzFind.i: pavlov/LzFind.c.i .PHONY : pavlov/LzFind.i # target to preprocess a source file pavlov/LzFind.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/LzFind.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/LzFind.c.i .PHONY : pavlov/LzFind.c.i pavlov/LzFind.s: pavlov/LzFind.c.s .PHONY : pavlov/LzFind.s # target to generate assembly for a file pavlov/LzFind.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/LzFind.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/LzFind.c.s .PHONY : pavlov/LzFind.c.s pavlov/LzmaDec.o: pavlov/LzmaDec.c.o .PHONY : pavlov/LzmaDec.o # target to build an object file pavlov/LzmaDec.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/LzmaDec.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/LzmaDec.c.o .PHONY : pavlov/LzmaDec.c.o pavlov/LzmaDec.i: pavlov/LzmaDec.c.i .PHONY : pavlov/LzmaDec.i # target to preprocess a source file pavlov/LzmaDec.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/LzmaDec.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/LzmaDec.c.i .PHONY : pavlov/LzmaDec.c.i pavlov/LzmaDec.s: pavlov/LzmaDec.c.s .PHONY : pavlov/LzmaDec.s # target to generate assembly for a file pavlov/LzmaDec.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/LzmaDec.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/LzmaDec.c.s .PHONY : pavlov/LzmaDec.c.s pavlov/LzmaEnc.o: pavlov/LzmaEnc.c.o .PHONY : pavlov/LzmaEnc.o # target to build an object file pavlov/LzmaEnc.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/LzmaEnc.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/LzmaEnc.c.o .PHONY : pavlov/LzmaEnc.c.o pavlov/LzmaEnc.i: pavlov/LzmaEnc.c.i .PHONY : pavlov/LzmaEnc.i # target to preprocess a source file pavlov/LzmaEnc.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/LzmaEnc.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/LzmaEnc.c.i .PHONY : pavlov/LzmaEnc.c.i pavlov/LzmaEnc.s: pavlov/LzmaEnc.c.s .PHONY : pavlov/LzmaEnc.s # target to generate assembly for a file pavlov/LzmaEnc.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/LzmaEnc.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/LzmaEnc.c.s .PHONY : pavlov/LzmaEnc.c.s pavlov/LzmaLib.o: pavlov/LzmaLib.c.o .PHONY : pavlov/LzmaLib.o # target to build an object file pavlov/LzmaLib.c.o: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/LzmaLib.c.o cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/LzmaLib.c.o .PHONY : pavlov/LzmaLib.c.o pavlov/LzmaLib.i: pavlov/LzmaLib.c.i .PHONY : pavlov/LzmaLib.i # target to preprocess a source file pavlov/LzmaLib.c.i: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/LzmaLib.c.i cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/LzmaLib.c.i .PHONY : pavlov/LzmaLib.c.i pavlov/LzmaLib.s: pavlov/LzmaLib.c.s .PHONY : pavlov/LzmaLib.s # target to generate assembly for a file pavlov/LzmaLib.c.s: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma_s.dir/build.make src/CMakeFiles/easylzma_s.dir/pavlov/LzmaLib.c.s cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(MAKE) -f src/CMakeFiles/easylzma.dir/build.make src/CMakeFiles/easylzma.dir/pavlov/LzmaLib.c.s .PHONY : pavlov/LzmaLib.c.s # Help Target help: @echo "The following are some of the valid targets for this Makefile:" @echo "... all (the default if no target is provided)" @echo "... clean" @echo "... depend" @echo "... rebuild_cache" @echo "... edit_cache" @echo "... easylzma_s" @echo "... easylzma" @echo "... common_internal.o" @echo "... common_internal.i" @echo "... common_internal.s" @echo "... compress.o" @echo "... compress.i" @echo "... compress.s" @echo "... decompress.o" @echo "... decompress.i" @echo "... decompress.s" @echo "... lzip_header.o" @echo "... lzip_header.i" @echo "... lzip_header.s" @echo "... lzma_header.o" @echo "... lzma_header.i" @echo "... lzma_header.s" @echo "... pavlov/7zBuf.o" @echo "... pavlov/7zBuf.i" @echo "... pavlov/7zBuf.s" @echo "... pavlov/7zBuf2.o" @echo "... pavlov/7zBuf2.i" @echo "... pavlov/7zBuf2.s" @echo "... pavlov/7zCrc.o" @echo "... pavlov/7zCrc.i" @echo "... pavlov/7zCrc.s" @echo "... pavlov/7zFile.o" @echo "... pavlov/7zFile.i" @echo "... pavlov/7zFile.s" @echo "... pavlov/7zStream.o" @echo "... pavlov/7zStream.i" @echo "... pavlov/7zStream.s" @echo "... pavlov/Alloc.o" @echo "... pavlov/Alloc.i" @echo "... pavlov/Alloc.s" @echo "... pavlov/Bcj2.o" @echo "... pavlov/Bcj2.i" @echo "... pavlov/Bcj2.s" @echo "... pavlov/Bra.o" @echo "... pavlov/Bra.i" @echo "... pavlov/Bra.s" @echo "... pavlov/Bra86.o" @echo "... pavlov/Bra86.i" @echo "... pavlov/Bra86.s" @echo "... pavlov/BraIA64.o" @echo "... pavlov/BraIA64.i" @echo "... pavlov/BraIA64.s" @echo "... pavlov/LzFind.o" @echo "... pavlov/LzFind.i" @echo "... pavlov/LzFind.s" @echo "... pavlov/LzmaDec.o" @echo "... pavlov/LzmaDec.i" @echo "... pavlov/LzmaDec.s" @echo "... pavlov/LzmaEnc.o" @echo "... pavlov/LzmaEnc.i" @echo "... pavlov/LzmaEnc.s" @echo "... pavlov/LzmaLib.o" @echo "... pavlov/LzmaLib.i" @echo "... pavlov/LzmaLib.s" .PHONY : help #============================================================================= # Special targets to cleanup operation of make. # Special rule to run CMake to check the build system integrity. # No rule that depends on this can have commands that come from listfiles # because they might be regenerated. cmake_check_build_system: cd /home/fangq/space/git/Project/github/zmat/src/easylzma && $(CMAKE_COMMAND) -H$(CMAKE_SOURCE_DIR) -B$(CMAKE_BINARY_DIR) --check-build-system CMakeFiles/Makefile.cmake 0 .PHONY : cmake_check_build_system zmat-0.9.9/src/easylzma/common_internal.h0000644000175200007730000000412014515254731020705 0ustar rlaboissrlaboiss#ifndef __ELZMA_COMMON_INTERNAL_H__ #define __ELZMA_COMMON_INTERNAL_H__ #include "easylzma/common.h" /** a structure which may be cast and passed into Igor's allocate * routines */ struct elzma_alloc_struct { void *(*Alloc)(void *p, size_t size); void (*Free)(void *p, void *address); /* address can be 0 */ elzma_malloc clientMallocFunc; void * clientMallocContext; elzma_free clientFreeFunc; void * clientFreeContext; }; /* initialize an allocation structure, may be called safely multiple * times */ void init_alloc_struct(struct elzma_alloc_struct * allocStruct, elzma_malloc clientMallocFunc, void * clientMallocContext, elzma_free clientFreeFunc, void * clientFreeContext); /** superset representation of a compressed file header */ struct elzma_file_header { unsigned char pb; unsigned char lp; unsigned char lc; unsigned char isStreamed; long long unsigned int uncompressedSize; unsigned int dictSize; }; /** superset representation of a compressed file footer */ struct elzma_file_footer { unsigned int crc32; long long unsigned int uncompressedSize; }; /** a structure which encapsulates information about the particular * file header and footer in use (lzip vs lzma vs (eventually) xz. * The intention of this structure is to simplify compression and * decompression logic by abstracting the file format details a bit. */ struct elzma_format_handler { unsigned int header_size; void (*init_header)(struct elzma_file_header * hdr); int (*parse_header)(const unsigned char * hdrBuf, struct elzma_file_header * hdr); int (*serialize_header)(unsigned char * hdrBuf, const struct elzma_file_header * hdr); unsigned int footer_size; int (*serialize_footer)(struct elzma_file_footer * ftr, unsigned char * ftrBuf); int (*parse_footer)(const unsigned char * ftrBuf, struct elzma_file_footer * ftr); }; #endif zmat-0.9.9/src/easylzma/compress.c0000644000175200007730000002227014515254731017355 0ustar rlaboissrlaboiss/* * Written in 2009 by Lloyd Hilaiel * * License * * All the cruft you find here is public domain. You don't have to credit * anyone to use this code, but my personal request is that you mention * Igor Pavlov for his hard, high quality work. */ #include "easylzma/compress.h" #include "lzma_header.h" #include "lzip_header.h" #include "common_internal.h" #include "pavlov/Types.h" #include "pavlov/LzmaEnc.h" #include "pavlov/7zCrc.h" #include struct _elzma_compress_handle { CLzmaEncProps props; CLzmaEncHandle encHand; unsigned long long uncompressedSize; elzma_file_format format; struct elzma_alloc_struct allocStruct; struct elzma_format_handler formatHandler; }; elzma_compress_handle elzma_compress_alloc() { elzma_compress_handle hand = malloc(sizeof(struct _elzma_compress_handle)); memset((void *) hand, 0, sizeof(struct _elzma_compress_handle)); /* "reasonable" defaults for props */ LzmaEncProps_Init(&(hand->props)); hand->props.lc = 3; hand->props.lp = 0; hand->props.pb = 2; hand->props.level = 5; hand->props.algo = 1; hand->props.fb = 32; hand->props.dictSize = 1 << 24; hand->props.btMode = 1; hand->props.numHashBytes = 4; hand->props.mc = 32; hand->props.numThreads = 1; hand->props.writeEndMark = 1; init_alloc_struct(&(hand->allocStruct), NULL, NULL, NULL, NULL); /* default format is LZMA-Alone */ initializeLZMAFormatHandler(&(hand->formatHandler)); return hand; } void elzma_compress_free(elzma_compress_handle * hand) { if (hand && *hand) { if ((*hand)->encHand) { LzmaEnc_Destroy((*hand)->encHand, (ISzAlloc *) &((*hand)->allocStruct), (ISzAlloc *) &((*hand)->allocStruct)); } free(*hand); } *hand = NULL; } int elzma_compress_config(elzma_compress_handle hand, unsigned char lc, unsigned char lp, unsigned char pb, unsigned char level, unsigned int dictionarySize, elzma_file_format format, unsigned long long uncompressedSize) { /* XXX: validate arguments are in valid ranges */ hand->props.lc = lc; hand->props.lp = lp; hand->props.pb = pb; hand->props.level = level; hand->props.dictSize = dictionarySize; hand->uncompressedSize = uncompressedSize; hand->format = format; /* default of LZMA-Alone is set at alloc time, and there are only * two possible formats */ if (format == ELZMA_lzip) { initializeLZIPFormatHandler(&(hand->formatHandler)); } return ELZMA_E_OK; } /* use Igor's stream hooks for compression. */ struct elzmaInStream { SRes (*ReadPtr)(void *p, void *buf, size_t *size); elzma_read_callback inputStream; void * inputContext; unsigned int crc32; unsigned int crc32a; unsigned int crc32b; unsigned int crc32c; int calculateCRC; }; static SRes elzmaReadFunc(void *p, void *buf, size_t *size) { int rv; struct elzmaInStream * is = (struct elzmaInStream *) p; rv = is->inputStream(is->inputContext, buf, size); if (rv == 0 && *size > 0 && is->calculateCRC) { is->crc32 = CrcUpdate(is->crc32, buf, *size); } return rv; } struct elzmaOutStream { size_t (*WritePtr)(void *p, const void *buf, size_t size); elzma_write_callback outputStream; void * outputContext; }; static size_t elzmaWriteFunc(void *p, const void *buf, size_t size) { struct elzmaOutStream * os = (struct elzmaOutStream *) p; return os->outputStream(os->outputContext, buf, size); } /* use Igor's stream hooks for compression. */ struct elzmaProgressStruct { SRes (*Progress)(void *p, UInt64 inSize, UInt64 outSize); long long unsigned int uncompressedSize; elzma_progress_callback progressCallback; void * progressContext; }; #include static SRes elzmaProgress(void *p, UInt64 inSize, UInt64 outSize) { struct elzmaProgressStruct * ps = (struct elzmaProgressStruct *) p; if (ps->progressCallback) { ps->progressCallback(ps->progressContext, inSize, ps->uncompressedSize); } return SZ_OK; } void elzma_compress_set_allocation_callbacks( elzma_compress_handle hand, elzma_malloc mallocFunc, void * mallocFuncContext, elzma_free freeFunc, void * freeFuncContext) { if (hand) { init_alloc_struct(&(hand->allocStruct), mallocFunc, mallocFuncContext, freeFunc, freeFuncContext); } } int elzma_compress_run(elzma_compress_handle hand, elzma_read_callback inputStream, void * inputContext, elzma_write_callback outputStream, void * outputContext, elzma_progress_callback progressCallback, void * progressContext) { struct elzmaInStream inStreamStruct; struct elzmaOutStream outStreamStruct; struct elzmaProgressStruct progressStruct; SRes r; CrcGenerateTable(); if (hand == NULL || inputStream == NULL) return ELZMA_E_BAD_PARAMS; /* initialize stream structrures */ inStreamStruct.ReadPtr = elzmaReadFunc; inStreamStruct.inputStream = inputStream; inStreamStruct.inputContext = inputContext; inStreamStruct.crc32 = CRC_INIT_VAL; inStreamStruct.calculateCRC = (hand->formatHandler.serialize_footer != NULL); outStreamStruct.WritePtr = elzmaWriteFunc; outStreamStruct.outputStream = outputStream; outStreamStruct.outputContext = outputContext; progressStruct.Progress = elzmaProgress; progressStruct.uncompressedSize = hand->uncompressedSize; progressStruct.progressCallback = progressCallback; progressStruct.progressContext = progressContext; /* create an encoding object */ hand->encHand = LzmaEnc_Create((ISzAlloc *) &(hand->allocStruct)); if (hand->encHand == NULL) { return ELZMA_E_COMPRESS_ERROR; } /* inintialize with compression parameters */ if (SZ_OK != LzmaEnc_SetProps(hand->encHand, &(hand->props))) { return ELZMA_E_BAD_PARAMS; } /* verify format is sane */ if (ELZMA_lzma != hand->format && ELZMA_lzip != hand->format) { return ELZMA_E_UNSUPPORTED_FORMAT; } /* now write the compression header header */ { unsigned char * hdr = hand->allocStruct.Alloc(&(hand->allocStruct), hand->formatHandler.header_size); struct elzma_file_header h; size_t wt; hand->formatHandler.init_header(&h); h.pb = (unsigned char) hand->props.pb; h.lp = (unsigned char) hand->props.lp; h.lc = (unsigned char) hand->props.lc; h.dictSize = hand->props.dictSize; h.isStreamed = (unsigned char) (hand->uncompressedSize == 0); h.uncompressedSize = hand->uncompressedSize; hand->formatHandler.serialize_header(hdr, &h); wt = outputStream(outputContext, (void *) hdr, hand->formatHandler.header_size); hand->allocStruct.Free(&(hand->allocStruct), hdr); if (wt != hand->formatHandler.header_size) { return ELZMA_E_OUTPUT_ERROR; } } /* begin LZMA encoding */ /* XXX: expose encoding progress */ r = LzmaEnc_Encode(hand->encHand, (ISeqOutStream *) &outStreamStruct, (ISeqInStream *) &inStreamStruct, (ICompressProgress *) &progressStruct, (ISzAlloc *) &(hand->allocStruct), (ISzAlloc *) &(hand->allocStruct)); if (r != SZ_OK) return ELZMA_E_COMPRESS_ERROR; /* support a footer! (lzip) */ if (hand->formatHandler.serialize_footer != NULL && hand->formatHandler.footer_size > 0) { size_t wt; unsigned char * ftrBuf = hand->allocStruct.Alloc(&(hand->allocStruct), hand->formatHandler.footer_size); struct elzma_file_footer ftr; ftr.crc32 = inStreamStruct.crc32 ^ 0xFFFFFFFF; ftr.uncompressedSize = hand->uncompressedSize; hand->formatHandler.serialize_footer(&ftr, ftrBuf); wt = outputStream(outputContext, (void *) ftrBuf, hand->formatHandler.footer_size); hand->allocStruct.Free(&(hand->allocStruct), ftrBuf); if (wt != hand->formatHandler.footer_size) { return ELZMA_E_OUTPUT_ERROR; } } return ELZMA_E_OK; } unsigned int elzma_get_dict_size(unsigned long long size) { int i = 13; /* 16k dict is minimum */ /* now we'll find the closes power of two with a max at 16< * * if the size is greater than 8m, we'll divide by two, all of this * is based on a quick set of emperical tests on hopefully * representative sample data */ if ( size > ( 1 << 23 ) ) size >>= 1; while (size >> i) i++; if (i > 23) return 1 << 23; /* now 1 << i is greater than size, let's return either 1< (size - (1 << (i-1)))) ? i-1 : i); } zmat-0.9.9/src/easylzma/README0000644000175200007730000000027214515254731016234 0ustar rlaboissrlaboisspavlov/ - contains original lzma compress/decompress source from Igor Pavlov easylzma/ - contains the public api of this library ./ - contains the implementation of this wrapper library zmat-0.9.9/src/easylzma/lzma_header.h0000644000175200007730000000036014515254731017776 0ustar rlaboissrlaboiss#ifndef __EASYLZMA_LZMA_HEADER__ #define __EASYLZMA_LZMA_HEADER__ #include "common_internal.h" /* LZMA-Alone header format gleaned from reading Igor's code */ void initializeLZMAFormatHandler(struct elzma_format_handler * hand); #endif zmat-0.9.9/src/easylzma/pavlov/0000755000175200007730000000000014515254731016662 5ustar rlaboissrlaboisszmat-0.9.9/src/easylzma/pavlov/LzFind.h0000755000175200007730000000622414515254731020230 0ustar rlaboissrlaboiss/* LzFind.h -- Match finder for LZ algorithms 2008-10-04 : Igor Pavlov : Public domain */ #ifndef __LZFIND_H #define __LZFIND_H #include "Types.h" typedef UInt32 CLzRef; typedef struct _CMatchFinder { Byte *buffer; UInt32 pos; UInt32 posLimit; UInt32 streamPos; UInt32 lenLimit; UInt32 cyclicBufferPos; UInt32 cyclicBufferSize; /* it must be = (historySize + 1) */ UInt32 matchMaxLen; CLzRef *hash; CLzRef *son; UInt32 hashMask; UInt32 cutValue; Byte *bufferBase; ISeqInStream *stream; int streamEndWasReached; UInt32 blockSize; UInt32 keepSizeBefore; UInt32 keepSizeAfter; UInt32 numHashBytes; int directInput; int btMode; /* int skipModeBits; */ int bigHash; UInt32 historySize; UInt32 fixedHashSize; UInt32 hashSizeSum; UInt32 numSons; SRes result; UInt32 crc[256]; } CMatchFinder; #define Inline_MatchFinder_GetPointerToCurrentPos(p) ((p)->buffer) #define Inline_MatchFinder_GetIndexByte(p, index) ((p)->buffer[(Int32)(index)]) #define Inline_MatchFinder_GetNumAvailableBytes(p) ((p)->streamPos - (p)->pos) int MatchFinder_NeedMove(CMatchFinder *p); Byte *MatchFinder_GetPointerToCurrentPos(CMatchFinder *p); void MatchFinder_MoveBlock(CMatchFinder *p); void MatchFinder_ReadIfRequired(CMatchFinder *p); void MatchFinder_Construct(CMatchFinder *p); /* Conditions: historySize <= 3 GB keepAddBufferBefore + matchMaxLen + keepAddBufferAfter < 511MB */ int MatchFinder_Create(CMatchFinder *p, UInt32 historySize, UInt32 keepAddBufferBefore, UInt32 matchMaxLen, UInt32 keepAddBufferAfter, ISzAlloc *alloc); void MatchFinder_Free(CMatchFinder *p, ISzAlloc *alloc); void MatchFinder_Normalize3(UInt32 subValue, CLzRef *items, UInt32 numItems); void MatchFinder_ReduceOffsets(CMatchFinder *p, UInt32 subValue); UInt32 * GetMatchesSpec1(UInt32 lenLimit, UInt32 curMatch, UInt32 pos, const Byte *buffer, CLzRef *son, UInt32 _cyclicBufferPos, UInt32 _cyclicBufferSize, UInt32 _cutValue, UInt32 *distances, UInt32 maxLen); /* Conditions: Mf_GetNumAvailableBytes_Func must be called before each Mf_GetMatchLen_Func. Mf_GetPointerToCurrentPos_Func's result must be used only before any other function */ typedef void (*Mf_Init_Func)(void *object); typedef Byte (*Mf_GetIndexByte_Func)(void *object, Int32 index); typedef UInt32 (*Mf_GetNumAvailableBytes_Func)(void *object); typedef const Byte * (*Mf_GetPointerToCurrentPos_Func)(void *object); typedef UInt32 (*Mf_GetMatches_Func)(void *object, UInt32 *distances); typedef void (*Mf_Skip_Func)(void *object, UInt32); typedef struct _IMatchFinder { Mf_Init_Func Init; Mf_GetIndexByte_Func GetIndexByte; Mf_GetNumAvailableBytes_Func GetNumAvailableBytes; Mf_GetPointerToCurrentPos_Func GetPointerToCurrentPos; Mf_GetMatches_Func GetMatches; Mf_Skip_Func Skip; } IMatchFinder; void MatchFinder_CreateVTable(CMatchFinder *p, IMatchFinder *vTable); void MatchFinder_Init(CMatchFinder *p); UInt32 Bt3Zip_MatchFinder_GetMatches(CMatchFinder *p, UInt32 *distances); UInt32 Hc3Zip_MatchFinder_GetMatches(CMatchFinder *p, UInt32 *distances); void Bt3Zip_MatchFinder_Skip(CMatchFinder *p, UInt32 num); void Hc3Zip_MatchFinder_Skip(CMatchFinder *p, UInt32 num); #endif zmat-0.9.9/src/easylzma/pavlov/LzFind.c0000755000175200007730000004604514515254731020230 0ustar rlaboissrlaboiss/* LzFind.c -- Match finder for LZ algorithms 2008-10-04 : Igor Pavlov : Public domain */ #include #include "LzFind.h" #include "LzHash.h" #define kEmptyHashValue 0 #define kMaxValForNormalize ((UInt32)0xFFFFFFFF) #define kNormalizeStepMin (1 << 10) /* it must be power of 2 */ #define kNormalizeMask (~(kNormalizeStepMin - 1)) #define kMaxHistorySize ((UInt32)3 << 30) #define kStartMaxLen 3 static void LzInWindow_Free(CMatchFinder *p, ISzAlloc *alloc) { if (!p->directInput) { alloc->Free(alloc, p->bufferBase); p->bufferBase = 0; } } /* keepSizeBefore + keepSizeAfter + keepSizeReserv must be < 4G) */ static int LzInWindow_Create(CMatchFinder *p, UInt32 keepSizeReserv, ISzAlloc *alloc) { UInt32 blockSize = p->keepSizeBefore + p->keepSizeAfter + keepSizeReserv; if (p->directInput) { p->blockSize = blockSize; return 1; } if (p->bufferBase == 0 || p->blockSize != blockSize) { LzInWindow_Free(p, alloc); p->blockSize = blockSize; p->bufferBase = (Byte *)alloc->Alloc(alloc, (size_t)blockSize); } return (p->bufferBase != 0); } Byte *MatchFinder_GetPointerToCurrentPos(CMatchFinder *p) { return p->buffer; } Byte MatchFinder_GetIndexByte(CMatchFinder *p, Int32 index) { return p->buffer[index]; } UInt32 MatchFinder_GetNumAvailableBytes(CMatchFinder *p) { return p->streamPos - p->pos; } void MatchFinder_ReduceOffsets(CMatchFinder *p, UInt32 subValue) { p->posLimit -= subValue; p->pos -= subValue; p->streamPos -= subValue; } static void MatchFinder_ReadBlock(CMatchFinder *p) { if (p->streamEndWasReached || p->result != SZ_OK) return; for (;;) { Byte *dest = p->buffer + (p->streamPos - p->pos); size_t size = (p->bufferBase + p->blockSize - dest); if (size == 0) return; p->result = p->stream->Read(p->stream, dest, &size); if (p->result != SZ_OK) return; if (size == 0) { p->streamEndWasReached = 1; return; } p->streamPos += (UInt32)size; if (p->streamPos - p->pos > p->keepSizeAfter) return; } } void MatchFinder_MoveBlock(CMatchFinder *p) { memmove(p->bufferBase, p->buffer - p->keepSizeBefore, (size_t)(p->streamPos - p->pos + p->keepSizeBefore)); p->buffer = p->bufferBase + p->keepSizeBefore; } int MatchFinder_NeedMove(CMatchFinder *p) { /* if (p->streamEndWasReached) return 0; */ return ((size_t)(p->bufferBase + p->blockSize - p->buffer) <= p->keepSizeAfter); } void MatchFinder_ReadIfRequired(CMatchFinder *p) { if (p->streamEndWasReached) return; if (p->keepSizeAfter >= p->streamPos - p->pos) MatchFinder_ReadBlock(p); } static void MatchFinder_CheckAndMoveAndRead(CMatchFinder *p) { if (MatchFinder_NeedMove(p)) MatchFinder_MoveBlock(p); MatchFinder_ReadBlock(p); } static void MatchFinder_SetDefaultSettings(CMatchFinder *p) { p->cutValue = 32; p->btMode = 1; p->numHashBytes = 4; /* p->skipModeBits = 0; */ p->directInput = 0; p->bigHash = 0; } #define kCrcPoly 0xEDB88320 void MatchFinder_Construct(CMatchFinder *p) { UInt32 i; p->bufferBase = 0; p->directInput = 0; p->hash = 0; MatchFinder_SetDefaultSettings(p); for (i = 0; i < 256; i++) { UInt32 r = i; int j; for (j = 0; j < 8; j++) r = (r >> 1) ^ (kCrcPoly & ~((r & 1) - 1)); p->crc[i] = r; } } static void MatchFinder_FreeThisClassMemory(CMatchFinder *p, ISzAlloc *alloc) { alloc->Free(alloc, p->hash); p->hash = 0; } void MatchFinder_Free(CMatchFinder *p, ISzAlloc *alloc) { MatchFinder_FreeThisClassMemory(p, alloc); LzInWindow_Free(p, alloc); } static CLzRef* AllocRefs(UInt32 num, ISzAlloc *alloc) { size_t sizeInBytes = (size_t)num * sizeof(CLzRef); if (sizeInBytes / sizeof(CLzRef) != num) return 0; return (CLzRef *)alloc->Alloc(alloc, sizeInBytes); } int MatchFinder_Create(CMatchFinder *p, UInt32 historySize, UInt32 keepAddBufferBefore, UInt32 matchMaxLen, UInt32 keepAddBufferAfter, ISzAlloc *alloc) { UInt32 sizeReserv; if (historySize > kMaxHistorySize) { MatchFinder_Free(p, alloc); return 0; } sizeReserv = historySize >> 1; if (historySize > ((UInt32)2 << 30)) sizeReserv = historySize >> 2; sizeReserv += (keepAddBufferBefore + matchMaxLen + keepAddBufferAfter) / 2 + (1 << 19); p->keepSizeBefore = historySize + keepAddBufferBefore + 1; p->keepSizeAfter = matchMaxLen + keepAddBufferAfter; /* we need one additional byte, since we use MoveBlock after pos++ and before dictionary using */ if (LzInWindow_Create(p, sizeReserv, alloc)) { UInt32 newCyclicBufferSize = (historySize /* >> p->skipModeBits */) + 1; UInt32 hs; p->matchMaxLen = matchMaxLen; { p->fixedHashSize = 0; if (p->numHashBytes == 2) hs = (1 << 16) - 1; else { hs = historySize - 1; hs |= (hs >> 1); hs |= (hs >> 2); hs |= (hs >> 4); hs |= (hs >> 8); hs >>= 1; /* hs >>= p->skipModeBits; */ hs |= 0xFFFF; /* don't change it! It's required for Deflate */ if (hs > (1 << 24)) { if (p->numHashBytes == 3) hs = (1 << 24) - 1; else hs >>= 1; } } p->hashMask = hs; hs++; if (p->numHashBytes > 2) p->fixedHashSize += kHash2Size; if (p->numHashBytes > 3) p->fixedHashSize += kHash3Size; if (p->numHashBytes > 4) p->fixedHashSize += kHash4Size; hs += p->fixedHashSize; } { UInt32 prevSize = p->hashSizeSum + p->numSons; UInt32 newSize; p->historySize = historySize; p->hashSizeSum = hs; p->cyclicBufferSize = newCyclicBufferSize; p->numSons = (p->btMode ? newCyclicBufferSize * 2 : newCyclicBufferSize); newSize = p->hashSizeSum + p->numSons; if (p->hash != 0 && prevSize == newSize) return 1; MatchFinder_FreeThisClassMemory(p, alloc); p->hash = AllocRefs(newSize, alloc); if (p->hash != 0) { p->son = p->hash + p->hashSizeSum; return 1; } } } MatchFinder_Free(p, alloc); return 0; } static void MatchFinder_SetLimits(CMatchFinder *p) { UInt32 limit = kMaxValForNormalize - p->pos; UInt32 limit2 = p->cyclicBufferSize - p->cyclicBufferPos; if (limit2 < limit) limit = limit2; limit2 = p->streamPos - p->pos; if (limit2 <= p->keepSizeAfter) { if (limit2 > 0) limit2 = 1; } else limit2 -= p->keepSizeAfter; if (limit2 < limit) limit = limit2; { UInt32 lenLimit = p->streamPos - p->pos; if (lenLimit > p->matchMaxLen) lenLimit = p->matchMaxLen; p->lenLimit = lenLimit; } p->posLimit = p->pos + limit; } void MatchFinder_Init(CMatchFinder *p) { UInt32 i; for (i = 0; i < p->hashSizeSum; i++) p->hash[i] = kEmptyHashValue; p->cyclicBufferPos = 0; p->buffer = p->bufferBase; p->pos = p->streamPos = p->cyclicBufferSize; p->result = SZ_OK; p->streamEndWasReached = 0; MatchFinder_ReadBlock(p); MatchFinder_SetLimits(p); } static UInt32 MatchFinder_GetSubValue(CMatchFinder *p) { return (p->pos - p->historySize - 1) & kNormalizeMask; } void MatchFinder_Normalize3(UInt32 subValue, CLzRef *items, UInt32 numItems) { UInt32 i; for (i = 0; i < numItems; i++) { UInt32 value = items[i]; if (value <= subValue) value = kEmptyHashValue; else value -= subValue; items[i] = value; } } static void MatchFinder_Normalize(CMatchFinder *p) { UInt32 subValue = MatchFinder_GetSubValue(p); MatchFinder_Normalize3(subValue, p->hash, p->hashSizeSum + p->numSons); MatchFinder_ReduceOffsets(p, subValue); } static void MatchFinder_CheckLimits(CMatchFinder *p) { if (p->pos == kMaxValForNormalize) MatchFinder_Normalize(p); if (!p->streamEndWasReached && p->keepSizeAfter == p->streamPos - p->pos) MatchFinder_CheckAndMoveAndRead(p); if (p->cyclicBufferPos == p->cyclicBufferSize) p->cyclicBufferPos = 0; MatchFinder_SetLimits(p); } static UInt32 * Hc_GetMatchesSpec(UInt32 lenLimit, UInt32 curMatch, UInt32 pos, const Byte *cur, CLzRef *son, UInt32 _cyclicBufferPos, UInt32 _cyclicBufferSize, UInt32 cutValue, UInt32 *distances, UInt32 maxLen) { son[_cyclicBufferPos] = curMatch; for (;;) { UInt32 delta = pos - curMatch; if (cutValue-- == 0 || delta >= _cyclicBufferSize) return distances; { const Byte *pb = cur - delta; curMatch = son[_cyclicBufferPos - delta + ((delta > _cyclicBufferPos) ? _cyclicBufferSize : 0)]; if (pb[maxLen] == cur[maxLen] && *pb == *cur) { UInt32 len = 0; while (++len != lenLimit) if (pb[len] != cur[len]) break; if (maxLen < len) { *distances++ = maxLen = len; *distances++ = delta - 1; if (len == lenLimit) return distances; } } } } } UInt32 * GetMatchesSpec1(UInt32 lenLimit, UInt32 curMatch, UInt32 pos, const Byte *cur, CLzRef *son, UInt32 _cyclicBufferPos, UInt32 _cyclicBufferSize, UInt32 cutValue, UInt32 *distances, UInt32 maxLen) { CLzRef *ptr0 = son + (_cyclicBufferPos << 1) + 1; CLzRef *ptr1 = son + (_cyclicBufferPos << 1); UInt32 len0 = 0, len1 = 0; for (;;) { UInt32 delta = pos - curMatch; if (cutValue-- == 0 || delta >= _cyclicBufferSize) { *ptr0 = *ptr1 = kEmptyHashValue; return distances; } { CLzRef *pair = son + ((_cyclicBufferPos - delta + ((delta > _cyclicBufferPos) ? _cyclicBufferSize : 0)) << 1); const Byte *pb = cur - delta; UInt32 len = (len0 < len1 ? len0 : len1); if (pb[len] == cur[len]) { if (++len != lenLimit && pb[len] == cur[len]) while (++len != lenLimit) if (pb[len] != cur[len]) break; if (maxLen < len) { *distances++ = maxLen = len; *distances++ = delta - 1; if (len == lenLimit) { *ptr1 = pair[0]; *ptr0 = pair[1]; return distances; } } } if (pb[len] < cur[len]) { *ptr1 = curMatch; ptr1 = pair + 1; curMatch = *ptr1; len1 = len; } else { *ptr0 = curMatch; ptr0 = pair; curMatch = *ptr0; len0 = len; } } } } static void SkipMatchesSpec(UInt32 lenLimit, UInt32 curMatch, UInt32 pos, const Byte *cur, CLzRef *son, UInt32 _cyclicBufferPos, UInt32 _cyclicBufferSize, UInt32 cutValue) { CLzRef *ptr0 = son + (_cyclicBufferPos << 1) + 1; CLzRef *ptr1 = son + (_cyclicBufferPos << 1); UInt32 len0 = 0, len1 = 0; for (;;) { UInt32 delta = pos - curMatch; if (cutValue-- == 0 || delta >= _cyclicBufferSize) { *ptr0 = *ptr1 = kEmptyHashValue; return; } { CLzRef *pair = son + ((_cyclicBufferPos - delta + ((delta > _cyclicBufferPos) ? _cyclicBufferSize : 0)) << 1); const Byte *pb = cur - delta; UInt32 len = (len0 < len1 ? len0 : len1); if (pb[len] == cur[len]) { while (++len != lenLimit) if (pb[len] != cur[len]) break; { if (len == lenLimit) { *ptr1 = pair[0]; *ptr0 = pair[1]; return; } } } if (pb[len] < cur[len]) { *ptr1 = curMatch; ptr1 = pair + 1; curMatch = *ptr1; len1 = len; } else { *ptr0 = curMatch; ptr0 = pair; curMatch = *ptr0; len0 = len; } } } } #define MOVE_POS \ ++p->cyclicBufferPos; \ p->buffer++; \ if (++p->pos == p->posLimit) MatchFinder_CheckLimits(p); #define MOVE_POS_RET MOVE_POS return offset; static void MatchFinder_MovePos(CMatchFinder *p) { MOVE_POS; } #define GET_MATCHES_HEADER2(minLen, ret_op) \ UInt32 lenLimit; UInt32 hashValue; const Byte *cur; UInt32 curMatch; \ lenLimit = p->lenLimit; { if (lenLimit < minLen) { MatchFinder_MovePos(p); ret_op; }} \ cur = p->buffer; #define GET_MATCHES_HEADER(minLen) GET_MATCHES_HEADER2(minLen, return 0) #define SKIP_HEADER(minLen) GET_MATCHES_HEADER2(minLen, continue) #define MF_PARAMS(p) p->pos, p->buffer, p->son, p->cyclicBufferPos, p->cyclicBufferSize, p->cutValue #define GET_MATCHES_FOOTER(offset, maxLen) \ offset = (UInt32)(GetMatchesSpec1(lenLimit, curMatch, MF_PARAMS(p), \ distances + offset, maxLen) - distances); MOVE_POS_RET; #define SKIP_FOOTER \ SkipMatchesSpec(lenLimit, curMatch, MF_PARAMS(p)); MOVE_POS; static UInt32 Bt2_MatchFinder_GetMatches(CMatchFinder *p, UInt32 *distances) { UInt32 offset; GET_MATCHES_HEADER(2) HASH2_CALC; curMatch = p->hash[hashValue]; p->hash[hashValue] = p->pos; offset = 0; GET_MATCHES_FOOTER(offset, 1) } UInt32 Bt3Zip_MatchFinder_GetMatches(CMatchFinder *p, UInt32 *distances) { UInt32 offset; GET_MATCHES_HEADER(3) HASH_ZIP_CALC; curMatch = p->hash[hashValue]; p->hash[hashValue] = p->pos; offset = 0; GET_MATCHES_FOOTER(offset, 2) } static UInt32 Bt3_MatchFinder_GetMatches(CMatchFinder *p, UInt32 *distances) { UInt32 hash2Value, delta2, maxLen, offset; GET_MATCHES_HEADER(3) HASH3_CALC; delta2 = p->pos - p->hash[hash2Value]; curMatch = p->hash[kFix3HashSize + hashValue]; p->hash[hash2Value] = p->hash[kFix3HashSize + hashValue] = p->pos; maxLen = 2; offset = 0; if (delta2 < p->cyclicBufferSize && *(cur - delta2) == *cur) { for (; maxLen != lenLimit; maxLen++) if (cur[(ptrdiff_t)maxLen - delta2] != cur[maxLen]) break; distances[0] = maxLen; distances[1] = delta2 - 1; offset = 2; if (maxLen == lenLimit) { SkipMatchesSpec(lenLimit, curMatch, MF_PARAMS(p)); MOVE_POS_RET; } } GET_MATCHES_FOOTER(offset, maxLen) } static UInt32 Bt4_MatchFinder_GetMatches(CMatchFinder *p, UInt32 *distances) { UInt32 hash2Value, hash3Value, delta2, delta3, maxLen, offset; GET_MATCHES_HEADER(4) HASH4_CALC; delta2 = p->pos - p->hash[ hash2Value]; delta3 = p->pos - p->hash[kFix3HashSize + hash3Value]; curMatch = p->hash[kFix4HashSize + hashValue]; p->hash[ hash2Value] = p->hash[kFix3HashSize + hash3Value] = p->hash[kFix4HashSize + hashValue] = p->pos; maxLen = 1; offset = 0; if (delta2 < p->cyclicBufferSize && *(cur - delta2) == *cur) { distances[0] = maxLen = 2; distances[1] = delta2 - 1; offset = 2; } if (delta2 != delta3 && delta3 < p->cyclicBufferSize && *(cur - delta3) == *cur) { maxLen = 3; distances[offset + 1] = delta3 - 1; offset += 2; delta2 = delta3; } if (offset != 0) { for (; maxLen != lenLimit; maxLen++) if (cur[(ptrdiff_t)maxLen - delta2] != cur[maxLen]) break; distances[offset - 2] = maxLen; if (maxLen == lenLimit) { SkipMatchesSpec(lenLimit, curMatch, MF_PARAMS(p)); MOVE_POS_RET; } } if (maxLen < 3) maxLen = 3; GET_MATCHES_FOOTER(offset, maxLen) } static UInt32 Hc4_MatchFinder_GetMatches(CMatchFinder *p, UInt32 *distances) { UInt32 hash2Value, hash3Value, delta2, delta3, maxLen, offset; GET_MATCHES_HEADER(4) HASH4_CALC; delta2 = p->pos - p->hash[ hash2Value]; delta3 = p->pos - p->hash[kFix3HashSize + hash3Value]; curMatch = p->hash[kFix4HashSize + hashValue]; p->hash[ hash2Value] = p->hash[kFix3HashSize + hash3Value] = p->hash[kFix4HashSize + hashValue] = p->pos; maxLen = 1; offset = 0; if (delta2 < p->cyclicBufferSize && *(cur - delta2) == *cur) { distances[0] = maxLen = 2; distances[1] = delta2 - 1; offset = 2; } if (delta2 != delta3 && delta3 < p->cyclicBufferSize && *(cur - delta3) == *cur) { maxLen = 3; distances[offset + 1] = delta3 - 1; offset += 2; delta2 = delta3; } if (offset != 0) { for (; maxLen != lenLimit; maxLen++) if (cur[(ptrdiff_t)maxLen - delta2] != cur[maxLen]) break; distances[offset - 2] = maxLen; if (maxLen == lenLimit) { p->son[p->cyclicBufferPos] = curMatch; MOVE_POS_RET; } } if (maxLen < 3) maxLen = 3; offset = (UInt32)(Hc_GetMatchesSpec(lenLimit, curMatch, MF_PARAMS(p), distances + offset, maxLen) - (distances)); MOVE_POS_RET } UInt32 Hc3Zip_MatchFinder_GetMatches(CMatchFinder *p, UInt32 *distances) { UInt32 offset; GET_MATCHES_HEADER(3) HASH_ZIP_CALC; curMatch = p->hash[hashValue]; p->hash[hashValue] = p->pos; offset = (UInt32)(Hc_GetMatchesSpec(lenLimit, curMatch, MF_PARAMS(p), distances, 2) - (distances)); MOVE_POS_RET } static void Bt2_MatchFinder_Skip(CMatchFinder *p, UInt32 num) { do { SKIP_HEADER(2) HASH2_CALC; curMatch = p->hash[hashValue]; p->hash[hashValue] = p->pos; SKIP_FOOTER } while (--num != 0); } void Bt3Zip_MatchFinder_Skip(CMatchFinder *p, UInt32 num) { do { SKIP_HEADER(3) HASH_ZIP_CALC; curMatch = p->hash[hashValue]; p->hash[hashValue] = p->pos; SKIP_FOOTER } while (--num != 0); } static void Bt3_MatchFinder_Skip(CMatchFinder *p, UInt32 num) { do { UInt32 hash2Value; SKIP_HEADER(3) HASH3_CALC; curMatch = p->hash[kFix3HashSize + hashValue]; p->hash[hash2Value] = p->hash[kFix3HashSize + hashValue] = p->pos; SKIP_FOOTER } while (--num != 0); } static void Bt4_MatchFinder_Skip(CMatchFinder *p, UInt32 num) { do { UInt32 hash2Value, hash3Value; SKIP_HEADER(4) HASH4_CALC; curMatch = p->hash[kFix4HashSize + hashValue]; p->hash[ hash2Value] = p->hash[kFix3HashSize + hash3Value] = p->pos; p->hash[kFix4HashSize + hashValue] = p->pos; SKIP_FOOTER } while (--num != 0); } static void Hc4_MatchFinder_Skip(CMatchFinder *p, UInt32 num) { do { UInt32 hash2Value, hash3Value; SKIP_HEADER(4) HASH4_CALC; curMatch = p->hash[kFix4HashSize + hashValue]; p->hash[ hash2Value] = p->hash[kFix3HashSize + hash3Value] = p->hash[kFix4HashSize + hashValue] = p->pos; p->son[p->cyclicBufferPos] = curMatch; MOVE_POS } while (--num != 0); } void Hc3Zip_MatchFinder_Skip(CMatchFinder *p, UInt32 num) { do { SKIP_HEADER(3) HASH_ZIP_CALC; curMatch = p->hash[hashValue]; p->hash[hashValue] = p->pos; p->son[p->cyclicBufferPos] = curMatch; MOVE_POS } while (--num != 0); } void MatchFinder_CreateVTable(CMatchFinder *p, IMatchFinder *vTable) { vTable->Init = (Mf_Init_Func)MatchFinder_Init; vTable->GetIndexByte = (Mf_GetIndexByte_Func)MatchFinder_GetIndexByte; vTable->GetNumAvailableBytes = (Mf_GetNumAvailableBytes_Func)MatchFinder_GetNumAvailableBytes; vTable->GetPointerToCurrentPos = (Mf_GetPointerToCurrentPos_Func)MatchFinder_GetPointerToCurrentPos; if (!p->btMode) { vTable->GetMatches = (Mf_GetMatches_Func)Hc4_MatchFinder_GetMatches; vTable->Skip = (Mf_Skip_Func)Hc4_MatchFinder_Skip; } else if (p->numHashBytes == 2) { vTable->GetMatches = (Mf_GetMatches_Func)Bt2_MatchFinder_GetMatches; vTable->Skip = (Mf_Skip_Func)Bt2_MatchFinder_Skip; } else if (p->numHashBytes == 3) { vTable->GetMatches = (Mf_GetMatches_Func)Bt3_MatchFinder_GetMatches; vTable->Skip = (Mf_Skip_Func)Bt3_MatchFinder_Skip; } else { vTable->GetMatches = (Mf_GetMatches_Func)Bt4_MatchFinder_GetMatches; vTable->Skip = (Mf_Skip_Func)Bt4_MatchFinder_Skip; } } zmat-0.9.9/src/easylzma/pavlov/Bra.h0000755000175200007730000000344614515254731017551 0ustar rlaboissrlaboiss/* Bra.h -- Branch converters for executables 2008-10-04 : Igor Pavlov : Public domain */ #ifndef __BRA_H #define __BRA_H #include "Types.h" /* These functions convert relative addresses to absolute addresses in CALL instructions to increase the compression ratio. In: data - data buffer size - size of data ip - current virtual Instruction Pinter (IP) value state - state variable for x86 converter encoding - 0 (for decoding), 1 (for encoding) Out: state - state variable for x86 converter Returns: The number of processed bytes. If you call these functions with multiple calls, you must start next call with first byte after block of processed bytes. Type Endian Alignment LookAhead x86 little 1 4 ARMT little 2 2 ARM little 4 0 PPC big 4 0 SPARC big 4 0 IA64 little 16 0 size must be >= Alignment + LookAhead, if it's not last block. If (size < Alignment + LookAhead), converter returns 0. Example: UInt32 ip = 0; for () { ; size must be >= Alignment + LookAhead, if it's not last block SizeT processed = Convert(data, size, ip, 1); data += processed; size -= processed; ip += processed; } */ #define x86_Convert_Init(state) { state = 0; } SizeT x86_Convert(Byte *data, SizeT size, UInt32 ip, UInt32 *state, int encoding); SizeT ARM_Convert(Byte *data, SizeT size, UInt32 ip, int encoding); SizeT ARMT_Convert(Byte *data, SizeT size, UInt32 ip, int encoding); SizeT PPC_Convert(Byte *data, SizeT size, UInt32 ip, int encoding); SizeT SPARC_Convert(Byte *data, SizeT size, UInt32 ip, int encoding); SizeT IA64_Convert(Byte *data, SizeT size, UInt32 ip, int encoding); #endif zmat-0.9.9/src/easylzma/pavlov/LzmaEnc.h0000755000175200007730000000542214515254731020372 0ustar rlaboissrlaboiss/* LzmaEnc.h -- LZMA Encoder 2008-10-04 : Igor Pavlov : Public domain */ #ifndef __LZMAENC_H #define __LZMAENC_H #include "Types.h" #define LZMA_PROPS_SIZE 5 typedef struct _CLzmaEncProps { int level; /* 0 <= level <= 9 */ UInt32 dictSize; /* (1 << 12) <= dictSize <= (1 << 27) for 32-bit version (1 << 12) <= dictSize <= (1 << 30) for 64-bit version default = (1 << 24) */ int lc; /* 0 <= lc <= 8, default = 3 */ int lp; /* 0 <= lp <= 4, default = 0 */ int pb; /* 0 <= pb <= 4, default = 2 */ int algo; /* 0 - fast, 1 - normal, default = 1 */ int fb; /* 5 <= fb <= 273, default = 32 */ int btMode; /* 0 - hashChain Mode, 1 - binTree mode - normal, default = 1 */ int numHashBytes; /* 2, 3 or 4, default = 4 */ UInt32 mc; /* 1 <= mc <= (1 << 30), default = 32 */ unsigned writeEndMark; /* 0 - do not write EOPM, 1 - write EOPM, default = 0 */ int numThreads; /* 1 or 2, default = 2 */ } CLzmaEncProps; void LzmaEncProps_Init(CLzmaEncProps *p); void LzmaEncProps_Normalize(CLzmaEncProps *p); UInt32 LzmaEncProps_GetDictSize(const CLzmaEncProps *props2); /* ---------- CLzmaEncHandle Interface ---------- */ /* LzmaEnc_* functions can return the following exit codes: Returns: SZ_OK - OK SZ_ERROR_MEM - Memory allocation error SZ_ERROR_PARAM - Incorrect paramater in props SZ_ERROR_WRITE - Write callback error. SZ_ERROR_PROGRESS - some break from progress callback SZ_ERROR_THREAD - errors in multithreading functions (only for Mt version) */ typedef void * CLzmaEncHandle; CLzmaEncHandle LzmaEnc_Create(ISzAlloc *alloc); void LzmaEnc_Destroy(CLzmaEncHandle p, ISzAlloc *alloc, ISzAlloc *allocBig); SRes LzmaEnc_SetProps(CLzmaEncHandle p, const CLzmaEncProps *props); SRes LzmaEnc_WriteProperties(CLzmaEncHandle p, Byte *properties, SizeT *size); SRes LzmaEnc_Encode(CLzmaEncHandle p, ISeqOutStream *outStream, ISeqInStream *inStream, ICompressProgress *progress, ISzAlloc *alloc, ISzAlloc *allocBig); SRes LzmaEnc_MemEncode(CLzmaEncHandle p, Byte *dest, SizeT *destLen, const Byte *src, SizeT srcLen, int writeEndMark, ICompressProgress *progress, ISzAlloc *alloc, ISzAlloc *allocBig); /* ---------- One Call Interface ---------- */ /* LzmaEncode Return code: SZ_OK - OK SZ_ERROR_MEM - Memory allocation error SZ_ERROR_PARAM - Incorrect paramater SZ_ERROR_OUTPUT_EOF - output buffer overflow SZ_ERROR_THREAD - errors in multithreading functions (only for Mt version) */ SRes LzmaEncode(Byte *dest, SizeT *destLen, const Byte *src, SizeT srcLen, const CLzmaEncProps *props, Byte *propsEncoded, SizeT *propsSize, int writeEndMark, ICompressProgress *progress, ISzAlloc *alloc, ISzAlloc *allocBig); #endif zmat-0.9.9/src/easylzma/pavlov/LzmaEnc.c0000755000175200007730000017255214515254731020376 0ustar rlaboissrlaboiss/* LzmaEnc.c -- LZMA Encoder 2008-10-04 : Igor Pavlov : Public domain */ #include /* #define SHOW_STAT */ /* #define SHOW_STAT2 */ #if defined(SHOW_STAT) || defined(SHOW_STAT2) #include #endif #include "LzmaEnc.h" #include "LzFind.h" #ifdef COMPRESS_MF_MT #include "LzFindMt.h" #endif #ifdef SHOW_STAT static int ttt = 0; #endif #define kBlockSizeMax ((1 << LZMA_NUM_BLOCK_SIZE_BITS) - 1) #define kBlockSize (9 << 10) #define kUnpackBlockSize (1 << 18) #define kMatchArraySize (1 << 21) #define kMatchRecordMaxSize ((LZMA_MATCH_LEN_MAX * 2 + 3) * LZMA_MATCH_LEN_MAX) #define kNumMaxDirectBits (31) #define kNumTopBits 24 #define kTopValue ((UInt32)1 << kNumTopBits) #define kNumBitModelTotalBits 11 #define kBitModelTotal (1 << kNumBitModelTotalBits) #define kNumMoveBits 5 #define kProbInitValue (kBitModelTotal >> 1) #define kNumMoveReducingBits 4 #define kNumBitPriceShiftBits 4 #define kBitPrice (1 << kNumBitPriceShiftBits) void LzmaEncProps_Init(CLzmaEncProps *p) { p->level = 5; p->dictSize = p->mc = 0; p->lc = p->lp = p->pb = p->algo = p->fb = p->btMode = p->numHashBytes = p->numThreads = -1; p->writeEndMark = 0; } void LzmaEncProps_Normalize(CLzmaEncProps *p) { int level = p->level; if (level < 0) level = 5; p->level = level; if (p->dictSize == 0) p->dictSize = (level <= 5 ? (1 << (level * 2 + 14)) : (level == 6 ? (1 << 25) : (1 << 26))); if (p->lc < 0) p->lc = 3; if (p->lp < 0) p->lp = 0; if (p->pb < 0) p->pb = 2; if (p->algo < 0) p->algo = (level < 5 ? 0 : 1); if (p->fb < 0) p->fb = (level < 7 ? 32 : 64); if (p->btMode < 0) p->btMode = (p->algo == 0 ? 0 : 1); if (p->numHashBytes < 0) p->numHashBytes = 4; if (p->mc == 0) p->mc = (16 + (p->fb >> 1)) >> (p->btMode ? 0 : 1); if (p->numThreads < 0) p->numThreads = ((p->btMode && p->algo) ? 2 : 1); } UInt32 LzmaEncProps_GetDictSize(const CLzmaEncProps *props2) { CLzmaEncProps props = *props2; LzmaEncProps_Normalize(&props); return props.dictSize; } /* #define LZMA_LOG_BSR */ /* Define it for Intel's CPU */ #ifdef LZMA_LOG_BSR #define kDicLogSizeMaxCompress 30 #define BSR2_RET(pos, res) { unsigned long i; _BitScanReverse(&i, (pos)); res = (i + i) + ((pos >> (i - 1)) & 1); } UInt32 GetPosSlot1(UInt32 pos) { UInt32 res; BSR2_RET(pos, res); return res; } #define GetPosSlot2(pos, res) { BSR2_RET(pos, res); } #define GetPosSlot(pos, res) { if (pos < 2) res = pos; else BSR2_RET(pos, res); } #else #define kNumLogBits (9 + (int)sizeof(size_t) / 2) #define kDicLogSizeMaxCompress ((kNumLogBits - 1) * 2 + 7) void LzmaEnc_FastPosInit(Byte *g_FastPos) { int c = 2, slotFast; g_FastPos[0] = 0; g_FastPos[1] = 1; for (slotFast = 2; slotFast < kNumLogBits * 2; slotFast++) { UInt32 k = (1 << ((slotFast >> 1) - 1)); UInt32 j; for (j = 0; j < k; j++, c++) g_FastPos[c] = (Byte)slotFast; } } #define BSR2_RET(pos, res) { UInt32 i = 6 + ((kNumLogBits - 1) & \ (0 - (((((UInt32)1 << (kNumLogBits + 6)) - 1) - pos) >> 31))); \ res = p->g_FastPos[pos >> i] + (i * 2); } /* #define BSR2_RET(pos, res) { res = (pos < (1 << (kNumLogBits + 6))) ? \ p->g_FastPos[pos >> 6] + 12 : \ p->g_FastPos[pos >> (6 + kNumLogBits - 1)] + (6 + (kNumLogBits - 1)) * 2; } */ #define GetPosSlot1(pos) p->g_FastPos[pos] #define GetPosSlot2(pos, res) { BSR2_RET(pos, res); } #define GetPosSlot(pos, res) { if (pos < kNumFullDistances) res = p->g_FastPos[pos]; else BSR2_RET(pos, res); } #endif #define LZMA_NUM_REPS 4 typedef unsigned CState; typedef struct _COptimal { UInt32 price; CState state; int prev1IsChar; int prev2; UInt32 posPrev2; UInt32 backPrev2; UInt32 posPrev; UInt32 backPrev; UInt32 backs[LZMA_NUM_REPS]; } COptimal; #define kNumOpts (1 << 12) #define kNumLenToPosStates 4 #define kNumPosSlotBits 6 #define kDicLogSizeMin 0 #define kDicLogSizeMax 32 #define kDistTableSizeMax (kDicLogSizeMax * 2) #define kNumAlignBits 4 #define kAlignTableSize (1 << kNumAlignBits) #define kAlignMask (kAlignTableSize - 1) #define kStartPosModelIndex 4 #define kEndPosModelIndex 14 #define kNumPosModels (kEndPosModelIndex - kStartPosModelIndex) #define kNumFullDistances (1 << (kEndPosModelIndex / 2)) #ifdef _LZMA_PROB32 #define CLzmaProb UInt32 #else #define CLzmaProb UInt16 #endif #define LZMA_PB_MAX 4 #define LZMA_LC_MAX 8 #define LZMA_LP_MAX 4 #define LZMA_NUM_PB_STATES_MAX (1 << LZMA_PB_MAX) #define kLenNumLowBits 3 #define kLenNumLowSymbols (1 << kLenNumLowBits) #define kLenNumMidBits 3 #define kLenNumMidSymbols (1 << kLenNumMidBits) #define kLenNumHighBits 8 #define kLenNumHighSymbols (1 << kLenNumHighBits) #define kLenNumSymbolsTotal (kLenNumLowSymbols + kLenNumMidSymbols + kLenNumHighSymbols) #define LZMA_MATCH_LEN_MIN 2 #define LZMA_MATCH_LEN_MAX (LZMA_MATCH_LEN_MIN + kLenNumSymbolsTotal - 1) #define kNumStates 12 typedef struct { CLzmaProb choice; CLzmaProb choice2; CLzmaProb low[LZMA_NUM_PB_STATES_MAX << kLenNumLowBits]; CLzmaProb mid[LZMA_NUM_PB_STATES_MAX << kLenNumMidBits]; CLzmaProb high[kLenNumHighSymbols]; } CLenEnc; typedef struct { CLenEnc p; UInt32 prices[LZMA_NUM_PB_STATES_MAX][kLenNumSymbolsTotal]; UInt32 tableSize; UInt32 counters[LZMA_NUM_PB_STATES_MAX]; } CLenPriceEnc; typedef struct _CRangeEnc { UInt32 range; Byte cache; UInt64 low; UInt64 cacheSize; Byte *buf; Byte *bufLim; Byte *bufBase; ISeqOutStream *outStream; UInt64 processed; SRes res; } CRangeEnc; typedef struct _CSeqInStreamBuf { ISeqInStream funcTable; const Byte *data; SizeT rem; } CSeqInStreamBuf; static SRes MyRead(void *pp, void *data, size_t *size) { size_t curSize = *size; CSeqInStreamBuf *p = (CSeqInStreamBuf *)pp; if (p->rem < curSize) curSize = p->rem; memcpy(data, p->data, curSize); p->rem -= curSize; p->data += curSize; *size = curSize; return SZ_OK; } typedef struct { CLzmaProb *litProbs; CLzmaProb isMatch[kNumStates][LZMA_NUM_PB_STATES_MAX]; CLzmaProb isRep[kNumStates]; CLzmaProb isRepG0[kNumStates]; CLzmaProb isRepG1[kNumStates]; CLzmaProb isRepG2[kNumStates]; CLzmaProb isRep0Long[kNumStates][LZMA_NUM_PB_STATES_MAX]; CLzmaProb posSlotEncoder[kNumLenToPosStates][1 << kNumPosSlotBits]; CLzmaProb posEncoders[kNumFullDistances - kEndPosModelIndex]; CLzmaProb posAlignEncoder[1 << kNumAlignBits]; CLenPriceEnc lenEnc; CLenPriceEnc repLenEnc; UInt32 reps[LZMA_NUM_REPS]; UInt32 state; } CSaveState; typedef struct _CLzmaEnc { IMatchFinder matchFinder; void *matchFinderObj; #ifdef COMPRESS_MF_MT Bool mtMode; CMatchFinderMt matchFinderMt; #endif CMatchFinder matchFinderBase; #ifdef COMPRESS_MF_MT Byte pad[128]; #endif UInt32 optimumEndIndex; UInt32 optimumCurrentIndex; UInt32 longestMatchLength; UInt32 numPairs; UInt32 numAvail; COptimal opt[kNumOpts]; #ifndef LZMA_LOG_BSR Byte g_FastPos[1 << kNumLogBits]; #endif UInt32 ProbPrices[kBitModelTotal >> kNumMoveReducingBits]; UInt32 matches[LZMA_MATCH_LEN_MAX * 2 + 2 + 1]; UInt32 numFastBytes; UInt32 additionalOffset; UInt32 reps[LZMA_NUM_REPS]; UInt32 state; UInt32 posSlotPrices[kNumLenToPosStates][kDistTableSizeMax]; UInt32 distancesPrices[kNumLenToPosStates][kNumFullDistances]; UInt32 alignPrices[kAlignTableSize]; UInt32 alignPriceCount; UInt32 distTableSize; unsigned lc, lp, pb; unsigned lpMask, pbMask; CLzmaProb *litProbs; CLzmaProb isMatch[kNumStates][LZMA_NUM_PB_STATES_MAX]; CLzmaProb isRep[kNumStates]; CLzmaProb isRepG0[kNumStates]; CLzmaProb isRepG1[kNumStates]; CLzmaProb isRepG2[kNumStates]; CLzmaProb isRep0Long[kNumStates][LZMA_NUM_PB_STATES_MAX]; CLzmaProb posSlotEncoder[kNumLenToPosStates][1 << kNumPosSlotBits]; CLzmaProb posEncoders[kNumFullDistances - kEndPosModelIndex]; CLzmaProb posAlignEncoder[1 << kNumAlignBits]; CLenPriceEnc lenEnc; CLenPriceEnc repLenEnc; unsigned lclp; Bool fastMode; CRangeEnc rc; Bool writeEndMark; UInt64 nowPos64; UInt32 matchPriceCount; Bool finished; Bool multiThread; SRes result; UInt32 dictSize; UInt32 matchFinderCycles; ISeqInStream *inStream; CSeqInStreamBuf seqBufInStream; CSaveState saveState; } CLzmaEnc; void LzmaEnc_SaveState(CLzmaEncHandle pp) { CLzmaEnc *p = (CLzmaEnc *)pp; CSaveState *dest = &p->saveState; int i; dest->lenEnc = p->lenEnc; dest->repLenEnc = p->repLenEnc; dest->state = p->state; for (i = 0; i < kNumStates; i++) { memcpy(dest->isMatch[i], p->isMatch[i], sizeof(p->isMatch[i])); memcpy(dest->isRep0Long[i], p->isRep0Long[i], sizeof(p->isRep0Long[i])); } for (i = 0; i < kNumLenToPosStates; i++) memcpy(dest->posSlotEncoder[i], p->posSlotEncoder[i], sizeof(p->posSlotEncoder[i])); memcpy(dest->isRep, p->isRep, sizeof(p->isRep)); memcpy(dest->isRepG0, p->isRepG0, sizeof(p->isRepG0)); memcpy(dest->isRepG1, p->isRepG1, sizeof(p->isRepG1)); memcpy(dest->isRepG2, p->isRepG2, sizeof(p->isRepG2)); memcpy(dest->posEncoders, p->posEncoders, sizeof(p->posEncoders)); memcpy(dest->posAlignEncoder, p->posAlignEncoder, sizeof(p->posAlignEncoder)); memcpy(dest->reps, p->reps, sizeof(p->reps)); memcpy(dest->litProbs, p->litProbs, (0x300 << p->lclp) * sizeof(CLzmaProb)); } void LzmaEnc_RestoreState(CLzmaEncHandle pp) { CLzmaEnc *dest = (CLzmaEnc *)pp; const CSaveState *p = &dest->saveState; int i; dest->lenEnc = p->lenEnc; dest->repLenEnc = p->repLenEnc; dest->state = p->state; for (i = 0; i < kNumStates; i++) { memcpy(dest->isMatch[i], p->isMatch[i], sizeof(p->isMatch[i])); memcpy(dest->isRep0Long[i], p->isRep0Long[i], sizeof(p->isRep0Long[i])); } for (i = 0; i < kNumLenToPosStates; i++) memcpy(dest->posSlotEncoder[i], p->posSlotEncoder[i], sizeof(p->posSlotEncoder[i])); memcpy(dest->isRep, p->isRep, sizeof(p->isRep)); memcpy(dest->isRepG0, p->isRepG0, sizeof(p->isRepG0)); memcpy(dest->isRepG1, p->isRepG1, sizeof(p->isRepG1)); memcpy(dest->isRepG2, p->isRepG2, sizeof(p->isRepG2)); memcpy(dest->posEncoders, p->posEncoders, sizeof(p->posEncoders)); memcpy(dest->posAlignEncoder, p->posAlignEncoder, sizeof(p->posAlignEncoder)); memcpy(dest->reps, p->reps, sizeof(p->reps)); memcpy(dest->litProbs, p->litProbs, (0x300 << dest->lclp) * sizeof(CLzmaProb)); } SRes LzmaEnc_SetProps(CLzmaEncHandle pp, const CLzmaEncProps *props2) { CLzmaEnc *p = (CLzmaEnc *)pp; CLzmaEncProps props = *props2; LzmaEncProps_Normalize(&props); if (props.lc > LZMA_LC_MAX || props.lp > LZMA_LP_MAX || props.pb > LZMA_PB_MAX || props.dictSize > (1 << kDicLogSizeMaxCompress) || props.dictSize > (1 << 30)) return SZ_ERROR_PARAM; p->dictSize = props.dictSize; p->matchFinderCycles = props.mc; { unsigned fb = props.fb; if (fb < 5) fb = 5; if (fb > LZMA_MATCH_LEN_MAX) fb = LZMA_MATCH_LEN_MAX; p->numFastBytes = fb; } p->lc = props.lc; p->lp = props.lp; p->pb = props.pb; p->fastMode = (props.algo == 0); p->matchFinderBase.btMode = props.btMode; { UInt32 numHashBytes = 4; if (props.btMode) { if (props.numHashBytes < 2) numHashBytes = 2; else if (props.numHashBytes < 4) numHashBytes = props.numHashBytes; } p->matchFinderBase.numHashBytes = numHashBytes; } p->matchFinderBase.cutValue = props.mc; p->writeEndMark = props.writeEndMark; #ifdef COMPRESS_MF_MT /* if (newMultiThread != _multiThread) { ReleaseMatchFinder(); _multiThread = newMultiThread; } */ p->multiThread = (props.numThreads > 1); #endif return SZ_OK; } static const int kLiteralNextStates[kNumStates] = {0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 4, 5}; static const int kMatchNextStates[kNumStates] = {7, 7, 7, 7, 7, 7, 7, 10, 10, 10, 10, 10}; static const int kRepNextStates[kNumStates] = {8, 8, 8, 8, 8, 8, 8, 11, 11, 11, 11, 11}; static const int kShortRepNextStates[kNumStates]= {9, 9, 9, 9, 9, 9, 9, 11, 11, 11, 11, 11}; #define IsCharState(s) ((s) < 7) #define GetLenToPosState(len) (((len) < kNumLenToPosStates + 1) ? (len) - 2 : kNumLenToPosStates - 1) #define kInfinityPrice (1 << 30) static void RangeEnc_Construct(CRangeEnc *p) { p->outStream = 0; p->bufBase = 0; } #define RangeEnc_GetProcessed(p) ((p)->processed + ((p)->buf - (p)->bufBase) + (p)->cacheSize) #define RC_BUF_SIZE (1 << 16) static int RangeEnc_Alloc(CRangeEnc *p, ISzAlloc *alloc) { if (p->bufBase == 0) { p->bufBase = (Byte *)alloc->Alloc(alloc, RC_BUF_SIZE); if (p->bufBase == 0) return 0; p->bufLim = p->bufBase + RC_BUF_SIZE; } return 1; } static void RangeEnc_Free(CRangeEnc *p, ISzAlloc *alloc) { alloc->Free(alloc, p->bufBase); p->bufBase = 0; } static void RangeEnc_Init(CRangeEnc *p) { /* Stream.Init(); */ p->low = 0; p->range = 0xFFFFFFFF; p->cacheSize = 1; p->cache = 0; p->buf = p->bufBase; p->processed = 0; p->res = SZ_OK; } static void RangeEnc_FlushStream(CRangeEnc *p) { size_t num; if (p->res != SZ_OK) return; num = p->buf - p->bufBase; if (num != p->outStream->Write(p->outStream, p->bufBase, num)) p->res = SZ_ERROR_WRITE; p->processed += num; p->buf = p->bufBase; } static void MY_FAST_CALL RangeEnc_ShiftLow(CRangeEnc *p) { if ((UInt32)p->low < (UInt32)0xFF000000 || (int)(p->low >> 32) != 0) { Byte temp = p->cache; do { Byte *buf = p->buf; *buf++ = (Byte)(temp + (Byte)(p->low >> 32)); p->buf = buf; if (buf == p->bufLim) RangeEnc_FlushStream(p); temp = 0xFF; } while (--p->cacheSize != 0); p->cache = (Byte)((UInt32)p->low >> 24); } p->cacheSize++; p->low = (UInt32)p->low << 8; } static void RangeEnc_FlushData(CRangeEnc *p) { int i; for (i = 0; i < 5; i++) RangeEnc_ShiftLow(p); } static void RangeEnc_EncodeDirectBits(CRangeEnc *p, UInt32 value, int numBits) { do { p->range >>= 1; p->low += p->range & (0 - ((value >> --numBits) & 1)); if (p->range < kTopValue) { p->range <<= 8; RangeEnc_ShiftLow(p); } } while (numBits != 0); } static void RangeEnc_EncodeBit(CRangeEnc *p, CLzmaProb *prob, UInt32 symbol) { UInt32 ttt = *prob; UInt32 newBound = (p->range >> kNumBitModelTotalBits) * ttt; if (symbol == 0) { p->range = newBound; ttt += (kBitModelTotal - ttt) >> kNumMoveBits; } else { p->low += newBound; p->range -= newBound; ttt -= ttt >> kNumMoveBits; } *prob = (CLzmaProb)ttt; if (p->range < kTopValue) { p->range <<= 8; RangeEnc_ShiftLow(p); } } static void LitEnc_Encode(CRangeEnc *p, CLzmaProb *probs, UInt32 symbol) { symbol |= 0x100; do { RangeEnc_EncodeBit(p, probs + (symbol >> 8), (symbol >> 7) & 1); symbol <<= 1; } while (symbol < 0x10000); } static void LitEnc_EncodeMatched(CRangeEnc *p, CLzmaProb *probs, UInt32 symbol, UInt32 matchByte) { UInt32 offs = 0x100; symbol |= 0x100; do { matchByte <<= 1; RangeEnc_EncodeBit(p, probs + (offs + (matchByte & offs) + (symbol >> 8)), (symbol >> 7) & 1); symbol <<= 1; offs &= ~(matchByte ^ symbol); } while (symbol < 0x10000); } void LzmaEnc_InitPriceTables(UInt32 *ProbPrices) { UInt32 i; for (i = (1 << kNumMoveReducingBits) / 2; i < kBitModelTotal; i += (1 << kNumMoveReducingBits)) { const int kCyclesBits = kNumBitPriceShiftBits; UInt32 w = i; UInt32 bitCount = 0; int j; for (j = 0; j < kCyclesBits; j++) { w = w * w; bitCount <<= 1; while (w >= ((UInt32)1 << 16)) { w >>= 1; bitCount++; } } ProbPrices[i >> kNumMoveReducingBits] = ((kNumBitModelTotalBits << kCyclesBits) - 15 - bitCount); } } #define GET_PRICE(prob, symbol) \ p->ProbPrices[((prob) ^ (((-(int)(symbol))) & (kBitModelTotal - 1))) >> kNumMoveReducingBits]; #define GET_PRICEa(prob, symbol) \ ProbPrices[((prob) ^ ((-((int)(symbol))) & (kBitModelTotal - 1))) >> kNumMoveReducingBits]; #define GET_PRICE_0(prob) p->ProbPrices[(prob) >> kNumMoveReducingBits] #define GET_PRICE_1(prob) p->ProbPrices[((prob) ^ (kBitModelTotal - 1)) >> kNumMoveReducingBits] #define GET_PRICE_0a(prob) ProbPrices[(prob) >> kNumMoveReducingBits] #define GET_PRICE_1a(prob) ProbPrices[((prob) ^ (kBitModelTotal - 1)) >> kNumMoveReducingBits] static UInt32 LitEnc_GetPrice(const CLzmaProb *probs, UInt32 symbol, UInt32 *ProbPrices) { UInt32 price = 0; symbol |= 0x100; do { price += GET_PRICEa(probs[symbol >> 8], (symbol >> 7) & 1); symbol <<= 1; } while (symbol < 0x10000); return price; } static UInt32 LitEnc_GetPriceMatched(const CLzmaProb *probs, UInt32 symbol, UInt32 matchByte, UInt32 *ProbPrices) { UInt32 price = 0; UInt32 offs = 0x100; symbol |= 0x100; do { matchByte <<= 1; price += GET_PRICEa(probs[offs + (matchByte & offs) + (symbol >> 8)], (symbol >> 7) & 1); symbol <<= 1; offs &= ~(matchByte ^ symbol); } while (symbol < 0x10000); return price; } static void RcTree_Encode(CRangeEnc *rc, CLzmaProb *probs, int numBitLevels, UInt32 symbol) { UInt32 m = 1; int i; for (i = numBitLevels; i != 0;) { UInt32 bit; i--; bit = (symbol >> i) & 1; RangeEnc_EncodeBit(rc, probs + m, bit); m = (m << 1) | bit; } } static void RcTree_ReverseEncode(CRangeEnc *rc, CLzmaProb *probs, int numBitLevels, UInt32 symbol) { UInt32 m = 1; int i; for (i = 0; i < numBitLevels; i++) { UInt32 bit = symbol & 1; RangeEnc_EncodeBit(rc, probs + m, bit); m = (m << 1) | bit; symbol >>= 1; } } static UInt32 RcTree_GetPrice(const CLzmaProb *probs, int numBitLevels, UInt32 symbol, UInt32 *ProbPrices) { UInt32 price = 0; symbol |= (1 << numBitLevels); while (symbol != 1) { price += GET_PRICEa(probs[symbol >> 1], symbol & 1); symbol >>= 1; } return price; } static UInt32 RcTree_ReverseGetPrice(const CLzmaProb *probs, int numBitLevels, UInt32 symbol, UInt32 *ProbPrices) { UInt32 price = 0; UInt32 m = 1; int i; for (i = numBitLevels; i != 0; i--) { UInt32 bit = symbol & 1; symbol >>= 1; price += GET_PRICEa(probs[m], bit); m = (m << 1) | bit; } return price; } static void LenEnc_Init(CLenEnc *p) { unsigned i; p->choice = p->choice2 = kProbInitValue; for (i = 0; i < (LZMA_NUM_PB_STATES_MAX << kLenNumLowBits); i++) p->low[i] = kProbInitValue; for (i = 0; i < (LZMA_NUM_PB_STATES_MAX << kLenNumMidBits); i++) p->mid[i] = kProbInitValue; for (i = 0; i < kLenNumHighSymbols; i++) p->high[i] = kProbInitValue; } static void LenEnc_Encode(CLenEnc *p, CRangeEnc *rc, UInt32 symbol, UInt32 posState) { if (symbol < kLenNumLowSymbols) { RangeEnc_EncodeBit(rc, &p->choice, 0); RcTree_Encode(rc, p->low + (posState << kLenNumLowBits), kLenNumLowBits, symbol); } else { RangeEnc_EncodeBit(rc, &p->choice, 1); if (symbol < kLenNumLowSymbols + kLenNumMidSymbols) { RangeEnc_EncodeBit(rc, &p->choice2, 0); RcTree_Encode(rc, p->mid + (posState << kLenNumMidBits), kLenNumMidBits, symbol - kLenNumLowSymbols); } else { RangeEnc_EncodeBit(rc, &p->choice2, 1); RcTree_Encode(rc, p->high, kLenNumHighBits, symbol - kLenNumLowSymbols - kLenNumMidSymbols); } } } static void LenEnc_SetPrices(CLenEnc *p, UInt32 posState, UInt32 numSymbols, UInt32 *prices, UInt32 *ProbPrices) { UInt32 a0 = GET_PRICE_0a(p->choice); UInt32 a1 = GET_PRICE_1a(p->choice); UInt32 b0 = a1 + GET_PRICE_0a(p->choice2); UInt32 b1 = a1 + GET_PRICE_1a(p->choice2); UInt32 i = 0; for (i = 0; i < kLenNumLowSymbols; i++) { if (i >= numSymbols) return; prices[i] = a0 + RcTree_GetPrice(p->low + (posState << kLenNumLowBits), kLenNumLowBits, i, ProbPrices); } for (; i < kLenNumLowSymbols + kLenNumMidSymbols; i++) { if (i >= numSymbols) return; prices[i] = b0 + RcTree_GetPrice(p->mid + (posState << kLenNumMidBits), kLenNumMidBits, i - kLenNumLowSymbols, ProbPrices); } for (; i < numSymbols; i++) prices[i] = b1 + RcTree_GetPrice(p->high, kLenNumHighBits, i - kLenNumLowSymbols - kLenNumMidSymbols, ProbPrices); } static void MY_FAST_CALL LenPriceEnc_UpdateTable(CLenPriceEnc *p, UInt32 posState, UInt32 *ProbPrices) { LenEnc_SetPrices(&p->p, posState, p->tableSize, p->prices[posState], ProbPrices); p->counters[posState] = p->tableSize; } static void LenPriceEnc_UpdateTables(CLenPriceEnc *p, UInt32 numPosStates, UInt32 *ProbPrices) { UInt32 posState; for (posState = 0; posState < numPosStates; posState++) LenPriceEnc_UpdateTable(p, posState, ProbPrices); } static void LenEnc_Encode2(CLenPriceEnc *p, CRangeEnc *rc, UInt32 symbol, UInt32 posState, Bool updatePrice, UInt32 *ProbPrices) { LenEnc_Encode(&p->p, rc, symbol, posState); if (updatePrice) if (--p->counters[posState] == 0) LenPriceEnc_UpdateTable(p, posState, ProbPrices); } static void MovePos(CLzmaEnc *p, UInt32 num) { #ifdef SHOW_STAT ttt += num; printf("\n MovePos %d", num); #endif if (num != 0) { p->additionalOffset += num; p->matchFinder.Skip(p->matchFinderObj, num); } } static UInt32 ReadMatchDistances(CLzmaEnc *p, UInt32 *numDistancePairsRes) { UInt32 lenRes = 0, numPairs; p->numAvail = p->matchFinder.GetNumAvailableBytes(p->matchFinderObj); numPairs = p->matchFinder.GetMatches(p->matchFinderObj, p->matches); #ifdef SHOW_STAT printf("\n i = %d numPairs = %d ", ttt, numPairs / 2); ttt++; { UInt32 i; for (i = 0; i < numPairs; i += 2) printf("%2d %6d | ", p->matches[i], p->matches[i + 1]); } #endif if (numPairs > 0) { lenRes = p->matches[numPairs - 2]; if (lenRes == p->numFastBytes) { const Byte *pby = p->matchFinder.GetPointerToCurrentPos(p->matchFinderObj) - 1; UInt32 distance = p->matches[numPairs - 1] + 1; UInt32 numAvail = p->numAvail; if (numAvail > LZMA_MATCH_LEN_MAX) numAvail = LZMA_MATCH_LEN_MAX; { const Byte *pby2 = pby - distance; for (; lenRes < numAvail && pby[lenRes] == pby2[lenRes]; lenRes++); } } } p->additionalOffset++; *numDistancePairsRes = numPairs; return lenRes; } #define MakeAsChar(p) (p)->backPrev = (UInt32)(-1); (p)->prev1IsChar = False; #define MakeAsShortRep(p) (p)->backPrev = 0; (p)->prev1IsChar = False; #define IsShortRep(p) ((p)->backPrev == 0) static UInt32 GetRepLen1Price(CLzmaEnc *p, UInt32 state, UInt32 posState) { return GET_PRICE_0(p->isRepG0[state]) + GET_PRICE_0(p->isRep0Long[state][posState]); } static UInt32 GetPureRepPrice(CLzmaEnc *p, UInt32 repIndex, UInt32 state, UInt32 posState) { UInt32 price; if (repIndex == 0) { price = GET_PRICE_0(p->isRepG0[state]); price += GET_PRICE_1(p->isRep0Long[state][posState]); } else { price = GET_PRICE_1(p->isRepG0[state]); if (repIndex == 1) price += GET_PRICE_0(p->isRepG1[state]); else { price += GET_PRICE_1(p->isRepG1[state]); price += GET_PRICE(p->isRepG2[state], repIndex - 2); } } return price; } static UInt32 GetRepPrice(CLzmaEnc *p, UInt32 repIndex, UInt32 len, UInt32 state, UInt32 posState) { return p->repLenEnc.prices[posState][len - LZMA_MATCH_LEN_MIN] + GetPureRepPrice(p, repIndex, state, posState); } static UInt32 Backward(CLzmaEnc *p, UInt32 *backRes, UInt32 cur) { UInt32 posMem = p->opt[cur].posPrev; UInt32 backMem = p->opt[cur].backPrev; p->optimumEndIndex = cur; do { if (p->opt[cur].prev1IsChar) { MakeAsChar(&p->opt[posMem]) p->opt[posMem].posPrev = posMem - 1; if (p->opt[cur].prev2) { p->opt[posMem - 1].prev1IsChar = False; p->opt[posMem - 1].posPrev = p->opt[cur].posPrev2; p->opt[posMem - 1].backPrev = p->opt[cur].backPrev2; } } { UInt32 posPrev = posMem; UInt32 backCur = backMem; backMem = p->opt[posPrev].backPrev; posMem = p->opt[posPrev].posPrev; p->opt[posPrev].backPrev = backCur; p->opt[posPrev].posPrev = cur; cur = posPrev; } } while (cur != 0); *backRes = p->opt[0].backPrev; p->optimumCurrentIndex = p->opt[0].posPrev; return p->optimumCurrentIndex; } #define LIT_PROBS(pos, prevByte) (p->litProbs + ((((pos) & p->lpMask) << p->lc) + ((prevByte) >> (8 - p->lc))) * 0x300) static UInt32 GetOptimum(CLzmaEnc *p, UInt32 position, UInt32 *backRes) { UInt32 numAvail, mainLen, numPairs, repMaxIndex, i, posState, lenEnd, len, cur; UInt32 matchPrice, repMatchPrice, normalMatchPrice; UInt32 reps[LZMA_NUM_REPS], repLens[LZMA_NUM_REPS]; UInt32 *matches; const Byte *data; Byte curByte, matchByte; if (p->optimumEndIndex != p->optimumCurrentIndex) { const COptimal *opt = &p->opt[p->optimumCurrentIndex]; UInt32 lenRes = opt->posPrev - p->optimumCurrentIndex; *backRes = opt->backPrev; p->optimumCurrentIndex = opt->posPrev; return lenRes; } p->optimumCurrentIndex = p->optimumEndIndex = 0; if (p->additionalOffset == 0) mainLen = ReadMatchDistances(p, &numPairs); else { mainLen = p->longestMatchLength; numPairs = p->numPairs; } numAvail = p->numAvail; if (numAvail < 2) { *backRes = (UInt32)(-1); return 1; } if (numAvail > LZMA_MATCH_LEN_MAX) numAvail = LZMA_MATCH_LEN_MAX; data = p->matchFinder.GetPointerToCurrentPos(p->matchFinderObj) - 1; repMaxIndex = 0; for (i = 0; i < LZMA_NUM_REPS; i++) { UInt32 lenTest; const Byte *data2; reps[i] = p->reps[i]; data2 = data - (reps[i] + 1); if (data[0] != data2[0] || data[1] != data2[1]) { repLens[i] = 0; continue; } for (lenTest = 2; lenTest < numAvail && data[lenTest] == data2[lenTest]; lenTest++); repLens[i] = lenTest; if (lenTest > repLens[repMaxIndex]) repMaxIndex = i; } if (repLens[repMaxIndex] >= p->numFastBytes) { UInt32 lenRes; *backRes = repMaxIndex; lenRes = repLens[repMaxIndex]; MovePos(p, lenRes - 1); return lenRes; } matches = p->matches; if (mainLen >= p->numFastBytes) { *backRes = matches[numPairs - 1] + LZMA_NUM_REPS; MovePos(p, mainLen - 1); return mainLen; } curByte = *data; matchByte = *(data - (reps[0] + 1)); if (mainLen < 2 && curByte != matchByte && repLens[repMaxIndex] < 2) { *backRes = (UInt32)-1; return 1; } p->opt[0].state = (CState)p->state; posState = (position & p->pbMask); { const CLzmaProb *probs = LIT_PROBS(position, *(data - 1)); p->opt[1].price = GET_PRICE_0(p->isMatch[p->state][posState]) + (!IsCharState(p->state) ? LitEnc_GetPriceMatched(probs, curByte, matchByte, p->ProbPrices) : LitEnc_GetPrice(probs, curByte, p->ProbPrices)); } MakeAsChar(&p->opt[1]); matchPrice = GET_PRICE_1(p->isMatch[p->state][posState]); repMatchPrice = matchPrice + GET_PRICE_1(p->isRep[p->state]); if (matchByte == curByte) { UInt32 shortRepPrice = repMatchPrice + GetRepLen1Price(p, p->state, posState); if (shortRepPrice < p->opt[1].price) { p->opt[1].price = shortRepPrice; MakeAsShortRep(&p->opt[1]); } } lenEnd = ((mainLen >= repLens[repMaxIndex]) ? mainLen : repLens[repMaxIndex]); if (lenEnd < 2) { *backRes = p->opt[1].backPrev; return 1; } p->opt[1].posPrev = 0; for (i = 0; i < LZMA_NUM_REPS; i++) p->opt[0].backs[i] = reps[i]; len = lenEnd; do p->opt[len--].price = kInfinityPrice; while (len >= 2); for (i = 0; i < LZMA_NUM_REPS; i++) { UInt32 repLen = repLens[i]; UInt32 price; if (repLen < 2) continue; price = repMatchPrice + GetPureRepPrice(p, i, p->state, posState); do { UInt32 curAndLenPrice = price + p->repLenEnc.prices[posState][repLen - 2]; COptimal *opt = &p->opt[repLen]; if (curAndLenPrice < opt->price) { opt->price = curAndLenPrice; opt->posPrev = 0; opt->backPrev = i; opt->prev1IsChar = False; } } while (--repLen >= 2); } normalMatchPrice = matchPrice + GET_PRICE_0(p->isRep[p->state]); len = ((repLens[0] >= 2) ? repLens[0] + 1 : 2); if (len <= mainLen) { UInt32 offs = 0; while (len > matches[offs]) offs += 2; for (; ; len++) { COptimal *opt; UInt32 distance = matches[offs + 1]; UInt32 curAndLenPrice = normalMatchPrice + p->lenEnc.prices[posState][len - LZMA_MATCH_LEN_MIN]; UInt32 lenToPosState = GetLenToPosState(len); if (distance < kNumFullDistances) curAndLenPrice += p->distancesPrices[lenToPosState][distance]; else { UInt32 slot; GetPosSlot2(distance, slot); curAndLenPrice += p->alignPrices[distance & kAlignMask] + p->posSlotPrices[lenToPosState][slot]; } opt = &p->opt[len]; if (curAndLenPrice < opt->price) { opt->price = curAndLenPrice; opt->posPrev = 0; opt->backPrev = distance + LZMA_NUM_REPS; opt->prev1IsChar = False; } if (len == matches[offs]) { offs += 2; if (offs == numPairs) break; } } } cur = 0; #ifdef SHOW_STAT2 if (position >= 0) { unsigned i; printf("\n pos = %4X", position); for (i = cur; i <= lenEnd; i++) printf("\nprice[%4X] = %d", position - cur + i, p->opt[i].price); } #endif for (;;) { UInt32 numAvailFull, newLen, numPairs, posPrev, state, posState, startLen; UInt32 curPrice, curAnd1Price, matchPrice, repMatchPrice; Bool nextIsChar; Byte curByte, matchByte; const Byte *data; COptimal *curOpt; COptimal *nextOpt; cur++; if (cur == lenEnd) return Backward(p, backRes, cur); newLen = ReadMatchDistances(p, &numPairs); if (newLen >= p->numFastBytes) { p->numPairs = numPairs; p->longestMatchLength = newLen; return Backward(p, backRes, cur); } position++; curOpt = &p->opt[cur]; posPrev = curOpt->posPrev; if (curOpt->prev1IsChar) { posPrev--; if (curOpt->prev2) { state = p->opt[curOpt->posPrev2].state; if (curOpt->backPrev2 < LZMA_NUM_REPS) state = kRepNextStates[state]; else state = kMatchNextStates[state]; } else state = p->opt[posPrev].state; state = kLiteralNextStates[state]; } else state = p->opt[posPrev].state; if (posPrev == cur - 1) { if (IsShortRep(curOpt)) state = kShortRepNextStates[state]; else state = kLiteralNextStates[state]; } else { UInt32 pos; const COptimal *prevOpt; if (curOpt->prev1IsChar && curOpt->prev2) { posPrev = curOpt->posPrev2; pos = curOpt->backPrev2; state = kRepNextStates[state]; } else { pos = curOpt->backPrev; if (pos < LZMA_NUM_REPS) state = kRepNextStates[state]; else state = kMatchNextStates[state]; } prevOpt = &p->opt[posPrev]; if (pos < LZMA_NUM_REPS) { UInt32 i; reps[0] = prevOpt->backs[pos]; for (i = 1; i <= pos; i++) reps[i] = prevOpt->backs[i - 1]; for (; i < LZMA_NUM_REPS; i++) reps[i] = prevOpt->backs[i]; } else { UInt32 i; reps[0] = (pos - LZMA_NUM_REPS); for (i = 1; i < LZMA_NUM_REPS; i++) reps[i] = prevOpt->backs[i - 1]; } } curOpt->state = (CState)state; curOpt->backs[0] = reps[0]; curOpt->backs[1] = reps[1]; curOpt->backs[2] = reps[2]; curOpt->backs[3] = reps[3]; curPrice = curOpt->price; nextIsChar = False; data = p->matchFinder.GetPointerToCurrentPos(p->matchFinderObj) - 1; curByte = *data; matchByte = *(data - (reps[0] + 1)); posState = (position & p->pbMask); curAnd1Price = curPrice + GET_PRICE_0(p->isMatch[state][posState]); { const CLzmaProb *probs = LIT_PROBS(position, *(data - 1)); curAnd1Price += (!IsCharState(state) ? LitEnc_GetPriceMatched(probs, curByte, matchByte, p->ProbPrices) : LitEnc_GetPrice(probs, curByte, p->ProbPrices)); } nextOpt = &p->opt[cur + 1]; if (curAnd1Price < nextOpt->price) { nextOpt->price = curAnd1Price; nextOpt->posPrev = cur; MakeAsChar(nextOpt); nextIsChar = True; } matchPrice = curPrice + GET_PRICE_1(p->isMatch[state][posState]); repMatchPrice = matchPrice + GET_PRICE_1(p->isRep[state]); if (matchByte == curByte && !(nextOpt->posPrev < cur && nextOpt->backPrev == 0)) { UInt32 shortRepPrice = repMatchPrice + GetRepLen1Price(p, state, posState); if (shortRepPrice <= nextOpt->price) { nextOpt->price = shortRepPrice; nextOpt->posPrev = cur; MakeAsShortRep(nextOpt); nextIsChar = True; } } numAvailFull = p->numAvail; { UInt32 temp = kNumOpts - 1 - cur; if (temp < numAvailFull) numAvailFull = temp; } if (numAvailFull < 2) continue; numAvail = (numAvailFull <= p->numFastBytes ? numAvailFull : p->numFastBytes); if (!nextIsChar && matchByte != curByte) /* speed optimization */ { /* try Literal + rep0 */ UInt32 temp; UInt32 lenTest2; const Byte *data2 = data - (reps[0] + 1); UInt32 limit = p->numFastBytes + 1; if (limit > numAvailFull) limit = numAvailFull; for (temp = 1; temp < limit && data[temp] == data2[temp]; temp++); lenTest2 = temp - 1; if (lenTest2 >= 2) { UInt32 state2 = kLiteralNextStates[state]; UInt32 posStateNext = (position + 1) & p->pbMask; UInt32 nextRepMatchPrice = curAnd1Price + GET_PRICE_1(p->isMatch[state2][posStateNext]) + GET_PRICE_1(p->isRep[state2]); /* for (; lenTest2 >= 2; lenTest2--) */ { UInt32 curAndLenPrice; COptimal *opt; UInt32 offset = cur + 1 + lenTest2; while (lenEnd < offset) p->opt[++lenEnd].price = kInfinityPrice; curAndLenPrice = nextRepMatchPrice + GetRepPrice(p, 0, lenTest2, state2, posStateNext); opt = &p->opt[offset]; if (curAndLenPrice < opt->price) { opt->price = curAndLenPrice; opt->posPrev = cur + 1; opt->backPrev = 0; opt->prev1IsChar = True; opt->prev2 = False; } } } } startLen = 2; /* speed optimization */ { UInt32 repIndex; for (repIndex = 0; repIndex < LZMA_NUM_REPS; repIndex++) { UInt32 lenTest; UInt32 lenTestTemp; UInt32 price; const Byte *data2 = data - (reps[repIndex] + 1); if (data[0] != data2[0] || data[1] != data2[1]) continue; for (lenTest = 2; lenTest < numAvail && data[lenTest] == data2[lenTest]; lenTest++); while (lenEnd < cur + lenTest) p->opt[++lenEnd].price = kInfinityPrice; lenTestTemp = lenTest; price = repMatchPrice + GetPureRepPrice(p, repIndex, state, posState); do { UInt32 curAndLenPrice = price + p->repLenEnc.prices[posState][lenTest - 2]; COptimal *opt = &p->opt[cur + lenTest]; if (curAndLenPrice < opt->price) { opt->price = curAndLenPrice; opt->posPrev = cur; opt->backPrev = repIndex; opt->prev1IsChar = False; } } while (--lenTest >= 2); lenTest = lenTestTemp; if (repIndex == 0) startLen = lenTest + 1; /* if (_maxMode) */ { UInt32 lenTest2 = lenTest + 1; UInt32 limit = lenTest2 + p->numFastBytes; UInt32 nextRepMatchPrice; if (limit > numAvailFull) limit = numAvailFull; for (; lenTest2 < limit && data[lenTest2] == data2[lenTest2]; lenTest2++); lenTest2 -= lenTest + 1; if (lenTest2 >= 2) { UInt32 state2 = kRepNextStates[state]; UInt32 posStateNext = (position + lenTest) & p->pbMask; UInt32 curAndLenCharPrice = price + p->repLenEnc.prices[posState][lenTest - 2] + GET_PRICE_0(p->isMatch[state2][posStateNext]) + LitEnc_GetPriceMatched(LIT_PROBS(position + lenTest, data[lenTest - 1]), data[lenTest], data2[lenTest], p->ProbPrices); state2 = kLiteralNextStates[state2]; posStateNext = (position + lenTest + 1) & p->pbMask; nextRepMatchPrice = curAndLenCharPrice + GET_PRICE_1(p->isMatch[state2][posStateNext]) + GET_PRICE_1(p->isRep[state2]); /* for (; lenTest2 >= 2; lenTest2--) */ { UInt32 curAndLenPrice; COptimal *opt; UInt32 offset = cur + lenTest + 1 + lenTest2; while (lenEnd < offset) p->opt[++lenEnd].price = kInfinityPrice; curAndLenPrice = nextRepMatchPrice + GetRepPrice(p, 0, lenTest2, state2, posStateNext); opt = &p->opt[offset]; if (curAndLenPrice < opt->price) { opt->price = curAndLenPrice; opt->posPrev = cur + lenTest + 1; opt->backPrev = 0; opt->prev1IsChar = True; opt->prev2 = True; opt->posPrev2 = cur; opt->backPrev2 = repIndex; } } } } } } /* for (UInt32 lenTest = 2; lenTest <= newLen; lenTest++) */ if (newLen > numAvail) { newLen = numAvail; for (numPairs = 0; newLen > matches[numPairs]; numPairs += 2); matches[numPairs] = newLen; numPairs += 2; } if (newLen >= startLen) { UInt32 normalMatchPrice = matchPrice + GET_PRICE_0(p->isRep[state]); UInt32 offs, curBack, posSlot; UInt32 lenTest; while (lenEnd < cur + newLen) p->opt[++lenEnd].price = kInfinityPrice; offs = 0; while (startLen > matches[offs]) offs += 2; curBack = matches[offs + 1]; GetPosSlot2(curBack, posSlot); for (lenTest = /*2*/ startLen; ; lenTest++) { UInt32 curAndLenPrice = normalMatchPrice + p->lenEnc.prices[posState][lenTest - LZMA_MATCH_LEN_MIN]; UInt32 lenToPosState = GetLenToPosState(lenTest); COptimal *opt; if (curBack < kNumFullDistances) curAndLenPrice += p->distancesPrices[lenToPosState][curBack]; else curAndLenPrice += p->posSlotPrices[lenToPosState][posSlot] + p->alignPrices[curBack & kAlignMask]; opt = &p->opt[cur + lenTest]; if (curAndLenPrice < opt->price) { opt->price = curAndLenPrice; opt->posPrev = cur; opt->backPrev = curBack + LZMA_NUM_REPS; opt->prev1IsChar = False; } if (/*_maxMode && */lenTest == matches[offs]) { /* Try Match + Literal + Rep0 */ const Byte *data2 = data - (curBack + 1); UInt32 lenTest2 = lenTest + 1; UInt32 limit = lenTest2 + p->numFastBytes; UInt32 nextRepMatchPrice; if (limit > numAvailFull) limit = numAvailFull; for (; lenTest2 < limit && data[lenTest2] == data2[lenTest2]; lenTest2++); lenTest2 -= lenTest + 1; if (lenTest2 >= 2) { UInt32 state2 = kMatchNextStates[state]; UInt32 posStateNext = (position + lenTest) & p->pbMask; UInt32 curAndLenCharPrice = curAndLenPrice + GET_PRICE_0(p->isMatch[state2][posStateNext]) + LitEnc_GetPriceMatched(LIT_PROBS(position + lenTest, data[lenTest - 1]), data[lenTest], data2[lenTest], p->ProbPrices); state2 = kLiteralNextStates[state2]; posStateNext = (posStateNext + 1) & p->pbMask; nextRepMatchPrice = curAndLenCharPrice + GET_PRICE_1(p->isMatch[state2][posStateNext]) + GET_PRICE_1(p->isRep[state2]); /* for (; lenTest2 >= 2; lenTest2--) */ { UInt32 offset = cur + lenTest + 1 + lenTest2; UInt32 curAndLenPrice; COptimal *opt; while (lenEnd < offset) p->opt[++lenEnd].price = kInfinityPrice; curAndLenPrice = nextRepMatchPrice + GetRepPrice(p, 0, lenTest2, state2, posStateNext); opt = &p->opt[offset]; if (curAndLenPrice < opt->price) { opt->price = curAndLenPrice; opt->posPrev = cur + lenTest + 1; opt->backPrev = 0; opt->prev1IsChar = True; opt->prev2 = True; opt->posPrev2 = cur; opt->backPrev2 = curBack + LZMA_NUM_REPS; } } } offs += 2; if (offs == numPairs) break; curBack = matches[offs + 1]; if (curBack >= kNumFullDistances) GetPosSlot2(curBack, posSlot); } } } } } #define ChangePair(smallDist, bigDist) (((bigDist) >> 7) > (smallDist)) static UInt32 GetOptimumFast(CLzmaEnc *p, UInt32 *backRes) { UInt32 numAvail, mainLen, mainDist, numPairs, repIndex, repLen, i; const Byte *data; const UInt32 *matches; if (p->additionalOffset == 0) mainLen = ReadMatchDistances(p, &numPairs); else { mainLen = p->longestMatchLength; numPairs = p->numPairs; } numAvail = p->numAvail; *backRes = (UInt32)-1; if (numAvail < 2) return 1; if (numAvail > LZMA_MATCH_LEN_MAX) numAvail = LZMA_MATCH_LEN_MAX; data = p->matchFinder.GetPointerToCurrentPos(p->matchFinderObj) - 1; repLen = repIndex = 0; for (i = 0; i < LZMA_NUM_REPS; i++) { UInt32 len; const Byte *data2 = data - (p->reps[i] + 1); if (data[0] != data2[0] || data[1] != data2[1]) continue; for (len = 2; len < numAvail && data[len] == data2[len]; len++); if (len >= p->numFastBytes) { *backRes = i; MovePos(p, len - 1); return len; } if (len > repLen) { repIndex = i; repLen = len; } } matches = p->matches; if (mainLen >= p->numFastBytes) { *backRes = matches[numPairs - 1] + LZMA_NUM_REPS; MovePos(p, mainLen - 1); return mainLen; } mainDist = 0; /* for GCC */ if (mainLen >= 2) { mainDist = matches[numPairs - 1]; while (numPairs > 2 && mainLen == matches[numPairs - 4] + 1) { if (!ChangePair(matches[numPairs - 3], mainDist)) break; numPairs -= 2; mainLen = matches[numPairs - 2]; mainDist = matches[numPairs - 1]; } if (mainLen == 2 && mainDist >= 0x80) mainLen = 1; } if (repLen >= 2 && ( (repLen + 1 >= mainLen) || (repLen + 2 >= mainLen && mainDist >= (1 << 9)) || (repLen + 3 >= mainLen && mainDist >= (1 << 15)))) { *backRes = repIndex; MovePos(p, repLen - 1); return repLen; } if (mainLen < 2 || numAvail <= 2) return 1; p->longestMatchLength = ReadMatchDistances(p, &p->numPairs); if (p->longestMatchLength >= 2) { UInt32 newDistance = matches[p->numPairs - 1]; if ((p->longestMatchLength >= mainLen && newDistance < mainDist) || (p->longestMatchLength == mainLen + 1 && !ChangePair(mainDist, newDistance)) || (p->longestMatchLength > mainLen + 1) || (p->longestMatchLength + 1 >= mainLen && mainLen >= 3 && ChangePair(newDistance, mainDist))) return 1; } data = p->matchFinder.GetPointerToCurrentPos(p->matchFinderObj) - 1; for (i = 0; i < LZMA_NUM_REPS; i++) { UInt32 len, limit; const Byte *data2 = data - (p->reps[i] + 1); if (data[0] != data2[0] || data[1] != data2[1]) continue; limit = mainLen - 1; for (len = 2; len < limit && data[len] == data2[len]; len++); if (len >= limit) return 1; } *backRes = mainDist + LZMA_NUM_REPS; MovePos(p, mainLen - 2); return mainLen; } static void WriteEndMarker(CLzmaEnc *p, UInt32 posState) { UInt32 len; RangeEnc_EncodeBit(&p->rc, &p->isMatch[p->state][posState], 1); RangeEnc_EncodeBit(&p->rc, &p->isRep[p->state], 0); p->state = kMatchNextStates[p->state]; len = LZMA_MATCH_LEN_MIN; LenEnc_Encode2(&p->lenEnc, &p->rc, len - LZMA_MATCH_LEN_MIN, posState, !p->fastMode, p->ProbPrices); RcTree_Encode(&p->rc, p->posSlotEncoder[GetLenToPosState(len)], kNumPosSlotBits, (1 << kNumPosSlotBits) - 1); RangeEnc_EncodeDirectBits(&p->rc, (((UInt32)1 << 30) - 1) >> kNumAlignBits, 30 - kNumAlignBits); RcTree_ReverseEncode(&p->rc, p->posAlignEncoder, kNumAlignBits, kAlignMask); } static SRes CheckErrors(CLzmaEnc *p) { if (p->result != SZ_OK) return p->result; if (p->rc.res != SZ_OK) p->result = SZ_ERROR_WRITE; if (p->matchFinderBase.result != SZ_OK) p->result = SZ_ERROR_READ; if (p->result != SZ_OK) p->finished = True; return p->result; } static SRes Flush(CLzmaEnc *p, UInt32 nowPos) { /* ReleaseMFStream(); */ p->finished = True; if (p->writeEndMark) WriteEndMarker(p, nowPos & p->pbMask); RangeEnc_FlushData(&p->rc); RangeEnc_FlushStream(&p->rc); return CheckErrors(p); } static void FillAlignPrices(CLzmaEnc *p) { UInt32 i; for (i = 0; i < kAlignTableSize; i++) p->alignPrices[i] = RcTree_ReverseGetPrice(p->posAlignEncoder, kNumAlignBits, i, p->ProbPrices); p->alignPriceCount = 0; } static void FillDistancesPrices(CLzmaEnc *p) { UInt32 tempPrices[kNumFullDistances]; UInt32 i, lenToPosState; for (i = kStartPosModelIndex; i < kNumFullDistances; i++) { UInt32 posSlot = GetPosSlot1(i); UInt32 footerBits = ((posSlot >> 1) - 1); UInt32 base = ((2 | (posSlot & 1)) << footerBits); tempPrices[i] = RcTree_ReverseGetPrice(p->posEncoders + base - posSlot - 1, footerBits, i - base, p->ProbPrices); } for (lenToPosState = 0; lenToPosState < kNumLenToPosStates; lenToPosState++) { UInt32 posSlot; const CLzmaProb *encoder = p->posSlotEncoder[lenToPosState]; UInt32 *posSlotPrices = p->posSlotPrices[lenToPosState]; for (posSlot = 0; posSlot < p->distTableSize; posSlot++) posSlotPrices[posSlot] = RcTree_GetPrice(encoder, kNumPosSlotBits, posSlot, p->ProbPrices); for (posSlot = kEndPosModelIndex; posSlot < p->distTableSize; posSlot++) posSlotPrices[posSlot] += ((((posSlot >> 1) - 1) - kNumAlignBits) << kNumBitPriceShiftBits); { UInt32 *distancesPrices = p->distancesPrices[lenToPosState]; UInt32 i; for (i = 0; i < kStartPosModelIndex; i++) distancesPrices[i] = posSlotPrices[i]; for (; i < kNumFullDistances; i++) distancesPrices[i] = posSlotPrices[GetPosSlot1(i)] + tempPrices[i]; } } p->matchPriceCount = 0; } void LzmaEnc_Construct(CLzmaEnc *p) { RangeEnc_Construct(&p->rc); MatchFinder_Construct(&p->matchFinderBase); #ifdef COMPRESS_MF_MT MatchFinderMt_Construct(&p->matchFinderMt); p->matchFinderMt.MatchFinder = &p->matchFinderBase; #endif { CLzmaEncProps props; LzmaEncProps_Init(&props); LzmaEnc_SetProps(p, &props); } #ifndef LZMA_LOG_BSR LzmaEnc_FastPosInit(p->g_FastPos); #endif LzmaEnc_InitPriceTables(p->ProbPrices); p->litProbs = 0; p->saveState.litProbs = 0; } CLzmaEncHandle LzmaEnc_Create(ISzAlloc *alloc) { void *p; p = alloc->Alloc(alloc, sizeof(CLzmaEnc)); if (p != 0) LzmaEnc_Construct((CLzmaEnc *)p); return p; } void LzmaEnc_FreeLits(CLzmaEnc *p, ISzAlloc *alloc) { alloc->Free(alloc, p->litProbs); alloc->Free(alloc, p->saveState.litProbs); p->litProbs = 0; p->saveState.litProbs = 0; } void LzmaEnc_Destruct(CLzmaEnc *p, ISzAlloc *alloc, ISzAlloc *allocBig) { #ifdef COMPRESS_MF_MT MatchFinderMt_Destruct(&p->matchFinderMt, allocBig); #endif MatchFinder_Free(&p->matchFinderBase, allocBig); LzmaEnc_FreeLits(p, alloc); RangeEnc_Free(&p->rc, alloc); } void LzmaEnc_Destroy(CLzmaEncHandle p, ISzAlloc *alloc, ISzAlloc *allocBig) { LzmaEnc_Destruct((CLzmaEnc *)p, alloc, allocBig); alloc->Free(alloc, p); } static SRes LzmaEnc_CodeOneBlock(CLzmaEnc *p, Bool useLimits, UInt32 maxPackSize, UInt32 maxUnpackSize) { UInt32 nowPos32, startPos32; if (p->inStream != 0) { p->matchFinderBase.stream = p->inStream; p->matchFinder.Init(p->matchFinderObj); p->inStream = 0; } if (p->finished) return p->result; RINOK(CheckErrors(p)); nowPos32 = (UInt32)p->nowPos64; startPos32 = nowPos32; if (p->nowPos64 == 0) { UInt32 numPairs; Byte curByte; if (p->matchFinder.GetNumAvailableBytes(p->matchFinderObj) == 0) return Flush(p, nowPos32); ReadMatchDistances(p, &numPairs); RangeEnc_EncodeBit(&p->rc, &p->isMatch[p->state][0], 0); p->state = kLiteralNextStates[p->state]; curByte = p->matchFinder.GetIndexByte(p->matchFinderObj, 0 - p->additionalOffset); LitEnc_Encode(&p->rc, p->litProbs, curByte); p->additionalOffset--; nowPos32++; } if (p->matchFinder.GetNumAvailableBytes(p->matchFinderObj) != 0) for (;;) { UInt32 pos, len, posState; if (p->fastMode) len = GetOptimumFast(p, &pos); else len = GetOptimum(p, nowPos32, &pos); #ifdef SHOW_STAT2 printf("\n pos = %4X, len = %d pos = %d", nowPos32, len, pos); #endif posState = nowPos32 & p->pbMask; if (len == 1 && pos == (UInt32)-1) { Byte curByte; CLzmaProb *probs; const Byte *data; RangeEnc_EncodeBit(&p->rc, &p->isMatch[p->state][posState], 0); data = p->matchFinder.GetPointerToCurrentPos(p->matchFinderObj) - p->additionalOffset; curByte = *data; probs = LIT_PROBS(nowPos32, *(data - 1)); if (IsCharState(p->state)) LitEnc_Encode(&p->rc, probs, curByte); else LitEnc_EncodeMatched(&p->rc, probs, curByte, *(data - p->reps[0] - 1)); p->state = kLiteralNextStates[p->state]; } else { RangeEnc_EncodeBit(&p->rc, &p->isMatch[p->state][posState], 1); if (pos < LZMA_NUM_REPS) { RangeEnc_EncodeBit(&p->rc, &p->isRep[p->state], 1); if (pos == 0) { RangeEnc_EncodeBit(&p->rc, &p->isRepG0[p->state], 0); RangeEnc_EncodeBit(&p->rc, &p->isRep0Long[p->state][posState], ((len == 1) ? 0 : 1)); } else { UInt32 distance = p->reps[pos]; RangeEnc_EncodeBit(&p->rc, &p->isRepG0[p->state], 1); if (pos == 1) RangeEnc_EncodeBit(&p->rc, &p->isRepG1[p->state], 0); else { RangeEnc_EncodeBit(&p->rc, &p->isRepG1[p->state], 1); RangeEnc_EncodeBit(&p->rc, &p->isRepG2[p->state], pos - 2); if (pos == 3) p->reps[3] = p->reps[2]; p->reps[2] = p->reps[1]; } p->reps[1] = p->reps[0]; p->reps[0] = distance; } if (len == 1) p->state = kShortRepNextStates[p->state]; else { LenEnc_Encode2(&p->repLenEnc, &p->rc, len - LZMA_MATCH_LEN_MIN, posState, !p->fastMode, p->ProbPrices); p->state = kRepNextStates[p->state]; } } else { UInt32 posSlot; RangeEnc_EncodeBit(&p->rc, &p->isRep[p->state], 0); p->state = kMatchNextStates[p->state]; LenEnc_Encode2(&p->lenEnc, &p->rc, len - LZMA_MATCH_LEN_MIN, posState, !p->fastMode, p->ProbPrices); pos -= LZMA_NUM_REPS; GetPosSlot(pos, posSlot); RcTree_Encode(&p->rc, p->posSlotEncoder[GetLenToPosState(len)], kNumPosSlotBits, posSlot); if (posSlot >= kStartPosModelIndex) { UInt32 footerBits = ((posSlot >> 1) - 1); UInt32 base = ((2 | (posSlot & 1)) << footerBits); UInt32 posReduced = pos - base; if (posSlot < kEndPosModelIndex) RcTree_ReverseEncode(&p->rc, p->posEncoders + base - posSlot - 1, footerBits, posReduced); else { RangeEnc_EncodeDirectBits(&p->rc, posReduced >> kNumAlignBits, footerBits - kNumAlignBits); RcTree_ReverseEncode(&p->rc, p->posAlignEncoder, kNumAlignBits, posReduced & kAlignMask); p->alignPriceCount++; } } p->reps[3] = p->reps[2]; p->reps[2] = p->reps[1]; p->reps[1] = p->reps[0]; p->reps[0] = pos; p->matchPriceCount++; } } p->additionalOffset -= len; nowPos32 += len; if (p->additionalOffset == 0) { UInt32 processed; if (!p->fastMode) { if (p->matchPriceCount >= (1 << 7)) FillDistancesPrices(p); if (p->alignPriceCount >= kAlignTableSize) FillAlignPrices(p); } if (p->matchFinder.GetNumAvailableBytes(p->matchFinderObj) == 0) break; processed = nowPos32 - startPos32; if (useLimits) { if (processed + kNumOpts + 300 >= maxUnpackSize || RangeEnc_GetProcessed(&p->rc) + kNumOpts * 2 >= maxPackSize) break; } else if (processed >= (1 << 15)) { p->nowPos64 += nowPos32 - startPos32; return CheckErrors(p); } } } p->nowPos64 += nowPos32 - startPos32; return Flush(p, nowPos32); } #define kBigHashDicLimit ((UInt32)1 << 24) static SRes LzmaEnc_Alloc(CLzmaEnc *p, UInt32 keepWindowSize, ISzAlloc *alloc, ISzAlloc *allocBig) { UInt32 beforeSize = kNumOpts; #ifdef COMPRESS_MF_MT Bool btMode; #endif if (!RangeEnc_Alloc(&p->rc, alloc)) return SZ_ERROR_MEM; #ifdef COMPRESS_MF_MT btMode = (p->matchFinderBase.btMode != 0); p->mtMode = (p->multiThread && !p->fastMode && btMode); #endif { unsigned lclp = p->lc + p->lp; if (p->litProbs == 0 || p->saveState.litProbs == 0 || p->lclp != lclp) { LzmaEnc_FreeLits(p, alloc); p->litProbs = (CLzmaProb *)alloc->Alloc(alloc, (0x300 << lclp) * sizeof(CLzmaProb)); p->saveState.litProbs = (CLzmaProb *)alloc->Alloc(alloc, (0x300 << lclp) * sizeof(CLzmaProb)); if (p->litProbs == 0 || p->saveState.litProbs == 0) { LzmaEnc_FreeLits(p, alloc); return SZ_ERROR_MEM; } p->lclp = lclp; } } p->matchFinderBase.bigHash = (p->dictSize > kBigHashDicLimit); if (beforeSize + p->dictSize < keepWindowSize) beforeSize = keepWindowSize - p->dictSize; #ifdef COMPRESS_MF_MT if (p->mtMode) { RINOK(MatchFinderMt_Create(&p->matchFinderMt, p->dictSize, beforeSize, p->numFastBytes, LZMA_MATCH_LEN_MAX, allocBig)); p->matchFinderObj = &p->matchFinderMt; MatchFinderMt_CreateVTable(&p->matchFinderMt, &p->matchFinder); } else #endif { if (!MatchFinder_Create(&p->matchFinderBase, p->dictSize, beforeSize, p->numFastBytes, LZMA_MATCH_LEN_MAX, allocBig)) return SZ_ERROR_MEM; p->matchFinderObj = &p->matchFinderBase; MatchFinder_CreateVTable(&p->matchFinderBase, &p->matchFinder); } return SZ_OK; } void LzmaEnc_Init(CLzmaEnc *p) { UInt32 i; p->state = 0; for (i = 0 ; i < LZMA_NUM_REPS; i++) p->reps[i] = 0; RangeEnc_Init(&p->rc); for (i = 0; i < kNumStates; i++) { UInt32 j; for (j = 0; j < LZMA_NUM_PB_STATES_MAX; j++) { p->isMatch[i][j] = kProbInitValue; p->isRep0Long[i][j] = kProbInitValue; } p->isRep[i] = kProbInitValue; p->isRepG0[i] = kProbInitValue; p->isRepG1[i] = kProbInitValue; p->isRepG2[i] = kProbInitValue; } { UInt32 num = 0x300 << (p->lp + p->lc); for (i = 0; i < num; i++) p->litProbs[i] = kProbInitValue; } { for (i = 0; i < kNumLenToPosStates; i++) { CLzmaProb *probs = p->posSlotEncoder[i]; UInt32 j; for (j = 0; j < (1 << kNumPosSlotBits); j++) probs[j] = kProbInitValue; } } { for (i = 0; i < kNumFullDistances - kEndPosModelIndex; i++) p->posEncoders[i] = kProbInitValue; } LenEnc_Init(&p->lenEnc.p); LenEnc_Init(&p->repLenEnc.p); for (i = 0; i < (1 << kNumAlignBits); i++) p->posAlignEncoder[i] = kProbInitValue; p->optimumEndIndex = 0; p->optimumCurrentIndex = 0; p->additionalOffset = 0; p->pbMask = (1 << p->pb) - 1; p->lpMask = (1 << p->lp) - 1; } void LzmaEnc_InitPrices(CLzmaEnc *p) { if (!p->fastMode) { FillDistancesPrices(p); FillAlignPrices(p); } p->lenEnc.tableSize = p->repLenEnc.tableSize = p->numFastBytes + 1 - LZMA_MATCH_LEN_MIN; LenPriceEnc_UpdateTables(&p->lenEnc, 1 << p->pb, p->ProbPrices); LenPriceEnc_UpdateTables(&p->repLenEnc, 1 << p->pb, p->ProbPrices); } static SRes LzmaEnc_AllocAndInit(CLzmaEnc *p, UInt32 keepWindowSize, ISzAlloc *alloc, ISzAlloc *allocBig) { UInt32 i; for (i = 0; i < (UInt32)kDicLogSizeMaxCompress; i++) if (p->dictSize <= ((UInt32)1 << i)) break; p->distTableSize = i * 2; p->finished = False; p->result = SZ_OK; RINOK(LzmaEnc_Alloc(p, keepWindowSize, alloc, allocBig)); LzmaEnc_Init(p); LzmaEnc_InitPrices(p); p->nowPos64 = 0; return SZ_OK; } static SRes LzmaEnc_Prepare(CLzmaEncHandle pp, ISeqInStream *inStream, ISeqOutStream *outStream, ISzAlloc *alloc, ISzAlloc *allocBig) { CLzmaEnc *p = (CLzmaEnc *)pp; p->inStream = inStream; p->rc.outStream = outStream; return LzmaEnc_AllocAndInit(p, 0, alloc, allocBig); } SRes LzmaEnc_PrepareForLzma2(CLzmaEncHandle pp, ISeqInStream *inStream, UInt32 keepWindowSize, ISzAlloc *alloc, ISzAlloc *allocBig) { CLzmaEnc *p = (CLzmaEnc *)pp; p->inStream = inStream; return LzmaEnc_AllocAndInit(p, keepWindowSize, alloc, allocBig); } static void LzmaEnc_SetInputBuf(CLzmaEnc *p, const Byte *src, SizeT srcLen) { p->seqBufInStream.funcTable.Read = MyRead; p->seqBufInStream.data = src; p->seqBufInStream.rem = srcLen; } SRes LzmaEnc_MemPrepare(CLzmaEncHandle pp, const Byte *src, SizeT srcLen, UInt32 keepWindowSize, ISzAlloc *alloc, ISzAlloc *allocBig) { CLzmaEnc *p = (CLzmaEnc *)pp; LzmaEnc_SetInputBuf(p, src, srcLen); p->inStream = &p->seqBufInStream.funcTable; return LzmaEnc_AllocAndInit(p, keepWindowSize, alloc, allocBig); } void LzmaEnc_Finish(CLzmaEncHandle pp) { #ifdef COMPRESS_MF_MT CLzmaEnc *p = (CLzmaEnc *)pp; if (p->mtMode) MatchFinderMt_ReleaseStream(&p->matchFinderMt); #else pp = pp; #endif } typedef struct _CSeqOutStreamBuf { ISeqOutStream funcTable; Byte *data; SizeT rem; Bool overflow; } CSeqOutStreamBuf; static size_t MyWrite(void *pp, const void *data, size_t size) { CSeqOutStreamBuf *p = (CSeqOutStreamBuf *)pp; if (p->rem < size) { size = p->rem; p->overflow = True; } memcpy(p->data, data, size); p->rem -= size; p->data += size; return size; } UInt32 LzmaEnc_GetNumAvailableBytes(CLzmaEncHandle pp) { const CLzmaEnc *p = (CLzmaEnc *)pp; return p->matchFinder.GetNumAvailableBytes(p->matchFinderObj); } const Byte *LzmaEnc_GetCurBuf(CLzmaEncHandle pp) { const CLzmaEnc *p = (CLzmaEnc *)pp; return p->matchFinder.GetPointerToCurrentPos(p->matchFinderObj) - p->additionalOffset; } SRes LzmaEnc_CodeOneMemBlock(CLzmaEncHandle pp, Bool reInit, Byte *dest, size_t *destLen, UInt32 desiredPackSize, UInt32 *unpackSize) { CLzmaEnc *p = (CLzmaEnc *)pp; UInt64 nowPos64; SRes res; CSeqOutStreamBuf outStream; outStream.funcTable.Write = MyWrite; outStream.data = dest; outStream.rem = *destLen; outStream.overflow = False; p->writeEndMark = False; p->finished = False; p->result = SZ_OK; if (reInit) LzmaEnc_Init(p); LzmaEnc_InitPrices(p); nowPos64 = p->nowPos64; RangeEnc_Init(&p->rc); p->rc.outStream = &outStream.funcTable; res = LzmaEnc_CodeOneBlock(p, True, desiredPackSize, *unpackSize); *unpackSize = (UInt32)(p->nowPos64 - nowPos64); *destLen -= outStream.rem; if (outStream.overflow) return SZ_ERROR_OUTPUT_EOF; return res; } SRes LzmaEnc_Encode(CLzmaEncHandle pp, ISeqOutStream *outStream, ISeqInStream *inStream, ICompressProgress *progress, ISzAlloc *alloc, ISzAlloc *allocBig) { CLzmaEnc *p = (CLzmaEnc *)pp; SRes res = SZ_OK; #ifdef COMPRESS_MF_MT Byte allocaDummy[0x300]; int i = 0; for (i = 0; i < 16; i++) allocaDummy[i] = (Byte)i; #endif RINOK(LzmaEnc_Prepare(pp, inStream, outStream, alloc, allocBig)); for (;;) { res = LzmaEnc_CodeOneBlock(p, False, 0, 0); if (res != SZ_OK || p->finished != 0) break; if (progress != 0) { res = progress->Progress(progress, p->nowPos64, RangeEnc_GetProcessed(&p->rc)); if (res != SZ_OK) { res = SZ_ERROR_PROGRESS; break; } } } LzmaEnc_Finish(pp); return res; } SRes LzmaEnc_WriteProperties(CLzmaEncHandle pp, Byte *props, SizeT *size) { CLzmaEnc *p = (CLzmaEnc *)pp; int i; UInt32 dictSize = p->dictSize; if (*size < LZMA_PROPS_SIZE) return SZ_ERROR_PARAM; *size = LZMA_PROPS_SIZE; props[0] = (Byte)((p->pb * 5 + p->lp) * 9 + p->lc); for (i = 11; i <= 30; i++) { if (dictSize <= ((UInt32)2 << i)) { dictSize = (2 << i); break; } if (dictSize <= ((UInt32)3 << i)) { dictSize = (3 << i); break; } } for (i = 0; i < 4; i++) props[1 + i] = (Byte)(dictSize >> (8 * i)); return SZ_OK; } SRes LzmaEnc_MemEncode(CLzmaEncHandle pp, Byte *dest, SizeT *destLen, const Byte *src, SizeT srcLen, int writeEndMark, ICompressProgress *progress, ISzAlloc *alloc, ISzAlloc *allocBig) { SRes res; CLzmaEnc *p = (CLzmaEnc *)pp; CSeqOutStreamBuf outStream; LzmaEnc_SetInputBuf(p, src, srcLen); outStream.funcTable.Write = MyWrite; outStream.data = dest; outStream.rem = *destLen; outStream.overflow = False; p->writeEndMark = writeEndMark; res = LzmaEnc_Encode(pp, &outStream.funcTable, &p->seqBufInStream.funcTable, progress, alloc, allocBig); *destLen -= outStream.rem; if (outStream.overflow) return SZ_ERROR_OUTPUT_EOF; return res; } SRes LzmaEncode(Byte *dest, SizeT *destLen, const Byte *src, SizeT srcLen, const CLzmaEncProps *props, Byte *propsEncoded, SizeT *propsSize, int writeEndMark, ICompressProgress *progress, ISzAlloc *alloc, ISzAlloc *allocBig) { CLzmaEnc *p = (CLzmaEnc *)LzmaEnc_Create(alloc); SRes res; if (p == 0) return SZ_ERROR_MEM; res = LzmaEnc_SetProps(p, props); if (res == SZ_OK) { res = LzmaEnc_WriteProperties(p, propsEncoded, propsSize); if (res == SZ_OK) res = LzmaEnc_MemEncode(p, dest, destLen, src, srcLen, writeEndMark, progress, alloc, allocBig); } LzmaEnc_Destroy(p, alloc, allocBig); return res; } zmat-0.9.9/src/easylzma/pavlov/Bcj2.c0000755000175200007730000000614214515254731017614 0ustar rlaboissrlaboiss/* Bcj2.c -- Converter for x86 code (BCJ2) 2008-10-04 : Igor Pavlov : Public domain */ #include "Bcj2.h" #ifdef _LZMA_PROB32 #define CProb UInt32 #else #define CProb UInt16 #endif #define IsJcc(b0, b1) ((b0) == 0x0F && ((b1) & 0xF0) == 0x80) #define IsJ(b0, b1) ((b1 & 0xFE) == 0xE8 || IsJcc(b0, b1)) #define kNumTopBits 24 #define kTopValue ((UInt32)1 << kNumTopBits) #define kNumBitModelTotalBits 11 #define kBitModelTotal (1 << kNumBitModelTotalBits) #define kNumMoveBits 5 #define RC_READ_BYTE (*buffer++) #define RC_TEST { if (buffer == bufferLim) return SZ_ERROR_DATA; } #define RC_INIT2 code = 0; range = 0xFFFFFFFF; \ { int i; for (i = 0; i < 5; i++) { RC_TEST; code = (code << 8) | RC_READ_BYTE; }} #define NORMALIZE if (range < kTopValue) { RC_TEST; range <<= 8; code = (code << 8) | RC_READ_BYTE; } #define IF_BIT_0(p) ttt = *(p); bound = (range >> kNumBitModelTotalBits) * ttt; if (code < bound) #define UPDATE_0(p) range = bound; *(p) = (CProb)(ttt + ((kBitModelTotal - ttt) >> kNumMoveBits)); NORMALIZE; #define UPDATE_1(p) range -= bound; code -= bound; *(p) = (CProb)(ttt - (ttt >> kNumMoveBits)); NORMALIZE; int Bcj2_Decode( const Byte *buf0, SizeT size0, const Byte *buf1, SizeT size1, const Byte *buf2, SizeT size2, const Byte *buf3, SizeT size3, Byte *outBuf, SizeT outSize) { CProb p[256 + 2]; SizeT inPos = 0, outPos = 0; const Byte *buffer, *bufferLim; UInt32 range, code; Byte prevByte = 0; unsigned int i; for (i = 0; i < sizeof(p) / sizeof(p[0]); i++) p[i] = kBitModelTotal >> 1; buffer = buf3; bufferLim = buffer + size3; RC_INIT2 if (outSize == 0) return SZ_OK; for (;;) { Byte b; CProb *prob; UInt32 bound; UInt32 ttt; SizeT limit = size0 - inPos; if (outSize - outPos < limit) limit = outSize - outPos; while (limit != 0) { Byte b = buf0[inPos]; outBuf[outPos++] = b; if (IsJ(prevByte, b)) break; inPos++; prevByte = b; limit--; } if (limit == 0 || outPos == outSize) break; b = buf0[inPos++]; if (b == 0xE8) prob = p + prevByte; else if (b == 0xE9) prob = p + 256; else prob = p + 257; IF_BIT_0(prob) { UPDATE_0(prob) prevByte = b; } else { UInt32 dest; const Byte *v; UPDATE_1(prob) if (b == 0xE8) { v = buf1; if (size1 < 4) return SZ_ERROR_DATA; buf1 += 4; size1 -= 4; } else { v = buf2; if (size2 < 4) return SZ_ERROR_DATA; buf2 += 4; size2 -= 4; } dest = (((UInt32)v[0] << 24) | ((UInt32)v[1] << 16) | ((UInt32)v[2] << 8) | ((UInt32)v[3])) - ((UInt32)outPos + 4); outBuf[outPos++] = (Byte)dest; if (outPos == outSize) break; outBuf[outPos++] = (Byte)(dest >> 8); if (outPos == outSize) break; outBuf[outPos++] = (Byte)(dest >> 16); if (outPos == outSize) break; outBuf[outPos++] = prevByte = (Byte)(dest >> 24); } } return (outPos == outSize) ? SZ_OK : SZ_ERROR_DATA; } zmat-0.9.9/src/easylzma/pavlov/Types.h0000755000175200007730000001107114515254731020142 0ustar rlaboissrlaboiss/* Types.h -- Basic types 2008-11-23 : Igor Pavlov : Public domain */ #ifndef __7Z_TYPES_H #define __7Z_TYPES_H #include #ifdef _WIN32 #include #endif #define SZ_OK 0 #define SZ_ERROR_DATA 1 #define SZ_ERROR_MEM 2 #define SZ_ERROR_CRC 3 #define SZ_ERROR_UNSUPPORTED 4 #define SZ_ERROR_PARAM 5 #define SZ_ERROR_INPUT_EOF 6 #define SZ_ERROR_OUTPUT_EOF 7 #define SZ_ERROR_READ 8 #define SZ_ERROR_WRITE 9 #define SZ_ERROR_PROGRESS 10 #define SZ_ERROR_FAIL 11 #define SZ_ERROR_THREAD 12 #define SZ_ERROR_ARCHIVE 16 #define SZ_ERROR_NO_ARCHIVE 17 typedef int SRes; #ifdef _WIN32 typedef DWORD WRes; #else typedef int WRes; #endif #ifndef RINOK #define RINOK(x) { int __result__ = (x); if (__result__ != 0) return __result__; } #endif typedef unsigned char Byte; typedef short Int16; typedef unsigned short UInt16; #ifdef _LZMA_UINT32_IS_ULONG typedef long Int32; typedef unsigned long UInt32; #else typedef int Int32; typedef unsigned int UInt32; #endif #ifdef _SZ_NO_INT_64 /* define _SZ_NO_INT_64, if your compiler doesn't support 64-bit integers. NOTES: Some code will work incorrectly in that case! */ typedef long Int64; typedef unsigned long UInt64; #else #if defined(_MSC_VER) || defined(__BORLANDC__) typedef __int64 Int64; typedef unsigned __int64 UInt64; #else typedef long long int Int64; typedef unsigned long long int UInt64; #endif #endif #ifdef _LZMA_NO_SYSTEM_SIZE_T typedef UInt32 SizeT; #else typedef size_t SizeT; #endif typedef int Bool; #define True 1 #define False 0 #ifdef _MSC_VER #if _MSC_VER >= 1300 #define MY_NO_INLINE __declspec(noinline) #else #define MY_NO_INLINE #endif #define MY_CDECL __cdecl #define MY_STD_CALL __stdcall #define MY_FAST_CALL MY_NO_INLINE __fastcall #else #define MY_CDECL #define MY_STD_CALL #define MY_FAST_CALL #endif /* The following interfaces use first parameter as pointer to structure */ typedef struct { SRes (*Read)(void *p, void *buf, size_t *size); /* if (input(*size) != 0 && output(*size) == 0) means end_of_stream. (output(*size) < input(*size)) is allowed */ } ISeqInStream; /* it can return SZ_ERROR_INPUT_EOF */ SRes SeqInStream_Read(ISeqInStream *stream, void *buf, size_t size); SRes SeqInStream_Read2(ISeqInStream *stream, void *buf, size_t size, SRes errorType); SRes SeqInStream_ReadByte(ISeqInStream *stream, Byte *buf); typedef struct { size_t (*Write)(void *p, const void *buf, size_t size); /* Returns: result - the number of actually written bytes. (result < size) means error */ } ISeqOutStream; typedef enum { SZ_SEEK_SET = 0, SZ_SEEK_CUR = 1, SZ_SEEK_END = 2 } ESzSeek; typedef struct { SRes (*Read)(void *p, void *buf, size_t *size); /* same as ISeqInStream::Read */ SRes (*Seek)(void *p, Int64 *pos, ESzSeek origin); } ISeekInStream; typedef struct { SRes (*Look)(void *p, void **buf, size_t *size); /* if (input(*size) != 0 && output(*size) == 0) means end_of_stream. (output(*size) > input(*size)) is not allowed (output(*size) < input(*size)) is allowed */ SRes (*Skip)(void *p, size_t offset); /* offset must be <= output(*size) of Look */ SRes (*Read)(void *p, void *buf, size_t *size); /* reads directly (without buffer). It's same as ISeqInStream::Read */ SRes (*Seek)(void *p, Int64 *pos, ESzSeek origin); } ILookInStream; SRes LookInStream_LookRead(ILookInStream *stream, void *buf, size_t *size); SRes LookInStream_SeekTo(ILookInStream *stream, UInt64 offset); /* reads via ILookInStream::Read */ SRes LookInStream_Read2(ILookInStream *stream, void *buf, size_t size, SRes errorType); SRes LookInStream_Read(ILookInStream *stream, void *buf, size_t size); #define LookToRead_BUF_SIZE (1 << 14) typedef struct { ILookInStream s; ISeekInStream *realStream; size_t pos; size_t size; Byte buf[LookToRead_BUF_SIZE]; } CLookToRead; void LookToRead_CreateVTable(CLookToRead *p, int lookahead); void LookToRead_Init(CLookToRead *p); typedef struct { ISeqInStream s; ILookInStream *realStream; } CSecToLook; void SecToLook_CreateVTable(CSecToLook *p); typedef struct { ISeqInStream s; ILookInStream *realStream; } CSecToRead; void SecToRead_CreateVTable(CSecToRead *p); typedef struct { SRes (*Progress)(void *p, UInt64 inSize, UInt64 outSize); /* Returns: result. (result != SZ_OK) means break. Value (UInt64)(Int64)-1 for size means unknown value. */ } ICompressProgress; typedef struct { void *(*Alloc)(void *p, size_t size); void (*Free)(void *p, void *address); /* address can be 0 */ } ISzAlloc; #define IAlloc_Alloc(p, size) (p)->Alloc((p), size) #define IAlloc_Free(p, a) (p)->Free((p), a) #endif zmat-0.9.9/src/easylzma/pavlov/Bcj2.h0000755000175200007730000000117614515254731017623 0ustar rlaboissrlaboiss/* Bcj2.h -- Converter for x86 code (BCJ2) 2008-10-04 : Igor Pavlov : Public domain */ #ifndef __BCJ2_H #define __BCJ2_H #include "Types.h" /* Conditions: outSize <= FullOutputSize, where FullOutputSize is full size of output stream of x86_2 filter. If buf0 overlaps outBuf, there are two required conditions: 1) (buf0 >= outBuf) 2) (buf0 + size0 >= outBuf + FullOutputSize). Returns: SZ_OK SZ_ERROR_DATA - Data error */ int Bcj2_Decode( const Byte *buf0, SizeT size0, const Byte *buf1, SizeT size1, const Byte *buf2, SizeT size2, const Byte *buf3, SizeT size3, Byte *outBuf, SizeT outSize); #endif zmat-0.9.9/src/easylzma/pavlov/LzmaDec.h0000755000175200007730000001521214515254731020356 0ustar rlaboissrlaboiss/* LzmaDec.h -- LZMA Decoder 2008-10-04 : Igor Pavlov : Public domain */ #ifndef __LZMADEC_H #define __LZMADEC_H #include "Types.h" /* #define _LZMA_PROB32 */ /* _LZMA_PROB32 can increase the speed on some CPUs, but memory usage for CLzmaDec::probs will be doubled in that case */ #ifdef _LZMA_PROB32 #define CLzmaProb UInt32 #else #define CLzmaProb UInt16 #endif /* ---------- LZMA Properties ---------- */ #define LZMA_PROPS_SIZE 5 typedef struct _CLzmaProps { unsigned lc, lp, pb; UInt32 dicSize; } CLzmaProps; /* LzmaProps_Decode - decodes properties Returns: SZ_OK SZ_ERROR_UNSUPPORTED - Unsupported properties */ SRes LzmaProps_Decode(CLzmaProps *p, const Byte *data, unsigned size); /* ---------- LZMA Decoder state ---------- */ /* LZMA_REQUIRED_INPUT_MAX = number of required input bytes for worst case. Num bits = log2((2^11 / 31) ^ 22) + 26 < 134 + 26 = 160; */ #define LZMA_REQUIRED_INPUT_MAX 20 typedef struct { CLzmaProps prop; CLzmaProb *probs; Byte *dic; const Byte *buf; UInt32 range, code; SizeT dicPos; SizeT dicBufSize; UInt32 processedPos; UInt32 checkDicSize; unsigned state; UInt32 reps[4]; unsigned remainLen; int needFlush; int needInitState; UInt32 numProbs; unsigned tempBufSize; Byte tempBuf[LZMA_REQUIRED_INPUT_MAX]; } CLzmaDec; #define LzmaDec_Construct(p) { (p)->dic = 0; (p)->probs = 0; } void LzmaDec_Init(CLzmaDec *p); /* There are two types of LZMA streams: 0) Stream with end mark. That end mark adds about 6 bytes to compressed size. 1) Stream without end mark. You must know exact uncompressed size to decompress such stream. */ typedef enum { LZMA_FINISH_ANY, /* finish at any point */ LZMA_FINISH_END /* block must be finished at the end */ } ELzmaFinishMode; /* ELzmaFinishMode has meaning only if the decoding reaches output limit !!! You must use LZMA_FINISH_END, when you know that current output buffer covers last bytes of block. In other cases you must use LZMA_FINISH_ANY. If LZMA decoder sees end marker before reaching output limit, it returns SZ_OK, and output value of destLen will be less than output buffer size limit. You can check status result also. You can use multiple checks to test data integrity after full decompression: 1) Check Result and "status" variable. 2) Check that output(destLen) = uncompressedSize, if you know real uncompressedSize. 3) Check that output(srcLen) = compressedSize, if you know real compressedSize. You must use correct finish mode in that case. */ typedef enum { LZMA_STATUS_NOT_SPECIFIED, /* use main error code instead */ LZMA_STATUS_FINISHED_WITH_MARK, /* stream was finished with end mark. */ LZMA_STATUS_NOT_FINISHED, /* stream was not finished */ LZMA_STATUS_NEEDS_MORE_INPUT, /* you must provide more input bytes */ LZMA_STATUS_MAYBE_FINISHED_WITHOUT_MARK /* there is probability that stream was finished without end mark */ } ELzmaStatus; /* ELzmaStatus is used only as output value for function call */ /* ---------- Interfaces ---------- */ /* There are 3 levels of interfaces: 1) Dictionary Interface 2) Buffer Interface 3) One Call Interface You can select any of these interfaces, but don't mix functions from different groups for same object. */ /* There are two variants to allocate state for Dictionary Interface: 1) LzmaDec_Allocate / LzmaDec_Free 2) LzmaDec_AllocateProbs / LzmaDec_FreeProbs You can use variant 2, if you set dictionary buffer manually. For Buffer Interface you must always use variant 1. LzmaDec_Allocate* can return: SZ_OK SZ_ERROR_MEM - Memory allocation error SZ_ERROR_UNSUPPORTED - Unsupported properties */ SRes LzmaDec_AllocateProbs(CLzmaDec *p, const Byte *props, unsigned propsSize, ISzAlloc *alloc); void LzmaDec_FreeProbs(CLzmaDec *p, ISzAlloc *alloc); SRes LzmaDec_Allocate(CLzmaDec *state, const Byte *prop, unsigned propsSize, ISzAlloc *alloc); void LzmaDec_Free(CLzmaDec *state, ISzAlloc *alloc); /* ---------- Dictionary Interface ---------- */ /* You can use it, if you want to eliminate the overhead for data copying from dictionary to some other external buffer. You must work with CLzmaDec variables directly in this interface. STEPS: LzmaDec_Constr() LzmaDec_Allocate() for (each new stream) { LzmaDec_Init() while (it needs more decompression) { LzmaDec_DecodeToDic() use data from CLzmaDec::dic and update CLzmaDec::dicPos } } LzmaDec_Free() */ /* LzmaDec_DecodeToDic The decoding to internal dictionary buffer (CLzmaDec::dic). You must manually update CLzmaDec::dicPos, if it reaches CLzmaDec::dicBufSize !!! finishMode: It has meaning only if the decoding reaches output limit (dicLimit). LZMA_FINISH_ANY - Decode just dicLimit bytes. LZMA_FINISH_END - Stream must be finished after dicLimit. Returns: SZ_OK status: LZMA_STATUS_FINISHED_WITH_MARK LZMA_STATUS_NOT_FINISHED LZMA_STATUS_NEEDS_MORE_INPUT LZMA_STATUS_MAYBE_FINISHED_WITHOUT_MARK SZ_ERROR_DATA - Data error */ SRes LzmaDec_DecodeToDic(CLzmaDec *p, SizeT dicLimit, const Byte *src, SizeT *srcLen, ELzmaFinishMode finishMode, ELzmaStatus *status); /* ---------- Buffer Interface ---------- */ /* It's zlib-like interface. See LzmaDec_DecodeToDic description for information about STEPS and return results, but you must use LzmaDec_DecodeToBuf instead of LzmaDec_DecodeToDic and you don't need to work with CLzmaDec variables manually. finishMode: It has meaning only if the decoding reaches output limit (*destLen). LZMA_FINISH_ANY - Decode just destLen bytes. LZMA_FINISH_END - Stream must be finished after (*destLen). */ SRes LzmaDec_DecodeToBuf(CLzmaDec *p, Byte *dest, SizeT *destLen, const Byte *src, SizeT *srcLen, ELzmaFinishMode finishMode, ELzmaStatus *status); /* ---------- One Call Interface ---------- */ /* LzmaDecode finishMode: It has meaning only if the decoding reaches output limit (*destLen). LZMA_FINISH_ANY - Decode just destLen bytes. LZMA_FINISH_END - Stream must be finished after (*destLen). Returns: SZ_OK status: LZMA_STATUS_FINISHED_WITH_MARK LZMA_STATUS_NOT_FINISHED LZMA_STATUS_MAYBE_FINISHED_WITHOUT_MARK SZ_ERROR_DATA - Data error SZ_ERROR_MEM - Memory allocation error SZ_ERROR_UNSUPPORTED - Unsupported properties SZ_ERROR_INPUT_EOF - It needs more bytes in input buffer (src). */ SRes LzmaDecode(Byte *dest, SizeT *destLen, const Byte *src, SizeT *srcLen, const Byte *propData, unsigned propSize, ELzmaFinishMode finishMode, ELzmaStatus *status, ISzAlloc *alloc); #endif zmat-0.9.9/src/easylzma/pavlov/7zVersion.h0000755000175200007730000000037614515254731020752 0ustar rlaboissrlaboiss#define MY_VER_MAJOR 4 #define MY_VER_MINOR 63 #define MY_VER_BUILD 0 #define MY_VERSION "4.63" #define MY_DATE "2008-12-31" #define MY_COPYRIGHT ": Igor Pavlov : Public domain" #define MY_VERSION_COPYRIGHT_DATE MY_VERSION " " MY_COPYRIGHT " : " MY_DATE zmat-0.9.9/src/easylzma/pavlov/7zStream.c0000755000175200007730000000760714515254731020557 0ustar rlaboissrlaboiss/* 7zStream.c -- 7z Stream functions 2008-11-23 : Igor Pavlov : Public domain */ #include #include "Types.h" SRes SeqInStream_Read2(ISeqInStream *stream, void *buf, size_t size, SRes errorType) { while (size != 0) { size_t processed = size; RINOK(stream->Read(stream, buf, &processed)); if (processed == 0) return errorType; buf = (void *)((Byte *)buf + processed); size -= processed; } return SZ_OK; } SRes SeqInStream_Read(ISeqInStream *stream, void *buf, size_t size) { return SeqInStream_Read2(stream, buf, size, SZ_ERROR_INPUT_EOF); } SRes SeqInStream_ReadByte(ISeqInStream *stream, Byte *buf) { size_t processed = 1; RINOK(stream->Read(stream, buf, &processed)); return (processed == 1) ? SZ_OK : SZ_ERROR_INPUT_EOF; } SRes LookInStream_SeekTo(ILookInStream *stream, UInt64 offset) { Int64 t = offset; return stream->Seek(stream, &t, SZ_SEEK_SET); } SRes LookInStream_LookRead(ILookInStream *stream, void *buf, size_t *size) { void *lookBuf; if (*size == 0) return SZ_OK; RINOK(stream->Look(stream, &lookBuf, size)); memcpy(buf, lookBuf, *size); return stream->Skip(stream, *size); } SRes LookInStream_Read2(ILookInStream *stream, void *buf, size_t size, SRes errorType) { while (size != 0) { size_t processed = size; RINOK(stream->Read(stream, buf, &processed)); if (processed == 0) return errorType; buf = (void *)((Byte *)buf + processed); size -= processed; } return SZ_OK; } SRes LookInStream_Read(ILookInStream *stream, void *buf, size_t size) { return LookInStream_Read2(stream, buf, size, SZ_ERROR_INPUT_EOF); } static SRes LookToRead_Look_Lookahead(void *pp, void **buf, size_t *size) { SRes res = SZ_OK; CLookToRead *p = (CLookToRead *)pp; size_t size2 = p->size - p->pos; if (size2 == 0 && *size > 0) { p->pos = 0; size2 = LookToRead_BUF_SIZE; res = p->realStream->Read(p->realStream, p->buf, &size2); p->size = size2; } if (size2 < *size) *size = size2; *buf = p->buf + p->pos; return res; } static SRes LookToRead_Look_Exact(void *pp, void **buf, size_t *size) { SRes res = SZ_OK; CLookToRead *p = (CLookToRead *)pp; size_t size2 = p->size - p->pos; if (size2 == 0 && *size > 0) { p->pos = 0; if (*size > LookToRead_BUF_SIZE) *size = LookToRead_BUF_SIZE; res = p->realStream->Read(p->realStream, p->buf, size); size2 = p->size = *size; } if (size2 < *size) *size = size2; *buf = p->buf + p->pos; return res; } static SRes LookToRead_Skip(void *pp, size_t offset) { CLookToRead *p = (CLookToRead *)pp; p->pos += offset; return SZ_OK; } static SRes LookToRead_Read(void *pp, void *buf, size_t *size) { CLookToRead *p = (CLookToRead *)pp; size_t rem = p->size - p->pos; if (rem == 0) return p->realStream->Read(p->realStream, buf, size); if (rem > *size) rem = *size; memcpy(buf, p->buf + p->pos, rem); p->pos += rem; *size = rem; return SZ_OK; } static SRes LookToRead_Seek(void *pp, Int64 *pos, ESzSeek origin) { CLookToRead *p = (CLookToRead *)pp; p->pos = p->size = 0; return p->realStream->Seek(p->realStream, pos, origin); } void LookToRead_CreateVTable(CLookToRead *p, int lookahead) { p->s.Look = lookahead ? LookToRead_Look_Lookahead : LookToRead_Look_Exact; p->s.Skip = LookToRead_Skip; p->s.Read = LookToRead_Read; p->s.Seek = LookToRead_Seek; } void LookToRead_Init(CLookToRead *p) { p->pos = p->size = 0; } static SRes SecToLook_Read(void *pp, void *buf, size_t *size) { CSecToLook *p = (CSecToLook *)pp; return LookInStream_LookRead(p->realStream, buf, size); } void SecToLook_CreateVTable(CSecToLook *p) { p->s.Read = SecToLook_Read; } static SRes SecToRead_Read(void *pp, void *buf, size_t *size) { CSecToRead *p = (CSecToRead *)pp; return p->realStream->Read(p->realStream, buf, size); } void SecToRead_CreateVTable(CSecToRead *p) { p->s.Read = SecToRead_Read; } zmat-0.9.9/src/easylzma/pavlov/Alloc.h0000755000175200007730000000104514515254731020070 0ustar rlaboissrlaboiss/* Alloc.h -- Memory allocation functions 2008-03-13 Igor Pavlov Public domain */ #ifndef __COMMON_ALLOC_H #define __COMMON_ALLOC_H #include void *MyAlloc(size_t size); void MyFree(void *address); #ifdef _WIN32 void SetLargePageSize(); void *MidAlloc(size_t size); void MidFree(void *address); void *BigAlloc(size_t size); void BigFree(void *address); #else #define MidAlloc(size) MyAlloc(size) #define MidFree(address) MyFree(address) #define BigAlloc(size) MyAlloc(size) #define BigFree(address) MyFree(address) #endif #endif zmat-0.9.9/src/easylzma/pavlov/LzmaLib.h0000755000175200007730000001043014515254731020366 0ustar rlaboissrlaboiss/* LzmaLib.h -- LZMA library interface 2008-08-05 Igor Pavlov Public domain */ #ifndef __LZMALIB_H #define __LZMALIB_H #include "Types.h" #ifdef __cplusplus #define MY_EXTERN_C extern "C" #else #define MY_EXTERN_C extern #endif #define MY_STDAPI MY_EXTERN_C int MY_STD_CALL #define LZMA_PROPS_SIZE 5 /* RAM requirements for LZMA: for compression: (dictSize * 11.5 + 6 MB) + state_size for decompression: dictSize + state_size state_size = (4 + (1.5 << (lc + lp))) KB by default (lc=3, lp=0), state_size = 16 KB. LZMA properties (5 bytes) format Offset Size Description 0 1 lc, lp and pb in encoded form. 1 4 dictSize (little endian). */ /* LzmaCompress ------------ outPropsSize - In: the pointer to the size of outProps buffer; *outPropsSize = LZMA_PROPS_SIZE = 5. Out: the pointer to the size of written properties in outProps buffer; *outPropsSize = LZMA_PROPS_SIZE = 5. LZMA Encoder will use defult values for any parameter, if it is -1 for any from: level, loc, lp, pb, fb, numThreads 0 for dictSize level - compression level: 0 <= level <= 9; level dictSize algo fb 0: 16 KB 0 32 1: 64 KB 0 32 2: 256 KB 0 32 3: 1 MB 0 32 4: 4 MB 0 32 5: 16 MB 1 32 6: 32 MB 1 32 7+: 64 MB 1 64 The default value for "level" is 5. algo = 0 means fast method algo = 1 means normal method dictSize - The dictionary size in bytes. The maximum value is 128 MB = (1 << 27) bytes for 32-bit version 1 GB = (1 << 30) bytes for 64-bit version The default value is 16 MB = (1 << 24) bytes. It's recommended to use the dictionary that is larger than 4 KB and that can be calculated as (1 << N) or (3 << N) sizes. lc - The number of literal context bits (high bits of previous literal). It can be in the range from 0 to 8. The default value is 3. Sometimes lc=4 gives the gain for big files. lp - The number of literal pos bits (low bits of current position for literals). It can be in the range from 0 to 4. The default value is 0. The lp switch is intended for periodical data when the period is equal to 2^lp. For example, for 32-bit (4 bytes) periodical data you can use lp=2. Often it's better to set lc=0, if you change lp switch. pb - The number of pos bits (low bits of current position). It can be in the range from 0 to 4. The default value is 2. The pb switch is intended for periodical data when the period is equal 2^pb. fb - Word size (the number of fast bytes). It can be in the range from 5 to 273. The default value is 32. Usually, a big number gives a little bit better compression ratio and slower compression process. numThreads - The number of thereads. 1 or 2. The default value is 2. Fast mode (algo = 0) can use only 1 thread. Out: destLen - processed output size Returns: SZ_OK - OK SZ_ERROR_MEM - Memory allocation error SZ_ERROR_PARAM - Incorrect paramater SZ_ERROR_OUTPUT_EOF - output buffer overflow SZ_ERROR_THREAD - errors in multithreading functions (only for Mt version) */ MY_STDAPI LzmaCompress(unsigned char *dest, size_t *destLen, const unsigned char *src, size_t srcLen, unsigned char *outProps, size_t *outPropsSize, /* *outPropsSize must be = 5 */ int level, /* 0 <= level <= 9, default = 5 */ unsigned dictSize, /* default = (1 << 24) */ int lc, /* 0 <= lc <= 8, default = 3 */ int lp, /* 0 <= lp <= 4, default = 0 */ int pb, /* 0 <= pb <= 4, default = 2 */ int fb, /* 5 <= fb <= 273, default = 32 */ int numThreads /* 1 or 2, default = 2 */ ); /* LzmaUncompress -------------- In: dest - output data destLen - output data size src - input data srcLen - input data size Out: destLen - processed output size srcLen - processed input size Returns: SZ_OK - OK SZ_ERROR_DATA - Data error SZ_ERROR_MEM - Memory allocation arror SZ_ERROR_UNSUPPORTED - Unsupported properties SZ_ERROR_INPUT_EOF - it needs more bytes in input buffer (src) */ MY_STDAPI LzmaUncompress(unsigned char *dest, size_t *destLen, const unsigned char *src, SizeT *srcLen, const unsigned char *props, size_t propsSize); #endif zmat-0.9.9/src/easylzma/pavlov/7zBuf.c0000755000175200007730000000076514515254731020036 0ustar rlaboissrlaboiss/* 7zBuf.c -- Byte Buffer 2008-03-28 Igor Pavlov Public domain */ #include "7zBuf.h" void Buf_Init(CBuf *p) { p->data = 0; p->size = 0; } int Buf_Create(CBuf *p, size_t size, ISzAlloc *alloc) { p->size = 0; if (size == 0) { p->data = 0; return 1; } p->data = (Byte *)alloc->Alloc(alloc, size); if (p->data != 0) { p->size = size; return 1; } return 0; } void Buf_Free(CBuf *p, ISzAlloc *alloc) { alloc->Free(alloc, p->data); p->data = 0; p->size = 0; } zmat-0.9.9/src/easylzma/pavlov/Alloc.c0000755000175200007730000000517414515254731020072 0ustar rlaboissrlaboiss/* Alloc.c -- Memory allocation functions 2008-09-24 Igor Pavlov Public domain */ #ifdef _WIN32 #include #endif #include #include "Alloc.h" /* #define _SZ_ALLOC_DEBUG */ /* use _SZ_ALLOC_DEBUG to debug alloc/free operations */ #ifdef _SZ_ALLOC_DEBUG #include int g_allocCount = 0; int g_allocCountMid = 0; int g_allocCountBig = 0; #endif void *MyAlloc(size_t size) { if (size == 0) return 0; #ifdef _SZ_ALLOC_DEBUG { void *p = malloc(size); fprintf(stderr, "\nAlloc %10d bytes, count = %10d, addr = %8X", size, g_allocCount++, (unsigned)p); return p; } #else return malloc(size); #endif } void MyFree(void *address) { #ifdef _SZ_ALLOC_DEBUG if (address != 0) fprintf(stderr, "\nFree; count = %10d, addr = %8X", --g_allocCount, (unsigned)address); #endif free(address); } #ifdef _WIN32 void *MidAlloc(size_t size) { if (size == 0) return 0; #ifdef _SZ_ALLOC_DEBUG fprintf(stderr, "\nAlloc_Mid %10d bytes; count = %10d", size, g_allocCountMid++); #endif return VirtualAlloc(0, size, MEM_COMMIT, PAGE_READWRITE); } void MidFree(void *address) { #ifdef _SZ_ALLOC_DEBUG if (address != 0) fprintf(stderr, "\nFree_Mid; count = %10d", --g_allocCountMid); #endif if (address == 0) return; VirtualFree(address, 0, MEM_RELEASE); } #ifndef MEM_LARGE_PAGES #undef _7ZIP_LARGE_PAGES #endif #ifdef _7ZIP_LARGE_PAGES SIZE_T g_LargePageSize = 0; typedef SIZE_T (WINAPI *GetLargePageMinimumP)(); #endif void SetLargePageSize() { #ifdef _7ZIP_LARGE_PAGES SIZE_T size = 0; GetLargePageMinimumP largePageMinimum = (GetLargePageMinimumP) GetProcAddress(GetModuleHandle(TEXT("kernel32.dll")), "GetLargePageMinimum"); if (largePageMinimum == 0) return; size = largePageMinimum(); if (size == 0 || (size & (size - 1)) != 0) return; g_LargePageSize = size; #endif } void *BigAlloc(size_t size) { if (size == 0) return 0; #ifdef _SZ_ALLOC_DEBUG fprintf(stderr, "\nAlloc_Big %10d bytes; count = %10d", size, g_allocCountBig++); #endif #ifdef _7ZIP_LARGE_PAGES if (g_LargePageSize != 0 && g_LargePageSize <= (1 << 30) && size >= (1 << 18)) { void *res = VirtualAlloc(0, (size + g_LargePageSize - 1) & (~(g_LargePageSize - 1)), MEM_COMMIT | MEM_LARGE_PAGES, PAGE_READWRITE); if (res != 0) return res; } #endif return VirtualAlloc(0, size, MEM_COMMIT, PAGE_READWRITE); } void BigFree(void *address) { #ifdef _SZ_ALLOC_DEBUG if (address != 0) fprintf(stderr, "\nFree_Big; count = %10d", --g_allocCountBig); #endif if (address == 0) return; VirtualFree(address, 0, MEM_RELEASE); } #endif zmat-0.9.9/src/easylzma/pavlov/BraIA64.c0000755000175200007730000000332414515254731020123 0ustar rlaboissrlaboiss/* BraIA64.c -- Converter for IA-64 code 2008-10-04 : Igor Pavlov : Public domain */ #include "Bra.h" static const Byte kBranchTable[32] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 4, 6, 6, 0, 0, 7, 7, 4, 4, 0, 0, 4, 4, 0, 0 }; SizeT IA64_Convert(Byte *data, SizeT size, UInt32 ip, int encoding) { SizeT i; if (size < 16) return 0; size -= 16; for (i = 0; i <= size; i += 16) { UInt32 instrTemplate = data[i] & 0x1F; UInt32 mask = kBranchTable[instrTemplate]; UInt32 bitPos = 5; int slot; for (slot = 0; slot < 3; slot++, bitPos += 41) { UInt32 bytePos, bitRes; UInt64 instruction, instNorm; int j; if (((mask >> slot) & 1) == 0) continue; bytePos = (bitPos >> 3); bitRes = bitPos & 0x7; instruction = 0; for (j = 0; j < 6; j++) instruction += (UInt64)data[i + j + bytePos] << (8 * j); instNorm = instruction >> bitRes; if (((instNorm >> 37) & 0xF) == 0x5 && ((instNorm >> 9) & 0x7) == 0) { UInt32 src = (UInt32)((instNorm >> 13) & 0xFFFFF); UInt32 dest; src |= ((UInt32)(instNorm >> 36) & 1) << 20; src <<= 4; if (encoding) dest = ip + (UInt32)i + src; else dest = src - (ip + (UInt32)i); dest >>= 4; instNorm &= ~((UInt64)(0x8FFFFF) << 13); instNorm |= ((UInt64)(dest & 0xFFFFF) << 13); instNorm |= ((UInt64)(dest & 0x100000) << (36 - 20)); instruction &= (1 << bitRes) - 1; instruction |= (instNorm << bitRes); for (j = 0; j < 6; j++) data[i + j + bytePos] = (Byte)(instruction >> (8 * j)); } } } return i; } zmat-0.9.9/src/easylzma/pavlov/7zCrc.h0000755000175200007730000000102314515254731020022 0ustar rlaboissrlaboiss/* 7zCrc.h -- CRC32 calculation 2008-03-13 Igor Pavlov Public domain */ #ifndef __7Z_CRC_H #define __7Z_CRC_H #include #include "Types.h" extern UInt32 g_CrcTable[]; void MY_FAST_CALL CrcGenerateTable(void); #define CRC_INIT_VAL 0xFFFFFFFF #define CRC_GET_DIGEST(crc) ((crc) ^ 0xFFFFFFFF) #define CRC_UPDATE_BYTE(crc, b) (g_CrcTable[((crc) ^ (b)) & 0xFF] ^ ((crc) >> 8)) UInt32 MY_FAST_CALL CrcUpdate(UInt32 crc, const void *data, size_t size); UInt32 MY_FAST_CALL CrcCalc(const void *data, size_t size); #endif zmat-0.9.9/src/easylzma/pavlov/7zFile.h0000755000175200007730000000237314515254731020203 0ustar rlaboissrlaboiss/* 7zFile.h -- File IO 2008-11-22 : Igor Pavlov : Public domain */ #ifndef __7Z_FILE_H #define __7Z_FILE_H #ifdef _WIN32 #define USE_WINDOWS_FILE #endif #ifdef USE_WINDOWS_FILE #include #else #include #endif #include "Types.h" /* ---------- File ---------- */ typedef struct { #ifdef USE_WINDOWS_FILE HANDLE handle; #else FILE *file; #endif } CSzFile; void File_Construct(CSzFile *p); WRes InFile_Open(CSzFile *p, const char *name); WRes OutFile_Open(CSzFile *p, const char *name); WRes File_Close(CSzFile *p); /* reads max(*size, remain file's size) bytes */ WRes File_Read(CSzFile *p, void *data, size_t *size); /* writes *size bytes */ WRes File_Write(CSzFile *p, const void *data, size_t *size); WRes File_Seek(CSzFile *p, Int64 *pos, ESzSeek origin); WRes File_GetLength(CSzFile *p, UInt64 *length); /* ---------- FileInStream ---------- */ typedef struct { ISeqInStream s; CSzFile file; } CFileSeqInStream; void FileSeqInStream_CreateVTable(CFileSeqInStream *p); typedef struct { ISeekInStream s; CSzFile file; } CFileInStream; void FileInStream_CreateVTable(CFileInStream *p); typedef struct { ISeqOutStream s; CSzFile file; } CFileOutStream; void FileOutStream_CreateVTable(CFileOutStream *p); #endif zmat-0.9.9/src/easylzma/pavlov/LzHash.h0000755000175200007730000000365214515254731020235 0ustar rlaboissrlaboiss/* LzHash.h -- HASH functions for LZ algorithms 2008-10-04 : Igor Pavlov : Public domain */ #ifndef __LZHASH_H #define __LZHASH_H #define kHash2Size (1 << 10) #define kHash3Size (1 << 16) #define kHash4Size (1 << 20) #define kFix3HashSize (kHash2Size) #define kFix4HashSize (kHash2Size + kHash3Size) #define kFix5HashSize (kHash2Size + kHash3Size + kHash4Size) #define HASH2_CALC hashValue = cur[0] | ((UInt32)cur[1] << 8); #define HASH3_CALC { \ UInt32 temp = p->crc[cur[0]] ^ cur[1]; \ hash2Value = temp & (kHash2Size - 1); \ hashValue = (temp ^ ((UInt32)cur[2] << 8)) & p->hashMask; } #define HASH4_CALC { \ UInt32 temp = p->crc[cur[0]] ^ cur[1]; \ hash2Value = temp & (kHash2Size - 1); \ hash3Value = (temp ^ ((UInt32)cur[2] << 8)) & (kHash3Size - 1); \ hashValue = (temp ^ ((UInt32)cur[2] << 8) ^ (p->crc[cur[3]] << 5)) & p->hashMask; } #define HASH5_CALC { \ UInt32 temp = p->crc[cur[0]] ^ cur[1]; \ hash2Value = temp & (kHash2Size - 1); \ hash3Value = (temp ^ ((UInt32)cur[2] << 8)) & (kHash3Size - 1); \ hash4Value = (temp ^ ((UInt32)cur[2] << 8) ^ (p->crc[cur[3]] << 5)); \ hashValue = (hash4Value ^ (p->crc[cur[4]] << 3)) & p->hashMask; \ hash4Value &= (kHash4Size - 1); } /* #define HASH_ZIP_CALC hashValue = ((cur[0] | ((UInt32)cur[1] << 8)) ^ p->crc[cur[2]]) & 0xFFFF; */ #define HASH_ZIP_CALC hashValue = ((cur[2] | ((UInt32)cur[0] << 8)) ^ p->crc[cur[1]]) & 0xFFFF; #define MT_HASH2_CALC \ hash2Value = (p->crc[cur[0]] ^ cur[1]) & (kHash2Size - 1); #define MT_HASH3_CALC { \ UInt32 temp = p->crc[cur[0]] ^ cur[1]; \ hash2Value = temp & (kHash2Size - 1); \ hash3Value = (temp ^ ((UInt32)cur[2] << 8)) & (kHash3Size - 1); } #define MT_HASH4_CALC { \ UInt32 temp = p->crc[cur[0]] ^ cur[1]; \ hash2Value = temp & (kHash2Size - 1); \ hash3Value = (temp ^ ((UInt32)cur[2] << 8)) & (kHash3Size - 1); \ hash4Value = (temp ^ ((UInt32)cur[2] << 8) ^ (p->crc[cur[3]] << 5)) & (kHash4Size - 1); } #endif zmat-0.9.9/src/easylzma/pavlov/7zBuf2.c0000755000175200007730000000150514515254731020111 0ustar rlaboissrlaboiss/* 7zBuf2.c -- Byte Buffer 2008-10-04 : Igor Pavlov : Public domain */ #include #include "7zBuf.h" void DynBuf_Construct(CDynBuf *p) { p->data = 0; p->size = 0; p->pos = 0; } void DynBuf_SeekToBeg(CDynBuf *p) { p->pos = 0; } int DynBuf_Write(CDynBuf *p, const Byte *buf, size_t size, ISzAlloc *alloc) { if (size > p->size - p->pos) { size_t newSize = p->pos + size; Byte *data; newSize += newSize / 4; data = (Byte *)alloc->Alloc(alloc, newSize); if (data == 0) return 0; p->size = newSize; memcpy(data, p->data, p->pos); alloc->Free(alloc, p->data); p->data = data; } memcpy(p->data + p->pos, buf, size); p->pos += size; return 1; } void DynBuf_Free(CDynBuf *p, ISzAlloc *alloc) { alloc->Free(alloc, p->data); p->data = 0; p->size = 0; p->pos = 0; } zmat-0.9.9/src/easylzma/pavlov/7zCrc.c0000755000175200007730000000125314515254731020022 0ustar rlaboissrlaboiss/* 7zCrc.c -- CRC32 calculation 2008-08-05 Igor Pavlov Public domain */ #include "7zCrc.h" #define kCrcPoly 0xEDB88320 UInt32 g_CrcTable[256]; void MY_FAST_CALL CrcGenerateTable(void) { UInt32 i; for (i = 0; i < 256; i++) { UInt32 r = i; int j; for (j = 0; j < 8; j++) r = (r >> 1) ^ (kCrcPoly & ~((r & 1) - 1)); g_CrcTable[i] = r; } } UInt32 MY_FAST_CALL CrcUpdate(UInt32 v, const void *data, size_t size) { const Byte *p = (const Byte *)data; for (; size > 0 ; size--, p++) v = CRC_UPDATE_BYTE(v, *p); return v; } UInt32 MY_FAST_CALL CrcCalc(const void *data, size_t size) { return CrcUpdate(CRC_INIT_VAL, data, size) ^ 0xFFFFFFFF; } zmat-0.9.9/src/easylzma/pavlov/7zFile.c0000755000175200007730000001356014515254731020176 0ustar rlaboissrlaboiss/* 7zFile.c -- File IO 2008-11-22 : Igor Pavlov : Public domain */ #include "7zFile.h" #ifndef USE_WINDOWS_FILE #include #endif #ifdef USE_WINDOWS_FILE /* ReadFile and WriteFile functions in Windows have BUG: If you Read or Write 64MB or more (probably min_failure_size = 64MB - 32KB + 1) from/to Network file, it returns ERROR_NO_SYSTEM_RESOURCES (Insufficient system resources exist to complete the requested service). Probably in some version of Windows there are problems with other sizes: for 32 MB (maybe also for 16 MB). And message can be "Network connection was lost" */ #define kChunkSizeMax (1 << 22) #endif void File_Construct(CSzFile *p) { #ifdef USE_WINDOWS_FILE p->handle = INVALID_HANDLE_VALUE; #else p->file = NULL; #endif } static WRes File_Open(CSzFile *p, const char *name, int writeMode) { #ifdef USE_WINDOWS_FILE p->handle = CreateFileA(name, writeMode ? GENERIC_WRITE : GENERIC_READ, FILE_SHARE_READ, NULL, writeMode ? CREATE_ALWAYS : OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL); return (p->handle != INVALID_HANDLE_VALUE) ? 0 : GetLastError(); #else p->file = fopen(name, writeMode ? "wb+" : "rb"); return (p->file != 0) ? 0 : errno; #endif } WRes InFile_Open(CSzFile *p, const char *name) { return File_Open(p, name, 0); } WRes OutFile_Open(CSzFile *p, const char *name) { return File_Open(p, name, 1); } WRes File_Close(CSzFile *p) { #ifdef USE_WINDOWS_FILE if (p->handle != INVALID_HANDLE_VALUE) { if (!CloseHandle(p->handle)) return GetLastError(); p->handle = INVALID_HANDLE_VALUE; } #else if (p->file != NULL) { int res = fclose(p->file); if (res != 0) return res; p->file = NULL; } #endif return 0; } WRes File_Read(CSzFile *p, void *data, size_t *size) { size_t originalSize = *size; if (originalSize == 0) return 0; #ifdef USE_WINDOWS_FILE *size = 0; do { DWORD curSize = (originalSize > kChunkSizeMax) ? kChunkSizeMax : (DWORD)originalSize; DWORD processed = 0; BOOL res = ReadFile(p->handle, data, curSize, &processed, NULL); data = (void *)((Byte *)data + processed); originalSize -= processed; *size += processed; if (!res) return GetLastError(); if (processed == 0) break; } while (originalSize > 0); return 0; #else *size = fread(data, 1, originalSize, p->file); if (*size == originalSize) return 0; return ferror(p->file); #endif } WRes File_Write(CSzFile *p, const void *data, size_t *size) { size_t originalSize = *size; if (originalSize == 0) return 0; #ifdef USE_WINDOWS_FILE *size = 0; do { DWORD curSize = (originalSize > kChunkSizeMax) ? kChunkSizeMax : (DWORD)originalSize; DWORD processed = 0; BOOL res = WriteFile(p->handle, data, curSize, &processed, NULL); data = (void *)((Byte *)data + processed); originalSize -= processed; *size += processed; if (!res) return GetLastError(); if (processed == 0) break; } while (originalSize > 0); return 0; #else *size = fwrite(data, 1, originalSize, p->file); if (*size == originalSize) return 0; return ferror(p->file); #endif } WRes File_Seek(CSzFile *p, Int64 *pos, ESzSeek origin) { #ifdef USE_WINDOWS_FILE LARGE_INTEGER value; DWORD moveMethod; value.LowPart = (DWORD)*pos; value.HighPart = (LONG)((UInt64)*pos >> 16 >> 16); /* for case when UInt64 is 32-bit only */ switch (origin) { case SZ_SEEK_SET: moveMethod = FILE_BEGIN; break; case SZ_SEEK_CUR: moveMethod = FILE_CURRENT; break; case SZ_SEEK_END: moveMethod = FILE_END; break; default: return ERROR_INVALID_PARAMETER; } value.LowPart = SetFilePointer(p->handle, value.LowPart, &value.HighPart, moveMethod); if (value.LowPart == 0xFFFFFFFF) { WRes res = GetLastError(); if (res != NO_ERROR) return res; } *pos = ((Int64)value.HighPart << 32) | value.LowPart; return 0; #else int moveMethod; int res; switch (origin) { case SZ_SEEK_SET: moveMethod = SEEK_SET; break; case SZ_SEEK_CUR: moveMethod = SEEK_CUR; break; case SZ_SEEK_END: moveMethod = SEEK_END; break; default: return 1; } res = fseek(p->file, (long)*pos, moveMethod); *pos = ftell(p->file); return res; #endif } WRes File_GetLength(CSzFile *p, UInt64 *length) { #ifdef USE_WINDOWS_FILE DWORD sizeHigh; DWORD sizeLow = GetFileSize(p->handle, &sizeHigh); if (sizeLow == 0xFFFFFFFF) { DWORD res = GetLastError(); if (res != NO_ERROR) return res; } *length = (((UInt64)sizeHigh) << 32) + sizeLow; return 0; #else long pos = ftell(p->file); int res = fseek(p->file, 0, SEEK_END); *length = ftell(p->file); fseek(p->file, pos, SEEK_SET); return res; #endif } /* ---------- FileSeqInStream ---------- */ static SRes FileSeqInStream_Read(void *pp, void *buf, size_t *size) { CFileSeqInStream *p = (CFileSeqInStream *)pp; return File_Read(&p->file, buf, size) == 0 ? SZ_OK : SZ_ERROR_READ; } void FileSeqInStream_CreateVTable(CFileSeqInStream *p) { p->s.Read = FileSeqInStream_Read; } /* ---------- FileInStream ---------- */ static SRes FileInStream_Read(void *pp, void *buf, size_t *size) { CFileInStream *p = (CFileInStream *)pp; return (File_Read(&p->file, buf, size) == 0) ? SZ_OK : SZ_ERROR_READ; } static SRes FileInStream_Seek(void *pp, Int64 *pos, ESzSeek origin) { CFileInStream *p = (CFileInStream *)pp; return File_Seek(&p->file, pos, origin); } void FileInStream_CreateVTable(CFileInStream *p) { p->s.Read = FileInStream_Read; p->s.Seek = FileInStream_Seek; } /* ---------- FileOutStream ---------- */ static size_t FileOutStream_Write(void *pp, const void *data, size_t size) { CFileOutStream *p = (CFileOutStream *)pp; File_Write(&p->file, data, &size); return size; } void FileOutStream_CreateVTable(CFileOutStream *p) { p->s.Write = FileOutStream_Write; } zmat-0.9.9/src/easylzma/pavlov/CpuArch.h0000755000175200007730000000363514515254731020372 0ustar rlaboissrlaboiss/* CpuArch.h 2008-08-05 Igor Pavlov Public domain */ #ifndef __CPUARCH_H #define __CPUARCH_H /* LITTLE_ENDIAN_UNALIGN means: 1) CPU is LITTLE_ENDIAN 2) it's allowed to make unaligned memory accesses if LITTLE_ENDIAN_UNALIGN is not defined, it means that we don't know about these properties of platform. */ #if defined(_M_IX86) || defined(_M_X64) || defined(_M_AMD64) || defined(__i386__) || defined(__x86_64__) #define LITTLE_ENDIAN_UNALIGN #endif #ifdef LITTLE_ENDIAN_UNALIGN #define GetUi16(p) (*(const UInt16 *)(p)) #define GetUi32(p) (*(const UInt32 *)(p)) #define GetUi64(p) (*(const UInt64 *)(p)) #define SetUi32(p, d) *(UInt32 *)(p) = (d); #else #define GetUi16(p) (((const Byte *)(p))[0] | ((UInt16)((const Byte *)(p))[1] << 8)) #define GetUi32(p) ( \ ((const Byte *)(p))[0] | \ ((UInt32)((const Byte *)(p))[1] << 8) | \ ((UInt32)((const Byte *)(p))[2] << 16) | \ ((UInt32)((const Byte *)(p))[3] << 24)) #define GetUi64(p) (GetUi32(p) | ((UInt64)GetUi32(((const Byte *)(p)) + 4) << 32)) #define SetUi32(p, d) { UInt32 _x_ = (d); \ ((Byte *)(p))[0] = (Byte)_x_; \ ((Byte *)(p))[1] = (Byte)(_x_ >> 8); \ ((Byte *)(p))[2] = (Byte)(_x_ >> 16); \ ((Byte *)(p))[3] = (Byte)(_x_ >> 24); } #endif #if defined(LITTLE_ENDIAN_UNALIGN) && defined(_WIN64) && (_MSC_VER >= 1300) #pragma intrinsic(_byteswap_ulong) #pragma intrinsic(_byteswap_uint64) #define GetBe32(p) _byteswap_ulong(*(const UInt32 *)(const Byte *)(p)) #define GetBe64(p) _byteswap_uint64(*(const UInt64 *)(const Byte *)(p)) #else #define GetBe32(p) ( \ ((UInt32)((const Byte *)(p))[0] << 24) | \ ((UInt32)((const Byte *)(p))[1] << 16) | \ ((UInt32)((const Byte *)(p))[2] << 8) | \ ((const Byte *)(p))[3] ) #define GetBe64(p) (((UInt64)GetBe32(p) << 32) | GetBe32(((const Byte *)(p)) + 4)) #endif #define GetBe16(p) (((UInt16)((const Byte *)(p))[0] << 8) | ((const Byte *)(p))[1]) #endif zmat-0.9.9/src/easylzma/pavlov/LzmaDec.c0000755000175200007730000006542114515254731020360 0ustar rlaboissrlaboiss/* LzmaDec.c -- LZMA Decoder 2008-11-06 : Igor Pavlov : Public domain */ #include "LzmaDec.h" #include #define kNumTopBits 24 #define kTopValue ((UInt32)1 << kNumTopBits) #define kNumBitModelTotalBits 11 #define kBitModelTotal (1 << kNumBitModelTotalBits) #define kNumMoveBits 5 #define RC_INIT_SIZE 5 #define NORMALIZE if (range < kTopValue) { range <<= 8; code = (code << 8) | (*buf++); } #define IF_BIT_0(p) ttt = *(p); NORMALIZE; bound = (range >> kNumBitModelTotalBits) * ttt; if (code < bound) #define UPDATE_0(p) range = bound; *(p) = (CLzmaProb)(ttt + ((kBitModelTotal - ttt) >> kNumMoveBits)); #define UPDATE_1(p) range -= bound; code -= bound; *(p) = (CLzmaProb)(ttt - (ttt >> kNumMoveBits)); #define GET_BIT2(p, i, A0, A1) IF_BIT_0(p) \ { UPDATE_0(p); i = (i + i); A0; } else \ { UPDATE_1(p); i = (i + i) + 1; A1; } #define GET_BIT(p, i) GET_BIT2(p, i, ; , ;) #define TREE_GET_BIT(probs, i) { GET_BIT((probs + i), i); } #define TREE_DECODE(probs, limit, i) \ { i = 1; do { TREE_GET_BIT(probs, i); } while (i < limit); i -= limit; } /* #define _LZMA_SIZE_OPT */ #ifdef _LZMA_SIZE_OPT #define TREE_6_DECODE(probs, i) TREE_DECODE(probs, (1 << 6), i) #else #define TREE_6_DECODE(probs, i) \ { i = 1; \ TREE_GET_BIT(probs, i); \ TREE_GET_BIT(probs, i); \ TREE_GET_BIT(probs, i); \ TREE_GET_BIT(probs, i); \ TREE_GET_BIT(probs, i); \ TREE_GET_BIT(probs, i); \ i -= 0x40; } #endif #define NORMALIZE_CHECK if (range < kTopValue) { if (buf >= bufLimit) return DUMMY_ERROR; range <<= 8; code = (code << 8) | (*buf++); } #define IF_BIT_0_CHECK(p) ttt = *(p); NORMALIZE_CHECK; bound = (range >> kNumBitModelTotalBits) * ttt; if (code < bound) #define UPDATE_0_CHECK range = bound; #define UPDATE_1_CHECK range -= bound; code -= bound; #define GET_BIT2_CHECK(p, i, A0, A1) IF_BIT_0_CHECK(p) \ { UPDATE_0_CHECK; i = (i + i); A0; } else \ { UPDATE_1_CHECK; i = (i + i) + 1; A1; } #define GET_BIT_CHECK(p, i) GET_BIT2_CHECK(p, i, ; , ;) #define TREE_DECODE_CHECK(probs, limit, i) \ { i = 1; do { GET_BIT_CHECK(probs + i, i) } while (i < limit); i -= limit; } #define kNumPosBitsMax 4 #define kNumPosStatesMax (1 << kNumPosBitsMax) #define kLenNumLowBits 3 #define kLenNumLowSymbols (1 << kLenNumLowBits) #define kLenNumMidBits 3 #define kLenNumMidSymbols (1 << kLenNumMidBits) #define kLenNumHighBits 8 #define kLenNumHighSymbols (1 << kLenNumHighBits) #define LenChoice 0 #define LenChoice2 (LenChoice + 1) #define LenLow (LenChoice2 + 1) #define LenMid (LenLow + (kNumPosStatesMax << kLenNumLowBits)) #define LenHigh (LenMid + (kNumPosStatesMax << kLenNumMidBits)) #define kNumLenProbs (LenHigh + kLenNumHighSymbols) #define kNumStates 12 #define kNumLitStates 7 #define kStartPosModelIndex 4 #define kEndPosModelIndex 14 #define kNumFullDistances (1 << (kEndPosModelIndex >> 1)) #define kNumPosSlotBits 6 #define kNumLenToPosStates 4 #define kNumAlignBits 4 #define kAlignTableSize (1 << kNumAlignBits) #define kMatchMinLen 2 #define kMatchSpecLenStart (kMatchMinLen + kLenNumLowSymbols + kLenNumMidSymbols + kLenNumHighSymbols) #define IsMatch 0 #define IsRep (IsMatch + (kNumStates << kNumPosBitsMax)) #define IsRepG0 (IsRep + kNumStates) #define IsRepG1 (IsRepG0 + kNumStates) #define IsRepG2 (IsRepG1 + kNumStates) #define IsRep0Long (IsRepG2 + kNumStates) #define PosSlot (IsRep0Long + (kNumStates << kNumPosBitsMax)) #define SpecPos (PosSlot + (kNumLenToPosStates << kNumPosSlotBits)) #define Align (SpecPos + kNumFullDistances - kEndPosModelIndex) #define LenCoder (Align + kAlignTableSize) #define RepLenCoder (LenCoder + kNumLenProbs) #define Literal (RepLenCoder + kNumLenProbs) #define LZMA_BASE_SIZE 1846 #define LZMA_LIT_SIZE 768 #define LzmaProps_GetNumProbs(p) ((UInt32)LZMA_BASE_SIZE + (LZMA_LIT_SIZE << ((p)->lc + (p)->lp))) #if Literal != LZMA_BASE_SIZE StopCompilingDueBUG #endif static const Byte kLiteralNextStates[kNumStates * 2] = { 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 4, 5, 7, 7, 7, 7, 7, 7, 7, 10, 10, 10, 10, 10 }; #define LZMA_DIC_MIN (1 << 12) /* First LZMA-symbol is always decoded. And it decodes new LZMA-symbols while (buf < bufLimit), but "buf" is without last normalization Out: Result: SZ_OK - OK SZ_ERROR_DATA - Error p->remainLen: < kMatchSpecLenStart : normal remain = kMatchSpecLenStart : finished = kMatchSpecLenStart + 1 : Flush marker = kMatchSpecLenStart + 2 : State Init Marker */ static int MY_FAST_CALL LzmaDec_DecodeReal(CLzmaDec *p, SizeT limit, const Byte *bufLimit) { CLzmaProb *probs = p->probs; unsigned state = p->state; UInt32 rep0 = p->reps[0], rep1 = p->reps[1], rep2 = p->reps[2], rep3 = p->reps[3]; unsigned pbMask = ((unsigned)1 << (p->prop.pb)) - 1; unsigned lpMask = ((unsigned)1 << (p->prop.lp)) - 1; unsigned lc = p->prop.lc; Byte *dic = p->dic; SizeT dicBufSize = p->dicBufSize; SizeT dicPos = p->dicPos; UInt32 processedPos = p->processedPos; UInt32 checkDicSize = p->checkDicSize; unsigned len = 0; const Byte *buf = p->buf; UInt32 range = p->range; UInt32 code = p->code; do { CLzmaProb *prob; UInt32 bound; unsigned ttt; unsigned posState = processedPos & pbMask; prob = probs + IsMatch + (state << kNumPosBitsMax) + posState; IF_BIT_0(prob) { unsigned symbol; UPDATE_0(prob); prob = probs + Literal; if (checkDicSize != 0 || processedPos != 0) prob += (LZMA_LIT_SIZE * (((processedPos & lpMask) << lc) + (dic[(dicPos == 0 ? dicBufSize : dicPos) - 1] >> (8 - lc)))); if (state < kNumLitStates) { symbol = 1; do { GET_BIT(prob + symbol, symbol) } while (symbol < 0x100); } else { unsigned matchByte = p->dic[(dicPos - rep0) + ((dicPos < rep0) ? dicBufSize : 0)]; unsigned offs = 0x100; symbol = 1; do { unsigned bit; CLzmaProb *probLit; matchByte <<= 1; bit = (matchByte & offs); probLit = prob + offs + bit + symbol; GET_BIT2(probLit, symbol, offs &= ~bit, offs &= bit) } while (symbol < 0x100); } dic[dicPos++] = (Byte)symbol; processedPos++; state = kLiteralNextStates[state]; /* if (state < 4) state = 0; else if (state < 10) state -= 3; else state -= 6; */ continue; } else { UPDATE_1(prob); prob = probs + IsRep + state; IF_BIT_0(prob) { UPDATE_0(prob); state += kNumStates; prob = probs + LenCoder; } else { UPDATE_1(prob); if (checkDicSize == 0 && processedPos == 0) return SZ_ERROR_DATA; prob = probs + IsRepG0 + state; IF_BIT_0(prob) { UPDATE_0(prob); prob = probs + IsRep0Long + (state << kNumPosBitsMax) + posState; IF_BIT_0(prob) { UPDATE_0(prob); dic[dicPos] = dic[(dicPos - rep0) + ((dicPos < rep0) ? dicBufSize : 0)]; dicPos++; processedPos++; state = state < kNumLitStates ? 9 : 11; continue; } UPDATE_1(prob); } else { UInt32 distance; UPDATE_1(prob); prob = probs + IsRepG1 + state; IF_BIT_0(prob) { UPDATE_0(prob); distance = rep1; } else { UPDATE_1(prob); prob = probs + IsRepG2 + state; IF_BIT_0(prob) { UPDATE_0(prob); distance = rep2; } else { UPDATE_1(prob); distance = rep3; rep3 = rep2; } rep2 = rep1; } rep1 = rep0; rep0 = distance; } state = state < kNumLitStates ? 8 : 11; prob = probs + RepLenCoder; } { unsigned limit, offset; CLzmaProb *probLen = prob + LenChoice; IF_BIT_0(probLen) { UPDATE_0(probLen); probLen = prob + LenLow + (posState << kLenNumLowBits); offset = 0; limit = (1 << kLenNumLowBits); } else { UPDATE_1(probLen); probLen = prob + LenChoice2; IF_BIT_0(probLen) { UPDATE_0(probLen); probLen = prob + LenMid + (posState << kLenNumMidBits); offset = kLenNumLowSymbols; limit = (1 << kLenNumMidBits); } else { UPDATE_1(probLen); probLen = prob + LenHigh; offset = kLenNumLowSymbols + kLenNumMidSymbols; limit = (1 << kLenNumHighBits); } } TREE_DECODE(probLen, limit, len); len += offset; } if (state >= kNumStates) { UInt32 distance; prob = probs + PosSlot + ((len < kNumLenToPosStates ? len : kNumLenToPosStates - 1) << kNumPosSlotBits); TREE_6_DECODE(prob, distance); if (distance >= kStartPosModelIndex) { unsigned posSlot = (unsigned)distance; int numDirectBits = (int)(((distance >> 1) - 1)); distance = (2 | (distance & 1)); if (posSlot < kEndPosModelIndex) { distance <<= numDirectBits; prob = probs + SpecPos + distance - posSlot - 1; { UInt32 mask = 1; unsigned i = 1; do { GET_BIT2(prob + i, i, ; , distance |= mask); mask <<= 1; } while (--numDirectBits != 0); } } else { numDirectBits -= kNumAlignBits; do { NORMALIZE range >>= 1; { UInt32 t; code -= range; t = (0 - ((UInt32)code >> 31)); /* (UInt32)((Int32)code >> 31) */ distance = (distance << 1) + (t + 1); code += range & t; } /* distance <<= 1; if (code >= range) { code -= range; distance |= 1; } */ } while (--numDirectBits != 0); prob = probs + Align; distance <<= kNumAlignBits; { unsigned i = 1; GET_BIT2(prob + i, i, ; , distance |= 1); GET_BIT2(prob + i, i, ; , distance |= 2); GET_BIT2(prob + i, i, ; , distance |= 4); GET_BIT2(prob + i, i, ; , distance |= 8); } if (distance == (UInt32)0xFFFFFFFF) { len += kMatchSpecLenStart; state -= kNumStates; break; } } } rep3 = rep2; rep2 = rep1; rep1 = rep0; rep0 = distance + 1; if (checkDicSize == 0) { if (distance >= processedPos) return SZ_ERROR_DATA; } else if (distance >= checkDicSize) return SZ_ERROR_DATA; state = (state < kNumStates + kNumLitStates) ? kNumLitStates : kNumLitStates + 3; /* state = kLiteralNextStates[state]; */ } len += kMatchMinLen; if (limit == dicPos) return SZ_ERROR_DATA; { SizeT rem = limit - dicPos; unsigned curLen = ((rem < len) ? (unsigned)rem : len); SizeT pos = (dicPos - rep0) + ((dicPos < rep0) ? dicBufSize : 0); processedPos += curLen; len -= curLen; if (pos + curLen <= dicBufSize) { Byte *dest = dic + dicPos; ptrdiff_t src = (ptrdiff_t)pos - (ptrdiff_t)dicPos; const Byte *lim = dest + curLen; dicPos += curLen; do *(dest) = (Byte)*(dest + src); while (++dest != lim); } else { do { dic[dicPos++] = dic[pos]; if (++pos == dicBufSize) pos = 0; } while (--curLen != 0); } } } } while (dicPos < limit && buf < bufLimit); NORMALIZE; p->buf = buf; p->range = range; p->code = code; p->remainLen = len; p->dicPos = dicPos; p->processedPos = processedPos; p->reps[0] = rep0; p->reps[1] = rep1; p->reps[2] = rep2; p->reps[3] = rep3; p->state = state; return SZ_OK; } static void MY_FAST_CALL LzmaDec_WriteRem(CLzmaDec *p, SizeT limit) { if (p->remainLen != 0 && p->remainLen < kMatchSpecLenStart) { Byte *dic = p->dic; SizeT dicPos = p->dicPos; SizeT dicBufSize = p->dicBufSize; unsigned len = p->remainLen; UInt32 rep0 = p->reps[0]; if (limit - dicPos < len) len = (unsigned)(limit - dicPos); if (p->checkDicSize == 0 && p->prop.dicSize - p->processedPos <= len) p->checkDicSize = p->prop.dicSize; p->processedPos += len; p->remainLen -= len; while (len-- != 0) { dic[dicPos] = dic[(dicPos - rep0) + ((dicPos < rep0) ? dicBufSize : 0)]; dicPos++; } p->dicPos = dicPos; } } static int MY_FAST_CALL LzmaDec_DecodeReal2(CLzmaDec *p, SizeT limit, const Byte *bufLimit) { do { SizeT limit2 = limit; if (p->checkDicSize == 0) { UInt32 rem = p->prop.dicSize - p->processedPos; if (limit - p->dicPos > rem) limit2 = p->dicPos + rem; } RINOK(LzmaDec_DecodeReal(p, limit2, bufLimit)); if (p->processedPos >= p->prop.dicSize) p->checkDicSize = p->prop.dicSize; LzmaDec_WriteRem(p, limit); } while (p->dicPos < limit && p->buf < bufLimit && p->remainLen < kMatchSpecLenStart); if (p->remainLen > kMatchSpecLenStart) { p->remainLen = kMatchSpecLenStart; } return 0; } typedef enum { DUMMY_ERROR, /* unexpected end of input stream */ DUMMY_LIT, DUMMY_MATCH, DUMMY_REP } ELzmaDummy; static ELzmaDummy LzmaDec_TryDummy(const CLzmaDec *p, const Byte *buf, SizeT inSize) { UInt32 range = p->range; UInt32 code = p->code; const Byte *bufLimit = buf + inSize; CLzmaProb *probs = p->probs; unsigned state = p->state; ELzmaDummy res; { CLzmaProb *prob; UInt32 bound; unsigned ttt; unsigned posState = (p->processedPos) & ((1 << p->prop.pb) - 1); prob = probs + IsMatch + (state << kNumPosBitsMax) + posState; IF_BIT_0_CHECK(prob) { UPDATE_0_CHECK /* if (bufLimit - buf >= 7) return DUMMY_LIT; */ prob = probs + Literal; if (p->checkDicSize != 0 || p->processedPos != 0) prob += (LZMA_LIT_SIZE * ((((p->processedPos) & ((1 << (p->prop.lp)) - 1)) << p->prop.lc) + (p->dic[(p->dicPos == 0 ? p->dicBufSize : p->dicPos) - 1] >> (8 - p->prop.lc)))); if (state < kNumLitStates) { unsigned symbol = 1; do { GET_BIT_CHECK(prob + symbol, symbol) } while (symbol < 0x100); } else { unsigned matchByte = p->dic[p->dicPos - p->reps[0] + ((p->dicPos < p->reps[0]) ? p->dicBufSize : 0)]; unsigned offs = 0x100; unsigned symbol = 1; do { unsigned bit; CLzmaProb *probLit; matchByte <<= 1; bit = (matchByte & offs); probLit = prob + offs + bit + symbol; GET_BIT2_CHECK(probLit, symbol, offs &= ~bit, offs &= bit) } while (symbol < 0x100); } res = DUMMY_LIT; } else { unsigned len; UPDATE_1_CHECK; prob = probs + IsRep + state; IF_BIT_0_CHECK(prob) { UPDATE_0_CHECK; state = 0; prob = probs + LenCoder; res = DUMMY_MATCH; } else { UPDATE_1_CHECK; res = DUMMY_REP; prob = probs + IsRepG0 + state; IF_BIT_0_CHECK(prob) { UPDATE_0_CHECK; prob = probs + IsRep0Long + (state << kNumPosBitsMax) + posState; IF_BIT_0_CHECK(prob) { UPDATE_0_CHECK; NORMALIZE_CHECK; return DUMMY_REP; } else { UPDATE_1_CHECK; } } else { UPDATE_1_CHECK; prob = probs + IsRepG1 + state; IF_BIT_0_CHECK(prob) { UPDATE_0_CHECK; } else { UPDATE_1_CHECK; prob = probs + IsRepG2 + state; IF_BIT_0_CHECK(prob) { UPDATE_0_CHECK; } else { UPDATE_1_CHECK; } } } state = kNumStates; prob = probs + RepLenCoder; } { unsigned limit, offset; CLzmaProb *probLen = prob + LenChoice; IF_BIT_0_CHECK(probLen) { UPDATE_0_CHECK; probLen = prob + LenLow + (posState << kLenNumLowBits); offset = 0; limit = 1 << kLenNumLowBits; } else { UPDATE_1_CHECK; probLen = prob + LenChoice2; IF_BIT_0_CHECK(probLen) { UPDATE_0_CHECK; probLen = prob + LenMid + (posState << kLenNumMidBits); offset = kLenNumLowSymbols; limit = 1 << kLenNumMidBits; } else { UPDATE_1_CHECK; probLen = prob + LenHigh; offset = kLenNumLowSymbols + kLenNumMidSymbols; limit = 1 << kLenNumHighBits; } } TREE_DECODE_CHECK(probLen, limit, len); len += offset; } if (state < 4) { unsigned posSlot; prob = probs + PosSlot + ((len < kNumLenToPosStates ? len : kNumLenToPosStates - 1) << kNumPosSlotBits); TREE_DECODE_CHECK(prob, 1 << kNumPosSlotBits, posSlot); if (posSlot >= kStartPosModelIndex) { int numDirectBits = ((posSlot >> 1) - 1); /* if (bufLimit - buf >= 8) return DUMMY_MATCH; */ if (posSlot < kEndPosModelIndex) { prob = probs + SpecPos + ((2 | (posSlot & 1)) << numDirectBits) - posSlot - 1; } else { numDirectBits -= kNumAlignBits; do { NORMALIZE_CHECK range >>= 1; code -= range & (((code - range) >> 31) - 1); /* if (code >= range) code -= range; */ } while (--numDirectBits != 0); prob = probs + Align; numDirectBits = kNumAlignBits; } { unsigned i = 1; do { GET_BIT_CHECK(prob + i, i); } while (--numDirectBits != 0); } } } } } NORMALIZE_CHECK; return res; } static void LzmaDec_InitRc(CLzmaDec *p, const Byte *data) { p->code = ((UInt32)data[1] << 24) | ((UInt32)data[2] << 16) | ((UInt32)data[3] << 8) | ((UInt32)data[4]); p->range = 0xFFFFFFFF; p->needFlush = 0; } void LzmaDec_InitDicAndState(CLzmaDec *p, Bool initDic, Bool initState) { p->needFlush = 1; p->remainLen = 0; p->tempBufSize = 0; if (initDic) { p->processedPos = 0; p->checkDicSize = 0; p->needInitState = 1; } if (initState) p->needInitState = 1; } void LzmaDec_Init(CLzmaDec *p) { p->dicPos = 0; LzmaDec_InitDicAndState(p, True, True); } static void LzmaDec_InitStateReal(CLzmaDec *p) { UInt32 numProbs = Literal + ((UInt32)LZMA_LIT_SIZE << (p->prop.lc + p->prop.lp)); UInt32 i; CLzmaProb *probs = p->probs; for (i = 0; i < numProbs; i++) probs[i] = kBitModelTotal >> 1; p->reps[0] = p->reps[1] = p->reps[2] = p->reps[3] = 1; p->state = 0; p->needInitState = 0; } SRes LzmaDec_DecodeToDic(CLzmaDec *p, SizeT dicLimit, const Byte *src, SizeT *srcLen, ELzmaFinishMode finishMode, ELzmaStatus *status) { SizeT inSize = *srcLen; (*srcLen) = 0; LzmaDec_WriteRem(p, dicLimit); *status = LZMA_STATUS_NOT_SPECIFIED; while (p->remainLen != kMatchSpecLenStart) { int checkEndMarkNow; if (p->needFlush != 0) { for (; inSize > 0 && p->tempBufSize < RC_INIT_SIZE; (*srcLen)++, inSize--) p->tempBuf[p->tempBufSize++] = *src++; if (p->tempBufSize < RC_INIT_SIZE) { *status = LZMA_STATUS_NEEDS_MORE_INPUT; return SZ_OK; } if (p->tempBuf[0] != 0) return SZ_ERROR_DATA; LzmaDec_InitRc(p, p->tempBuf); p->tempBufSize = 0; } checkEndMarkNow = 0; if (p->dicPos >= dicLimit) { if (p->remainLen == 0 && p->code == 0) { *status = LZMA_STATUS_MAYBE_FINISHED_WITHOUT_MARK; return SZ_OK; } if (finishMode == LZMA_FINISH_ANY) { *status = LZMA_STATUS_NOT_FINISHED; return SZ_OK; } if (p->remainLen != 0) { *status = LZMA_STATUS_NOT_FINISHED; return SZ_ERROR_DATA; } checkEndMarkNow = 1; } if (p->needInitState) LzmaDec_InitStateReal(p); if (p->tempBufSize == 0) { SizeT processed; const Byte *bufLimit; if (inSize < LZMA_REQUIRED_INPUT_MAX || checkEndMarkNow) { int dummyRes = LzmaDec_TryDummy(p, src, inSize); if (dummyRes == DUMMY_ERROR) { memcpy(p->tempBuf, src, inSize); p->tempBufSize = (unsigned)inSize; (*srcLen) += inSize; *status = LZMA_STATUS_NEEDS_MORE_INPUT; return SZ_OK; } if (checkEndMarkNow && dummyRes != DUMMY_MATCH) { *status = LZMA_STATUS_NOT_FINISHED; return SZ_ERROR_DATA; } bufLimit = src; } else bufLimit = src + inSize - LZMA_REQUIRED_INPUT_MAX; p->buf = src; if (LzmaDec_DecodeReal2(p, dicLimit, bufLimit) != 0) return SZ_ERROR_DATA; processed = (SizeT)(p->buf - src); (*srcLen) += processed; src += processed; inSize -= processed; } else { unsigned rem = p->tempBufSize, lookAhead = 0; while (rem < LZMA_REQUIRED_INPUT_MAX && lookAhead < inSize) p->tempBuf[rem++] = src[lookAhead++]; p->tempBufSize = rem; if (rem < LZMA_REQUIRED_INPUT_MAX || checkEndMarkNow) { int dummyRes = LzmaDec_TryDummy(p, p->tempBuf, rem); if (dummyRes == DUMMY_ERROR) { (*srcLen) += lookAhead; *status = LZMA_STATUS_NEEDS_MORE_INPUT; return SZ_OK; } if (checkEndMarkNow && dummyRes != DUMMY_MATCH) { *status = LZMA_STATUS_NOT_FINISHED; return SZ_ERROR_DATA; } } p->buf = p->tempBuf; if (LzmaDec_DecodeReal2(p, dicLimit, p->buf) != 0) return SZ_ERROR_DATA; lookAhead -= (rem - (unsigned)(p->buf - p->tempBuf)); (*srcLen) += lookAhead; src += lookAhead; inSize -= lookAhead; p->tempBufSize = 0; } } if (p->code == 0) *status = LZMA_STATUS_FINISHED_WITH_MARK; return (p->code == 0) ? SZ_OK : SZ_ERROR_DATA; } SRes LzmaDec_DecodeToBuf(CLzmaDec *p, Byte *dest, SizeT *destLen, const Byte *src, SizeT *srcLen, ELzmaFinishMode finishMode, ELzmaStatus *status) { SizeT outSize = *destLen; SizeT inSize = *srcLen; *srcLen = *destLen = 0; for (;;) { SizeT inSizeCur = inSize, outSizeCur, dicPos; ELzmaFinishMode curFinishMode; SRes res; if (p->dicPos == p->dicBufSize) p->dicPos = 0; dicPos = p->dicPos; if (outSize > p->dicBufSize - dicPos) { outSizeCur = p->dicBufSize; curFinishMode = LZMA_FINISH_ANY; } else { outSizeCur = dicPos + outSize; curFinishMode = finishMode; } res = LzmaDec_DecodeToDic(p, outSizeCur, src, &inSizeCur, curFinishMode, status); src += inSizeCur; inSize -= inSizeCur; *srcLen += inSizeCur; outSizeCur = p->dicPos - dicPos; memcpy(dest, p->dic + dicPos, outSizeCur); dest += outSizeCur; outSize -= outSizeCur; *destLen += outSizeCur; if (res != 0) return res; if (outSizeCur == 0 || outSize == 0) return SZ_OK; } } void LzmaDec_FreeProbs(CLzmaDec *p, ISzAlloc *alloc) { alloc->Free(alloc, p->probs); p->probs = 0; } static void LzmaDec_FreeDict(CLzmaDec *p, ISzAlloc *alloc) { alloc->Free(alloc, p->dic); p->dic = 0; } void LzmaDec_Free(CLzmaDec *p, ISzAlloc *alloc) { LzmaDec_FreeProbs(p, alloc); LzmaDec_FreeDict(p, alloc); } SRes LzmaProps_Decode(CLzmaProps *p, const Byte *data, unsigned size) { UInt32 dicSize; Byte d; if (size < LZMA_PROPS_SIZE) return SZ_ERROR_UNSUPPORTED; else dicSize = data[1] | ((UInt32)data[2] << 8) | ((UInt32)data[3] << 16) | ((UInt32)data[4] << 24); if (dicSize < LZMA_DIC_MIN) dicSize = LZMA_DIC_MIN; p->dicSize = dicSize; d = data[0]; if (d >= (9 * 5 * 5)) return SZ_ERROR_UNSUPPORTED; p->lc = d % 9; d /= 9; p->pb = d / 5; p->lp = d % 5; return SZ_OK; } static SRes LzmaDec_AllocateProbs2(CLzmaDec *p, const CLzmaProps *propNew, ISzAlloc *alloc) { UInt32 numProbs = LzmaProps_GetNumProbs(propNew); if (p->probs == 0 || numProbs != p->numProbs) { LzmaDec_FreeProbs(p, alloc); p->probs = (CLzmaProb *)alloc->Alloc(alloc, numProbs * sizeof(CLzmaProb)); p->numProbs = numProbs; if (p->probs == 0) return SZ_ERROR_MEM; } return SZ_OK; } SRes LzmaDec_AllocateProbs(CLzmaDec *p, const Byte *props, unsigned propsSize, ISzAlloc *alloc) { CLzmaProps propNew; RINOK(LzmaProps_Decode(&propNew, props, propsSize)); RINOK(LzmaDec_AllocateProbs2(p, &propNew, alloc)); p->prop = propNew; return SZ_OK; } SRes LzmaDec_Allocate(CLzmaDec *p, const Byte *props, unsigned propsSize, ISzAlloc *alloc) { CLzmaProps propNew; SizeT dicBufSize; RINOK(LzmaProps_Decode(&propNew, props, propsSize)); RINOK(LzmaDec_AllocateProbs2(p, &propNew, alloc)); dicBufSize = propNew.dicSize; if (p->dic == 0 || dicBufSize != p->dicBufSize) { LzmaDec_FreeDict(p, alloc); p->dic = (Byte *)alloc->Alloc(alloc, dicBufSize); if (p->dic == 0) { LzmaDec_FreeProbs(p, alloc); return SZ_ERROR_MEM; } } p->dicBufSize = dicBufSize; p->prop = propNew; return SZ_OK; } SRes LzmaDecode(Byte *dest, SizeT *destLen, const Byte *src, SizeT *srcLen, const Byte *propData, unsigned propSize, ELzmaFinishMode finishMode, ELzmaStatus *status, ISzAlloc *alloc) { CLzmaDec p; SRes res; SizeT inSize = *srcLen; SizeT outSize = *destLen; *srcLen = *destLen = 0; if (inSize < RC_INIT_SIZE) return SZ_ERROR_INPUT_EOF; LzmaDec_Construct(&p); res = LzmaDec_AllocateProbs(&p, propData, propSize, alloc); if (res != 0) return res; p.dic = dest; p.dicBufSize = outSize; LzmaDec_Init(&p); *srcLen = inSize; res = LzmaDec_DecodeToDic(&p, outSize, src, srcLen, finishMode, status); if (res == SZ_OK && *status == LZMA_STATUS_NEEDS_MORE_INPUT) res = SZ_ERROR_INPUT_EOF; (*destLen) = p.dicPos; LzmaDec_FreeProbs(&p, alloc); return res; } zmat-0.9.9/src/easylzma/pavlov/Bra.c0000755000175200007730000000611014515254731017533 0ustar rlaboissrlaboiss/* Bra.c -- Converters for RISC code 2008-10-04 : Igor Pavlov : Public domain */ #include "Bra.h" SizeT ARM_Convert(Byte *data, SizeT size, UInt32 ip, int encoding) { SizeT i; if (size < 4) return 0; size -= 4; ip += 8; for (i = 0; i <= size; i += 4) { if (data[i + 3] == 0xEB) { UInt32 dest; UInt32 src = ((UInt32)data[i + 2] << 16) | ((UInt32)data[i + 1] << 8) | (data[i + 0]); src <<= 2; if (encoding) dest = ip + (UInt32)i + src; else dest = src - (ip + (UInt32)i); dest >>= 2; data[i + 2] = (Byte)(dest >> 16); data[i + 1] = (Byte)(dest >> 8); data[i + 0] = (Byte)dest; } } return i; } SizeT ARMT_Convert(Byte *data, SizeT size, UInt32 ip, int encoding) { SizeT i; if (size < 4) return 0; size -= 4; ip += 4; for (i = 0; i <= size; i += 2) { if ((data[i + 1] & 0xF8) == 0xF0 && (data[i + 3] & 0xF8) == 0xF8) { UInt32 dest; UInt32 src = (((UInt32)data[i + 1] & 0x7) << 19) | ((UInt32)data[i + 0] << 11) | (((UInt32)data[i + 3] & 0x7) << 8) | (data[i + 2]); src <<= 1; if (encoding) dest = ip + (UInt32)i + src; else dest = src - (ip + (UInt32)i); dest >>= 1; data[i + 1] = (Byte)(0xF0 | ((dest >> 19) & 0x7)); data[i + 0] = (Byte)(dest >> 11); data[i + 3] = (Byte)(0xF8 | ((dest >> 8) & 0x7)); data[i + 2] = (Byte)dest; i += 2; } } return i; } SizeT PPC_Convert(Byte *data, SizeT size, UInt32 ip, int encoding) { SizeT i; if (size < 4) return 0; size -= 4; for (i = 0; i <= size; i += 4) { if ((data[i] >> 2) == 0x12 && (data[i + 3] & 3) == 1) { UInt32 src = ((UInt32)(data[i + 0] & 3) << 24) | ((UInt32)data[i + 1] << 16) | ((UInt32)data[i + 2] << 8) | ((UInt32)data[i + 3] & (~3)); UInt32 dest; if (encoding) dest = ip + (UInt32)i + src; else dest = src - (ip + (UInt32)i); data[i + 0] = (Byte)(0x48 | ((dest >> 24) & 0x3)); data[i + 1] = (Byte)(dest >> 16); data[i + 2] = (Byte)(dest >> 8); data[i + 3] &= 0x3; data[i + 3] |= dest; } } return i; } SizeT SPARC_Convert(Byte *data, SizeT size, UInt32 ip, int encoding) { UInt32 i; if (size < 4) return 0; size -= 4; for (i = 0; i <= size; i += 4) { if ((data[i] == 0x40 && (data[i + 1] & 0xC0) == 0x00) || (data[i] == 0x7F && (data[i + 1] & 0xC0) == 0xC0)) { UInt32 src = ((UInt32)data[i + 0] << 24) | ((UInt32)data[i + 1] << 16) | ((UInt32)data[i + 2] << 8) | ((UInt32)data[i + 3]); UInt32 dest; src <<= 2; if (encoding) dest = ip + i + src; else dest = src - (ip + i); dest >>= 2; dest = (((0 - ((dest >> 22) & 1)) << 22) & 0x3FFFFFFF) | (dest & 0x3FFFFF) | 0x40000000; data[i + 0] = (Byte)(dest >> 24); data[i + 1] = (Byte)(dest >> 16); data[i + 2] = (Byte)(dest >> 8); data[i + 3] = (Byte)dest; } } return i; } zmat-0.9.9/src/easylzma/pavlov/Bra86.c0000755000175200007730000000421014515254731017710 0ustar rlaboissrlaboiss/* Bra86.c -- Converter for x86 code (BCJ) 2008-10-04 : Igor Pavlov : Public domain */ #include "Bra.h" #define Test86MSByte(b) ((b) == 0 || (b) == 0xFF) const Byte kMaskToAllowedStatus[8] = {1, 1, 1, 0, 1, 0, 0, 0}; const Byte kMaskToBitNumber[8] = {0, 1, 2, 2, 3, 3, 3, 3}; SizeT x86_Convert(Byte *data, SizeT size, UInt32 ip, UInt32 *state, int encoding) { SizeT bufferPos = 0, prevPosT; UInt32 prevMask = *state & 0x7; if (size < 5) return 0; ip += 5; prevPosT = (SizeT)0 - 1; for (;;) { Byte *p = data + bufferPos; Byte *limit = data + size - 4; for (; p < limit; p++) if ((*p & 0xFE) == 0xE8) break; bufferPos = (SizeT)(p - data); if (p >= limit) break; prevPosT = bufferPos - prevPosT; if (prevPosT > 3) prevMask = 0; else { prevMask = (prevMask << ((int)prevPosT - 1)) & 0x7; if (prevMask != 0) { Byte b = p[4 - kMaskToBitNumber[prevMask]]; if (!kMaskToAllowedStatus[prevMask] || Test86MSByte(b)) { prevPosT = bufferPos; prevMask = ((prevMask << 1) & 0x7) | 1; bufferPos++; continue; } } } prevPosT = bufferPos; if (Test86MSByte(p[4])) { UInt32 src = ((UInt32)p[4] << 24) | ((UInt32)p[3] << 16) | ((UInt32)p[2] << 8) | ((UInt32)p[1]); UInt32 dest; for (;;) { Byte b; int index; if (encoding) dest = (ip + (UInt32)bufferPos) + src; else dest = src - (ip + (UInt32)bufferPos); if (prevMask == 0) break; index = kMaskToBitNumber[prevMask] * 8; b = (Byte)(dest >> (24 - index)); if (!Test86MSByte(b)) break; src = dest ^ ((1 << (32 - index)) - 1); } p[4] = (Byte)(~(((dest >> 24) & 1) - 1)); p[3] = (Byte)(dest >> 16); p[2] = (Byte)(dest >> 8); p[1] = (Byte)dest; bufferPos += 5; } else { prevMask = ((prevMask << 1) & 0x7) | 1; bufferPos++; } } prevPosT = bufferPos - prevPosT; *state = ((prevPosT > 3) ? 0 : ((prevMask << ((int)prevPosT - 1)) & 0x7)); return bufferPos; } zmat-0.9.9/src/easylzma/pavlov/LzmaLib.c0000755000175200007730000000270614515254731020370 0ustar rlaboissrlaboiss/* LzmaLib.c -- LZMA library wrapper 2008-08-05 Igor Pavlov Public domain */ #include "LzmaEnc.h" #include "LzmaDec.h" #include "Alloc.h" #include "LzmaLib.h" static void *SzAlloc(void *p, size_t size) { p = p; return MyAlloc(size); } static void SzFree(void *p, void *address) { p = p; MyFree(address); } static ISzAlloc g_Alloc = { SzAlloc, SzFree }; MY_STDAPI LzmaCompress(unsigned char *dest, size_t *destLen, const unsigned char *src, size_t srcLen, unsigned char *outProps, size_t *outPropsSize, int level, /* 0 <= level <= 9, default = 5 */ unsigned dictSize, /* use (1 << N) or (3 << N). 4 KB < dictSize <= 128 MB */ int lc, /* 0 <= lc <= 8, default = 3 */ int lp, /* 0 <= lp <= 4, default = 0 */ int pb, /* 0 <= pb <= 4, default = 2 */ int fb, /* 5 <= fb <= 273, default = 32 */ int numThreads /* 1 or 2, default = 2 */ ) { CLzmaEncProps props; LzmaEncProps_Init(&props); props.level = level; props.dictSize = dictSize; props.lc = lc; props.lp = lp; props.pb = pb; props.fb = fb; props.numThreads = numThreads; return LzmaEncode(dest, destLen, src, srcLen, &props, outProps, outPropsSize, 0, NULL, &g_Alloc, &g_Alloc); } MY_STDAPI LzmaUncompress(unsigned char *dest, size_t *destLen, const unsigned char *src, size_t *srcLen, const unsigned char *props, size_t propsSize) { ELzmaStatus status; return LzmaDecode(dest, destLen, src, srcLen, props, (unsigned)propsSize, LZMA_FINISH_ANY, &status, &g_Alloc); } zmat-0.9.9/src/easylzma/pavlov/7zBuf.h0000755000175200007730000000110414515254731020027 0ustar rlaboissrlaboiss/* 7zBuf.h -- Byte Buffer 2008-10-04 : Igor Pavlov : Public domain */ #ifndef __7Z_BUF_H #define __7Z_BUF_H #include "Types.h" typedef struct { Byte *data; size_t size; } CBuf; void Buf_Init(CBuf *p); int Buf_Create(CBuf *p, size_t size, ISzAlloc *alloc); void Buf_Free(CBuf *p, ISzAlloc *alloc); typedef struct { Byte *data; size_t size; size_t pos; } CDynBuf; void DynBuf_Construct(CDynBuf *p); void DynBuf_SeekToBeg(CDynBuf *p); int DynBuf_Write(CDynBuf *p, const Byte *buf, size_t size, ISzAlloc *alloc); void DynBuf_Free(CDynBuf *p, ISzAlloc *alloc); #endif zmat-0.9.9/src/easylzma/easylzma/0000755000175200007730000000000014515254731017200 5ustar rlaboissrlaboisszmat-0.9.9/src/easylzma/easylzma/compress.h0000644000175200007730000000474314515254731021214 0ustar rlaboissrlaboiss/* * Written in 2009 by Lloyd Hilaiel * * License * * All the cruft you find here is public domain. You don't have to credit * anyone to use this code, but my personal request is that you mention * Igor Pavlov for his hard, high quality work. * * easylzma/compress.h - the API for LZMA compression using easylzma */ #ifndef __EASYLZMACOMPRESS_H__ #define __EASYLZMACOMPRESS_H__ #include "easylzma/common.h" #include #ifdef __cplusplus extern "C" { #endif /** suggested default values */ #define ELZMA_LC_DEFAULT 3 #define ELZMA_LP_DEFAULT 0 #define ELZMA_PB_DEFAULT 2 #define ELZMA_DICT_SIZE_DEFAULT_MAX (1 << 24) /** an opaque handle to an lzma compressor */ typedef struct _elzma_compress_handle * elzma_compress_handle; /** * Allocate a handle to an LZMA compressor object. */ elzma_compress_handle EASYLZMA_API elzma_compress_alloc(); /** * set allocation routines (optional, if not called malloc & free will * be used) */ void EASYLZMA_API elzma_compress_set_allocation_callbacks( elzma_compress_handle hand, elzma_malloc mallocFunc, void * mallocFuncContext, elzma_free freeFunc, void * freeFuncContext); /** * Free all data associated with an LZMA compressor object. */ void EASYLZMA_API elzma_compress_free(elzma_compress_handle * hand); /** * Set configuration paramters for a compression run. If not called, * reasonable defaults will be used. */ int EASYLZMA_API elzma_compress_config(elzma_compress_handle hand, unsigned char lc, unsigned char lp, unsigned char pb, unsigned char level, unsigned int dictionarySize, elzma_file_format format, unsigned long long uncompressedSize); /** * Run compression */ int EASYLZMA_API elzma_compress_run( elzma_compress_handle hand, elzma_read_callback inputStream, void * inputContext, elzma_write_callback outputStream, void * outputContext, elzma_progress_callback progressCallback, void * progressContext); /** * a heuristic utility routine to guess a dictionary size that gets near * optimal compression while reducing memory usage. * accepts a size in bytes, returns a proposed dictionary size */ unsigned int EASYLZMA_API elzma_get_dict_size(unsigned long long size); #ifdef __cplusplus }; #endif #endif zmat-0.9.9/src/easylzma/easylzma/common.h0000644000175200007730000001042714515254731020645 0ustar rlaboissrlaboiss/* * Written in 2009 by Lloyd Hilaiel * * License * * All the cruft you find here is public domain. You don't have to credit * anyone to use this code, but my personal request is that you mention * Igor Pavlov for his hard, high quality work. * * easylzma/common.h - definitions common to both compression and * decompression */ #ifndef __EASYLZMACOMMON_H__ #define __EASYLZMACOMMON_H__ #include #ifdef __cplusplus extern "C" { #endif /* msft dll export gunk. To build a DLL on windows, you * must define WIN32, EASYLZMA_SHARED, and EASYLZMA_BUILD. To use a * DLL, you must define EASYLZMA_SHARED and WIN32 */ #if defined(WIN32) && defined(EASYLZMA_SHARED) # ifdef EASYLZMA_BUILD # define EASYLZMA_API __declspec(dllexport) # else # define EASYLZMA_API __declspec(dllimport) # endif #else # define EASYLZMA_API #endif /** error codes */ /** no error */ #define ELZMA_E_OK 0 /** bad parameters passed to an ELZMA function */ #define ELZMA_E_BAD_PARAMS 10 /** could not initialize the encode with configured parameters. */ #define ELZMA_E_ENCODING_PROPERTIES_ERROR 11 /** an error occured during compression (XXX: be more specific) */ #define ELZMA_E_COMPRESS_ERROR 12 /** currently unsupported lzma file format was specified*/ #define ELZMA_E_UNSUPPORTED_FORMAT 13 /** an error occured when reading input */ #define ELZMA_E_INPUT_ERROR 14 /** an error occured when writing output */ #define ELZMA_E_OUTPUT_ERROR 15 /** LZMA header couldn't be parsed */ #define ELZMA_E_CORRUPT_HEADER 16 /** an error occured during decompression (XXX: be more specific) */ #define ELZMA_E_DECOMPRESS_ERROR 17 /** the input stream returns EOF before the decompression could complete */ #define ELZMA_E_INSUFFICIENT_INPUT 18 /** for formats which have an emebedded crc, this error would indicated that * what came out was not what went in, i.e. data corruption */ #define ELZMA_E_CRC32_MISMATCH 19 /** for formats which have an emebedded uncompressed content length, * this error indicates that the amount we read was not what we expected */ #define ELZMA_E_SIZE_MISMATCH 20 /** Supported file formats */ typedef enum { ELZMA_lzip, /**< the lzip format which includes a magic number and * CRC check */ ELZMA_lzma /**< the LZMA-Alone format, originally designed by * Igor Pavlov and in widespread use due to lzmautils, * lacking both aforementioned features of lzip */ /* XXX: future, potentially , ELZMA_xz */ } elzma_file_format; /** * A callback invoked during elzma_[de]compress_run when the [de]compression * process has generated [de]compressed output. * * the size parameter indicates how much data is in buf to be written. * it is required that the write callback consume all data, and a return * value not equal to input size indicates and error. */ typedef size_t (*elzma_write_callback)(void *ctx, const void *buf, size_t size); /** * A callback invoked during elzma_[de]compress_run when the [de]compression * process requires more [un]compressed input. * * the size parameter is an in/out argument. on input it indicates * the buffer size. on output it indicates the amount of data read into * buf. when *size is zero on output it indicates EOF. * * \returns the read callback should return nonzero on failure. */ typedef int (*elzma_read_callback)(void *ctx, void *buf, size_t *size); /** * A callback invoked during elzma_[de]compress_run to report progress * on the [de]compression. * * \returns the read callback should return nonzero on failure. */ typedef void (*elzma_progress_callback)(void *ctx, size_t complete, size_t total); /** pointer to a malloc function, supporting client overriding memory * allocation routines */ typedef void * (*elzma_malloc)(void *ctx, unsigned int sz); /** pointer to a free function, supporting client overriding memory * allocation routines */ typedef void (*elzma_free)(void *ctx, void * ptr); #ifdef __cplusplus }; #endif #endif zmat-0.9.9/src/easylzma/easylzma/decompress.h0000644000175200007730000000314014515254731021513 0ustar rlaboissrlaboiss/* * Written in 2009 by Lloyd Hilaiel * * License * * All the cruft you find here is public domain. You don't have to credit * anyone to use this code, but my personal request is that you mention * Igor Pavlov for his hard, high quality work. * * easylzma/decompress.h - The API for LZMA decompression using easylzma */ #ifndef __EASYLZMADECOMPRESS_H__ #define __EASYLZMADECOMPRESS_H__ #include "easylzma/common.h" #ifdef __cplusplus extern "C" { #endif /** an opaque handle to an lzma decompressor */ typedef struct _elzma_decompress_handle * elzma_decompress_handle; /** * Allocate a handle to an LZMA decompressor object. */ elzma_decompress_handle EASYLZMA_API elzma_decompress_alloc(); /** * set allocation routines (optional, if not called malloc & free will * be used) */ void EASYLZMA_API elzma_decompress_set_allocation_callbacks( elzma_decompress_handle hand, elzma_malloc mallocFunc, void * mallocFuncContext, elzma_free freeFunc, void * freeFuncContext); /** * Free all data associated with an LZMA decompressor object. */ void EASYLZMA_API elzma_decompress_free(elzma_decompress_handle * hand); /** * Perform decompression * * XXX: should the library automatically detect format by reading stream? * currently it's based on data external to stream (such as extension * or convention) */ int EASYLZMA_API elzma_decompress_run( elzma_decompress_handle hand, elzma_read_callback inputStream, void * inputContext, elzma_write_callback outputStream, void * outputContext, elzma_file_format format); #ifdef __cplusplus }; #endif #endif zmat-0.9.9/src/easylzma/lzma_header.c0000644000175200007730000000721314515254731017775 0ustar rlaboissrlaboiss/* * Written in 2009 by Lloyd Hilaiel * * License * * All the cruft you find here is public domain. You don't have to credit * anyone to use this code, but my personal request is that you mention * Igor Pavlov for his hard, high quality work. */ /* XXX: clean this up, it's mostly lifted from pavel */ #include "lzma_header.h" #include #include #define ELZMA_LZMA_HEADER_SIZE 13 #define ELZMA_LZMA_PROPSBUF_SIZE 5 /**************** Header parsing ****************/ #ifndef UINT64_MAX #define UINT64_MAX ((unsigned long long) -1) #endif /* Parse the properties byte */ static char lzmadec_header_properties ( unsigned char *pb, unsigned char *lp, unsigned char *lc, const unsigned char c) { /* pb, lp and lc are encoded into a single byte. */ if (c > (9 * 5 * 5)) return -1; *pb = c / (9 * 5); /* 0 <= pb <= 4 */ *lp = (c % (9 * 5)) / 9; /* 0 <= lp <= 4 */ *lc = c % 9; /* 0 <= lc <= 8 */ assert (*pb < 5 && *lp < 5 && *lc < 9); return 0; } /* Parse the dictionary size (4 bytes, little endian) */ static char lzmadec_header_dictionary (unsigned int *size, const unsigned char *buffer) { unsigned int i; *size = 0; for (i = 0; i < 4; i++) *size += (unsigned int)(*buffer++) << (i * 8); /* The dictionary size is limited to 256 MiB (checked from * LZMA SDK 4.30) */ if (*size > (1 << 28)) return -1; return 0; } /* Parse the uncompressed size field (8 bytes, little endian) */ static void lzmadec_header_uncompressed (unsigned long long *size, unsigned char *is_streamed, const unsigned char *buffer) { unsigned int i; /* Streamed files have all 64 bits set in the size field. * We don't know the uncompressed size beforehand. */ *is_streamed = 1; /* Assume streamed. */ *size = 0; for (i = 0; i < 8; i++) { *size += (unsigned long long)buffer[i] << (i * 8); if (buffer[i] != 255) *is_streamed = 0; } assert ((*is_streamed == 1 && *size == UINT64_MAX) || (*is_streamed == 0 && *size < UINT64_MAX)); } static void initLzmaHeader(struct elzma_file_header * hdr) { memset((void *) hdr, 0, sizeof(struct elzma_file_header)); } static int parseLzmaHeader(const unsigned char * hdrBuf, struct elzma_file_header * hdr) { if (lzmadec_header_properties(&(hdr->pb), &(hdr->lp), &(hdr->lc), *hdrBuf) || lzmadec_header_dictionary(&(hdr->dictSize), hdrBuf + 1)) { return 1; } lzmadec_header_uncompressed(&(hdr->uncompressedSize), &(hdr->isStreamed), hdrBuf + 5); return 0; } static int serializeLzmaHeader(unsigned char * hdrBuf, const struct elzma_file_header * hdr) { unsigned int i; memset((void *) hdrBuf, 0, ELZMA_LZMA_HEADER_SIZE); /* encode lc, pb, and lp */ *hdrBuf++ = hdr->lc + (hdr->pb * 45) + (hdr->lp * 45 * 9); /* encode dictionary size */ for (i = 0; i < 4; i++) { *(hdrBuf++) = (unsigned char) (hdr->dictSize >> (i * 8)); } /* encode uncompressed size */ for (i = 0; i < 8; i++) { if (hdr->isStreamed) { *(hdrBuf++) = 0xff; } else { *(hdrBuf++) = (unsigned char) (hdr->uncompressedSize >> (i * 8)); } } return 0; } void initializeLZMAFormatHandler(struct elzma_format_handler * hand) { hand->header_size = ELZMA_LZMA_HEADER_SIZE; hand->init_header = initLzmaHeader; hand->parse_header = parseLzmaHeader; hand->serialize_header = serializeLzmaHeader; hand->footer_size = 0; hand->serialize_footer = NULL; } zmat-0.9.9/src/easylzma/lzip_header.c0000644000175200007730000000463414515254731020014 0ustar rlaboissrlaboiss#include "lzip_header.h" #include #define ELZMA_LZIP_HEADER_SIZE 6 #define ELZMA_LZIP_FOOTER_SIZE 12 static void initLzipHeader(struct elzma_file_header * hdr) { memset((void *) hdr, 0, sizeof(struct elzma_file_header)); } static int parseLzipHeader(const unsigned char * hdrBuf, struct elzma_file_header * hdr) { if (0 != strncmp("LZIP", (char *) hdrBuf, 4)) return 1; /* XXX: ignore version for now */ hdr->pb = 2; hdr->lp = 0; hdr->lc = 3; /* unknown at this point */ hdr->isStreamed = 1; hdr->uncompressedSize = 0; hdr->dictSize = 1 << (hdrBuf[5] & 0x1F); return 0; } static int serializeLzipHeader(unsigned char * hdrBuf, const struct elzma_file_header * hdr) { hdrBuf[0] = 'L'; hdrBuf[1] = 'Z'; hdrBuf[2] = 'I'; hdrBuf[3] = 'P'; hdrBuf[4] = 0; { int r = 0; while ((hdr->dictSize >> r) != 0) r++; hdrBuf[5] = (unsigned char) (r-1) & 0x1F; } return 0; } static int serializeLzipFooter(struct elzma_file_footer * ftr, unsigned char * ftrBuf) { unsigned int i = 0; /* first crc32 */ for (i = 0; i < 4; i++) { *(ftrBuf++) = (unsigned char) (ftr->crc32 >> (i * 8)); } /* next data size */ for (i = 0; i < 8; i++) { *(ftrBuf++) = (unsigned char) (ftr->uncompressedSize >> (i * 8)); } /* write version 0 files, omit member length for now*/ return 0; } static int parseLzipFooter(const unsigned char * ftrBuf, struct elzma_file_footer * ftr) { unsigned int i = 0; ftr->crc32 = 0; ftr->uncompressedSize = 0; /* first crc32 */ for (i = 0; i < 4; i++) { ftr->crc32 += ((unsigned int) *(ftrBuf++) << (i * 8)); } /* next data size */ for (i = 0; i < 8; i++) { ftr->uncompressedSize += (unsigned long long) *(ftrBuf++) << (i * 8); } /* read version 0 files, omit member length for now*/ return 0; } void initializeLZIPFormatHandler(struct elzma_format_handler * hand) { hand->header_size = ELZMA_LZIP_HEADER_SIZE; hand->init_header = initLzipHeader; hand->parse_header = parseLzipHeader; hand->serialize_header = serializeLzipHeader; hand->footer_size = ELZMA_LZIP_FOOTER_SIZE; hand->serialize_footer = serializeLzipFooter; hand->parse_footer = parseLzipFooter; } zmat-0.9.9/src/easylzma/common_internal.c0000644000175200007730000000240714515254731020706 0ustar rlaboissrlaboiss/* * Written in 2009 by Lloyd Hilaiel * * License * * All the cruft you find here is public domain. You don't have to credit * anyone to use this code, but my personal request is that you mention * Igor Pavlov for his hard, high quality work. */ #include "common_internal.h" static void *elzmaAlloc(void *p, size_t size) { struct elzma_alloc_struct * as = (struct elzma_alloc_struct *) p; if (as->clientMallocFunc) { return as->clientMallocFunc(as->clientMallocContext, size); } return malloc(size); } static void elzmaFree(void *p, void *address) { struct elzma_alloc_struct * as = (struct elzma_alloc_struct *) p; if (as->clientFreeFunc) { as->clientFreeFunc(as->clientMallocContext, address); } else { free(address); } } void init_alloc_struct(struct elzma_alloc_struct * as, elzma_malloc clientMallocFunc, void * clientMallocContext, elzma_free clientFreeFunc, void * clientFreeContext) { as->Alloc = elzmaAlloc; as->Free = elzmaFree; as->clientMallocFunc = clientMallocFunc; as->clientMallocContext = clientMallocContext; as->clientFreeFunc = clientFreeFunc; as->clientFreeContext = clientFreeContext; } zmat-0.9.9/src/easylzma/decompress.c0000644000175200007730000001763014515254731017672 0ustar rlaboissrlaboiss/* * Written in 2009 by Lloyd Hilaiel * * License * * All the cruft you find here is public domain. You don't have to credit * anyone to use this code, but my personal request is that you mention * Igor Pavlov for his hard, high quality work. */ #include "easylzma/decompress.h" #include "pavlov/LzmaDec.h" #include "pavlov/7zCrc.h" #include "common_internal.h" #include "lzma_header.h" #include "lzip_header.h" #include #include #define ELZMA_DECOMPRESS_INPUT_BUFSIZE (1024 * 64) #define ELZMA_DECOMPRESS_OUTPUT_BUFSIZE (1024 * 256) /** an opaque handle to an lzma decompressor */ struct _elzma_decompress_handle { char inbuf[ELZMA_DECOMPRESS_INPUT_BUFSIZE]; char outbuf[ELZMA_DECOMPRESS_OUTPUT_BUFSIZE]; struct elzma_alloc_struct allocStruct; }; elzma_decompress_handle elzma_decompress_alloc() { elzma_decompress_handle hand = malloc(sizeof(struct _elzma_decompress_handle)); memset((void *) hand, 0, sizeof(struct _elzma_decompress_handle)); init_alloc_struct(&(hand->allocStruct), NULL, NULL, NULL, NULL); return hand; } void elzma_decompress_set_allocation_callbacks( elzma_decompress_handle hand, elzma_malloc mallocFunc, void * mallocFuncContext, elzma_free freeFunc, void * freeFuncContext) { if (hand) { init_alloc_struct(&(hand->allocStruct), mallocFunc, mallocFuncContext, freeFunc, freeFuncContext); } } void elzma_decompress_free(elzma_decompress_handle * hand) { if (*hand) free(*hand); *hand = NULL; } int elzma_decompress_run(elzma_decompress_handle hand, elzma_read_callback inputStream, void * inputContext, elzma_write_callback outputStream, void * outputContext, elzma_file_format format) { unsigned long long int totalRead = 0; /* total amount read from stream */ unsigned int crc32 = CRC_INIT_VAL; /* running crc32 (lzip case) */ CLzmaDec dec; unsigned int errorCode = ELZMA_E_OK; struct elzma_format_handler formatHandler; struct elzma_file_header h; struct elzma_file_footer f; /* switch between supported formats */ if (format == ELZMA_lzma) { initializeLZMAFormatHandler(&formatHandler); } else if (format == ELZMA_lzip) { CrcGenerateTable(); initializeLZIPFormatHandler(&formatHandler); } else { return ELZMA_E_BAD_PARAMS; } /* initialize footer */ f.crc32 = 0; f.uncompressedSize = 0; /* initialize decoder memory */ memset((void *) &dec, 0, sizeof(dec)); LzmaDec_Init(&dec); /* decode the header. */ { unsigned char * hdr = hand->allocStruct.Alloc(&(hand->allocStruct), formatHandler.header_size); size_t sz = formatHandler.header_size; formatHandler.init_header(&h); if (inputStream(inputContext, hdr, &sz) != 0 || sz != formatHandler.header_size) { hand->allocStruct.Free(&(hand->allocStruct), hdr); return ELZMA_E_INPUT_ERROR; } if (0 != formatHandler.parse_header(hdr, &h)) { hand->allocStruct.Free(&(hand->allocStruct), hdr); return ELZMA_E_CORRUPT_HEADER; } /* the LzmaDec_Allocate call requires 5 bytes which have * compression properties encoded in them. In the case of * lzip, the header format does not already contain what * LzmaDec_Allocate expects, so we must craft it, silly */ { unsigned char propsBuf[13]; const unsigned char * propsPtr = hdr; if (format == ELZMA_lzip) { struct elzma_format_handler lzmaHand; initializeLZMAFormatHandler(&lzmaHand); lzmaHand.serialize_header(propsBuf, &h); propsPtr = propsBuf; } /* now we're ready to allocate the decoder */ LzmaDec_Allocate(&dec, propsPtr, 5, (ISzAlloc *) &(hand->allocStruct)); } hand->allocStruct.Free(&(hand->allocStruct), hdr); } /* perform the decoding */ for (;;) { size_t dstLen = ELZMA_DECOMPRESS_OUTPUT_BUFSIZE; size_t srcLen = ELZMA_DECOMPRESS_INPUT_BUFSIZE; size_t amt = 0; size_t bufOff = 0; ELzmaStatus stat; if (0 != inputStream(inputContext, hand->inbuf, &srcLen)) { errorCode = ELZMA_E_INPUT_ERROR; goto decompressEnd; } /* handle the case where the input prematurely finishes */ if (srcLen == 0) { errorCode = ELZMA_E_INSUFFICIENT_INPUT; goto decompressEnd; } amt = srcLen; /* handle the case where a single read buffer of compressed bytes * will translate into multiple buffers of uncompressed bytes, * with this inner loop */ stat = LZMA_STATUS_NOT_SPECIFIED; while (bufOff < srcLen) { SRes r = LzmaDec_DecodeToBuf(&dec, (Byte *) hand->outbuf, &dstLen, ((Byte *) hand->inbuf + bufOff), &amt, LZMA_FINISH_ANY, &stat); /* XXX deal with result code more granularly*/ if (r != SZ_OK) { errorCode = ELZMA_E_DECOMPRESS_ERROR; goto decompressEnd; } /* write what we've read */ { size_t wt; /* if decoding lzip, update our crc32 value */ if (format == ELZMA_lzip && dstLen > 0) { crc32 = CrcUpdate(crc32, hand->outbuf, dstLen); } totalRead += dstLen; wt = outputStream(outputContext, hand->outbuf, dstLen); if (wt != dstLen) { errorCode = ELZMA_E_OUTPUT_ERROR; goto decompressEnd; } } /* do we have more data on the input buffer? */ bufOff += amt; assert( bufOff <= srcLen ); if (bufOff >= srcLen) break; amt = srcLen - bufOff; /* with lzip, we will have the footer left on the buffer! */ if (stat == LZMA_STATUS_FINISHED_WITH_MARK) { break; } } /* now check status */ if (stat == LZMA_STATUS_FINISHED_WITH_MARK) { /* read a footer if one is expected and * present */ if (formatHandler.footer_size > 0 && amt >= formatHandler.footer_size && formatHandler.parse_footer != NULL) { formatHandler.parse_footer( (unsigned char *) hand->inbuf + bufOff, &f); } break; } /* for LZMA utils, we don't always have a finished mark */ if (!h.isStreamed && totalRead >= h.uncompressedSize) { break; } } /* finish the calculated crc32 */ crc32 ^= 0xFFFFFFFF; /* if we have a footer, check that the calculated crc32 matches * the encoded crc32, and that the sizes match */ if (formatHandler.footer_size) { if (f.crc32 != crc32) { errorCode = ELZMA_E_CRC32_MISMATCH; } else if (f.uncompressedSize != totalRead) { errorCode = ELZMA_E_SIZE_MISMATCH; } } else if (!h.isStreamed) { /* if the format does not support a footer and has an uncompressed * size in the header, let's compare that with how much we actually * read */ if (h.uncompressedSize != totalRead) { errorCode = ELZMA_E_SIZE_MISMATCH; } } decompressEnd: LzmaDec_Free(&dec, (ISzAlloc *) &(hand->allocStruct)); return errorCode; } zmat-0.9.9/src/easylzma/lzip_header.h0000644000175200007730000000043514515254731020014 0ustar rlaboissrlaboiss#ifndef __EASYLZMA_LZIP_HEADER__ #define __EASYLZMA_LZIP_HEADER__ #include "common_internal.h" /* lzip file format documented here: * http://download.savannah.gnu.org/releases-noredirect/lzip/manual/ */ void initializeLZIPFormatHandler(struct elzma_format_handler * hand); #endif zmat-0.9.9/.gitmodules0000644000175200007730000000000014515254731015102 0ustar rlaboissrlaboisszmat-0.9.9/COPYING0000644000175200007730000010451414515254731013777 0ustar rlaboissrlaboiss GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: Copyright (C) This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read .